Files
openclaw/docs/platforms/mac/canvas.md

122 lines
3.6 KiB
Markdown
Raw Normal View History

---
summary: "Agent-controlled Canvas panel embedded via WKWebView + custom URL scheme"
read_when:
- Implementing the macOS Canvas panel
- Adding agent controls for visual workspace
- Debugging WKWebView canvas loads
---
# Canvas (macOS app)
2026-01-08 23:06:56 +01:00
The macOS app embeds an agentcontrolled **Canvas panel** using `WKWebView`. It
is a lightweight visual workspace for HTML/CSS/JS, A2UI, and small interactive
UI surfaces.
2026-01-08 23:06:56 +01:00
## Where Canvas lives
2026-01-08 23:06:56 +01:00
Canvas state is stored under Application Support:
2026-01-08 23:06:56 +01:00
- `~/Library/Application Support/Clawdbot/canvas/<session>/...`
2026-01-08 23:06:56 +01:00
The Canvas panel serves those files via a **custom URL scheme**:
2026-01-04 14:32:47 +00:00
- `clawdbot-canvas://<session>/<path>`
2026-01-08 23:06:56 +01:00
Examples:
- `clawdbot-canvas://main/``<canvasRoot>/main/index.html`
2026-01-04 14:32:47 +00:00
- `clawdbot-canvas://main/assets/app.css``<canvasRoot>/main/assets/app.css`
2026-01-08 23:06:56 +01:00
- `clawdbot-canvas://main/widgets/todo/``<canvasRoot>/main/widgets/todo/index.html`
2026-01-08 23:06:56 +01:00
If no `index.html` exists at the root, the app shows a **builtin scaffold page**.
2026-01-08 23:06:56 +01:00
## Panel behavior
2026-01-08 23:06:56 +01:00
- Borderless, resizable panel anchored near the menu bar (or mouse cursor).
- Remembers size/position per session.
- Autoreloads when local canvas files change.
- Only one Canvas panel is visible at a time (session is switched as needed).
2026-01-08 23:06:56 +01:00
Canvas can be disabled from Settings → **Allow Canvas**. When disabled, canvas
node commands return `CANVAS_DISABLED`.
2026-01-08 23:06:56 +01:00
## Agent API surface
2026-01-08 23:06:56 +01:00
Canvas is exposed via the **node bridge**, so the agent can:
2026-01-08 23:06:56 +01:00
- show/hide the panel
- navigate to a path or URL
- evaluate JavaScript
- capture a snapshot image
2026-01-08 23:06:56 +01:00
CLI examples:
2026-01-08 23:06:56 +01:00
```bash
clawdbot nodes canvas present --node <id>
clawdbot nodes canvas navigate --node <id> --url "/"
clawdbot nodes canvas eval --node <id> --js "document.title"
clawdbot nodes canvas snapshot --node <id>
```
2025-12-17 11:35:06 +01:00
2026-01-08 23:06:56 +01:00
Notes:
- `canvas.navigate` accepts **local canvas paths**, `http(s)` URLs, and `file://` URLs.
- If you pass `"/"`, the Canvas shows the local scaffold or `index.html`.
2025-12-17 11:35:06 +01:00
2026-01-08 23:06:56 +01:00
## A2UI in Canvas
2025-12-17 11:35:06 +01:00
2026-01-08 23:06:56 +01:00
A2UI is hosted by the Gateway canvas host and rendered inside the Canvas panel.
When the Gateway advertises a Canvas host, the macOS app autonavigates to the
A2UI host page on first open.
2025-12-17 11:35:06 +01:00
2026-01-08 23:06:56 +01:00
Default A2UI host URL:
2025-12-20 12:17:39 +00:00
```
2026-01-04 14:32:47 +00:00
http://<gateway-host>:18793/__clawdbot__/a2ui/
2025-12-20 12:17:39 +00:00
```
2026-01-08 23:06:56 +01:00
### A2UI commands (v0.8)
Canvas currently accepts **A2UI v0.8** server→client messages:
2025-12-17 11:35:06 +01:00
2026-01-08 23:06:56 +01:00
- `beginRendering`
- `surfaceUpdate`
- `dataModelUpdate`
- `deleteSurface`
2025-12-17 11:35:06 +01:00
2026-01-08 23:06:56 +01:00
`createSurface` (v0.9) is not supported.
2025-12-17 11:35:06 +01:00
2026-01-08 23:06:56 +01:00
CLI example:
2025-12-17 13:20:27 +01:00
```bash
2026-01-08 23:06:56 +01:00
cat > /tmp/a2ui-v0.8.jsonl <<'EOFA2'
{"surfaceUpdate":{"surfaceId":"main","components":[{"id":"root","component":{"Column":{"children":{"explicitList":["title","content"]}}}},{"id":"title","component":{"Text":{"text":{"literalString":"Canvas (A2UI v0.8)"},"usageHint":"h1"}}},{"id":"content","component":{"Text":{"text":{"literalString":"If you can read this, A2UI push works."},"usageHint":"body"}}}]}}
2025-12-17 13:20:27 +01:00
{"beginRendering":{"surfaceId":"main","root":"root"}}
2026-01-08 23:06:56 +01:00
EOFA2
2025-12-17 13:20:27 +01:00
2026-01-08 07:16:05 +01:00
clawdbot nodes canvas a2ui push --jsonl /tmp/a2ui-v0.8.jsonl --node <id>
2025-12-17 13:20:27 +01:00
```
2026-01-08 23:06:56 +01:00
Quick smoke:
2026-01-08 23:06:56 +01:00
```bash
clawdbot nodes canvas a2ui push --node <id> --text "Hello from A2UI"
```
2026-01-08 23:06:56 +01:00
## Triggering agent runs from Canvas
2026-01-08 23:06:56 +01:00
Canvas can trigger new agent runs via deep links:
2026-01-08 23:06:56 +01:00
- `clawdbot://agent?...`
2026-01-08 23:06:56 +01:00
Example (in JS):
2026-01-08 23:06:56 +01:00
```js
window.location.href = "clawdbot://agent?message=Review%20this%20design";
```
2026-01-08 23:06:56 +01:00
The app prompts for confirmation unless a valid key is provided.
2026-01-08 23:06:56 +01:00
## Security notes
2026-01-08 23:06:56 +01:00
- Canvas scheme blocks directory traversal; files must live under the session root.
- Local Canvas content uses a custom scheme (no loopback server required).
- External `http(s)` URLs are allowed only when explicitly navigated.