# Desktop & Computer Use

A sandbox with a graphical Linux desktop that your agent can see and control: it takes screenshots and moves the mouse, clicks, scrolls and types. You can watch it live in the portal and take over at any time.

**Beta.** Desktop & Computer Use is off by default and is enabled per team on request – contact support. Until it is enabled, the endpoints below answer `403` with `"code": "feature_disabled"`, and so does creating a sandbox, job or batch from the `desktop` template – the template is not listed in `GET /v1/templates` or in the portal either. If the feature is turned off later, running desktop sandboxes keep running until their timeout (which can no longer be extended), but sleeping or archived ones can no longer be woken up. A desktop sandbox is billed like any other sandbox, by its vCPU and memory.

## The desktop template

Create the sandbox from the `desktop` template: an XFCE desktop on a virtual 1280×800 screen (Xvfb on display `:0`), Chromium, Python 3.12 with Playwright, and xdotool and ImageMagick for input and screenshots. The desktop runs as the unprivileged user `user` (uid 1000), and the screen is only reachable through the API and the portal – no port of the sandbox is opened to the internet. We recommend 2 vCPU and 2 GB of memory (the template default). See [Templates](https://sandbox.wbx.cz/docs/templates.md). What is installed and how to start programs on the screen: [Desktop environment (XFCE)](https://sandbox.wbx.cz/docs/sandboxes/desktop-environment.md).

**Python**

```
from wbxsandbox import Sandbox

sb = Sandbox.create(template="desktop", timeout=1800)
sb.run("su user -c 'DISPLAY=:0 nohup chromium https://example.com >/dev/null 2>&1 &'")
print(sb.computer.status())          # available, width, height, controller …

sb.computer.click(640, 120)
sb.computer.type("hello world")
sb.computer.key("Return")
shot = sb.computer.screenshot(format="jpeg", quality=70)
shot.save("screen.jpg")              # shot.width, shot.height
```

**curl**

```
curl -s https://sandbox.wbx.cz/v1/sandboxes/SB_ID/computer/status -H "Authorization: Bearer YOUR_API_KEY"

curl -s -o screen.png "https://sandbox.wbx.cz/v1/sandboxes/SB_ID/computer/screenshot?format=png" \
  -H "Authorization: Bearer YOUR_API_KEY"

curl -s -X POST https://sandbox.wbx.cz/v1/sandboxes/SB_ID/computer/actions \
  -H "Authorization: Bearer YOUR_API_KEY" -H "Content-Type: application/json" \
  -d '{"actions": [
        {"type": "click", "x": 640, "y": 120},
        {"type": "type", "text": "hello world"},
        {"type": "key", "keys": "Return"},
        {"type": "wait", "ms": 500}
      ], "screenshot": true}'
```

## Endpoints

| Endpoint | What it does |
| --- | --- |
| GET /v1/sandboxes/{id}/computer/status | `{"available": true, "display": ":0", "width": 1280, "height": 800, "controller": "agent", "controller_until": null, "viewers": 0}`. When `available` is false, `reason` says why: `sandbox_sleeping`, `sandbox_archived`, `sandbox_not_running`, `display_unavailable` (no desktop in this template, or it is still starting) or `agent_unavailable`. Does not wake a sleeping sandbox. |
| GET /v1/sandboxes/{id}/computer/screenshot?format=&quality=&max_width= | The image itself (`image/png` or `image/jpeg`). `format` png (default) or jpeg, `quality` 1–100 for JPEG (default 80), `max_width` scales the image down. Headers `X-WBX-Width` / `X-WBX-Height` give the image size, `X-WBX-Screen-Width` / `X-WBX-Screen-Height` the screen size. Does not wake a sleeping sandbox (`409 sandbox_sleeping`). |
| POST /v1/sandboxes/{id}/computer/actions | Runs up to 50 actions in order: `{"actions": [...], "screenshot": false}`. Answers `{"ok": true, "results": [{"type": "click", "ok": true}, …]}`; with `"screenshot": true` also `"screenshot": {"format", "width", "height", "screen_width", "screen_height", "data_base64"}` (options `screenshot_format`, `screenshot_quality`, `screenshot_max_width`). If an action fails, the rest is skipped and the answer has `"ok": false`, `"error"` and `"failed_index"`. Wakes a sleeping sandbox, like a command. |
| POST /v1/sandboxes/{id}/computer/control | `{"mode": "human" \| "agent", "ttl_s": 600}` – who controls the desktop; answers with the status. A person takes control only in the portal; with an API key `"human"` is refused (`403 human_only`). The agent cannot take control away from a person: `"agent"` answers `409 human_in_control` until the person releases it or the time runs out. |

### Actions

Coordinates are screen pixels, `(0, 0)` is the top left corner; a point outside the screen is refused with `400` before anything runs. If you scaled the screenshot down with `max_width`, multiply its coordinates by `screen_width / width`.

| Action | Fields |
| --- | --- |
| `move` | `x`, `y` |
| `click` | `x`, `y`, `button` left (default), right or middle, `count` 1–3 (double and triple click) |
| `mouse_down` / `mouse_up` | `button`; optionally `x`, `y` to move first |
| `drag` | `x`, `y` → `to_x`, `to_y`, `button` |
| `scroll` | `x`, `y`, `dy` wheel steps down (negative = up), `dx` to the right (negative = left), each −50 to 50. The SDK shortcuts take them in a different order – Python `scroll(x, y, dy, dx)`, TypeScript `scroll(x, y, dx, dy)` – so in Python prefer `scroll(640, 400, dy=5)`. |
| `type` | `text` – up to 4096 characters; newlines (`\n`, `\r\n` or `\r`) press Enter, tabs are typed; other control characters are refused |
| `key` | `keys` – a key or combination such as `Return`, `Escape`, `ctrl+c`, `ctrl+shift+t`, `alt+F4`, `super`; several separated by spaces. Key names use letters, digits and `_` (X keysym names); aliases `enter`, `esc`, `cmd`, `pageup` … are accepted |
| `wait` | `ms` 0–5000 |

## MCP tools

AI assistants connected over MCP get two tools (see [MCP server and tools](https://sandbox.wbx.cz/docs/mcp.md)):

- Both tools appear in the MCP tool list only for accounts with the desktop feature enabled.
- `computer_screenshot` – a screenshot returned to the model as an image, JPEG quality 70 and at most 1280 pixels wide by default, with the image and screen size.
- `computer_action` – a batch of the actions above, followed by a screenshot unless `screenshot` is false.

Both need the `desktop` template and the feature enabled for your team; otherwise they return a clear error. Without `sandbox_id` they use the last sandbox of the conversation – unlike the other tools they never create one, so let the assistant start it first with `create_sandbox` and the template `desktop`.

## Watching and taking control in the portal

Step-by-step guide with all controls, tips and troubleshooting: [Live console](https://sandbox.wbx.cz/docs/sandboxes/console.md). What is on the desktop and how to start programs: [Desktop environment (XFCE)](https://sandbox.wbx.cz/docs/sandboxes/desktop-environment.md).

In the portal, *Desktop* next to a running desktop sandbox on the *Sandboxes* page opens a **live console** of the screen – a continuous VNC stream shown in your browser (noVNC), similar to a virtual machine console. The picture is scaled to fit the window; *Fullscreen* uses the whole screen.

- **Watching** is the default: you see everything the agent does, and your mouse and keyboard are not passed to the desktop.
- **Take control** switches the console to control mode, and the mouse and keyboard work directly on the desktop. While you are in control the agent's actions get `409` with `"code": "human_in_control"` and `controller_until` (screenshots and status keep working, so the agent can wait and look). Control goes back to the agent when you click *Release control*, after 2 minutes without input from you, or after `ttl_s` (10 minutes by default, at most 60); the console then switches back to watching. Taking and releasing control is recorded in your team's access log.
- **Keys** sends shortcuts that your browser keeps for itself, such as `Ctrl+W`, `Ctrl+T`, `Alt+Tab`, `Alt+F4` or the `Super` key.
- **Clipboard**: text copied on the desktop appears in the *Clipboard* box; while you are in control, *Send to desktop* puts the text from the box into the desktop clipboard, and you paste it there with `Ctrl+V`.
- If the connection closes – the sandbox fell asleep or ended, or the network dropped – the console shows why, and *Reconnect* opens it again. Watching does not keep the sandbox awake; your mouse and keyboard input while in control does, like any other activity. A console connection lasts at most an hour, then the page reconnects. A sandbox can have up to 4 consoles open at once, a team up to 8.

The console connects over a WebSocket on the portal's own address with a short-lived ticket that the portal issues only to a signed-in member of the team that owns the sandbox. If the live console cannot be used – a sandbox from an older build of the template without the VNC server, or a browser or network that cannot keep the connection open – the page falls back to screenshots refreshed about once a second (0.5–2 per second, only while the tab is visible) and says *Live console unavailable – showing screenshots*; you can still take control and click, type and scroll in the picture.

Who is a person and who is an agent is decided on our side: only requests from a signed-in portal user count as a person – an API key or an MCP connection is always the agent, whatever headers it sends. A live view stops after 30 minutes (reload the page to continue). Status and screenshots do not count as sandbox activity; actions do.

## Limits

| What | Limit |
| --- | --- |
| Actions per request | 50; text in all `type` actions at most 8192 characters; waits at most 30 s in total |
| Requests per sandbox | actions 20 per second, screenshots 5 per second, status and control 10 per second – above that `429` with `Retry-After`. The limits count per account: the live view in the portal has its own allowance, so people watching never use up your agent's screenshots |
| At the same time | 2 screenshots of one sandbox (each at most 10 s); one batch of actions runs at a time and at most 2 more wait for it – above that `429` with `Retry-After` |
| Screenshot | at most 8 MB and 8192×8192 pixels – use JPEG or `max_width` for big screens |
| Control by a person | 10 minutes by default, at most 60; ends after 2 minutes without input |
| Live console (portal) | 4 open consoles per sandbox and 8 per team; one connection lasts at most an hour (the page then reconnects) |

## Security

- Your input never becomes part of a shell command: coordinates and buttons are checked and passed as numbers, key names must match a strict pattern, and text is typed from standard input.
- Input runs as the desktop user, not as root. The desktop is reachable only through the API and the portal – no port of the sandbox is open to the internet, the portal's live console needs a short-lived ticket issued to a signed-in member of your team, and screenshots are never available at a public URL.
- A screenshot comes from inside the sandbox, so we check that it really is a PNG or JPEG of a sane size before passing it on, and send it with `X-Content-Type-Options: nosniff` and `Cache-Control: no-store`.
- Treat what is on the screen as untrusted, like command output: a web page can show text that tries to instruct your agent.

## Known limitations

- Through the API the agent sees screenshots, not video. The live console is only in the portal.
- The clipboard works only in the portal's live console and only for text; depending on the desktop's VNC server, characters outside Western European (Latin-1) may arrive as `?` – type such text instead. In the screenshot fallback there is no clipboard.
- Control by a person is kept in the memory of the server running the sandbox: after that server restarts, or after the sandbox sleeps and wakes up, the agent is in control again.
- After the sandbox starts or wakes up, the desktop needs a few seconds before `available` is true.
- No sound, one screen of a fixed size (1280×800).

Source: https://sandbox.wbx.cz/docs/sandboxes/desktop
