Desktop & Computer Use
A sandbox with a graphical Linux desktop that your agent can see and control: it takes screenshots and moves the mouse, clicks, scrolls and types. You can watch it live in the portal and take over at any time.
Beta. Desktop & Computer Use is off by default and is enabled per team on request – contact support. Until it is enabled, the endpoints below answer 403 with "code": "feature_disabled", and so does creating a sandbox, job or batch from the desktop template – the template is not listed in GET /v1/templates or in the portal either. If the feature is turned off later, running desktop sandboxes keep running until their timeout (which can no longer be extended), but sleeping or archived ones can no longer be woken up. A desktop sandbox is billed like any other sandbox, by its vCPU and memory.
The desktop template
Create the sandbox from the desktop template: an XFCE desktop on a virtual 1280×800 screen (Xvfb on display :0), Chromium, Python 3.12 with Playwright, and xdotool and ImageMagick for input and screenshots. The desktop runs as the unprivileged user user (uid 1000), and the screen is only reachable through the API and the portal – no port of the sandbox is opened to the internet. We recommend 2 vCPU and 2 GB of memory (the template default). See Templates. What is installed and how to start programs on the screen: Desktop environment (XFCE).
from wbxsandbox import Sandbox
sb = Sandbox.create(template="desktop", timeout=1800)
sb.run("su user -c 'DISPLAY=:0 nohup chromium https://example.com >/dev/null 2>&1 &'")
print(sb.computer.status()) # available, width, height, controller …
sb.computer.click(640, 120)
sb.computer.type("hello world")
sb.computer.key("Return")
shot = sb.computer.screenshot(format="jpeg", quality=70)
shot.save("screen.jpg") # shot.width, shot.height
curl -s https://sandbox.wbx.cz/v1/sandboxes/SB_ID/computer/status -H "Authorization: Bearer YOUR_API_KEY"
curl -s -o screen.png "https://sandbox.wbx.cz/v1/sandboxes/SB_ID/computer/screenshot?format=png" \
-H "Authorization: Bearer YOUR_API_KEY"
curl -s -X POST https://sandbox.wbx.cz/v1/sandboxes/SB_ID/computer/actions \
-H "Authorization: Bearer YOUR_API_KEY" -H "Content-Type: application/json" \
-d '{"actions": [
{"type": "click", "x": 640, "y": 120},
{"type": "type", "text": "hello world"},
{"type": "key", "keys": "Return"},
{"type": "wait", "ms": 500}
], "screenshot": true}'
Endpoints
| Endpoint | What it does |
|---|---|
| GET /v1/sandboxes/{id}/computer/status | {"available": true, "display": ":0", "width": 1280, "height": 800, "controller": "agent", "controller_until": null, "viewers": 0}. When available is false, reason says why: sandbox_sleeping, sandbox_archived, sandbox_not_running, display_unavailable (no desktop in this template, or it is still starting) or agent_unavailable. Does not wake a sleeping sandbox. |
| GET /v1/sandboxes/{id}/computer/screenshot?format=&quality=&max_width= | The image itself (image/png or image/jpeg). format png (default) or jpeg, quality 1–100 for JPEG (default 80), max_width scales the image down. Headers X-WBX-Width / X-WBX-Height give the image size, X-WBX-Screen-Width / X-WBX-Screen-Height the screen size. Does not wake a sleeping sandbox (409 sandbox_sleeping). |
| POST /v1/sandboxes/{id}/computer/actions | Runs up to 50 actions in order: {"actions": [...], "screenshot": false}. Answers {"ok": true, "results": [{"type": "click", "ok": true}, …]}; with "screenshot": true also "screenshot": {"format", "width", "height", "screen_width", "screen_height", "data_base64"} (options screenshot_format, screenshot_quality, screenshot_max_width). If an action fails, the rest is skipped and the answer has "ok": false, "error" and "failed_index". Wakes a sleeping sandbox, like a command. |
| POST /v1/sandboxes/{id}/computer/control | {"mode": "human" | "agent", "ttl_s": 600} – who controls the desktop; answers with the status. A person takes control only in the portal; with an API key "human" is refused (403 human_only). The agent cannot take control away from a person: "agent" answers 409 human_in_control until the person releases it or the time runs out. |
Actions
Coordinates are screen pixels, (0, 0) is the top left corner; a point outside the screen is refused with 400 before anything runs. If you scaled the screenshot down with max_width, multiply its coordinates by screen_width / width.
| Action | Fields |
|---|---|
move | x, y |
click | x, y, button left (default), right or middle, count 1–3 (double and triple click) |
mouse_down / mouse_up | button; optionally x, y to move first |
drag | x, y → to_x, to_y, button |
scroll | x, y, dy wheel steps down (negative = up), dx to the right (negative = left), each −50 to 50. The SDK shortcuts take them in a different order – Python scroll(x, y, dy, dx), TypeScript scroll(x, y, dx, dy) – so in Python prefer scroll(640, 400, dy=5). |
type | text – up to 4096 characters; newlines (\n, \r\n or \r) press Enter, tabs are typed; other control characters are refused |
key | keys – a key or combination such as Return, Escape, ctrl+c, ctrl+shift+t, alt+F4, super; several separated by spaces. Key names use letters, digits and _ (X keysym names); aliases enter, esc, cmd, pageup … are accepted |
wait | ms 0–5000 |
MCP tools
AI assistants connected over MCP get two tools (see MCP server and tools):
- Both tools appear in the MCP tool list only for accounts with the desktop feature enabled.
computer_screenshot– a screenshot returned to the model as an image, JPEG quality 70 and at most 1280 pixels wide by default, with the image and screen size.computer_action– a batch of the actions above, followed by a screenshot unlessscreenshotis false.
Both need the desktop template and the feature enabled for your team; otherwise they return a clear error. Without sandbox_id they use the last sandbox of the conversation – unlike the other tools they never create one, so let the assistant start it first with create_sandbox and the template desktop.
Watching and taking control in the portal
Step-by-step guide with all controls, tips and troubleshooting: Live console. What is on the desktop and how to start programs: Desktop environment (XFCE).
In the portal, Desktop next to a running desktop sandbox on the Sandboxes page opens a live console of the screen – a continuous VNC stream shown in your browser (noVNC), similar to a virtual machine console. The picture is scaled to fit the window; Fullscreen uses the whole screen.
- Watching is the default: you see everything the agent does, and your mouse and keyboard are not passed to the desktop.
- Take control switches the console to control mode, and the mouse and keyboard work directly on the desktop. While you are in control the agent's actions get
409with"code": "human_in_control"andcontroller_until(screenshots and status keep working, so the agent can wait and look). Control goes back to the agent when you click Release control, after 2 minutes without input from you, or afterttl_s(10 minutes by default, at most 60); the console then switches back to watching. Taking and releasing control is recorded in your team's access log. - Keys sends shortcuts that your browser keeps for itself, such as
Ctrl+W,Ctrl+T,Alt+Tab,Alt+F4or theSuperkey. - Clipboard: text copied on the desktop appears in the Clipboard box; while you are in control, Send to desktop puts the text from the box into the desktop clipboard, and you paste it there with
Ctrl+V. - If the connection closes – the sandbox fell asleep or ended, or the network dropped – the console shows why, and Reconnect opens it again. Watching does not keep the sandbox awake; your mouse and keyboard input while in control does, like any other activity. A console connection lasts at most an hour, then the page reconnects. A sandbox can have up to 4 consoles open at once, a team up to 8.
The console connects over a WebSocket on the portal's own address with a short-lived ticket that the portal issues only to a signed-in member of the team that owns the sandbox. If the live console cannot be used – a sandbox from an older build of the template without the VNC server, or a browser or network that cannot keep the connection open – the page falls back to screenshots refreshed about once a second (0.5–2 per second, only while the tab is visible) and says Live console unavailable – showing screenshots; you can still take control and click, type and scroll in the picture.
Who is a person and who is an agent is decided on our side: only requests from a signed-in portal user count as a person – an API key or an MCP connection is always the agent, whatever headers it sends. A live view stops after 30 minutes (reload the page to continue). Status and screenshots do not count as sandbox activity; actions do.
Limits
| What | Limit |
|---|---|
| Actions per request | 50; text in all type actions at most 8192 characters; waits at most 30 s in total |
| Requests per sandbox | actions 20 per second, screenshots 5 per second, status and control 10 per second – above that 429 with Retry-After. The limits count per account: the live view in the portal has its own allowance, so people watching never use up your agent's screenshots |
| At the same time | 2 screenshots of one sandbox (each at most 10 s); one batch of actions runs at a time and at most 2 more wait for it – above that 429 with Retry-After |
| Screenshot | at most 8 MB and 8192×8192 pixels – use JPEG or max_width for big screens |
| Control by a person | 10 minutes by default, at most 60; ends after 2 minutes without input |
| Live console (portal) | 4 open consoles per sandbox and 8 per team; one connection lasts at most an hour (the page then reconnects) |
Security
- Your input never becomes part of a shell command: coordinates and buttons are checked and passed as numbers, key names must match a strict pattern, and text is typed from standard input.
- Input runs as the desktop user, not as root. The desktop is reachable only through the API and the portal – no port of the sandbox is open to the internet, the portal's live console needs a short-lived ticket issued to a signed-in member of your team, and screenshots are never available at a public URL.
- A screenshot comes from inside the sandbox, so we check that it really is a PNG or JPEG of a sane size before passing it on, and send it with
X-Content-Type-Options: nosniffandCache-Control: no-store. - Treat what is on the screen as untrusted, like command output: a web page can show text that tries to instruct your agent.
Known limitations
- Through the API the agent sees screenshots, not video. The live console is only in the portal.
- The clipboard works only in the portal's live console and only for text; depending on the desktop's VNC server, characters outside Western European (Latin-1) may arrive as
?– type such text instead. In the screenshot fallback there is no clipboard. - Control by a person is kept in the memory of the server running the sandbox: after that server restarts, or after the sandbox sleeps and wakes up, the agent is in control again.
- After the sandbox starts or wakes up, the desktop needs a few seconds before
availableis true. - No sound, one screen of a fixed size (1280×800).