Computer Use and Desktop Agents

You want an AI to do a task: book the cheapest refundable hotel room, or file an expense in the company portal.

If the service has an API (a programmatic door: a documented way for software to talk to software), this is easy — a script calls it.

But much of the real world has no API:

• a 2003 government portal that only speaks to a browser,
• an ERP thin client (a corporate app that runs in a locked-down window),
• Citrix-remoted desktop software,
• an in-house admin tool no one has time to wrap.

The only interface those programs offer is the one built for humans: pixels and a mouse.

So the question:

That is a computer-use agent. It needs no cooperation from the software it drives. It works wherever a human could work.

The catch — and this topic is mostly about the catch — is that the human interface is a terrible interface for a machine: slow to read, ambiguous, invisible in half its state, and full of traps.

A computer-use agent is simply

Four moving parts:

• Perception. The screen is captured (screenshot, sometimes plus the accessibility tree — the OS/browser's own list of "here are the buttons and what they mean"). A vision model reads it.
• Action space. The moves the agent may output: click(x, y), type(text), scroll, key presses, drag, wait, done. That is the entire vocabulary — the model is playing the computer through a mouse and keyboard, nothing else.
• Grounding. The hard middle step: turning "click the checkout button" into this box, at these coordinates, now. Where most failures live (section 5).
• Environment. A real VM or browser profile where actions have real effects — which is also why mistakes are real.

Compare with a scripted automation tool (old RPA — Robotic Process Automation, brittle click-recorder macros): RPA replays fixed selectors; break on any redesign. The computer-use agent sees the screen semantically, so it can survive a redesign — at the cost of speed, price, and certainty.

One sentence to keep:

Task: "Book a room, check-in 2026-03-14." Watch the loop at work:

code
Copy
1  screenshot          → sees a booking form, dates blank
2  find date field     → SoM labels the widgets: [47] is "Check-in date"
3  click [47]          → a calendar popup appears
4  screenshot          → calendar shows March 2026, tiny day cells
5  decide: type instead of clicking the tiny cell
6  type "2026-03-14"   → field shows the text
7  screenshot          → popup still open, field shows 2026-03-15 ✗
8  diagnose: the combobox needs Enter after typing
9  press Enter         → popup closes
10 screenshot          → field now reads March 14 ✓ (verify, do not assume)
11 proceed             → next widget: guests
...

Three lessons hide in those twelve lines:

• Step 7 is why verification exists. The agent thought it typed correctly. Only re-observation caught the off-by-one-day. "Screenshot → check → then continue" is the difference between agents and confident disasters.
• Step 5 is the hybrid instinct. Typing beat clicking a 14-pixel cell. A good agent constantly looks for the more structured move.
• Step 3 is grounding. "[47]" refers to a labeled box the harness drew; the runtime maps 47 back to real coordinates. The model never guessed pixels.

Each step costs roughly 1.5-5 seconds of real time. A task a human finishes in three minutes can be 40+ steps and several minutes — plus hundreds of thousands of re-read tokens. That is the economics you design around.

The right analogy: a gamer playing a game they have never seen, from screenshots only.

Picture an expert gamer sitting at your computer. But they are not inside the machine. They see only a live video of your screen (the screenshot stream), and they control only a mouse and keyboard — no cheat menu, no save editor.

• They are smart about games: they know what forms, buttons, dialogs, and checkout flows are, because they have seen ten thousand UIs. That is the model's prior knowledge.
• They cannot see state the screen hides: the loading spinner means the page is still fetching; the spreadsheet is a canvas (one undifferentiated image to them, not cells); the list is virtualized — rows off-screen simply do not exist to be seen. (Section 6's "state invisibility".)
• Their actions are irreversible and consequential: a click on "Pay now" is a click on "Pay now". There is no pause button. Hence the safety layer.
• Their speed is bounded by frame-taking: look, decide, act, look again. 1.5-5 seconds a move.

The loop, drawn from the gamer's seat:

code
Copy
   ┌──────────────┐   "screenshot"    ┌──────────────┐
   │   The screen │ ────────────────► │ expert gamer │
   └──────────────┘   (pixels only)   │  (the model) │
          ▲                            └──────┬───────┘
          │    click/type/scroll              │
          └───────────────────────────────────┘
     between moves: ground "checkout button" → (x,y) or box [47]
                     verify: did the screen change as predicted?

Production systems hand the gamer better tools whenever possible: the DOM/accessibility tree is like getting the game's debug overlay; MCP or an API is like being allowed to use the console. The whole engineering argument of section 7 is: give the gamer the overlay whenever it exists, and reserve raw pixels for when it does not.

Early GUI automation was deterministic: selectors, macros, RPA scripts, accessibility-tree walks. They were fast and brittle — a redesign broke every flow. The multimodal shift:

• 2023 (proof of concept): CogAgent (a 18B VLM trained for GUI perception), SeeClick and similar zero-shot grounding work, and WebArena (July 2023) as a reproducible browser-environment benchmark showed that agents could complete real web tasks with a vision model in the loop.
• 2024 (productized): Anthropic shipped computer use with Claude 3.5 Sonnet (Oct 2024): a loop where the model receives screenshots and emits mouse/keyboard actions executed in a container. OpenAI followed with Operator and the computer-using agent (CUA) model in January 2025, aimed at browser tasks with takeover-for-login (the human takes over at password/captcha moments); and 2025 brought native OS-level copilots (Gemini 2.x computer-use mode, Apple/macOS-adjacent automation frameworks) and Windows-focused agents (e.g., UFO-style UI automation).
• 2025-2026 (reliability engineering): hybrid action spaces that prefer structured APIs and fall back to pixels, RL-trained long-horizon GUI policies, agents that install and drive CLIs/APIs instead of clicking, and the rise of MCP (Model Context Protocol — a standard plug-in surface exposing app tools to agents) as the plug-in surface between agent and application.

OSWorld (April 2024) is the reference hard benchmark: real Ubuntu desktop/OS tasks across apps with execution-based success checks. Human success is roughly 72%; the first published agent submissions sat near 12%, and by late 2025 frontier computer-use models were reported in the 40-60% band — still well short of human reliability on long multi-app flows.

Read that spread honestly: computer use is working, and it is still behind humans on exactly the long flows that were its selling point.

Grounding is the make-or-break step, and it is where most "the model is smart but the agent is useless" failures live. "Click the submit button" must become one concrete target among thousands of pixels. Techniques in production order:

1. Set-of-Marks (SoM) prompting. Overlay numbered boxes on interactive elements (from the DOM or a detector) and have the model output an index, not coordinates. Cheap, robust, and still the most-used trick in browser agents. The gamer gets the debug overlay with numbered hotspots.
2. Coordinate output with binning. The model emits normalized coordinates or discretized bins (0-1000 — the picture is a 1000×1000 grid in the model's head, so outputs are resolution-independent integers); the runtime de-normalizes to pixels. Simple, but resolution-sensitive; relative clicks and drags amplify small errors.
3. Detector-assisted grounding. A specialized grounding model or SAM-style segmenter (a model trained to outline objects in images) proposes element boxes; the VLM picks among candidates. Better small-element accuracy at extra latency.
4. Accessibility tree / DOM priors. For browsers, combine the a11y tree (roles, labels, test IDs) with the screenshot; the model chooses a stable selector when available and a pixel target when not.
5. Verification before commit. Zoom-and-inspect the intended target region, check hover/tooltip state, and require that the observed post-action state changed as predicted — otherwise do not proceed. The "screenshot after every move" discipline.

Scale sensitivity is a documented pathology: early computer-use models were notoriously sensitive to display scaling (a click landing 40 pixels off on a 2x retina surface), which is why normalized coordinates plus SoM plus verification became standard.

The hybrid action schema production systems actually emit — prefer structured, fall back to pixels, and state the check that must come true:

{
  "step": 12,
  "thought": "The date field is a React combobox; typing into the DOM node is more reliable than clicking the calendar.",
  "action": {
    "kind": "dom_fill",
    "selector": "[data-testid=checkin-date]",
    "value": "2026-03-14",
    "fallback": { "kind": "click_then_type", "som_id": 47, "coords_norm": [412, 663] }
  },
  "expect_observation": {
    "check": "input_value_changed",
    "target": "[data-testid=checkin-date]",
    "value_regex": "^2026-03-14$"
  },
  "policy": { "side_effect": "reversible", "requires_confirm": false }
}

Every weakness of the loop, stated as the gamer experiences it:

• Latency stack: screenshot capture, image encoding, model decode, action dispatch, page settle. Realistically 1.5-5 s per step, so a 60-step task is several minutes with no parallelism.
• State invisibility: loading spinners, virtualized lists, deferred hydration (the page looks drawn but is still wiring itself up in the background), and canvas-drawn UIs (spreadsheets, PDF viewers, remote desktops — to a vision model, one big undifferentiated image) mean what the model sees is not the state.
• Error amplification: one wrong click changes the page, invalidating the plan; recovery often requires back-navigation that loses form data. The gamer pressed the wrong door and the level reloaded.
• Off-screen and temporal state: scroll-only content, session timeouts, captchas, cookie banners, and "takeover for login" flows break autonomy.
• Reward hacking on the web: models have been observed entering a mock/demo mode or reading the answer off a page rather than completing the flow (the gamer finds the strategy guide instead of playing the level); eval harnesses must restrict environment affordances.
• Safety: a general-purpose executor with typed credentials can purchase, delete, send, and post. Prompt injection via on-screen text (emails, comments, ad copy) is the dominant attack, and it is in-distribution for these agents: reading untrusted text on screens is literally the job. Unlike a text-only LLM, the agent acts on what it reads — that is what makes this class different and dangerous.

The honest engineering answer: use vision only where structure is absent.

Read the table top-down: every row is cheaper and more reliable than the one below, and the bottom row is "only when nothing else exists."

The 2025-2026 trend is agents that create structure: writing a script, installing a CLI, using a terminal, or asking for an API key rather than clicking through 40 dialogs. In the analogy: the smart gamer stops playing your level and writes a save-file editor. This is why "computer use" and "terminal use" converged into one automation product surface, and why benchmark suites added OS-app tasks alongside shell tasks.

The checklist for shipping, in plain-words → detail form:

• Sandbox first. Run the gamer in a borrowed machine, never your own: ephemeral VM or browser profile per task, no corporate credentials, egress allowlist, clipboard and filesystem controls, session TTL with kill switch.
• Confirm-on-execute. Classify actions (read/navigate vs purchase/send/delete/publish) and require human confirmation for irreversible ones, with a summary of what will change. "Pay 214 euros?" gets a human yes.
• Deterministic waits, not sleeps. Do not "wait 3 seconds and hope": poll for the expected observation (element present, value changed, URL matches) before the next action; retry with escalation on drift. The expect_observation field in the JSON above is exactly this contract.
• State journal + replay. Every step records screenshot, action, post-action observation; enables debugging, regression testing, and human takeover mid-task.
• Budgets and stall detection. Identical-action loops and oscillating navigation must trigger abort plus escalation (the gamer stuck on one door must raise a hand).
• Regression harness. Frozen task sets in disposable environments; run on every model or prompt change. Expect task success variance of several points between model versions — a "stronger" model can be worse at your flows.
• Metrics that matter: task success with human verification, steps-per-success, cost per accepted task, takeover rate, irreversible-action-blocked count, injection attempt detections.

Ship posture in one line: use pixels only where structure is absent, verify every action by re-observation, and gate every irreversible one behind a human.