ModelRefs / Computer Use — AI Glossary
Computer Use — AI Glossary
An agent capability where the LLM controls a computer — clicking, typing, and navigating GUIs — by observing screenshots.
Overview
Computer use (Anthropic's Claude computer use, OpenAI Operator, browser-use) enables agents to operate any desktop or web application without an API. It uses vision to read the screen and generates mouse/keyboard actions. More flexible than tool calling but slower and more error-prone.
Reference details
| Topic | agents |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: Every step is a screenshot, and screenshots are expensive
The loop is: capture the screen, decide, act, capture again. Each capture enters the context as image tokens — often on the order of 1,500 for a full screen — so a twenty-step task spends around 20 × 1,500 = 30,000 tokens on screenshots alone, before any reasoning text. Keep the whole visual history and that grows quadratically; keep only the latest frame and the agent forgets what it already tried. That tension, not model quality, is what makes these agents slow and expensive, and it is why production systems use an API wherever one exists and reserve screen control for the applications that offer none.
Commonly confused with
Computer use is a fallback interface, not an upgrade over tool calling. A tool call is a typed, validated request against a known contract; screen control is inference over pixels with no contract at all, so it is more general and strictly less reliable. Browser automation is the narrower case, where the DOM offers structure that a raw desktop does not.
When to use it
Reach for it when:
- Legacy or third-party applications with no API and no other integration path
- Exploratory and one-off automation, where building an integration is not worth it
- With human confirmation on any irreversible action, since misclicks are routine
Reach for something else when:
- Anywhere an API exists — it will be faster, cheaper and verifiable
- High-volume repetitive work, where per-step screenshot cost dominates
- Unattended operation on consequential actions, given error recovery is the weak point
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Computer Use — AI Glossary.
Frequently asked questions
What is Computer Use?
An agent capability where the LLM controls a computer — clicking, typing, and navigating GUIs — by observing screenshots.
What concepts are related to Computer Use?
Closely related concepts include agent, tool use, vision language model.