Computer use
A model operating a computer through screenshots, mouse and keyboard actions to complete tasks in apps that have no API.
Why it matters. Automates GUI workflows, but needs sandboxing and close supervision.
How it works
Computer use is a capability where a model controls a computer the way a person does: it looks at screenshots and issues actions such as clicks, typing, scrolling and key presses. Your code, or an environment the provider hosts, performs each action and returns a new screenshot, in a loop.
It is built on tool use, with the provider defining the action format. Makers differ in supported screen sizes, available actions and whether they host the environment. It is slower and less reliable than calling an API, because every step needs a screenshot and a model call.
Use it for apps with no API, inside a sandboxed virtual machine with limited accounts and permissions. Require human confirmation for purchases, deletions and anything sent outside.
Worked example
Illustrative: a 25-step task where each screenshot is about 1,500 tokens. If every step re-sends all earlier screenshots, total input is 1,500 x (1 + 2 + ... + 25) = 487,500 tokens, about $1.46 at $3 per 1M input tokens. Keeping only the last few screenshots cuts that sharply.
Common mistakes
- Running it on a real desktop with logged-in personal or admin accounts.
- Letting it act on instructions shown on web pages or in documents, which is prompt injection.
- Keeping every past screenshot in context, so cost and latency grow with each step.
See it on a real card: Feature support on Claude Opus 5.5.