Computer use agent:What it operates,what it may do

What does "computer use agent" mean?

A computer use agent runs a loop that is easy to follow. It receives a task in plain language, say transferring invoice data from a PDF into an entry form. It then sees a screenshot of the machine, picks the next action, performs it, and asks for the result as a fresh screenshot. OpenAI describes the pattern in its own guide as send task, execute returned actions, return screenshot.

The actions themselves are exactly what a person does with mouse and keyboard. OpenAI's documentation lists click, double click, drag, move, scroll, keypress, type, wait and screenshot. Anthropic ships a longer set of seventeen member tools, including right click, triple click, zooming into a region at full resolution and holding a key for a set duration. That is the whole vocabulary, and that is the appeal: software a person can operate by hand can be operated this way too.

The contrast with ordinary automation is therefore sharp. An API exchanges data in a fixed format. It is fast and dependable, but the vendor has to offer it. A computer use agent goes through the interface instead, so it needs nothing from the vendor, but it is slower and less dependable. Where an API exists, the API stays the better choice. What an agent is in general, and what separates it from a chatbot, is in our piece on the AI agent.

Which screen tasks suit a computer use agent

What a computer use agent is good for shows in the examples the vendors pick themselves, and those look strikingly alike. Microsoft lists automated data entry, invoice processing and data extraction, and supplies three sample instructions. Move invoice data from a PDF into a form. Add several items to an inventory system. Return two values from a table as plain text. These are not future visions but the cases the vendors demonstrate with (and you can tell, because all three are tame and forgiving).

That tells you what a suitable task looks like. It repeats, it always runs through the same screens, it has a checkable result, and a mistake would be cheap to correct. Conversely, anything that calls for case-by-case judgement, touches money or contracts, or cannot be withdrawn after submission is a poor fit. Microsoft adds that computer use works best for autonomous agents performing tasks in the background.

One practical note that follows from the same documentation: extraction is a good place to start. It changes nothing in the target system, the result can be held against the source, and a wrong turn costs only a second run. So start with reading, not with writing.

Where a person must approve before a computer use agent submits anything

On approvals the vendors of computer use agents are unusually direct. Consequential actions must be confirmed and users kept in control of purchases, data transmission and destructive changes, writes OpenAI, and the confirmation belongs at the point where the risk arises. Anthropic gets more specific and spells out what a human should agree to. Decisions that might result in meaningful real-world consequences, plus any task requiring affirmative consent, so accepting cookies, completing financial transactions or agreeing to terms of service.

Microsoft has built a mechanism for it. When the agent needs a confirmation or is missing a detail, it sends a review request by email to a named reviewer, carrying the agent name, a link to the run and a field for the answer. The workflow stays paused, and if the stated time limit passes without a response, the run ends. Worth reading too: turning this oversight off does not make the agent autonomous. It still pauses when it needs confirmation, and with no reviewer configured the session simply fails.

And now the sentence that matters. In the same documentation Microsoft writes that model behaviour is probabilistic, so these requests do not fire in every situation where a person would want a pause, and may fire when no pause is needed. It then warns plainly not to rely on human review or clarification requests as a safety measure, or as a guarantee that the system always asks for human input before proceeding. We think that is the most important line in the whole documentation set, because it moves the safeguard to where it is dependable: into permissions and the allow list. An agent that technically cannot submit an order needs no question about whether it should.

Where computer use agents still fall short, from prompt injection to Citrix

The first limit of a computer use agent is an attack route with a name of its own. Prompt injection means hidden instructions in a web page, an image or a screenshot steering the agent somewhere else. In some circumstances the model will follow commands found in content even when they conflict with your instructions, Anthropic describes, and therefore runs classifiers that scan what the tools return to flag potential injections and make the agent check back when something looks wrong. The same warning appears at Microsoft, there tied to a recommendation for trusted, isolated environments. OpenAI puts the underlying stance most briefly. Treat screen content as untrusted, because text in a page cannot grant permission.

The second limit is technical, and it bites mid-sized companies harder than expected. Microsoft writes that password fields are supported on all websites and most Windows applications, but names app types where that may not hold: Electron, Java, Unity, games, command-line interfaces, Citrix and other virtualised environments. If your line-of-business software is delivered over Citrix, check that before anything else. Anthropic adds that small UI elements are tricky to hit and that heavy downscaling of high-resolution displays hurts click precision.

The third limit is expectation. An agent driving an interface works more slowly than an API and makes mistakes an API would not, because it interprets a picture instead of reading a data field. So every deployment needs a check on the result, and OpenAI says as much: set step, time or cost limits, support cancellation, and check the actual outcome. Plan for that and you get a useful tool. Skip it and you get a source of errors nobody notices.

If you are wondering where your own company should start, the wider picture helps: which tasks belong with AI at all is sorted in AI in mid-sized companies, the line between goal-driven systems and simple helpers in agentic AI, and what an office assistant delivers today in AI assistant for business.