Computer use

Computer use lets the agent drive native desktop applications on macOS and Windows through real mouse, keyboard, and screen capture—when integrations, browser automation, and shell are not the right tool for the job.

Actions run on the computer where CoWork OS is running, using that computer's local permissions. Computer use does not provision a separate hosted computer or virtual machine; cloud sandbox tools are a separate execution surface.

This page is the authoritative product guide for the feature. For a short summary, see Features → Computer use.

What it is for

Use computer use when the task clearly requires a desktop GUI that is not exposed through a stable API or through the in-app browser tools—for example:

• Operating a native app’s windows, menus, or dialogs
• Filling fields or clicking controls that only exist in a desktop UI
• Short, bounded flows where scripting the UI is impractical

Prefer instead when possible:

• MCP connectors and APIs for SaaS and internal systems
• browser_* tools and the Browser Workbench for web surfaces, website testing, normal-user browser flows, Browser V2 snapshot refs, diagnostics, screenshots, annotation, and visible in-app browser automation
• run_command / scripts for file, git, and CLI workflows
• open_application to launch an app; pair with screenshot/click/type_text only when the agent must then interact with that app’s UI

The planner and tool policy treat the computer-use lane as a controlled, last-resort path: tools stay deferred unless the task signals native desktop GUI intent (for example Calculator, System Settings, or “click the OK button in the native dialog”). That keeps routine coding and web work from accidentally taking the desktop-control path.

If the task is not asking CoWork to click or type, but instead to understand a vague on-screen reference like what is this, what's on the right side, or why is this failing, see Chronicle. Chronicle shares the same Screen Recording prerequisite but resolves local screen context rather than driving the mouse and keyboard.

Platform requirements

• macOS and Windows desktop builds are supported. Linux/headless builds do not expose this tool family.
• macOS uses a bundled Swift helper with Accessibility and Screen Recording permissions.
• Windows v1 uses a bundled PowerShell/Win32 helper for visible, non-minimized windows. Some protected or elevated apps may block capture or input unless CoWork is running with comparable privileges.

Optional Codex Computer Use MCP connection on macOS

An installed Codex/ChatGPT desktop Computer Use plugin can also be connected as a local MCP server. This is an unofficial integration with an installed desktop runtime; it is not a bundled CoWork engine or the OpenAI computer-use API.

1. Find the newest installed unified-computer-use/<version>/.mcp.json under ~/.codex/plugins/cache/openai-bundled/ and read its mcpServers.cua_repl entry.
2. In CoWork's MCP Servers, add a stdio server named codex-cu, copying the entry's command, args, and env exactly. Enter environment values as KEY=value, one per line. Use the installed paths rather than a version or username copied from someone else's configuration.
3. Connect the server and verify that its js and js_reset tools appear.
4. Ask CoWork: Use the codex-cu MCP tools to operate Calculator in the background, calculate 12 × 12, and verify the displayed result. CoWork exposes the tools as mcp_js and mcp_js_reset with the default MCP prefix.

Keep the desktop runtime installed and available, including its Computer Use helper and macOS permissions. Updates can change its paths or protocol; refresh the connection from the installed manifest if needed. Registering this server does not automatically replace CoWork's built-in desktop tools. To prefer it, add a behavioral rule in Personality → Instructions for native macOS GUI work. CoWork recognizes js and js_reset from the locally configured @oai/cua-repl stdio entry when its enabled surfaces include computer. Unlike other connected MCP tools, which are always available, those two are offered only for native GUI requests, and they stay available within step budgets for those tasks.

CoWork supports the plugin's approval-only form elicitation over stdio. The Ask for approval profile offers Allow for this task on the first engine permission. Each app then asks once for consent covering reads, clicks, typing, and drags for that task. Subsequent calls reuse consent; different apps and tasks ask again. Grants are kept in memory and invalidated by changed configuration, permissions, or access profile. Hard denies, explicit rule asks, and semantic review remain enforced on each call. Audio consent and other forms remain per operation. No grant changes a workspace rule or survives a restart. General data forms and URL elicitation are cancelled. Calls are serialized so consent belongs to the calling task.

Under effective Full access, routine engine calls and validated app-consent requests proceed without CoWork approval cards. Policy denies and OS permissions still apply. Audio and other special consent forms do not gain this exception. The access choice belongs to each task: selecting Full access in the Home composer does not change a task already created with Ask for approval.

Access-profile and operating-system boundaries

Computer-use tools must be exposed by the task's effective access profile and remain subject to administrator policy, tool restrictions, hard key-combination blocks, and action-specific approval. Selecting Full access does not grant macOS Accessibility or Screen Recording; those are independent operating-system permissions. Conversely, granting those OS permissions does not widen the task's filesystem, network, or command-tool profile. A remote or fallback computer-use route cannot use its transport to bypass the local profile.

macOS permissions

Two system permissions gate computer use:

Where to enable in the product

1. Open Settings → Tools → Computer use.
2. Use the shortcuts into System Settings to enable Accessibility and Screen Recording for the computer-use helper binary shown in settings.
3. After changing Screen Recording, quit and restart CoWork OS if capture still fails—macOS sometimes caches the old state until restart.

If a tool returns an error mentioning screen capture timeout or permission, re-check Screen Recording for this app and restart.

Windows behavior

Windows computer use targets one visible, non-minimized top-level window at a time. The helper enumerates native windows, captures the selected window region, and sends mouse/keyboard input through Win32 APIs.

Known v1 limits:

• Minimized windows are not controlled; restore the target window first.
• Apps running as administrator may require CoWork to run as administrator.
• Some games, protected apps, or anti-cheat surfaces may block capture or input.
• If background capture/control is unavailable, actions may briefly use the foreground mouse and keyboard.

Session model (one active session)

Computer use runs under a single global session on the machine:

• Only one computer-use session is active at a time; starting control for a new task coordinates with the session manager.
• Esc aborts the active computer-use session so you can interrupt quickly without hunting in the UI.
• Sequential execution and a shortcut guard reduce the chance of overlapping actions or global hotkeys firing at the wrong time during automation.

The session is torn down when the task finishes or the session ends cleanly after abort.

Built-in tools

All of these are part of the computer_use built-in tool family. They are registered together and can be enabled or prioritized alongside other built-in categories in Settings → Tools → Built-in tools.

Key contract details:

• Call screenshot() first.
• All action coordinates are relative to the latest screenshot, not the whole desktop.
• Successful actions return a new screenshot and captureId.
• Passing a stale captureId fails with a refresh instruction.

Related tools: open_application

open_application can launch native apps by name or bundle id/path where supported. For native GUI workflows, policy may allow open_application in the same steps as screenshot/click/type_text so the agent can start the target app before driving it. On macOS, that is separate from shell-based AppleScript or one-off scripts: when the goal is GUI interaction, the product steers toward the dedicated computer-use tools (and open_application when needed) rather than run_applescript as a first choice.

Routing and planner behavior (operator mental model)

Rough order the stack encourages:

1. Structured integrations (MCP, APIs, mail, channels).
2. Browser tools for web UIs.
3. Shell and file tools for repo and CLI work.
4. open_application when the missing piece is “the app is not running.”
5. native_app (macOS, background; see Working with apps in the background) or screenshot + follow-up computer-use actions when the task still requires native GUI interaction.

Native desktop GUI intent is detected from the user goal and step text (not only from generic words like “browser” in strategy headers). Explicit mentions of native apps, windows, dialogs, or on-screen UI tend to unlock the computer-use lane; purely web or repo tasks should not.

Settings checklist

1. Built-in tools: Confirm the computer_use category is enabled if you want the agent to use this lane at all.
2. Permissions / platform status: On macOS, Accessibility + Screen Recording granted for the helper binary; restart after Screen Recording changes if needed. On Windows, confirm the helper is installed and the target window is visible/non-minimized.
3. Operational model: On macOS the default is Background only: native_app targets an exact app window without taking the foreground, pointer, keyboard or clipboard (see Local background app control). Foreground modes are explicit opt-ins; on Windows, expect the visible controlled-window loop centered on screenshot() and the latest captureId.
4. Chronicle: If your goal is contextual screen understanding rather than GUI control, enable Settings > Tools > Chronicle, keep the dedicated Chronicle built-in tool category enabled, and test screen_context_resolve before forcing a computer-use flow.

Security and abuse considerations

Computer use is high trust: a mistaken or malicious task could operate any UI your user can reach. Mitigations include:

• Esc abort and single-session sequential execution
• Helper-targeted permissions with inline bootstrap on macOS; visible-window constraints on Windows
• Policy that keeps the computer-use lane off the default path unless GUI intent is clear
• Blocklisted key combinations that could disrupt the session or OS
• Private Browser Workbench prompts and filled documents block native computer control, including captures that overlap the private flow. Wait for prompt clearance or full navigation/closing the protected tab; do not route around the guard. See private browser safeguards.

For how this fits the wider tool-risk model, see Security guide → Computer use.

Troubleshooting

Implementation map (for contributors)

---

See also: Chronicle, Architecture, Features, Security guide, Operator runtime visibility.