Agent harness audit — September 14, 2026

This audit tests concrete failure modes in CoWork's execution and evaluation harness. It does not establish a ranking against every competing product. All results refer to this local working checkout; publishing and live provider benchmarks require separate evidence.

Latest validation: 7,761 full-suite tests, 777 harness tests, and six replay fixtures passed. Electron, daemon, and CLI builds passed; isolated shutdown processes passed. See the final validation section for scope and limits.

Evaluation standard

The audit prioritizes durable progress, correct cancellation, one result per tool call, verification before success, and tests that cannot pass by skipping the behavior they claim to cover. CoWork's multi-provider support and existing session protocol remain the foundation.

The distinction between a transcript claim and an actual outcome follows Anthropic's agent evaluation guidance. The emphasis on inspectable environments and enforceable constraints is also consistent with OpenAI's harness engineering account. These sources inform the audit criteria; they do not validate CoWork's code.

Findings and approach comparisons

Repeatable checks

Run npm run qa:harness with Node 24+, native better-sqlite3 for that Node ABI, and the sqlite3 CLI installed. This runs runtime, verification, compaction, checkpoint, session, replay, timeline, risk, and evaluation-runner regressions, then the built-in deterministic replay fixtures. Temporary databases and local fake HTTP services isolate tests from the user's application data. Verifier/helper capability tests and security-scanner evidence-status regressions are included.

PR, release, and nightly workflows use this same deterministic harness gate. The optional live battery reports not_configured when all prerequisites are absent; partial configuration or a failing configured run fails the gate. No production corpus is built or implied on a fresh nightly runner. Executable workflow-shell tests cover those states and failed commands behind log piping. The PR path filter includes runtime, session, database, shared protocol, eval, and QA changes. A clean CI checkout has no production task corpus: it must not claim production replay coverage from an empty database. Real timeline metric enforcement remains available through qa:timeline:enforce against a populated database; the empty-database CI step is replaced by executable timeline tests.

Hook evaluation now aborts unresponsive HTTP requests and leaves pending approvals blocked by default. Only an explicit --mode hooks --auto-approve enables automatic approval responses; use that option only for isolated, intended test workspaces. --allow-empty can explicitly permit an empty run, but its recorded status is skipped, with zero executed coverage. Invalid modes, flags, and contradictory fixture selectors fail instead of silently selecting another mode.

Evidence and limits

The initial Electron TypeScript build and 126 focused tests passed. Adding adversarial verification tests exposed 13 failures before the fix. After the verification changes, the expanded baseline passed 222 tests in 25 files. The first integrated gate passed 295 tests in 29 files plus six replay fixtures. Seven additional malformed/unsupported assertion probes failed before validation was added and passed afterward. A full-suite run exposed five stale messaging expectations; those tests were updated to assert the current receipt and context contract, with no change to the corresponding production entrypoints. Original audit validation (before the continuation below):

The Electron build caught a usage-optional narrowing error that the root type check missed; all three backend builds passed after correction. Independent cross-review also found untracked document/canvas/report helper responses; six response-shape tests now exercise those actual helper paths.

The read-only helper boundary covers verifier and researcher roles, including entropy sweeps. It disables shell, writes, deletion, and external actions even under broad custom profiles and saved allow rules. External ACP runtimes are removed from these workers because they cannot enforce this process's policy boundary. These workers inspect supplied/local evidence and must return partial or failure when fresh command/build evidence is missing.

Detailed local logs, original-file snapshots, the ranking experiment, and a session-only patch are retained at /tmp/cowork-harness-audit-20260914/. A validation manifest records source fingerprints over the preexisting dirty checkout. No publish, live-provider benchmark, or released-client validation was performed.

The tracked JSON files under scripts/qa/eval-cases/ currently include scenario specifications with assertions that are not executable by the replay grader. They are not counted as measured coverage. A follow-up capability benchmark should provide isolated environments and outcome graders for coding, research, documents, browser work, and multi-agent handoffs; repeat trials per provider and report success rate, cost, latency, and confidence intervals. Runtime unit and replay tests do not substitute for those measurements.

Routing experiment

A fixed-candidate experiment compared the production ranker with two lexical alternatives on the 27 prompts in resources/skills/_evals/routing-cases.jsonl. All 152 bundled definitions were supplied as candidates; installed-tool eligibility, model choice, and end-to-end task success were not measured.

The hybrid improves aggregate retrieval but regresses a previously passing scope-control/bug-fix prompt. It is not enabled by this patch. Promoting a ranking change needs a larger separate test set, eligibility-aware checks, and preservation of the existing invocation/security boundaries. The old advisory skills:check score is also not a runtime benchmark: its standalone overlap scorer includes negative examples as positive matching text and differs from production ranking. Its observed 86.20% hit / 55.50% forbidden-misfire figures must not be interpreted as production task rates.

Continuation: remaining audit candidates

The follow-up audit reproduced an additional persistence gap in the fatal tool dispatch path: ordered tool results were appended to the local turn, but a fatal error bypassed the normal copy into conversation history. Both the main and follow-up loops now save that history before propagating the fatal error.

The executor regression checks the actual emitted snapshot, writes it through the filesystem checkpoint store in a subprocess, kills the writer after commit, and loads the checkpoint in a fresh process. A fresh executor restores both tool results without redispatching either call. All 138 executor failure tests passed in this continuation. This proves committed snapshot recovery, not a live daemon restart or recovery of external side effects.

Continuation evidence is retained at /tmp/cowork-harness-candidates-20260914/.

Provider arguments

OpenAI, generic compatible, Azure, OpenRouter, xAI, Ollama, and standalone Pi converters now preserve malformed calls as structured rejections. Explicit {} and native argument objects are valid; missing, blank, malformed, scalar, array, and null arguments are rejected. The scheduler rejects them before preparation, policy, approval, schema repair, or tool execution and preserves their call IDs and valid siblings.

The alternatives were conversion failure (which loses valid siblings), silent empty-object defaults (which can execute unintended operations), or an explicit rejection marker. The marker survives snapshot serialization and restore. A real executor test confirms that a malformed write never reaches preparation while its valid read sibling executes and both protocol results are retained. Six additional provider-specific tests exercise Azure chat/Responses streams, Azure and xAI Responses conversion, OpenRouter, and native Pi objects. They also exposed and fixed Azure's streamed fallback labelling tool calls as end_turn. Ollama's local batch limiter also keeps malformed mutations in the rejection path instead of reporting them as successfully deferred; valid calls retain their normal execution and deferral limits.

Independent checkpoint writers

The filesystem store now holds a per-task SQLite transaction across checkpoint replacement. Its zero busy timeout and asynchronous retry leave the event loop free. The OS releases the lock when a writer dies. Custom PID/stale-file lock reclamation was rejected after review exposed ownership and deletion races.

Locks live in the app-owned ~/.cowork/checkpoint-locks directory, keyed by canonical workspace path and task ID. Workspace-preseeded symlinks cannot redirect SQLite writes. COWORK_CHECKPOINT_LOCK_ROOT can select an isolated root; all processes sharing checkpoints must use the same root. Locks cover processes under the same user, including different profiles and Node/Electron entrypoints; these tests do not establish cross-host filesystem locking. Lock databases are retained to avoid replacing a lock inode while another process still holds it.

The store preserves the freshest valid generation, refuses a late older snapshot, fsyncs the recovery copy before replacement, and preserves a valid previous checkpoint when current is corrupt. Eighteen checkpoint/recall tests pass, including four simultaneous subprocess writers and a real writer killed after temporary-file sync but before current-file replacement.

Follow-up acceptance

Queue-only delivery now means that the native worker has durably incorporated the input into its transcript. It does not mean that the requested work has completed. The initial acceptance snapshot retains the full pending payload alongside a consumed message ID. Only then is the receipt acknowledged and the pending item removed. If acknowledgement fails, restart can retry only the receipt, without redispatching the provider turn. Keeping both in the first snapshot avoids depending on a second successful write after a receipt failure.

Acknowledging before sendMessage() was unsafe because incorporation could still fail; acknowledging only after the whole turn was unsafe because a late failure could cause duplicate execution. The explicit incorporation boundary separates those outcomes. Failed acceptance persistence rolls back the attempted input before saving a retry, preserving earlier user turns and the full image and quoted-message context. Pending queue-only items remain present during handoff, busy-worker races retain their original payload, and duplicate receipt retries are serialized under the executor's lifecycle mutex.

The boundary covers native regular and chat follow-ups, queued injection, pending skill input, and handled goal commands. External ACP runtimes do not expose durable prompt acceptance; new queue-only requests to those runtimes are rejected with an instruction to use a normal follow-up. Existing normal follow-ups retain their behavior. Target receipt persistence is authoritative; parent timeline projection failures do not undo accepted delivery.

Regression tests exercise snapshot failure, callback failure, ACK-only retry, receipt replay, preserved payloads, busy-worker handoff, chat incorporation, and ACP rejection. These checks use isolated state and fake providers. They do not establish exactly-once external side effects or a live daemon restart.

The three remaining audit candidates are addressed by these changes. The routing experiment still needs a separate holdout benchmark, and no local test suite can establish a universal best-in-class ranking.

Final continuation validation

Luna Max agents implemented and cross-reviewed the provider, acceptance, and checkpoint changes. Final validation includes the last acknowledgement-ordering fix. The expanded qa:harness command retains these regressions for future runs.

Logs, source fingerprints, and continuation-changes.patch are retained in the continuation evidence directory above. The patch compares against preserved before-files in this already dirty checkout. The missing continuation-start copy of SessionRuntime.ts was reconstructed from the prior audit patch and verified against its recorded SHA-256; baseline-provenance.json identifies that exception. No commit, publication, live-provider benchmark, or released client validation was performed.

Final audit: graceful desktop shutdown

A fresh baseline on the existing working checkout passed 702 harness tests in 49 files and six deterministic replay fixtures. The development log was stale (September 6), but it exposed a shutdown failure still present in current source: 2026-09-06T13:55:19.795Z reported The database connection is not open while AgentDaemon.shutdown() attempted to persist interrupted task state.

The desktop entrypoint closed the database before calling the async daemon shutdown, and its async before-quit listener did not defer Electron's quit. Simply swapping those calls would repair database ordering but would still let Electron exit before asynchronous cleanup. The selected fix explicitly prevents quit, serializes cleanup, and retries quit only when cleanup settles. Repeated quit requests join the existing shutdown. Each service failure is reported without skipping subsequent task persistence and storage cleanup. Agent shutdown runs before lifecycle listeners, MCP, memory, and the database are torn down.

Coordinator regressions cover delayed task writes, repeated quit requests, synchronous/asynchronous/hung services, and a failing error reporter. Daemon regressions verify interruption persistence after monitor stop failures, concurrent shutdown, and work still awaiting admission. Three isolated real Electron processes checked normal, failed, and timed-out shutdown while issuing repeated quit requests. Normal shutdown observed snapshot -> interrupted -> database close -> will-quit -> quit. A failed stop still attempted task persistence, but kept shared dependencies alive until exit. A timed-out worker likewise reached quit with the database still open. All processes exited with code zero. This validates the Electron quit boundary; it is not a full application restart with live provider work.

Shutdown cancellation now waits for the executor lifecycle mutex to drain. The daemon fences new execution and follow-ups, tracks task starters that are still awaiting setup, and prevents late completion callbacks from replacing interrupted state. Failure to reach quiescence within a bounded wait is explicit; MCP, memory, and database teardown is skipped until process exit. Ordinary user/system/tool cancellation stays responsive and does not wait on a mutex that a self-cancelling tool could already hold.

Evidence and before-file snapshots are retained in /tmp/cowork-harness-final-audit-20260914/. The initial smoke fixture incorrectly loaded Electron through its npm file path; that fixture-only import was corrected to Electron's built-in module before the successful run.

Current comparison criteria

Official product documentation was checked for Codex sandboxing, Codex approval boundaries, Claude Code security, and Cursor run modes. These support comparing enforceable permissions, execution environments, and human intervention. They do not establish comparative task success rates. For CoWork, the acceptance criteria are cancellation without new dispatch, independent sandbox and approval enforcement, bounded child concurrency, durable steering and attachment recovery, causally complete compaction, and success tied to current execution evidence. Each environment needs its own validation: mocked provider, local process, live app, remote daemon, and release.

Further reproduced boundary failures

Scheduler review also found that cancellation could prepare untouched sibling calls, and thrown preparation/finalization/summary hooks could discard protocol results. Cancellation now fences preparation. Hook failures retain a fatal error separately from one correlated tool result per call; already executed siblings keep their results and unstarted later batches are skipped. Fourteen scheduler tests cover this boundary, including later mutations after a fatal hook error.

Durable queued attachments and headless shutdown

Queued follow-ups now persist attachment bytes before acknowledging acceptance. Private files and manifests use opaque references bound to the task and message, atomic writes, checksums, validated byte counts, and MIME-compatible extensions. Recovery validates both receipt-only and snapshot-backed queues. Missing or corrupt media produces an explicit recovery error and omits the incomplete message, so text is not silently dispatched without its attachment. A corrected resend can proceed. Tests exercise the real executor image-loading path as well as receipt recovery, avoiding a metadata-only success assertion.

The store uses a 2.5 GiB bound with explicit backpressure; it never evicts undelivered attachments to accept new work. Synchronous persistence preserves queue acknowledgement ordering, at the cost of blocking during the bounded file copy. Delivered receipts release files. Task deletion and session pruning capture validated references before deleting the database record and release only after successful deletion. Conservative maintenance handles crash-left files after a 24-hour grace period, retaining authoritative receipt references and failing closed on uncertain ownership. This does not introduce a new restriction on legitimate user-selected upload source paths; broader upload provenance remains an existing application boundary, outside this store's opaque-reference checks.

The Node-only daemon also drains agent execution before releasing its control plane, channels, MCP servers, memory, or database. Repeated termination signals share one shutdown; failed or timed-out cleanup exits nonzero and skips dependent releases. Uncaught exceptions retain a nonzero exit status. An isolated SIGTERM subprocess exercises the shared coordinator's timeout path; this is helper-level process evidence, not a full daemon restart under live provider load.

Final local validation for this pass

The full suite was repeated after attachment cleanup and headless shutdown integration. The initial pass had 7,756 passing tests; the final 7,761 includes the last cleanup and shutdown regressions. Formatting-only changes were applied before the final harness pass. These counts measure deterministic local tests; fixture coverage is not production-corpus or live-provider coverage.

Evidence is in /tmp/cowork-harness-final-audit-20260914/, including logs, source-manifest.json, and session-changes.patch. Most original files were snapshotted before edits. Two test baselines were reconstructed from the previous pass and verified against its recorded hashes; three clean-file baselines were verified against HEAD. The already-dirty context-overflow test lacked a safe original snapshot: its final hash is recorded, and it is explicitly excluded from the session-only patch. This limitation does not affect its executed tests, but the patch must not be treated as a complete isolated commit. No unrelated working-tree changes were reset or staged.

The independent final security review found no remaining material regression in this pass. Its last finding concerned attachment cleanup using separate offset queries: task reordering could omit an owner. The final implementation reads the owner set in one SQLite query. A regression with more than 500 tasks exercises that omission scenario and confirms referenced bytes survive cleanup. The final full-suite and harness results above include this test.