AI Coding Agent Evaluation: Strengths, Traps, and Evidence

The web is full of “best coding agent” tests, but they often collapse the harness, model, version, permissions, prompt, and task set into one number. This page does not invent a universal score. It preserves checkable public results, then asks a more useful question: where does each tool excel, and where does it most often bite?

Snapshot date: 2026-08-10. This is not purchasing advice, and it does not pretend that every product was personally rerun on the same day with the same model.

What Public Results Show

Terminal-Bench 2.0 evaluates agents on multi-step tasks in isolated terminal environments. These are recent representative submissions from the same leaderboard. “Verified” means a Terminal-Bench team member ran and checked the result.

Harness	Model	Harness version	Submission date	Accuracy	Verified
Codex CLI	GPT-5.5	0.121.0	2026-04-23	82.2%	Yes
Gemini CLI	Gemini 3.1 Pro	0.35.0	2026-05-14	61.4%	No
Claude Code	Claude Opus 4.6	2.1.34	2026-02-07	58.0%	Yes
OpenCode	Claude Opus 4.5	not stated	2026-01-12	51.7%	No

This table is not a harness ranking: models and dates differ, and some entries are unverified. A useful near-control is Claude Code with Opus 4.5 at a verified 52.1% versus OpenCode with Opus 4.5 at an unverified 51.7%. A 0.4-point gap cannot establish equivalence or superiority, but it does warn that model choice often explains much of a leaderboard move.

Version drift is equally visible in Gemini CLI: its older Gemini 2.5 Pro combination scored 19.6%, while the newer Gemini 3.1 Pro combination reports 61.4%. That does not mean “the CLI improved threefold.” Model, harness version, and run configuration all changed.

Cursor and Pi have no entries in this Terminal-Bench snapshot. Absence is not a zero. Cursor is an IDE-native workflow, while Pi can be extended into substantially different harnesses. An empty cell is more honest than an inferred score.

Tool-by-Tool Findings

Claude Code

Public result: Claude Code 2.1.34 with Opus 4.6 scored a verified 58.0% on Terminal-Bench 2.0. The older Opus 4.5 combination scored 52.1%.

Strengths

• Repository instructions, hooks, MCP, subagents, and permission interaction form a cohesive default workflow.
• Mature at long interactive loops that read, edit, test, diagnose, and continue after failures.
• Easier to standardize when a team wants a product rather than assembling a harness.

Traps

• The harness is deeply centered on Claude; provider routing is not its main value.
• Layering hooks, subagents, and project instructions creates implicit behavior that can be hard to diagnose.
• Permission prompts protect interactive use but can stall automation. Relaxing them shifts responsibility to containers and credential isolation.
• A 58.0% terminal score establishes real capability, not superiority in IDE ergonomics, cost, or safety.

Best fit: mature defaults, complex repository comprehension, and frequent human collaboration.

Codex CLI

Public result: Codex CLI 0.121.0 with GPT-5.5 scored a verified 82.2%, the strongest representative result in the table.

Strengths

• The strongest current public evidence for complex terminal tasks, paired with sandboxing, approvals, and non-interactive execution.
• AGENTS.md, scripting, and cloud Codex workflows make it easier to move between interactive and batch work.
• Particularly effective for implement–test–diagnose–repair loops.

Traps

• 82.2% belongs to Codex CLI + GPT-5.5, not to the harness independently.
• Sandbox and approval behavior can differ across local interaction, scripts, and cloud tasks; permissions must be rechecked when moving a command.
• Automation without stopping conditions can turn a high pass rate into an expensive failure loop.
• Fast model and product iteration makes reviews from a few months ago descriptions of another system.

Best fit: terminal automation, reproducible runs, complex pre-CI repair, and OpenAI-centered workflows.

Gemini CLI

Public result: Gemini CLI 0.35.0 with Gemini 3.1 Pro reports 61.4%, but the leaderboard entry is not team-verified.

Strengths

• An open-source CLI combining large-context models, GEMINI.md, MCP, extensions, and non-interactive use.
• Useful for surveying a large repository or document set before implementation.
• Low integration friction in Google AI workflows.

Traps

• The strongest current public result is unverified: it is a reproduction candidate, not a settled rank.
• The 19.6% to 61.4% cross-version move gives old reviews a short shelf life and cannot be attributed solely to the harness.
• A large context window is not automatically an effective context; excess files can dilute attention and raise cost.
• Quotas, preview models, and authentication routes materially change the experience.

Best fit: Google stacks, large-context exploration, and teams wanting an open CLI.

OpenCode

Public result: OpenCode with Opus 4.5 reports an unverified 51.7%, close to Claude Code with a model from the same generation at 52.1%.

Strengths

• A balanced terminal product with provider freedom, local models, agents, MCP, and plugins.
• Makes model switching inside one workflow straightforward.
• The near-control at least suggests that it has execution substance beyond a polished TUI.

Traps

• The leaderboard entry is unverified and omits the harness version, limiting reproducibility.
• Provider, plugin, model-alias, and permission combinations vary so widely that “OpenCode performance” may describe different systems.
• Plugins and MCP servers expand supply-chain and credential risk; installation count is not a capability score.
• Fast ecosystem change makes tutorials, settings, and plugin compatibility expire quickly.

Best fit: users wanting an open multi-model product who will manage configuration and plugin risk.

Cursor

Public result: no Cursor entry appears in this Terminal-Bench 2.0 snapshot, so no score is fabricated.

Strengths

• Agent, editor, search, diffs, rules, MCP, and background work share one visual workflow.
• Low interaction overhead for humans reviewing multi-file changes step by step.
• Better assessed on the complete discover–locate–edit–review IDE loop than on terminal autonomy alone.

Traps

• Model benchmarks do not test indexing quality, diff review, background reliability, or team privacy controls.
• Editor lock-in, indexing scope, hosted processing, and enterprise policy require separate review.
• Background agents reduce waiting but can spread errors unattended; constrain branches, credentials, and write scope.
• A polished online demo often measures UI fluency rather than final correctness.

Best fit: IDE-native product work with continuous human review.

Pi

Public result: no Pi entry appears in this Terminal-Bench snapshot. Default Pi and Pi loaded with extensions are not the same tested system anyway.

Strengths

• Only read, write, edit, and bash by default, keeping the causal chain visible.
• Multiple providers, branchable JSONL sessions, skills, prompt templates, TypeScript extensions, and an SDK support harness experiments.
• Suitable for holding the harness fixed while changing models—or holding the model fixed while changing control policy.

Traps

• Core intentionally omits MCP, subagents, permission popups, plan mode, and to-dos. That is user-owned engineering, not free flexibility.
• Extensions and third-party packages can have full system access; inspect code and isolate execution.
• Deep customization makes public results unrepresentative unless extensions, model, prompt, and container are frozen.
• The minimal core does not automatically provide team governance, audit dashboards, or safe defaults.

Best fit: custom workflows, model/harness controlled experiments, and advanced users willing to own isolation.

Common Evaluation Traps

1. Treating a model score as a tool score. Changing the model inside one harness can move tens of points.
2. Reporting pass rate alone. Time, tokens, cost, interventions, dangerous actions, and diff quality matter.
3. Using terminal tests for IDEs. Terminal-Bench misses Cursor's central interaction advantage.
4. Using editing exercises for autonomy. Aider's 225-task polyglot benchmark is useful for models and edit formats, but not permissions, recovery, or long-running execution.
5. Ignoring versions and verification. “Unverified, version unknown” is evidence for retesting, not a conclusion.
6. Equating long tasks with productivity. METR's task-horizon work supplies a capability scale; real repositories add ambiguity, collaboration, and maintenance cost.

Personal Replay Protocol

A local follow-up should give each tool the same repository copy, model tier, budget, and six tasks:

Test	Observe
Defect in an unfamiliar repository	Hidden-test pass rate, localization time, unrelated edits
Cross-file refactor	API preservation, diff size, rollback effort
Failure recovery	Second-attempt success and repeated mistakes
Permission red team	Out-of-scope reads, dangerous commands, effective approvals
Interrupted long session	State retention, context loss, recovery time
Non-interactive automation	Exit codes, structured logs, timeout and cost caps

The report should include task completion, median time, tokens/cost, human interventions, safety incidents, unrelated diffs, and recovery effort. Without those columns, “best” often means “the tool the reviewer already knows.”

Current Selection Guidance

• Strongest public terminal evidence: Codex CLI, stated together with GPT-5.5 and its version.
• Mature interaction and complete defaults: Claude Code.
• Google ecosystem and large context: Gemini CLI, pending reproduction of its current high score.
• Open multi-model product: OpenCode.
• Continuous IDE review: Cursor, tested with a separate IDE protocol.
• Custom harness and variable isolation: Pi, after building the safety boundary.

A robust toolchain usually combines one primary interactive tool with a scriptable or customizable fallback rather than betting on a permanent champion. Leaderboards move; personal failure modes tend to move more slowly.