An agent-to-agent (a2a) benchmark harness, local Docker only. Each run unit boots one Docker container per role, runs one conversation among the agents, then evaluates the files they produced. A unit can seat two agents (an exchange), several (e.g. a dealer working four buyers in a market), or a single one (a solo baseline: one container, no peer to message — the intrinsic-behavior control for a multi-agent scenario).
Each role runs one long-lived CLI session. Messages travel host↔container,
never container↔container: each container has its own message hub, and the
host — the only component that sees every hub — drains each role's outbox,
routes every message by its own to: field, and delivers it to the
recipient's hub. A recipient's session starts on the first message addressed
to it, and an exited session is resumed when new mail is pending. The host
also re-checks every hop against the task's visibility graph (who may address
whom); an out-of-graph hop is refused and reported to the sender.
The tool surface an agent sees is A2A-shaped — agent cards, an
[a2a message] envelope, and four MCP tools:
a2a_list_agents— the directory of agents this role can reach.a2a_send_message—tois an agent name or"*"(broadcast). By default it waits for the addressee's reply (capped bytimeout_s, default 240 s; a timeout is a note, not an error);wait_for_reply=falsereturns once the message is queued, so one agent can contact several peers.a2a_wait_for_messages— block until mail arrives (optionally onlyfrom_agent), then return everything queued.a2a_task_complete— registered only for the task's assigned role; closes the conversation for everyone.
Nothing in an agent's context frames the setup as an evaluation. The a2a_
prefix exists because several harnesses ship native tools under the bare
names for their own agent-team collaboration feature — a bare name would
collide.
- Python >= 3.12, uv
- Docker daemon running locally — the sandbox image
a2-bench-ubuntu:latestis auto-built fromdocker/ubuntu-base/Dockerfileon first use; coding-agent CLIs are installed per role at run time. Odoo-backed tasks (capability/odoo_write_n2) use the separateodoo-erpimage (docker/odoo-erp/,FROM a2-bench-ubuntu) viaconfigs/environments/docker_odoo.yaml secret/.envwith agent credentials (copysecret/.env.example), e.g.OPENAI_API_KEYforcodex,CURSOR_API_KEYforcursor, orANTHROPIC_API_KEY/ bedrock / vertex / openrouter settings forclaude_code- Task data — fetched once from a private HuggingFace dataset (next section); it is not committed to git
- For the judge-backed interaction metrics only:
uv sync --extra metricsand a judge credential in the samesecret/.env. A plain run needs neither.
Task data (the input/software/reference files each scenario stages into the
containers) lives in the gated HF dataset a2a-salesforce/a2-bench, mirrored 1:1
as <domain>/<task>/<variant>/[<role>/]{input,software,reference}.
export HF_TOKEN=hf_... # or: hf auth login
# Fetch everything into ./task-data (re-run to pull updates)
scripts/fetch_task_data.shreference/ holds eval answers, so the dataset is private.
uv sync
scripts/fetch_task_data.sh # once (see "Task data" above)
# Show the run matrix (task × variant) without executing anything
uv run python -m a2_run run example_exp_a2a.yaml --dry-run
# Run for real
uv run python -m a2_run run example_exp_a2a.yaml
# Useful flags
# --task PATH only these tasks --resume skip completed units -v debug logsResults land under
.logs/<experiment>/<roles>/<models>/<domain>__<task>/<variant>/<timestamp>/
(hops.jsonl streamed as the conversation runs, conversation.json at
finalize, per-role trajectories + origin logs, and eval scores).
example_exp_a2a.yaml points at everything else:
agents:— a role → agent-preset mapping (one or more roles). It may name more roles than any one variant uses: each unit binds only the roles its variant declares, so one yaml runs a corpus whose scenarios seat two to five agents. A variant declaring a role the yaml does not bind is skipped.environment:— the env yaml (configs/environments/docker.yaml).tasks:— a task-list file underselected_tasks/(onedomain/taskper line; the runner keeps only variants whose declared roles are covered by the binding).secret_file,output.root,concurrency(units in flight — each boots one container per bound role, so size it to roles × units),wall_time_s(budget for the whole conversation),cleanup_mode(delete | stop | keep).a2a:block —max_messages(total delivered messages across all senders; one message = one hop, default 16 — it replacedmax_rounds, which now fails loudly),channel(the delivery envelope — see below).
A run ends on task_complete, max_messages, wall_time,
invalid_agent_response (the opener went silent), all_agents_exited,
all_agents_blocked (every live agent waits on something nothing can
satisfy), or agent_failed. A dead or silent non-assigned role does not end
the run: agents waiting on it get a relay notice and the rest continue.
Agents talk over an A2A v1.0 JSON-RPC binding: each role's hub serves
POST /rpc (SendMessage / GetTask / CancelTask) plus its own agent card
at /.well-known/agent-card.json, and the host is the A2A client on every
side. channel selects how much of that envelope the receiving model reads:
a2a(default) — the specMessagefields: sender name,role,messageId,contextId,taskId, then the body.direct— the body under a barefrom: <name>line. The pre-protocol baseline.
Both arms carry the same body over the same wire, so running a scenario twice isolates the envelope as a single variable:
uv run python -m a2_run run example_exp_a2a.yaml -S a2a.channel=direct
uv run python -m a2_run run example_exp_a2a.yaml -S a2a.channel=a2aNote what A2A does not define: there is no directory operation
(a2a_list_agents is direct-configuration discovery) and no client-side
completion operation (a2a_task_complete is the assigned agent declaring its
own task terminal). The MCP tools are a local affordance — how a model asks
its own hub to act — not protocol operations; A2A covers agent-to-agent
communication, MCP covers model-to-tool. Their a2a_ prefix keeps them from
colliding with a harness's own native agent-team tools of the same bare name.
Agent presets live under configs/agents/: claude_code, codex, and
cursor are the real CLI coding agents executed inside the container (incl.
direct / openrouter / bedrock / vertex routing for claude_code —
openrouter drives any OpenRouter model through the same Claude Code
harness, see below; cursor talks only to Cursor's backend, with the model
named by a Cursor catalog id such as cursor-grok-4.6-high); dummy
(configs/agents/dummy_a2a.yaml) is a no-LLM scripted responder — pair it
with itself to exercise the whole pipeline token-free.
Any experiment key can be overridden with repeatable --set / -S KEY=VALUE
flags, applied to the parsed yaml before validation — one base yaml covers a
whole sweep:
# swap one role's preset, the task list, and a knob:
uv run python -m a2_run run example_exp_a2a.yaml \
-S agents.oracle=configs/agents/codex.yaml \
-S tasks=selected_tasks/a2a_hello.txt \
-S a2a.max_messages=20 -S output.root=.logs/sweep1
# replace the whole role mapping inline (VALUE is parsed as yaml):
uv run python -m a2_run run example_exp_a2a.yaml \
-S "agents={shopper: configs/agents/claude_code_vertex.yaml, merchant: configs/agents/codex.yaml}"KEY is a dotted path into the yaml; relative paths resolve against the
yaml's directory, exactly as if written in the file. Overrides bypass no
validation. Combine with --dry-run to preview the resulting matrix.
A role entry can be a mapping instead of a path: preset: names the base
yaml and every other key overlays it (model, id, executor, or any
deployer config field such as effort_level). So a model variation needs no
new preset file — e.g. Claude Opus 5 via Vertex:
uv run python -m a2_run run example_exp_a2a.yaml \
-S "agents.asker={preset: configs/agents/claude_code_vertex.yaml, model: claude-opus-5}" \
-S "agents.oracle={preset: configs/agents/claude_code_vertex.yaml, model: claude-opus-5}"The dotted form works too, one key at a time:
-S agents.asker.preset=configs/agents/claude_code_vertex.yaml \
-S agents.asker.model=claude-opus-5 \
-S agents.asker.effort_level=highThe same mapping form is valid written directly in the experiment yaml's
agents: block. Config fields are still validated against the deployer's
config dataclass, so a typo'd field name fails at load time.
To benchmark GPT, Gemini, Llama, DeepSeek, and the rest without a new harness, run the claude_code deployer with provider: openrouter. OpenRouter exposes an Anthropic-compatible endpoint, so the Claude Code CLI drives any model on it unchanged; the deployer points ANTHROPIC_BASE_URL at https://openrouter.ai/api, authenticates with a Bearer token, and you name the model with OpenRouter's vendor/model ids.
# 1. Put your OpenRouter key in secret/.env → OPENROUTER_API_KEY=sk-or-...
# 2. Point one or both roles at the openrouter preset and pick models:
uv run python -m a2_run run example_exp_a2a.yaml \
-S "agents.asker={preset: configs/agents/claude_code_openrouter.yaml, model: deepseek/deepseek-v4-flash-0731}" \
-S "agents.oracle={preset: configs/agents/claude_code_openrouter.yaml, model: deepseek/deepseek-v4-flash-0731}"The key travels through the env passthrough (never the host-gathered _spec.json). For adaptive-thinking models over OpenRouter keep max_thinking_tokens: null (the preset's default) — see the field docs in a2_run/agents/claude_code/config.py. effort_level is Anthropic-native and a no-op on non-Claude models, so leave it null there.
serve starts a local, on-demand viewer (no files written; runs re-read on
every request, so an in-progress run shows its latest state on refresh):
uv run python -m a2_run serve .logs/<experiment> # a tree of runs
uv run python -m a2_run serve .logs/<...>/<timestamp> # a single run
uv run python -m a2_run serve .logs --port 9000The index is one sortable, filterable table of runs (filters live in the URL, so any view is shareable). Clicking a run opens its page: the transcript tab is the merged per-turn timeline (reasoning / tool calls / observations, then the reply), the judge tab holds the interaction metrics with the judge's evidence quoted and highlighted inline in the transcript, and a score card shows which rubric criteria earned or lost points.
A task's evaluate() scores its deliverable. The metrics/ package scores
the interaction instead — per agent, offline, from the recorded run (e.g.
did an agent leak its principal's reservation value, however good the final
paperwork). Pure functions of a finished run dir, so a new metric re-scores
archived runs without re-running them.
uv sync --extra metrics # judge-backed metrics need the SDK
uv run python -m metrics list # registered metrics
uv run python -m metrics score .logs/<experiment> # writes <run_dir>/metrics.jsonSee metrics/README.md for the metric list, the
Metric interface, and judge configuration. Scores appear automatically as
columns in the run viewer.
A task directory holds exactly two files, splitting declarative setup from scoring code:
tasks/<domain>/<task>/
task.yaml # everything declarative — read on its own, no Python needed
main.py # only the scoring code: load()/start()/evaluate() + helpers
task.yamldeclares the roles and how they meet: task identity (domain/task/variants), thevmblock sizing the containers, each role's privateroles:brief +agent_cards:(identity/skills), theassignmentfor the assigned role, andpaths:templates the scoring code reads. Two optional per-variant blocks are host-side setup only and never shown to agents:visibility:— who may address whom:mesh(default — everyone reaches everyone),star(the assigned role is the hub; other roles reach only it), or an explicit{role: [peers]}adjacency map. The graph is made symmetric, and an edge naming an unknown role fails loudly.egress:—{role: open | restricted}. A restricted role can reach only its own model backend; the allowlist is derived from the bound harness/provider (a2_run/environments/egress.py), never written in the task.
main.pystays thin:load()builds theTasks fromtask.yaml;start(cfg, sandboxes)seeds each role's container before the run (sandboxesis a dict role →SandboxHandle, so each role's world can differ);evaluate(cfg, sandboxes)scores the artifacts afterward, and may return ascored(...)breakdown (seetasks/task_spec.py).
To enable a task, add its domain/task line to a file in selected_tasks/
and reference that file from the experiment yaml.
a2_run/ framework package
cli.py `python -m a2_run run <spec.yaml>` / `serve <path>`
orchestration/ config loading, run matrix, unit lifecycle (pair_lifecycle.py,
one container per role) + N-agent conversation driver
protocols/ A2A v1.0: agent_card, message, jsonrpc binding, task_state, channel
environments/ docker provider, image registry, task-data staging,
egress.py (restricted-role backend allowlists)
base_interface/ SandboxHandle (docker exec/cp), executor + deployer ABCs
executors/ sandbox (agent runs in container), local (host; dummy agent)
agents/ claude_code, codex, cursor, dummy deployers
transcript/ the on-demand run viewer (`serve`)
docker/ubuntu-base/ sandbox image: Dockerfile, a2a_hub.py (PID 1; the role's
A2A endpoint), a2a_tools_mcp.py (the local tool surface:
a2a_list_agents / a2a_send_message / a2a_wait_for_messages /
a2a_task_complete)
docker/odoo-erp/ Odoo 19 + PostgreSQL image (FROM ubuntu-base) for odoo tasks
metrics/ offline interaction metrics (see metrics/README.md)
tasks/ task library (task.yaml declares briefs/vm; main.py = scoring)
task-data/ host-side task data, staged into the containers (fetched)
configs/ agents/ presets, environments/docker.yaml (+ docker_odoo.yaml)
selected_tasks/ plain-text task lists, one `domain/task` per line
example_exp_a2a.yaml experiment spec: roles + environment + task list + a2a knobs
secret/.env credentials (gitignored)