[arminazimi]

Closes #102

Stacked on #571 (feat(bench): 10 agent benchmark tasks) — this runner consumes planning/bench/tasks.json and tools/bench-tasks.mjs exactly as that PR defines them.

What changed

• tools/bench-run.mjs — the runner. For each task in planning/bench/tasks.json it builds that task's workspace in its own temp directory (task.start → <workspace>/site, every seed to <workspace>/<file>), runs each criterion through the real built CLI as an argv array with no shell (it imports runCriterion from tools/bench-tasks.mjs rather than re-implementing the contract), applies the task's scripted reference edit, re-runs the criteria, and prints a per-task score plus a summary line. The verdict is still the documented exit code alone: 0 clean at --fail-on, 1 findings at or above it, 2 usage, 4 crash.
• npm run bench:run — one script line in package.json. Deliberately not wired into npm run check: a benchmark is a measurement, not a gate.
• --json — the same report, deterministic: fixed key order, task-file order, workspace-relative locations, no timestamps, no temp paths, so CI can diff it. --task <id> scores one task; --list prints the plan (argv and expected exit per criterion) without running the CLI.
• A failing gate prints the findings still open at that criterion's --fail-on level, with check id, file, line and fix hint — so a red task is actionable, not just a number.
• Per task it also records a fix --dry-run preview (exit code and suggested-edit count) on the solved workspace, which keeps the "reports and suggests" boundary visible in the score.
• Immutability witness: the runner hashes every workspace file before and after the scored CLI runs and marks the task invalid if anything outside the engine's own .seaworthy/ run record changed. Seaworthy never rewrites the user's site, so the fix step is the runner's scripted edit inside the temp dir — never the CLI's.
• tools/bench-run.test.mjs — 8 tests through the repo's tools/*.test.mjs convention (matching tools/bench-tasks.test.mjs, not packages/verify/tests): a fixture that satisfies its criteria scores pass; a deliberately broken fixture scores fail and names its unresolved findings; a scripted reference edit turns a red gate green; a reference edit that no longer fits the fixture throws instead of passing silently; a task with no reference solution is reported unsolved with a reason; the plan and summary counts tie to tasks.json; two --json runs are byte-identical and leak no local paths or timestamps; the task file is unchanged after scoring.
• changelog.d/102.added.md.

Why these files

packages/verify owns the engine and the CLI; the benchmark is repo tooling that consumes built output, so it lives in tools/ beside the validator it follows, and it imports that validator's helpers so #101's workspace contract has exactly one definition. The scripted reference edits live in the runner because planning/bench/tasks.json is #101's file and is read-only here; keying them by task id means a new task without an edit is reported, never skipped silently.

What you will not find here

• No agent-under-test harness: nothing runs a model, applies its diff, and scores the result. That is the next issue in the epic; this runner produces the pass/fail signal it will consume.
• No change to the task definitions, the criteria, the gates or planning/bench/README.md — nothing in feat(bench): 10 agent benchmark tasks with exit-code-decided success criteria and a live validator #571's data was loosened to make the score look good, and all 10 tasks score solved on their own criteria.
• No new check, format, spec field, MCP tool or CLI flag; no registry.ts regeneration was needed.
• No CI wiring and no file under .github/workflows/.
• No parallelism (the suite is ~50 s of short offline CLI runs sequentially) and no history/cache of past scores.
• Two honest follow-ups, not fixed here: verify writes its run record to <workspace>/.seaworthy/history/, so scoring a workspace is not byte-pure and the runner exempts that path by name — a --no-history/history-outside-root option would be cleaner; and fix --dry-run currently suggests scripted edits for only a few checks (2 for fix-image-alternatives, 0 for the rest), so the preview column mostly reports zero even where findings remain.

Commands I ran

rm -rf packages/*/dist && npm run build && node tools/gen-registries.mjs && npm run check
# GATE EXIT=0
# # tests 1056 / # pass 1056 / # fail 0 / # skipped 0
# last line: OK: schema (2 built schemas in sync, 40 fragments)
# (check:startup "OK: startup (best of 5 runs: 24 ms, budget 150 ms, floor 17 ms)";
#  one earlier full run failed only `--watch prints the first report in full…` under parallel
#  load — re-ran packages/verify/dist/tests/watch.test.js alone: 7/7 pass, exit 0. No test edited.)

node --test tools/bench-run.test.mjs
# # tests 8 / # pass 8 / # fail 0

npm run bench:run --silent      # exit 0, verbatim below
npm run bench:run --silent -- --json --task fix-image-alternatives --task migrate-base-url  # twice, byte-identical
npm run bench:run --silent -- --list
node tools/bench-tasks.mjs      # OK: bench tasks (10 tasks, 11 criteria, 1 seeds, 22 offline CLI runs)

Real score output of npm run bench:run:

seaworthy bench run (offline) — 10 tasks from planning/bench/tasks.json

PASS  fix-broken-links           criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)
PASS  restore-missing-assets     criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)
PASS  fix-image-alternatives     criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 2 suggested edit(s)
PASS  add-not-found-page         criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)
PASS  add-missing-translation    criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)
PASS  mirror-rtl-page            criteria 2/2 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)
PASS  make-agent-readable        criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)
PASS  migrate-base-url           criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)
PASS  pass-baseline-gate         criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)
PASS  ship-zero-findings         criteria 1/1 green on the reference solution, red on the untouched start as #101 requires, fix --dry-run exit 0: 0 suggested edit(s)

bench run: 10/10 tasks solved, 11/11 criteria green — 22 verify runs, 10 fix previews, no network

Merge note

This PR stacks on PR #571 (feat(bench): 10 agent benchmark tasks, branch tip 00d73a7). It reads planning/bench/tasks.json and imports tools/bench-tasks.mjs, both added there, so it must merge after #571 — do not merge it into main first, and do not rebase it onto main. Once #571 lands, this branch fast-forwards to it; npm run check here already exercises #571's validator through npm test.

Plan comment: #102 (comment)