[undera]

Runs kept "converging" without the student learning anything. The causes turned out to be measurement, not coaching.

The stop rule

The teacher walked down to meet the student. It was re-run and re-graded every round on an unchanged prompt, so its mean drifted purely on evaluator noise — 0.875 → 0.813 → 0.738 on one run — and the student won by the bar falling rather than by climbing. The teacher is now graded once in round one and carried forward. Its mean is constant for the whole run, which also let teacher_baseline averaging go (a constant's running average is itself) and cuts a 4-case round from ~18 model calls to ~10.

One lucky round ended the loop. The student's single round was compared against the teacher's average, so across a 5-round budget it got five independent chances to clear the bar by luck. The gap now has to hold for two consecutive rounds. I chose consecutive rounds over comparing running averages because the student is supposed to be trending upward, and averaging its history would drag a genuine catch-up backwards.

Every round still carries the teacher's cells, so the score grid, the per-cell detail view and replay are unchanged.

The tickets could not tell the models apart

Every policy snippet stated the operative clause outright, with the date arithmetic pre-computed in the customer's message. Haiku scored 1.00 on the refund case the README builds its worked example around — better than Sonnet, twice.

Each snippet now buries the operative clause among near-misses. The rule that emerged while tuning these: a distractor must be excludable by one cheap, unambiguous check — a category, an account type, a size limit. A first attempt at compensation added a clause that Sonnet fell for too, which only lowers the teacher and leaves the gap flat.

Customer messages are unchanged. Only refund and multi-request changed their expected checklists.

Also in here

• AWS Bedrock documented as a provider. No Anthropic key — LiteLLM uses the boto3 credential chain — but AWS_REGION_NAME must be set in .env because it is not read from ~/.aws/config. The model ids are not guessable from the first-party ones: Sonnet 5 is us.anthropic.claude-sonnet-5 with no version suffix, Haiku 4.5 is us.anthropic.claude-haiku-4-5-20251001-v1:0 with one.
• Retries at the single model-call chokepoint in llm.py, for the provider 503s a burst of ~18 parallel calls provokes. One unretried failure used to abort a whole round.
• Malformed-JSON failures now carry finish_reason and length. "invalid JSON" alone cannot distinguish a truncated reply from a model ignoring the format, which made one real failure undiagnosable. No salvage path: the score would survive a truncation but the reason would not, and the written reasons are the part of this loop you actually read.
• two-questions → multi-request, now that it carries three requests.
• Run-id suffix widened to 24 bits. token_hex(2) collides ~1.8% of the time over 50 ids, against a docstring promising it never would.

Deliberately not included

config.yaml keeps its shipped models on anthropic/. Only min_rounds 2→3 and gap 0.05 ship from that file — making AWS mandatory would be a product change, and Bedrock is documented as an option instead.

Still stale after this

The README's "One real run, and how the prompt changed" section and the three screenshots describe a run against the old tickets under the old stop rule. Regenerating them needs a run under this loop that produces a story worth telling.

Testing

49 offline tests pass, ruff check and ruff format --check clean. Four tests encoded the old one-round contract and moved with it; the teacher-variance test is gone because its premise is now impossible, replaced by two covering the new invariants — one lucky round does not stop the loop, and the teacher is called exactly once per case per run.

🤖 Generated with Claude Code