[danjdewhurst]

Closes #56

Cause

I traced 39 progressive-arm runs on the dev split (13 fixtures × 3) with --output-format stream-json and logged every Read call. The model read no reference file in 33 of 39 runs. It read genre-tells.md and preflight.md only for commit-message and pr-review-reply, the two venues that SKILL.md names with a section pointer. Step 3 of SKILL.md only said to "use" each reference, so the model audited from memory.

[Your Name] is listed under Near-conclusive artefacts in references/ai-writing-patterns.md and in preflight.md. SKILL.md mentioned near-conclusive artefacts but never listed them, so a model that read no reference had no rule that removes a placeholder.

Change

Step 3 of SKILL.md now:

• tells the model to read ai-writing-patterns.md and structures-and-phrases.md before every rewrite, draft, or review;
• tells it to read its genre's section of genre-tells.md;
• lists the near-conclusive artefacts inline, because they are the one single-instance exception. The list keeps the reference's limits: only placeholders the writer forgot to fill, and only standalone AI-model disclaimers. A mail-merge slot and a legal or medical disclaimer stay.

scripts/validate.py now checks that every example in the inline list appears in references/ai-writing-patterns.md, with tests.

On the final text, 35 of 39 traced dev runs read at least one reference, and 20 read both audit files. The 4 that read none are light edits.

The claude-opus-5 reference summary is re-recorded at 17452ef, the commit with the measured skill text. evals/README.md describes both references.

Numbers

--arm both --repeats 5 with the judge, against the old reference (skill 1.2.1).

Committed reference, skill text at 17452ef:

Split	Arm	Old	New
dev	always-on	58/65	64/65
dev	progressive	50/65	58/65
held-out	always-on	35/45	37/45
held-out	progressive	27/45	33/45
all	always-on	93/110	101/110
all	progressive	77/110	91/110 (+0.13 [+0.01, +0.24])

The paired progressive-minus-always-on difference went from -0.15 [-0.24, -0.05] to -0.09 [-0.17, -0.01]. chatbot-artefacts progressive went from 1/5 to 5/5 (+0.80 [+0.19, +0.96]).

ui-microcopy went from 2/5 to 5/5 in both arms, but that gain comes from the new brief in #65, not from this PR. After the rebase on #65, I re-ran that fixture with the new brief and spliced it in, so the reference matches main.

An earlier run (almost the same text, before the review fixes, old ui-microcopy brief) passed 95/110 always-on and 96/110 progressive, a paired difference of +0.01 [-0.05, +0.07]. There chatbot-artefacts was also 5/5. The two runs agree that the progressive arm gains and that the [Your Name] failure is gone. Five repeats do not settle how much of the gap remains.

Tuning used only the dev split. The held-out split ran once on each text, for confirmation; nothing changed because of it.

The progressive arm of the committed run was re-run at lower concurrency, because 26 runs in the first attempt failed with claude exited 1. The always-on arm comes from the first attempt, on the same skill text. Each fixture ran in its own run_skill.py process, and the per-fixture summaries were merged with build_summary.

detector-request still fails in most runs (2/5 in each arm), for the judge reason that #57 covers.

Not changed

The progressive arm's harness footer says "read only the ones you need, never the whole set up front". I left it, because changing the instrument would break comparison with the old reference. The new instruction names two of six files, so the two do not conflict. A real harness adds no such footer.

🤖 Generated with Claude Code