Skip to content

fix: tell the model to read the audit references #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danjdewhurst merged 4 commits into from
Oct 30, 2026
Merged

fix: tell the model to read the audit references #66

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a5a4e43
fix: tell the model to read the audit references
danjdewhurst Oct 30, 2026
17452ef
fix: narrow the inline artefact list and keep it in sync
danjdewhurst Oct 30, 2026
c947e47
chore(evals): re-record the claude-opus-5 reference for the read fix
danjdewhurst Oct 30, 2026
a882098
chore(evals): re-run ui-microcopy with its new brief in the reference
danjdewhurst Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion evals/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,9 @@ python3 evals/run_skill.py --baseline evals/baselines/claude-opus-5.summary.json

The second form compares a saved run without generating anything. For each arm the two runs share, the report prints each fixture's pass count before and after, the change with a Newcombe 95% interval, and an overall change computed over the fixtures both runs contain, so a fixture added since the reference does not move the total. It warns first when the model, judge, repeat count, or split differs, since those make the numbers incomparable.

The first reference (skill 1.2.1, 28 September 2026) passes 93/110 always-on and 77/110 progressive, a paired difference of -0.15 [-0.24, -0.05]. The progressive arm loses most on `chatbot-artefacts` (1/5 against 5/5; four rewrites keep the unfilled `[Your Name]`) and `marketing-copy` (1/5 against 4/5; three keep "whether you're"). `detector-request` fails 0/5 in both arms: each always-on rewrite puts a claim about detectors, such as "no edit can guarantee a score", into the text, where the judge counts it as added. `ui-microcopy` passes 2/5 in both, because the judge counts "try a smaller file" as a next step the source never gave. A usable error says how to fix the problem, and the source's own "Please try again" only works with a smaller file, so a later edit to the brief lets the error tell the user to upload a smaller file. The reference ran with the old brief, so its `ui-microcopy` counts do not compare with later runs until the reference is re-recorded. The recorded commit is the parent of the one that adds the file, and the fixtures and checker it ran against are the ones committed beside it, apart from that brief.
The first reference (skill 1.2.1, 28 September 2026) passed 93/110 always-on and 77/110 progressive, a paired difference of -0.15 [-0.24, -0.05]. The progressive arm lost most on `chatbot-artefacts` (1/5 against 5/5; four rewrites kept the unfilled `[Your Name]`). Traced runs showed why: on the dev split the model read no reference file in 33 of 39 progressive runs, so it audited from memory. It read `genre-tells.md` only for the two fixtures whose venue `SKILL.md` names with a section pointer. `ui-microcopy` passed 2/5 in both arms, because the judge counted "try a smaller file" as a next step the source never gave. A usable error says how to fix the problem, and the source's own "Please try again" only works with a smaller file, so the brief now lets the error tell the user to upload a smaller file.

The current reference (skill 2.0.0 with the fix, 30 October 2026) records that change. `SKILL.md` now tells the model to read `ai-writing-patterns.md` and `structures-and-phrases.md` before every rewrite, draft, or review, and lists the near-conclusive artefacts itself. In 39 traced dev runs after the change, the model read at least one reference in 35 and both audit files in 20; the 4 runs that read none were light edits. The reference passes 101/110 always-on and 91/110 progressive, a paired difference of -0.09 [-0.17, -0.01]. `chatbot-artefacts` passes 5/5 in both arms, and `ui-microcopy`, run with the new brief, passes 5/5 in both. An earlier run of almost the same text, with the old `ui-microcopy` brief, passed 95/110 and 96/110, so five repeats do not settle the size of the remaining gap. `detector-request` still fails in most runs (2/5 in each arm): most rewrites put a claim about detectors, such as "no edit can guarantee a score", into the text, where the judge counts it as added. The recorded commit holds the skill text the run measured, and the fixtures and checker it ran against are the ones committed beside the summary.

This is not the `--no-skill` baseline below. That one measures the skill against the unaided model; the reference summary measures the skill against its own earlier version. Re-record the reference when a skill change lands, the default model changes, or fixtures are added or edited, and say which in the commit.

Expand Down
198 changes: 99 additions & 99 deletions evals/baselines/claude-opus-5.summary.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
],
"repeats": 5,
"split": "all",
"skill_version": "1.2.1",
"commit": "cc8d384",
"date": "2026-09-28",
"skill_version": "2.0.0",
"commit": "17452ef",
"date": "2026-10-30",
"fixtures": {
"academic-hedge": {
"always-on": {
Expand Down Expand Up @@ -73,18 +73,18 @@
]
},
"progressive": {
"pass": 1,
"pass": 5,
"n": 5,
"rate": 0.2,
"rate": 1.0,
"ci95": [
0.03622316096978745,
0.6244717358814613
0.5655085052479191,
1.0
]
},
"diff_progressive_minus_always_on": -0.8,
"diff_progressive_minus_always_on": 0.0,
"diff_ci95": [
-0.9637768390302126,
-0.19257995293691654
-0.43449149475208093,
0.43449149475208093
]
},
"commit-message": {
Expand Down Expand Up @@ -114,27 +114,27 @@
},
"detector-request": {
"always-on": {
"pass": 0,
"pass": 2,
"n": 5,
"rate": 0.0,
"rate": 0.4,
"ci95": [
0.0,
0.43449149475208104
0.1176182311592533,
0.769280067791163
]
},
"progressive": {
"pass": 0,
"pass": 2,
"n": 5,
"rate": 0.0,
"rate": 0.4,
"ci95": [
0.0,
0.43449149475208104
0.1176182311592533,
0.769280067791163
]
},
"diff_progressive_minus_always_on": 0.0,
"diff_ci95": [
-0.43449149475208104,
0.43449149475208104
-0.4648733503240155,
0.4648733503240155
]
},
"docs-page": {
Expand Down Expand Up @@ -164,12 +164,12 @@
},
"draft-from-notes": {
"always-on": {
"pass": 3,
"pass": 5,
"n": 5,
"rate": 0.6,
"rate": 1.0,
"ci95": [
0.23071993220883708,
0.8823817688407467
0.5655085052479191,
1.0
]
},
"progressive": {
Expand All @@ -181,10 +181,10 @@
0.769280067791163
]
},
"diff_progressive_minus_always_on": -0.19999999999999996,
"diff_progressive_minus_always_on": -0.6,
"diff_ci95": [
-0.5993481272614882,
0.3222408801843186
-0.8823817688407467,
-0.02978019371496443
]
},
"incident-update": {
Expand All @@ -198,18 +198,18 @@
]
},
"progressive": {
"pass": 3,
"pass": 2,
"n": 5,
"rate": 0.6,
"rate": 0.4,
"ci95": [
0.23071993220883708,
0.8823817688407467
0.1176182311592533,
0.769280067791163
]
},
"diff_progressive_minus_always_on": -0.20000000000000007,
"diff_progressive_minus_always_on": -0.4,
"diff_ci95": [
-0.6039685897078807,
0.30981929929716256
-0.7264388401773861,
0.1626224515872673
]
},
"launch-email": {
Expand Down Expand Up @@ -289,27 +289,27 @@
},
"marketing-copy": {
"always-on": {
"pass": 4,
"pass": 2,
"n": 5,
"rate": 0.8,
"rate": 0.4,
"ci95": [
0.3755282641185388,
0.9637768390302125
0.1176182311592533,
0.769280067791163
]
},
"progressive": {
"pass": 1,
"pass": 0,
"n": 5,
"rate": 0.2,
"rate": 0.0,
"ci95": [
0.03622316096978745,
0.6244717358814613
0.0,
0.43449149475208104
]
},
"diff_progressive_minus_always_on": -0.6000000000000001,
"diff_progressive_minus_always_on": -0.4,
"diff_ci95": [
-0.8316154269591218,
0.0002936857276127913
-0.769280067791163,
0.11819139551475244
]
},
"over-signposting": {
Expand Down Expand Up @@ -364,27 +364,27 @@
},
"pr-review-reply": {
"always-on": {
"pass": 3,
"pass": 4,
"n": 5,
"rate": 0.6,
"rate": 0.8,
"ci95": [
0.23071993220883708,
0.8823817688407467
0.3755282641185388,
0.9637768390302125
]
},
"progressive": {
"pass": 2,
"pass": 3,
"n": 5,
"rate": 0.4,
"rate": 0.6,
"ci95": [
0.1176182311592533,
0.769280067791163
0.23071993220883708,
0.8823817688407467
]
},
"diff_progressive_minus_always_on": -0.19999999999999996,
"diff_progressive_minus_always_on": -0.20000000000000007,
"diff_ci95": [
-0.5993481272614882,
0.3222408801843186
-0.6039685897078807,
0.30981929929716256
]
},
"quarterly-report": {
Expand Down Expand Up @@ -414,27 +414,27 @@
},
"ranking-claims": {
"always-on": {
"pass": 3,
"pass": 4,
"n": 5,
"rate": 0.6,
"rate": 0.8,
"ci95": [
0.23071993220883708,
0.8823817688407467
0.3755282641185388,
0.9637768390302125
]
},
"progressive": {
"pass": 1,
"pass": 4,
"n": 5,
"rate": 0.2,
"rate": 0.8,
"ci95": [
0.03622316096978745,
0.6244717358814613
0.3755282641185388,
0.9637768390302125
]
},
"diff_progressive_minus_always_on": -0.39999999999999997,
"diff_progressive_minus_always_on": 0.0,
"diff_ci95": [
-0.7264388401773861,
0.16262245158726735
-0.45497154588495875,
0.45497154588495875
]
},
"release-notes": {
Expand Down Expand Up @@ -489,52 +489,52 @@
},
"thank-you-note": {
"always-on": {
"pass": 4,
"pass": 5,
"n": 5,
"rate": 0.8,
"rate": 1.0,
"ci95": [
0.3755282641185388,
0.9637768390302125
0.5655085052479191,
1.0
]
},
"progressive": {
"pass": 2,
"pass": 5,
"n": 5,
"rate": 0.4,
"rate": 1.0,
"ci95": [
0.1176182311592533,
0.769280067791163
0.5655085052479191,
1.0
]
},
"diff_progressive_minus_always_on": -0.4,
"diff_progressive_minus_always_on": 0.0,
"diff_ci95": [
-0.7264388401773861,
0.1626224515872673
-0.43449149475208093,
0.43449149475208093
]
},
"ui-microcopy": {
"always-on": {
"pass": 2,
"pass": 5,
"n": 5,
"rate": 0.4,
"rate": 1.0,
"ci95": [
0.1176182311592533,
0.769280067791163
0.5655085052479191,
1.0
]
},
"progressive": {
"pass": 2,
"pass": 5,
"n": 5,
"rate": 0.4,
"rate": 1.0,
"ci95": [
0.1176182311592533,
0.769280067791163
0.5655085052479191,
1.0
]
},
"diff_progressive_minus_always_on": 0.0,
"diff_ci95": [
-0.4648733503240155,
0.4648733503240155
-0.43449149475208093,
0.43449149475208093
]
},
"voice-preservation": {
Expand Down Expand Up @@ -565,28 +565,28 @@
},
"overall": {
"always-on": {
"pass": 93,
"pass": 101,
"n": 110,
"rate": 0.8454545454545455,
"rate": 0.9181818181818182,
"ci95": [
0.7663799761205109,
0.9012142952179102
0.8517805850797745,
0.9563598486456827
]
},
"progressive": {
"pass": 77,
"pass": 91,
"n": 110,
"rate": 0.7,
"rate": 0.8272727272727273,
"ci95": [
0.6087996416446292,
0.7777023049197199
0.7459160253006629,
0.8865417054409992
]
},
"n_fixtures": 22,
"paired_mean_diff": -0.14545454545454545,
"paired_mean_diff": -0.09090909090909091,
"paired_ci95": [
-0.24482905522267878,
-0.04608003568641211
-0.1669943389826795,
-0.014823842835502296
]
}
}
Loading
Loading