Stop 4 of 10 · Weekly
The audit assistant and its evaluation
Fifty account snapshots with the findings written down first, a runner that scores the assistant against them, and a gate that decides whether this version ships.
The assistant, and why it needs a score
The audit assistant reads an account snapshot and returns findings: budget pacing, wasted spend, tracking that stopped, a campaign carrying the account. Nour's team ran it by hand all last year and it was good, most of the time, which is the worst thing an assistant can be. Most of the time is unmeasurable, so it cannot be improved and it cannot be trusted with twenty accounts.
An evaluation harness turns most of the time into a number that moves when you change the prompt.
The case format
One directory per case. Inside it, snapshot.json, which is a redacted metrics.json from a real account, and expected.yaml, which holds what a senior on your team found when they audited it.
| Field | What it holds |
|---|---|
| must_find | Findings a competent audit cannot miss, each with a short id |
| may_find | Real but secondary findings, scored as a bonus, never required |
| must_not_say | Claims that are wrong for this snapshot, like a pacing problem on an account that is on pace |
Fifty cases, and they are not evenly boring. Aim for roughly thirty ordinary accounts, ten with something broken in the data itself, five where the right answer is that nothing is wrong, and five where the tempting finding is the wrong one.
Attached are twelve of our audit write-ups from the past year with the account snapshots they were based on, all redacted. For each one, produce a case directory: snapshot.json unchanged, and expected.yaml with must_find, may_find and must_not_say. A finding goes in must_find only if the write-up names it and the snapshot rows support it. Give each finding a short stable id and one line describing the evidence. Put anything the write-up asserts that the rows do not support into a separate file called disputed.md rather than into the case, and tell me why.
That last instruction is the useful one. Building the test set is also an audit of how your team audits.
The runner
Write eval.py. It walks cases/, and for each case sends snapshot.json to our audit assistant prompt through the Claude API, parses the returned findings as JSON, and scores three axes. Recall: the fraction of must_find ids present in the output. Precision: the fraction of returned findings that match a must_find or a may_find id. Fabrication: any figure in the output that does not appear in snapshot.json, and any claim matching a must_not_say entry. Write results/<prompt_version>.csv with one row per case: case id, recall, precision, fabrication count, and the ids missed. Print a summary and exit non-zero when the gate fails: mean recall below 0.90 on must_find, or any fabrication anywhere in the fifty. Read the current API request shape from docs.claude.com. Take the prompt version from a constant in the assistant module so the results file is always stamped.
Write test_eval.py that runs eval.py against three deliberately broken assistants I will stub out: one that returns an empty findings list, one that returns every finding id in the case including ones it could not know, and one that returns a correct finding with a made-up dollar figure. Assert the first scores near zero recall, the second scores high recall and low precision, and the third fails the gate on fabrication despite good recall. A harness that passes all three of those is a harness that is not measuring anything.
The gate, and what to do when it fails
The gate is not advice. Nothing goes into the weekly run of twenty accounts unless the current prompt version cleared it on the fifty. When it fails, you read the missed ids, not the mean. Six cases missing the same finding is a prompt problem you can fix in one edit; six cases missing six different findings is a case-set problem or a model choice.