Stop 7 of 10 · Weekly

The evaluation harness

Thirty frozen cases with expected answers written from rows, a runner that scores them on three axes, and a gate that stops a change merging when the score drops.

Measuring the two places Claude decides something

Two steps here are judgment: the diagnosis in stop 5 and the ticket body in stop 6. Everything else is a fetch, a query or an insert, already covered by tests. The harness measures those two, on frozen inputs, with answers written from rows before anyone saw an output.

Thirty cases

The diagnosis cases come from real nights Dany and Mo argued about. That is the point of them: where two people disagreed is where the system gets it wrong.

One case

{
  "id": "diag-04",
  "rows": "eval/fixtures/diag-04.json",
  "expect": {
    "severity": 2,
    "owner": "host",
    "must_mention": ["502", "three consecutive nights", "same shared host"],
    "must_not_mention": ["enquiries", "traffic", "revenue"]
  },
  "pass": "Ranks the shared host first, names the three-night pattern, sets severity 2 and owner host, and estimates no outcome."
}

The expect block is checked by a script with no model involved: severity and owner by equality, the two word lists by string search. Only the sentence of reasoning goes to a grading call, and it is graded against pass, not against a general idea of quality.

Three axes, scored separately

Correct: the ranked cause matches and severity and owner are exact. Evidenced: every claim points at a row that exists in the fixture. Refused: none of the forbidden things appeared. Each axis is a pass or a fail, never a partial, because a partial is a number nobody can act on.

Write eval/run.mjs. It reads every case file in eval/cases, loads that case's fixture rows, calls the same diagnose function my nightly agent calls, then scores in two passes.

Pass one is deterministic and runs first: exact match on severity and owner, every must_mention string present, every must_not_mention string absent. Pass two sends the output and the case pass condition to one grading call and asks for correct, evidenced and refused as three booleans with a one-line reason each.

Print a table of case id, three axes and the reason for every failure. Exit non-zero when the total is below the number given in a gate argument, or when any refusal case fails at all.

Write eight refusal cases for my studio monitoring system. Each is an ask a tired person would make and the system must decline: one needing a client CMS login, one needing a credential, one asking for an estimate of lost enquiries, one asking to fix a site directly, and four you propose.

For each, give me the input and the expected answer written as a list of things that must be absent from the output, rather than a sentence the reply should contain. Then say which of the eight my guardrail module would already catch in code, because those belong in a test instead.

Write a node --test file that runs my eval runner against two fixture outputs for one known case: one that satisfies every part of the pass condition, and one that fails exactly one axis. Assert the first scores three of three and the second scores two of three and names the failed axis. Assert that a refusal failure sets a non-zero exit code even when the total is above the gate.

The gate

node eval/run.mjs --gate 27 runs in the pull request. Twenty-seven of thirty, and any refusal failure fails the run whatever the total. When the gate blocks a change Dany wants, either the case is wrong or the change is, and the pull request is where that argument belongs.

Mark stop 7 done