Stop 9 of 10 · Weekly
Cost, time and errors
One ledger row per system per week covering what it cost, what it saved and what it got wrong, read every Friday, with a written rule for killing a system.
The question nobody asks until it is expensive
Nour now runs six systems. Each one felt worth building. Only three are worth keeping, and without a ledger she cannot tell you which three, because every one of them produces something and every one of them costs something, in money, in the minutes her team spends reviewing output, and in the errors that get past everyone.
One row per system per week. Six numbers, all measured rather than remembered.
| Column | Where it comes from |
|---|---|
| Model and API cost | Token counts in the run manifests, priced at your current rate |
| Platform and host cost | The host bill, split per system by run time |
| Machine minutes | Wall time in the manifests |
| Human minutes | Timestamps between a draft entering the review queue and being approved |
| Errors caught | Checker failures, guardrail refusals, evaluation gate failures |
| Errors shipped | Anything wrong that a client read, whoever found it |
Build it from what the systems already write
Every stop so far wrote a manifest, a log or a queue entry. The ledger reads those. It does not ask anyone to fill in a form, because nobody fills in the form in week three.
Write ledger.py. Each week it reads every runs/<date>/manifest.json, changes.jsonl, the review queue timestamps and the evaluation results, and appends one row per system to ledger.csv: week, system, runs, model cost, host cost, machine minutes, human review minutes, errors caught, errors shipped, and a value line. The value line is the baseline minutes for the same work done by hand, which I will supply per system in baselines.yaml, minus the human review minutes, times our internal hourly rate. If a system has no baseline recorded, or the review queue has no timestamps for that week, write unknown rather than a number and say which input was missing. Write test_ledger.py: a system with no baseline reports unknown and not a saving, a week with no queue timestamps reports unknown, an error marked shipped never nets against a saving, and two systems sharing a run directory are attributed separately. Run it.
Refusing to guess is what makes the ledger worth reading. A saving column that fills itself with plausible numbers is the same failure mode as a report with invented figures, one layer up.
The Friday review
Twenty minutes, the whole team, the ledger on screen. Three questions per system and nothing else.
Read ledger.csv for the last eight weeks and write the review pack. For each system: cost per week and the trend, human review minutes and the trend, errors caught and errors shipped, and the net saving or the word unknown with the missing input named. Then three lists. Systems where review minutes are rising faster than runs, which means the output is getting worse or the reviewer is getting stricter. Systems with zero errors caught in eight weeks, which usually means the check is not checking. Systems where the net is negative or unknown for two weeks running. No recommendations. I want the lists.
The middle list surprises people. A checker that has never failed is not proof of quality, it is an untested assertion, and the fix is to feed it a deliberately broken input and confirm it fails.
Here is the ledger row and the eight-week history for [name the system]. Argue both sides in under three hundred words. Keep it: what it costs, what it saves, what it would cost to do this work by hand at our current account count. Kill it: what it costs, what breaks if it stops, who absorbs the work, and what we would lose that is not in the ledger. Then state which single fix would most change the numbers, what it would take, and what the row would have to read next month for the system to survive.