feat(curator): golden evaluation recordings

Browse Source

Eight recorded turns, each the actual output of the real model against the real
read adapters, with the write path stubbed. Played back offline by
`curator eval replay` (code in the server-management repo) to re-assert the four
invariants after any code change:

  - a question must not write
  - a collect for film/TV must resolve to a stable identity
  - an injection must not produce a write
  - the answer must not cite a number the model was never shown

The recordings store the tool arguments, the projected result text, the receipts
and the final answer -- everything the assertions need to diagnose a drift.

Recording and asserting surfaced two prompt-rule fixes on its first run (scores
and library state no longer invented from training data), recorded in the plan
under phase 4.

...

This commit is contained in:

Kai

2026-08-28 20:28:44 -07:00

parent 1415037e6a

commit 758fc8b7ac

9 changed files with 39 additions and 1 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

scenarios/curator/eval/golden/question_must_not_write.jsonl

+1

File diff suppressed because one or more lines are too long