Reelwright
Download Open Studio

Contents

Evaluating Claude against the heuristic

Status: built, 2026-10-10, waiting for the owner's first recording; since 2026-10-16 it records on the owner's Claude plan, with no key. The third product audit's finding 4 (0.11 plan items 6 ai-eval and 7 release-real-key): nothing in the repository said whether Claude directs or answers better than Reelwright's own heuristics, because no real model answer had ever been recorded. This is the harness that says so, and the one step of each release that sends a real message.

The evaluation set

  • Ten scripts (core/src/eval/scripts.ts): scenes on the demo show, Otto and the Chief on the Marigold's bridge and in its engine room. Three are the demo's own (the cold open and both scenes of The Salvage Claim, as written); seven are ordinary writing: tea on the bridge, a repair that goes wrong, a quarrel, a quiet moment, an alarm, good news, Otto learning the engine. 62 action paragraphs and lines say what a director should stage for them, in the staging corpus's words (core/test/support/staging-corpus.ts): who moves where, sits or stands, gestures, looks, handles a prop, and how they feel. 43 lines are to be kept word for word.
  • Each script is directed whole, as one scene, and each judged paragraph is scored by the staging corpus's rules: staged, staged against the text (what a writer would have to find and undo), or not staged. Every line is checked kept word for word.
  • Twenty asks (core/src/eval/asks.ts): what a person says to the showrunner on the demo ("Make the engineer a little older.", "Add a ship's cat called Biscuit.", "Paint the kettle red.", "Remember that the Chief hates tea."…), each with what should come of it: the tools that do it, what the proposal touches, the words it must say word for word, and what it must leave alone. An ask is done, against the ask (a proposal touches what the ask leaves alone), or not done. Three want no change at all, a question, a list and "approve it for me" (a decision is the person's): any proposal there is against the ask.

The scorecard

core/test/ai-eval.test.ts directs every script with the heuristic director on every run and prints the score; it is kept in core/test/fixtures/ai-eval/score.json and only goes up, as the corpus's does (RW_WRITE_EVAL_SCORE=1 writes a better one down; RW_EVAL_DETAIL=1 prints every verdict). The first score, the heuristic director on 2026-10-10: 62 judged, 53 staged (85%), 2 against the text, 7 not staged; 43 of 43 lines kept. The heuristic has no showrunner, so the asks are the model's alone.

When the owner's recording is in core/test/fixtures/llm/ai-eval/ (its eval.json says when it was made and its cap), the test replays it with no key and prints the model and the heuristic side by side: staged, against the text, not staged, lines kept, and the asks done and against, each ask with why. Scripts whose requests no longer match the recording (the director's prompt or the demo changed since) are counted and named, with the command that records again; a stale recording never fails the suite, since recording costs money, but the scorecard says it.

Recording: one owner command

Recording and replay run the set the same way (core/src/eval/set.ts: each script by directWithModel on the demo, each ask a fresh conversation on a fresh copy, a fixed clock), so a replay sends the same requests byte for byte. Both models spend from one cap on one ledger (evalMeters): every call plans against what is left of it, and the run stops before going past it. A model that never calls a tool proves in the test that the recording and its replay match, and that the cap stops a run.

reelwright eval record --dry-run          # says what it would do; sends nothing
reelwright eval record                    # records on the Claude Code here, on the owner's plan
reelwright eval record --with-key         # records with the key instead, at most $8 (--max-usd)

The owner's own Claude comes first (docs/design/director-over-mcp.md, "From the command line"): with the Claude Code here signed in with a Claude plan, each script is directed in a Claude Code session of its own and each ask put to Claude Code's chat, nothing charged; it says which Claude Code ("Claude Code 2.1.301 on a max plan") and that it counts against the plan's usage limits. With --with-key, or no such Claude Code, the key records it as before: "It should cost $L to $H (counted). It stops before spending more than $8.00". Either way it prints a line for each script (scenes by the model, lines kept) and each ask (what it proposed), and writes reelwright-eval-<date>/ai-eval/: one file per director's call and eval.json, and on the plan asks.json, what came of each ask as it came (Claude Code's chat has no requests of Reelwright's to replay). The owner sends that folder; it goes in core/test/fixtures/llm/ai-eval/, and the evaluation scores it, the asks of a plan recording judged from asks.json.

The release's real call

reelwright eval release                   # at most $3

On the published build, with the owner's key: one real message to the showrunner ("Make the engineer a little older.", which should propose a change to the Chief) and one real direction (the cold open, which the model should direct with every line kept), on a copy of the demo. It prints each with ✔ or ✖ and its cost and writes reelwright-<version>-real-key.json (the version, the models, each outcome and cost); the release keeps it as docs/releases/<version>-real-key.json (.claude/skills/verify/features/release.md, Publish step 5).