Re-analysis

items.jsonl is the observation. Everything else is derived from it, and can be derived again by anyone holding the bundle.

This is the property that makes the format worth the overhead. A reader who disagrees with how a number was computed does not have to take it up with whoever computed it; they can redo it.

Recompute the estimates

$ touchstone estimate ./run-004 --by language --by difficulty --out ./my-analysis

Runs against the bundle you were handed. No Docker, no network, nothing rerun. It reads items.jsonl from the bundle and writes estimates.json to ./my-analysis, so the bundle still verifies afterwards.

Split it differently, cross different keys, look at a stratum the original analysis did not report:

$ touchstone estimate ./run-004 --by region --out ./my-analysis

If the pack tagged its items with region, the cells are there whether or not the first analysis looked at them.

A sealed bundle is never written to

estimate and grade refuse to write into a directory that holds MANIFEST.json, because a new or rewritten file makes verify fail. Pass --out. To make your analysis a bundle of its own, copy the bundle, remove MANIFEST.json from the copy and seal it again with touchstone bundle. Its hash differs from the one you were handed, which is how anyone tells your analysis from the original.

Grade against your own card

$ touchstone grade ./run-004 --score-card my-card.yaml --out ./my-analysis

grade reads estimates.json from ./my-analysis when you re-estimated there, and from the bundle otherwise, and prints which one it read.

The most useful thing in the format. The bundle carries the numbers; the thresholds are yours.

A procurement team with a different risk appetite from the vendor's does not have to argue about the vendor's rubric. They apply their own to the same rows, and the disagreement becomes a disagreement about policy, which is where it belongs.

The access tier ceiling still applies, and still comes from the frozen plan rather than from your card's assumptions about it. You cannot grade a black-box run as though it were white-box by writing a more generous card.

Re-bin the calibration

$ touchstone estimate ./run-004 --calibrate refused

Overrides the outcome the pack declared its confidence was a claim about. Useful when you are re-examining a bundle and want the reliability curve against something else. Be aware that you have stepped outside what the pack said its confidence meant. See Calibration.

Do the arithmetic somewhere else

estimates.json carries the estimator, its parameters and a citation beside every number, so the sums can be redone in R, in a spreadsheet, or on paper.

$ jq -r '.[] | select(.metric=="correct") | [.stratum.language, .k, .n] | @tsv' \
    run-004/estimates.json
en   192  200
pcm  171  200
ha   124  200

That is k and n per cell. Everything in the interval follows from those two numbers and the published Wilson formula. Nothing about the result depends on trusting this implementation. That is the point of recording the method beside the answer.

A hand-assembled bundle

A directory of items.jsonl and a plan.lock.json, sealed without ever having been run, is legitimate. bundle records run_ledger: "absent" so it says what it is rather than passing for a run.

Use it for pooled re-analysis across several evaluations, or for reproducing published figures from rows somebody else collected. tests/test_estimate_credential.py does exactly this. It reproduces the flood-warning figures from item records, to the last bit of both bounds.