Touchstone¶
An open-source harness for AI evaluations that seal into a checkable evidence bundle.
Touchstone runs an evaluation in containers, works out every number itself rather than
taking the system's word for any of them, and writes the plan, one row per test item and a
SHA-256 of every file into a folder anyone can re-check with shasum.
It is built for people handed a result who have to decide whether to act on it: auditors, procurement, risk, regulators, and the teams producing evidence for them.
It is not built for iterating on a prompt
Freezing plans and sealing bundles are overhead in a loop where you change a line and rerun twenty times. Use Inspect while exploring, and this for the claim you publish.
What it does differently¶
Bundles checkable without this tool. Every file is JSON, JSON Lines or a hash. No database, no index, no proprietary format. See Bundle anatomy.
Every rate carries an interval. A bare percentage cannot be represented at all. See Rates and Wilson.
Grades can say indeterminate when the interval crosses a threshold, instead of
printing a letter the evidence does not support. See
Indeterminate.
Packs report facts, never scores. The arithmetic is Touchstone's, and the rows travel in the bundle so anyone can redo it. See Item records.
Containment. A pack reaches the hosts it declared and nothing else; the proxy that lets it out never decrypts anything. See Containment.
A score card is data. Levels, thresholds and access-tier ceilings are read from a YAML file, so a card with three levels and a card with eight both work. See Score card anatomy.
The 94% problem¶
Two systems are tested. Both get 94%. One was tested on 50 items, the other on 1,000:
94.0% (95% CI 83.5-98.8%, n=50) <- cannot back a claim about a 90% bar
94.0% (95% CI 92.4-95.4%, n=1000) <- can
Grades work the same way. If the error bar crosses the threshold, the honest answer is not the better letter:
headline_accuracy: indeterminate, A or C [0.91, 0.8783 to 0.9345, n=400]
the interval spans the A boundary of 0.9, so the evidence does not say which
The interval is sampling error, and only that. It is how far the number would move if
you drew another set of items the same way. Three larger errors are not in it: your items
are not a random sample of deployment, whatever decided correct has its own error rate
(correlated, not independent, when the judge is a model), and a leaked item set measures
recall rather than ability.
Two of the usual suspects are measured and reported next to the rate: between-replicate variance, which shows both how far the rate moved and how many individual items flipped, and calibration error against the system's own stated confidence.
It is precision. It is not accuracy. See what this does not prove for the limits a hash cannot fix.
The pipeline¶
| Command | Does | Needs |
|---|---|---|
validate |
check the plan against what each pack declares it needs | the plan and the packs |
freeze |
lock image versions, fix seeds, hash the plan | Docker |
run |
run the packs, write one row per test item | Docker |
estimate |
compute rates and intervals, split by group | the bundle |
grade |
apply a score card, grade each indicator | the bundle and a card |
bundle |
hash every file, write MANIFEST.json |
the run directory |
verify |
re-check a bundle against its manifest, offline | the bundle |
Only run needs a container. Everything after it reads files, so verify works on a plane
with the wifi off.
Where to start¶
- Installation.
pip install touchstone-dqi, and what needs Docker. - Tutorial. A plan through to a sealed bundle, in seven commands.
- Writing a pack. The container contract, in about seventy lines of standard library.
- What this does not prove. Read before citing anything this produces.
Status¶
Early, and saying so. 0.1.0 is the first release that is the code this documentation
describes, and all seven commands work and are tested doing it. What is not settled is the
score card format. See Status.
The package is touchstone-dqi because touchstone was taken on PyPI.