Contracts¶
Every file in a bundle is one of these models. They are the format: what bundle writes,
what verify reads, and what a third-party reader would implement against.
All are pydantic models with extra="forbid", so an unknown key raises rather than being
quietly dropped.
Plan¶
Manifest¶
What a pack declares about itself, read before it is run.
Bases: BaseModel
The blast radius of one pack, declared by the pack.
ASQI caps memory at 2g and CPU at two cores, and does it as a global default a pack cannot express a need for: a pack that genuinely wants 8g has nowhere to say so, and the operator raises the cap for every pack at once or not at all. Declaring it here makes the ceiling per pack, reviewable in the frozen plan, and the defaults below match ASQI's so a pack that says nothing behaves the same under both.
Bases: BaseModel
False means summary only. Such metrics carry no interval and are capped when graded.
Informational. The engine never branches on it.
Which outcome the confidence on an item record is a claim about.
An item carries one confidence, so it is a claim about one outcome, and only the pack knows which. Declared here rather than passed at analysis time because it is a fact about the pack's schema, not a choice a reader makes. Binning a confidence against an unrelated boolean gives an ECE that reads as authoritative and means nothing.
Lock¶
A plan with nothing left to interpret. What freeze produces and run consumes.
Bases: BaseModel
Bytes, not a label. A tag here means the lock was hand edited.
Hosts this pack declared, read from its manifest at freeze time and pinned here so a security review reads the frozen plan rather than the image. Empty means no network.
The outcome this pack's confidence is a claim about, read from its manifest at
freeze time and pinned here, so what was calibrated is part of the frozen plan rather
than a flag somebody typed.
False means the pack reported summaries and no observations. Read from its manifest
at freeze time, because a sealed bundle otherwise has no way to tell a rate computed
from items apart from one a container asserted, and grade caps the second. See
02-DESIGN.md section 3.4.
What the pack declared it needs, read from its manifest at freeze time. Pinned here
for the same reason as egress: a reviewer reads the ceiling off the frozen plan
rather than off an image they would have to pull.
One per replicate, derived from the root seed and recorded so a rerun matches.
Bases: BaseModel
Plan content only. No timestamp and no tool version, so freezing the same plan twice produces the same bytes and therefore the same hash. When it was frozen is an event, and events belong in the ledger.
2 added calibrates, 3 added emits_items, 4 added resources, 5 narrowed the
seeds to 53 bits. A format bump changes the bytes and therefore the hash of an
unchanged plan, which is the point.
Item records¶
The observation a pack emits, one per evaluated item.
This is the load-bearing contract. Packs report what happened; Touchstone computes every statistic from these records, so rates, intervals and stratified breakdowns are derivable by anyone holding the bundle.
Bases: BaseModel
Open dimensions to group by. Never an enum: locale belongs in packs, not here.
Booleans become rates with a denominator.
Continuous measures become means with intervals.
Enables calibration and the confident-and-wrong rate.
Which repeat this is. Between-replicate variance needs it.
Which pack produced this observation.
Written by the harness when it merges the per-unit files, never by the pack, and a value a pack writes for itself is overwritten. A pack that could name itself could name another one, and every rate computed downstream is grouped by this field.
None only in records that never went through a run, which is to say a hand-built
sample. Two packs both reporting correct are not measuring the same thing, so
pooling them into one denominator is the aggregate this tool exists to prevent.
One NaN score makes the mean NaN, and JSON has no way to write either.
Estimates¶
What the harness computes from the observations, and how it computed it.
02-DESIGN.md section 6 rule 3: the bundle is self-describing. An estimate that does not name its estimator is a number a reviewer has to take on trust, so every record here carries the method, its parameters and a citation, and the arithmetic can be redone in R, in a spreadsheet, or by hand without this code.
Bases: BaseModel
Which cell of the rollup. Empty is the whole sample.
Which pack this was computed over. None means it pools every pack that ran, which
is only meaningful where they measure the same thing. Kept off stratum so that
field stays what the pack declared and cannot collide with a key of the same name.
None where n is 0. A cell with no items has no estimate, and says so.
wilson, bootstrap_bca. Named so the arithmetic can be redone elsewhere.
Everything the estimator needed beyond the data. z, resamples, confidence.
The published source of the estimator. The citation is what makes it reviewable.
Bases: BaseModel
The weakest cell of a rollup, and the cells the minimum size rule kept out of it.
worst: Estimate | None = None
None when no cell reaches min_n. That is a finding, not a missing value.
excluded: list[Estimate] = Field(default_factory=list)
Cells below min_n. Reported rather than dropped: a rollup that quietly discards
its thin cells reads as coverage it does not have.
How many eligible cells the winner was ranked against. The interval on worst is
widened to hold over all of them, so this is the number that makes it readable, and a
reader who sees it rise knows the point estimate is being selected harder.
Bases: BaseModel
One rung of a reliability curve.
Bases: BaseModel
Whether a stated confidence means what it says.
Which pack declared this outcome as the one its confidence is a claim about.
None when nothing was scored. Expected calibration error, sample weighted.
Items carrying the outcome but no confidence. They cannot be binned, and are counted here rather than dropped without trace.
Bases: BaseModel
The two additive sources of variation in a rate, reported apart.
NIST AI 800-3 equation 22. Only one of them shrinks when a plan buys more trials, so a single total is the wrong number to hand a reader deciding where to spend.
Sampling of completions from the system: E[Pi_i (1 - Pi_i)] / t. This is the
term more replicates shrink, and it shrinks as 1 / t.
Sampling of items from the superpopulation the pack drew them from: Var[Pi_i].
More replicates do not move it at all. Only more items do.
completion + item, the variance of one item's replicate-averaged score. Divide
by items for the variance of the rate that was reported.
Observations per item, t. Fractional where a replicate lost an item, because
the estimator below is pooled over the items rather than assuming a balanced grid.
Distinct items, n. Below two there is no between-item variance to estimate.
anova_moment. Named so the arithmetic can be redone elsewhere.
Bases: BaseModel
How much of a result is the system and how much is the run.
replicate -> (successes, observations). Each carries its own denominator.
Sample standard deviation across replicate rates. None below two replicates: one run cannot report its own stability.
Highest replicate rate minus lowest. What a grade boundary has to clear.
Items seen in more than one replicate that did not answer the same way each time.
Items seen in more than one replicate. The denominator for the line above.
components: VarianceComponents | None = None
The variance of a per-item score split into the part more trials shrink and the
part they do not. None where the split is not identifiable, which is fewer than two
items or no item observed twice: sd above still describes the run, and a split
computed from one item would be a number with nothing behind it.
Bases: BaseModel
One source of uncertainty in a figure, with its size or the statement that it has none.
The shape is the uncertainty budget of JCGM 100:2008, the GUM: every component, its size
and how it was evaluated. A budget listing only the components that have numbers reads as
complete, so a source nothing here measures is listed as unquantified rather than
left out.
sampling, completion_sampling, item_sampling, marking, item_selection,
item_leakage, endpoint_identity.
A standard uncertainty on the scale of the figure. Quantified components are independent and combine in quadrature.
How the magnitude was obtained, or what would be needed to obtain one.
Bases: BaseModel
Every known source of uncertainty in one whole-sample figure.
Bases: BaseModel
A premise the estimators rest on, and what checking it found.
NIST AI 800-2 ipd Practice 3.1 item 1 asks for modelling assumptions to be reported with the results of any checks of them. NIST AI 800-3 section 6.2 names the ones a regression-free estimator fails silently on, which is why an unchecked one is listed rather than omitted.
Bases: BaseModel
The estimates.json of an evidence bundle.
The stratum keys the rollup used, in the order they were requested.
Several keys are rolled up one at a time and then crossed, so the cells below carry one key or all of them. Which shape an estimate is keyed by is on the estimate.
Which packs contributed records. More than one and every estimate carrying a null
pack_id pools them, which a reader has to be told rather than left to notice.
True when more than one pack contributed. Set so the flag survives into the bundle rather than living only in what the command printed.
calibration: list[Calibration] = Field(default_factory=list)
One per boolean outcome, where any item reported a confidence.
replicate_variance: list[ReplicateVariance] = Field(
default_factory=list
)
One per boolean outcome, where the plan asked for more than one replicate.
uncertainty: list[UncertaintyBudget] = Field(
default_factory=list
)
One per whole-sample figure. Empty in a bundle written before budgets existed.
assumptions: list[Assumption] = Field(default_factory=list)
Empty in a bundle written before assumptions were recorded.
Score cards¶
What an analyst asserts a grade means, and what the harness decided.
The rubric is data. levels is an ordered list the score card declares, best first, and
nothing here or in grade.py knows how many there are or what they are called. DQI uses
eight; ASQI uses five; a score card written for one client may use three. An engine that
hardcodes A to E cannot carry a standard that has not been finished yet, and this one is
not finished yet.
The conditions ending _ci_lower, _ci_upper and _by_interval are the ones ASQI cannot
express. See grade.py for what they do to a verdict.
INTERVAL_CONDITIONS = frozenset(
{
"greater_equal_ci_lower",
"less_equal_ci_upper",
"threshold_crossed_by_interval",
}
)
Conditions that read low or high. A source that carries no interval cannot satisfy
one, and asking is a plan error rather than a false answer.
The three that carry an interval: a Wilson or BCa one, and for a paired difference an interval over the change itself. An ECE and a replicate spread are single numbers: neither has a sampling distribution this codebase is willing to assert.
graded awarded a level. indeterminate is the interval straddling a boundary, which
is a finding. ungraded is no rule holding at all, or nothing to grade, which is a
different finding and is not the same as the worst level.
Bases: BaseModel
Which number in estimates.json an indicator is about.
Which evaluation to read it from.
prior reaches into the bundle before this one, which is how movement is graded: a
drift indicator is an expression over the same metric in both. Every other command
here is a pure function of one bundle and stays that way. This is the one reference
that is not, and it is explicit in the score card rather than implied by a flag, so a
reader can see which numbers came from where.
None selects the pooled figure. On a multi-pack run that is rarely what is meant,
and grade says so rather than grading two packs as though they measured one thing.
Empty is the whole sample. Ignored by worst_stratum, which searches cells.
worst_stratum only: search cells keyed by exactly these stratum keys.
Without it every cell carrying any stratum is a candidate, so two indicators that mean to ask about different dimensions ask the same question and return the same cell. An index with a geographic indicator and a language indicator needs them to differ.
A bundle whose rollup was crossed holds cells of more than one shape, and a coarse cell
contains the finer ones inside it, so ranking them together compares a group against
part of itself. grade refuses that rather than picking a shape on the author's
behalf, which makes this field required in practice on any run given several keys.
worst_stratum only: the smallest cell allowed to be the worst one.
worst_stratum only: which end of the ranking the weakest cell is at.
Bases: BaseModel
An indicator a person assesses, not one the engine computes.
It reads nothing from estimates.json and carries no ladder: the assessor supplies the
level and the engine's job is to check it against the score card's own levels and cap
it by the access tier, which is where artefact_provenance gets its limit. The
response arrives in a separate file, keyed by indicator id, so the judgment and the
rubric are written by different people at different times.
What the assessor was asked, in the card rather than in the responses, so two audits of the same index answered the same question.
Bases: BaseModel
One rung of the ladder: the level this awards, and what has to hold to award it.
Every condition here compares against a number, so this is required. A rule with no threshold would be a rule that always holds, which is a typo, not a rubric.
Bases: BaseModel
assessment: list[Rule] = Field(default_factory=list)
Ordered, best level first. The first rule that holds decides the grade.
Empty only for an audit indicator, where the ladder is the assessor's and a rule here would be a threshold applied to a judgment that never produced a number.
Overrides the score card's map for this indicator alone.
A tier mapped to null is one where this indicator is not assessable: it returns
ungraded naming the tier, and the metric it would have read is never looked for, so
a bundle that legitimately does not hold it is not an error.
Required because the ceiling is a property of the pair. A black box evaluation can measure headline accuracy completely and cannot measure calibration at all, and one ceiling for a whole card either caps the first for no reason or lets the second be claimed on evidence that does not exist.
Bases: BaseModel
The rubric, its ladder, and the ceilings that stop a claim exceeding its evidence.
Ordered best first. Every level named by a rule or a ceiling has to appear here.
Access tier to the best level it may reach. The tier vocabulary is the score card's,
not this engine's, so a new tier is a YAML change. A tier absent from this map is
uncapped, which has to be written down rather than assumed: an unrecognised tier is a
hard error in grade.py.
The best level a metric from a pack that emitted no items may reach. None leaves it uncapped, which 02-DESIGN.md section 3.4 advises against.
Position in the ladder. Lower is better, because levels is best first.
Bases: BaseModel
How a movement between two bundles got its interval.
paired where both bundles ran the same frozen plan and hold the same items, and the
interval is over the per-item differences. unpaired everywhere else, with the two
stored intervals combined as though independent.
What was joined, or why nothing could be. A reader comparing two figures needs this
before the number, so grade prints it under an indicator that fell back.
paired_clt, wilson_difference or limits_difference.
The two figures the difference was taken between, and for a paired one the items joined, the standard error and z.
Bases: BaseModel
The number an indicator was decided on, and where in the bundle it came from.
None where the source carries no interval, or where nothing was measured.
The cell this number came from. Copied from the estimate rather than from the
reference, because worst_stratum chooses the cell and a report that says a system
was weakest somewhere without saying where is not a finding.
True when the pack behind this number emitted no items. Its ceiling applies.
difference: Difference | None = None
Set where the source was paired_difference, saying whether the two bundles were
paired and why. An unpaired interval read as a paired one is a narrower interval with
nothing behind it, so the choice travels with the number.
Bases: BaseModel
One indicator's outcome, and enough of the working to argue with it.
The level awarded, after any ceiling. None unless the verdict is graded.
rule: Rule | None = None
The rule that decided it. None for ungraded, and for indeterminate this is the
rule whose boundary the interval straddles.
For indeterminate, the level refused and the next one down, in that order. The
honest statement is that the grade is one of these and the evidence does not say
which.
Why, in one line, for indeterminate and ungraded. Printed in the report.
What the ladder awarded before a ceiling applied. Equal to level when nothing
capped it, and kept so a reader can see the claim the evidence supported and the
reason it was not made.
access_tier or summary_only. Set only where a ceiling actually bit.
measured: list[Measured] = Field(default_factory=list)
Every number read. More than one where the metric was an expression.
The number the ladder was actually walked against.
Equal to the single measured value for a plain reference, and the computed result for an expression, where it is the only place that number appears. A report that grades a difference between two rates and prints one of the rates beside the grade is showing a number that did not decide anything.
The formula, where one was used. It carries no interval, by design: combining two intervals needs their correlation, and a bundle does not record it.
audit: AuditResponse | None = None
The assessor's answer, where this indicator was audited rather than computed. Kept whole, so the level and the evidence behind it travel together into the report.
Bases: BaseModel
The scorecard.json of an evidence bundle.
Copied from the frozen plan, not from a flag, so the ceiling that applied is part of the evidence.
The ladder this was graded on, carried so the bundle is readable without the score card that produced it.
The frozen plan these grades were asserted against. A grade is only meaningful beside the thresholds that were fixed before the run.
The plan behind the earlier bundle, where an indicator compared the two. Named beside this run's, because movement between two evaluations run under different plans is movement in the plan as much as in the system.
The audit responses these grades used, and the hash of the file they came from. An audited level is somebody's judgment, and the bundle names whose and which version.
Command output¶
What a command says when something else is reading.
Every other contract here describes a file in a bundle. This one describes what the
commands emit under --json, and it is in contracts/ for the same reason: the moment
anything parses a shape, that shape is a promise, and a promise kept in a format string
inside cli.py is one nobody can find.
The human output is unchanged and stays the default. A Problem carries the sentence that
was already printed, so the two cannot drift apart, and adds the three things prose cannot
give a machine: where the trouble is, what kind it is, and a code that survives somebody
improving the wording.
Bumped when a field is removed or changes meaning. Adding one is not a bump: a reader that ignores what it does not know keeps working, and every reader should.
error failed the command. warning is a finding the command still succeeded past, so
a caller that treats the two the same will fail runs that were fine.
Bases: BaseModel
One thing wrong, or one thing worth saying, in one place.
Stable, and the field to branch on. The message is written for a person and gets rewritten whenever it reads badly; this does not.
Exactly what the human output prints, so the two cannot say different things.
The file, relative to where the command was invoked. None where the problem is about a bundle rather than about one file in it.
1-indexed, counting the way an editor counts. Both absent rather than zero where the check cannot say where it is looking: a position of 0 is a position, and a reader that trusts it puts a squiggle on the first character of the file.
The pack id, indicator id or file name the problem is about, lifted out of the message so a caller can group by it without parsing English.
Bases: BaseModel
One command's whole answer.
False when any problem is an error. A caller should still read the exit code, which says the same thing and is what a shell sees.
What the command produced, which for estimate and grade is where it wrote the
numbers rather than the numbers themselves. estimates.json and scorecard.json are
already contracts with their own shapes, and serialising them a second way here would
be two answers to one question, drifting from the release they first disagree in.
Conformance¶
What a conformance statement says about a bundle.
The findings are computed from the bundle and nothing else. A reader who disagrees with a line can open the file it names and check it, which is the only reason to publish a statement of conformance at all.
Three, and no fourth. A partial credit column is where a conformance report stops being readable, because every line becomes partial and the reader has to grade the grader.
Bases: BaseModel
One practice item, and what the bundle holds against it.
Stable. The practice reference moves when a draft is revised and this does not.
The reference in the source, where the item has one. None for the four this project added, which the survey's summary of the practices missed.
What the bundle actually holds. Written to be argued with, so it names figures and files rather than restating the requirement in the past tense.
The files the finding was read from, relative to the bundle.
Audit responses¶
What a person assessed, for the indicators no bundle can answer on its own.
Two of the DQI 0.1 indicators are not computed. Whether someone subject to an adverse
decision can obtain the reason and challenge it, and whether the artefact evaluated is the
artefact deployed, are both read off an organisation rather than off items.jsonl. The
engine still refuses to grade them itself: it takes a level an assessor recorded, checks it
is on the ladder the score card declares, and applies the same access tier ceiling as
every computed indicator, so a human judgment cannot claim more than the access allowed.
Every response carries its evidence for the same reason every rate carries its denominator. A level with nothing behind it is an assertion, and this file travels into the bundle.
What the responses are called inside a bundle. grade copies them in, because a grade
that cannot be recomputed from the bundle is not evidence.
Bases: BaseModel
One indicator, as a person assessed it.
A level from the score card's own ladder. One that is not on it is an error rather than a rounding, because the assessor and the card would then disagree about the vocabulary and the grade would be meaningless.
What was examined. Required: an audit level with nothing behind it is an opinion, and this is the field a reviewer argues with.
Bases: BaseModel
The file an assessor fills in, keyed by indicator id.
Who assessed it. Named, because an audit outcome is a person's judgment and a judgment with no author cannot be questioned.
responses: dict[str, AuditResponse] = Field(min_length=1)
Indicator id to what was found. An id the score card does not declare is an error: it is either a typo or an audit of a different card, and both produce a bundle whose grades came from somewhere nobody can identify.
Environment¶
What the run actually ran on and what it took, recorded so a reader can judge the evidence.
02-DESIGN.md section 7.3: a runtime that contained the pack less well than a container states that here. This is the access-tier logic applied to the runtime, so it has to be machine readable rather than a sentence in a report.
The cost is here because the harness is the one party that saw every unit start and stop.
What a unit spent on the system under test only the pack can see, so the pack reports it on
each row's cost and the harness adds the rows up rather than taking a total from the pack.
Bases: BaseModel
What one pack's units took, and what its rows say they spent.
Measured by the harness on a monotonic clock around every unit, a failed one included, because a unit that died after an hour of API calls still spent the hour.
Rows whose cost holds at least one figure and nothing but figures. Below items,
the totals undercount by whatever the other rows spent, and nobody knows how much.
Each cost key summed over the costed rows. Within one pack only, since two packs
both reporting tokens need not be counting the same tokenizer's tokens.
Bases: BaseModel
What producing the result took, per pack.
Summed over every unit of every pack. The seconds spent merging rows and writing files after the last unit are left out.
Bases: BaseModel
Every image that actually ran, read back from the runtime.
The egress proxy image behind every enforced allowlist. Empty where no unit had
one, and in an environment.json written before run recorded it.
Across the whole run. False if any unit was granted a network it declared but the backend could not restrict. A claim that a pack was contained is not available then.
Bundle manifest¶
What a sealed bundle records about itself.
Written by bundle, read by verify, and readable by a person with a text editor and
no copy of this tool.
Where a run records what it did, and the event it writes when it got to the end. Part
of the bundle layout rather than of run, because the thing that has to know what a
finished run looks like is the thing that refuses to seal an unfinished one.
Bases: BaseModel
Bumped when the layout changes in a way an older verify cannot read.
Over the canonicalised file list alone, so it does not move when sealed_utc does. This is the one value a report quotes and an anchor timestamps.
Whether this bundle holds a run log that reached run_finished.
absent is a directory assembled by hand, which is legitimate and says so here rather
than passing for a run. There is no incomplete: a directory whose ledger stops before
the end is refused, because a run that failed part way through produced files that
hash perfectly well and mean nothing.