Contracts¶
Every file in a bundle is one of these models. They are the format: what bundle writes,
what verify reads, and what a third-party reader would implement against.
All are pydantic models with extra="forbid", so an unknown key raises rather than being
quietly dropped.
Plan¶
Manifest¶
What a pack declares about itself, read before it is run.
Bases: BaseModel
The blast radius of one pack, declared by the pack.
ASQI caps memory at 2g and CPU at two cores, and does it as a global default a pack cannot express a need for: a pack that genuinely wants 8g has nowhere to say so, and the operator raises the cap for every pack at once or not at all. Declaring it here makes the ceiling per pack, reviewable in the frozen plan, and the defaults below match ASQI's so a pack that says nothing behaves the same under both.
Bases: BaseModel
False means summary only. Such metrics carry no interval and are capped when graded.
Informational. The engine never branches on it.
Which outcome the confidence on an item record is a claim about.
An item carries one confidence, so it is a claim about one outcome, and only the pack knows which. Declared here rather than passed at analysis time because it is a fact about the pack's schema, not a choice a reader makes. Binning a confidence against an unrelated boolean gives an ECE that reads as authoritative and means nothing.
Lock¶
A plan with nothing left to interpret. What freeze produces and run consumes.
Bases: BaseModel
Bytes, not a label. A tag here means the lock was hand edited.
Hosts this pack declared, read from its manifest at freeze time and pinned here so a security review reads the frozen plan rather than the image. Empty means no network.
The outcome this pack's confidence is a claim about, read from its manifest at
freeze time and pinned here, so what was calibrated is part of the frozen plan rather
than a flag somebody typed.
False means the pack reported summaries and no observations. Read from its manifest
at freeze time, because a sealed bundle otherwise has no way to tell a rate computed
from items apart from one a container asserted, and grade caps the second. See
02-DESIGN.md section 3.4.
What the pack declared it needs, read from its manifest at freeze time. Pinned here
for the same reason as egress: a reviewer reads the ceiling off the frozen plan
rather than off an image they would have to pull.
One per replicate, derived from the root seed and recorded so a rerun matches.
Bases: BaseModel
Plan content only. No timestamp and no tool version, so freezing the same plan twice produces the same bytes and therefore the same hash. When it was frozen is an event, and events belong in the ledger.
2 added calibrates, 3 added emits_items, 4 added resources. A format bump
changes the bytes and therefore the hash of an unchanged plan, which is the point.
Item records¶
The observation a pack emits, one per evaluated item.
This is the load-bearing contract. Packs report what happened; Touchstone computes every statistic from these records, so rates, intervals and stratified breakdowns are derivable by anyone holding the bundle.
Bases: BaseModel
Open dimensions to group by. Never an enum: locale belongs in packs, not here.
Booleans become rates with a denominator.
Continuous measures become means with intervals.
Enables calibration and the confident-and-wrong rate.
Which repeat this is. Between-replicate variance needs it.
Which pack produced this observation.
Written by the harness when it merges the per-unit files, never by the pack, and a value a pack writes for itself is overwritten. A pack that could name itself could name another one, and every rate computed downstream is grouped by this field.
None only in records that never went through a run, which is to say a hand-built
sample. Two packs both reporting correct are not measuring the same thing, so
pooling them into one denominator is the aggregate this tool exists to prevent.
Estimates¶
What the harness computes from the observations, and how it computed it.
02-DESIGN.md section 6 rule 3: the bundle is self-describing. An estimate that does not name its estimator is a number a reviewer has to take on trust, so every record here carries the method, its parameters and a citation, and the arithmetic can be redone in R, in a spreadsheet, or by hand without this code.
Bases: BaseModel
Which cell of the rollup. Empty is the whole sample.
Which pack this was computed over. None means it pools every pack that ran, which
is only meaningful where they measure the same thing. Kept off stratum so that
field stays what the pack declared and cannot collide with a key of the same name.
None where n is 0. A cell with no items has no estimate, and says so.
wilson, bootstrap_bca. Named so the arithmetic can be redone elsewhere.
Everything the estimator needed beyond the data. z, resamples, confidence.
The published source of the estimator. The citation is what makes it reviewable.
Bases: BaseModel
The weakest cell of a rollup, and the cells the minimum size rule kept out of it.
worst: Estimate | None = None
None when no cell reaches min_n. That is a finding, not a missing value.
excluded: list[Estimate] = Field(default_factory=list)
Cells below min_n. Reported rather than dropped: a rollup that quietly discards
its thin cells reads as coverage it does not have.
How many eligible cells the winner was ranked against. The interval on worst is
widened to hold over all of them, so this is the number that makes it readable, and a
reader who sees it rise knows the point estimate is being selected harder.
Bases: BaseModel
One rung of a reliability curve.
Bases: BaseModel
Whether a stated confidence means what it says.
Which pack declared this outcome as the one its confidence is a claim about.
None when nothing was scored. Expected calibration error, sample weighted.
Items carrying the outcome but no confidence. They cannot be binned, and are counted here rather than dropped without trace.
Bases: BaseModel
How much of a result is the system and how much is the run.
replicate -> (successes, observations). Each carries its own denominator.
Sample standard deviation across replicate rates. None below two replicates: one run cannot report its own stability.
Highest replicate rate minus lowest. What a grade boundary has to clear.
Items seen in more than one replicate that did not answer the same way each time.
Items seen in more than one replicate. The denominator for the line above.
Bases: BaseModel
The estimates.json of an evidence bundle.
The stratum keys the rollup used, in the order they were requested.
Several keys are rolled up one at a time and then crossed, so the cells below carry one key or all of them. Which shape an estimate is keyed by is on the estimate.
Which packs contributed records. More than one and every estimate carrying a null
pack_id pools them, which a reader has to be told rather than left to notice.
True when more than one pack contributed. Set so the flag survives into the bundle rather than living only in what the command printed.
calibration: list[Calibration] = Field(default_factory=list)
One per boolean outcome, where any item reported a confidence.
replicate_variance: list[ReplicateVariance] = Field(
default_factory=list
)
One per boolean outcome, where the plan asked for more than one replicate.
Score cards¶
What an analyst asserts a grade means, and what the harness decided.
The rubric is data. levels is an ordered list the score card declares, best first, and
nothing here or in grade.py knows how many there are or what they are called. DQI uses
eight; ASQI uses five; a score card written for one client may use three. An engine that
hardcodes A to E cannot carry a standard that has not been finished yet, and this one is
not finished yet.
The conditions ending _ci_lower, _ci_upper and _by_interval are the ones ASQI cannot
express. See grade.py for what they do to a verdict.
INTERVAL_CONDITIONS = frozenset(
{
"greater_equal_ci_lower",
"less_equal_ci_upper",
"threshold_crossed_by_interval",
}
)
Conditions that read low or high. A source that carries no interval cannot satisfy
one, and asking is a plan error rather than a false answer.
The two that carry a Wilson or BCa interval. An ECE and a replicate spread are single numbers: neither has a sampling distribution this codebase is willing to assert.
graded awarded a level. indeterminate is the interval straddling a boundary, which
is a finding. ungraded is no rule holding at all, or nothing to grade, which is a
different finding and is not the same as the worst level.
Bases: BaseModel
Which number in estimates.json an indicator is about.
Which evaluation to read it from.
prior reaches into the bundle before this one, which is how movement is graded: a
drift indicator is an expression over the same metric in both. Every other command
here is a pure function of one bundle and stays that way. This is the one reference
that is not, and it is explicit in the score card rather than implied by a flag, so a
reader can see which numbers came from where.
None selects the pooled figure. On a multi-pack run that is rarely what is meant,
and grade says so rather than grading two packs as though they measured one thing.
Empty is the whole sample. Ignored by worst_stratum, which searches cells.
worst_stratum only: search cells keyed by exactly these stratum keys.
Without it every cell carrying any stratum is a candidate, so two indicators that mean to ask about different dimensions ask the same question and return the same cell. An index with a geographic indicator and a language indicator needs them to differ.
A bundle whose rollup was crossed holds cells of more than one shape, and a coarse cell
contains the finer ones inside it, so ranking them together compares a group against
part of itself. grade refuses that rather than picking a shape on the author's
behalf, which makes this field required in practice on any run given several keys.
worst_stratum only: the smallest cell allowed to be the worst one.
worst_stratum only: which end of the ranking the weakest cell is at.
Bases: BaseModel
An indicator a person assesses, not one the engine computes.
It reads nothing from estimates.json and carries no ladder: the assessor supplies the
level and the engine's job is to check it against the score card's own levels and cap
it by the access tier, which is where artefact_provenance gets its limit. The
response arrives in a separate file, keyed by indicator id, so the judgment and the
rubric are written by different people at different times.
What the assessor was asked, in the card rather than in the responses, so two audits of the same index answered the same question.
Bases: BaseModel
One rung of the ladder: the level this awards, and what has to hold to award it.
Every condition here compares against a number, so this is required. A rule with no threshold would be a rule that always holds, which is a typo, not a rubric.
Bases: BaseModel
assessment: list[Rule] = Field(default_factory=list)
Ordered, best level first. The first rule that holds decides the grade.
Empty only for an audit indicator, where the ladder is the assessor's and a rule here would be a threshold applied to a judgment that never produced a number.
Overrides the score card's map for this indicator alone.
A tier mapped to null is one where this indicator is not assessable: it returns
ungraded naming the tier, and the metric it would have read is never looked for, so
a bundle that legitimately does not hold it is not an error.
Required because the ceiling is a property of the pair. A black box evaluation can measure headline accuracy completely and cannot measure calibration at all, and one ceiling for a whole card either caps the first for no reason or lets the second be claimed on evidence that does not exist.
Bases: BaseModel
The rubric, its ladder, and the ceilings that stop a claim exceeding its evidence.
Ordered best first. Every level named by a rule or a ceiling has to appear here.
Access tier to the best level it may reach. The tier vocabulary is the score card's,
not this engine's, so a new tier is a YAML change. A tier absent from this map is
uncapped, which has to be written down rather than assumed: an unrecognised tier is a
hard error in grade.py.
The best level a metric from a pack that emitted no items may reach. None leaves it uncapped, which 02-DESIGN.md section 3.4 advises against.
Position in the ladder. Lower is better, because levels is best first.
Bases: BaseModel
The number an indicator was decided on, and where in the bundle it came from.
None where the source carries no interval, or where nothing was measured.
The cell this number came from. Copied from the estimate rather than from the
reference, because worst_stratum chooses the cell and a report that says a system
was weakest somewhere without saying where is not a finding.
True when the pack behind this number emitted no items. Its ceiling applies.
Bases: BaseModel
One indicator's outcome, and enough of the working to argue with it.
The level awarded, after any ceiling. None unless the verdict is graded.
rule: Rule | None = None
The rule that decided it. None for ungraded, and for indeterminate this is the
rule whose boundary the interval straddles.
For indeterminate, the level refused and the next one down, in that order. The
honest statement is that the grade is one of these and the evidence does not say
which.
Why, in one line, for indeterminate and ungraded. Printed in the report.
What the ladder awarded before a ceiling applied. Equal to level when nothing
capped it, and kept so a reader can see the claim the evidence supported and the
reason it was not made.
access_tier or summary_only. Set only where a ceiling actually bit.
measured: list[Measured] = Field(default_factory=list)
Every number read. More than one where the metric was an expression.
The number the ladder was actually walked against.
Equal to the single measured value for a plain reference, and the computed result for an expression, where it is the only place that number appears. A report that grades a difference between two rates and prints one of the rates beside the grade is showing a number that did not decide anything.
The formula, where one was used. It carries no interval, by design: combining two intervals needs their correlation, and a bundle does not record it.
audit: AuditResponse | None = None
The assessor's answer, where this indicator was audited rather than computed. Kept whole, so the level and the evidence behind it travel together into the report.
Bases: BaseModel
The scorecard.json of an evidence bundle.
Copied from the frozen plan, not from a flag, so the ceiling that applied is part of the evidence.
The ladder this was graded on, carried so the bundle is readable without the score card that produced it.
The frozen plan these grades were asserted against. A grade is only meaningful beside the thresholds that were fixed before the run.
The plan behind the earlier bundle, where an indicator compared the two. Named beside this run's, because movement between two evaluations run under different plans is movement in the plan as much as in the system.
The audit responses these grades used, and the hash of the file they came from. An audited level is somebody's judgment, and the bundle names whose and which version.
Audit responses¶
What a person assessed, for the indicators no bundle can answer on its own.
Two of the DQI 0.1 indicators are not computed. Whether someone subject to an adverse
decision can obtain the reason and challenge it, and whether the artefact evaluated is the
artefact deployed, are both read off an organisation rather than off items.jsonl. The
engine still refuses to grade them itself: it takes a level an assessor recorded, checks it
is on the ladder the score card declares, and applies the same access tier ceiling as
every computed indicator, so a human judgment cannot claim more than the access allowed.
Every response carries its evidence for the same reason every rate carries its denominator. A level with nothing behind it is an assertion, and this file travels into the bundle.
What the responses are called inside a bundle. grade copies them in, because a grade
that cannot be recomputed from the bundle is not evidence.
Bases: BaseModel
One indicator, as a person assessed it.
A level from the score card's own ladder. One that is not on it is an error rather than a rounding, because the assessor and the card would then disagree about the vocabulary and the grade would be meaningless.
What was examined. Required: an audit level with nothing behind it is an opinion, and this is the field a reviewer argues with.
Bases: BaseModel
The file an assessor fills in, keyed by indicator id.
Who assessed it. Named, because an audit outcome is a person's judgment and a judgment with no author cannot be questioned.
responses: dict[str, AuditResponse] = Field(min_length=1)
Indicator id to what was found. An id the score card does not declare is an error: it is either a typo or an audit of a different card, and both produce a bundle whose grades came from somewhere nobody can identify.
Environment¶
What the run actually ran on, recorded so a reader can judge the evidence.
02-DESIGN.md section 7.3: a runtime that contained the pack less well than a container states that here. This is the access-tier logic applied to the runtime, so it has to be machine readable rather than a sentence in a report.
Bundle manifest¶
What a sealed bundle records about itself.
Written by bundle, read by verify, and readable by a person with a text editor and
no copy of this tool.
Where a run records what it did, and the event it writes when it got to the end. Part
of the bundle layout rather than of run, because the thing that has to know what a
finished run looks like is the thing that refuses to seal an unfinished one.
Bases: BaseModel
Bumped when the layout changes in a way an older verify cannot read.
Over the canonicalised file list alone, so it does not move when sealed_utc does. This is the one value a report quotes and an anchor timestamps.
Whether this bundle holds a run log that reached run_finished.
absent is a directory assembled by hand, which is legitimate and says so here rather
than passing for a run. There is no incomplete: a directory whose ledger stops before
the end is refused, because a run that failed part way through produced files that
hash perfectly well and mean nothing.