Running packs¶

$ touchstone run ./run-004 -o ./run-004

run executes a frozen plan. It refuses one that was never frozen or has been edited.

It also refuses an output directory that already holds a run. A run that died part way is run again into a new directory.

Needs Docker. It is one of two commands that do.

Units¶

The unit of work is one pack at one replicate, the thing that either ran or did not.

A plan with two packs, one at replicates: 3 and one at replicates: 1, is four units. Each gets its own run_id, its own seed, its own container and its own output directory under runs/.

run-004/runs/
├── example_pack-0/
│   └── items.jsonl
└── example_pack-1/
    └── items.jsonl

Those are merged into the bundle's top-level items.jsonl. The per-unit files stay, so the merge is checkable.

`pack_id` is stamped at merge¶

When the per-unit files are merged, the harness writes pack_id onto every record and overwrites anything the pack put there. If it overwrote any, the ledger records it:

{"utc": "...", "event": "pack_id_overwritten", "records": 12}

That line is a finding about the pack, not an error in the run. See Item records.

What a non-zero exit means¶

exit 0        the pack completed
exit non-zero the run failed

A pack that finds the system under test is terrible still exits 0. It reports the failure in the rows.

The distinction matters because a pack that crashes produces files that hash perfectly well and mean nothing. bundle refuses to seal a run whose ledger never reached run_finished.

Termination is recorded separately from the exit code¶

Both Docker and a timeout report exit 137, which is also plain SIGKILL. An exit code alone cannot tell them apart, so RunResult carries termination beside it:

`termination`	Meaning
`null`	exited on its own
`out_of_memory`	killed by the memory cgroup
`timeout`	killed by the harness for taking too long
`cancelled`	stopped deliberately

A pack killed for memory is recorded as out_of_memory, not as a timeout. See Resource limits.

Exit 125 is Docker failing before the pack started, such as a container name already in use. The ledger records it as unit_failed with Docker's message, and no exit code.

What run writes¶

File	Contents
`runs/<run_id>/items.jsonl`	the rows one unit produced, untouched
`runs/<run_id>.egress.log`	every request that unit made, allowed or denied
`items.jsonl`	all units merged, `pack_id` stamped
`environment.json`	what it ran on, what it cost, whether egress was enforced
`ledger/RUNLOG.jsonl`	append-only, written as each event happened
`plan.lock.json`, `PLAN.sha256`	copied from the lock directory

`environment.json`¶

{
  "touchstone_version": "0.8.0",
  "python": "3.12.4",
  "platform": "Linux-6.8.0-x86_64",
  "backend": "docker",
  "isolation": "container",
  "plan_hash": "81c63db1…",
  "image_digests": ["example_pack@sha256:…"],
  "proxy_digests": ["ubuntu/squid@sha256:…"],
  "egress_enforced": true,
  "cost": {
    "wall_seconds": 34.412,
    "packs": [
      {
        "pack_id": "example_pack",
        "units": 2,
        "wall_seconds": 34.412,
        "items": 400,
        "items_costed": 0,
        "totals": {}
      }
    ]
  }
}

image_digests is read back from the runtime, so it holds the images that actually ran rather than the ones the lock asked for. proxy_digests is the egress proxy image behind every enforced allowlist, empty where no pack declared egress. isolation is what the backend could actually provide: a runtime that contained the pack less well than a container says so here, machine-readably, because that is the access-tier argument applied to the runtime.

egress_enforced is across the whole run. It is false if any unit was granted a network it declared but the backend could not restrict. A claim that a pack was contained is not available then. See Containment.

`cost`¶

The block has two parts, and they come from different places.

wall_seconds is the harness's own measurement, taken on a monotonic clock around every unit. A unit that failed still counts, because a unit that died after an hour of API calls spent the hour. The ledger carries the same figure on each unit's line.

totals comes from the pack. The harness cannot see tokens or spend, so a pack reports them on each row's cost and run sums each key over that pack's rows. It never sums across packs, since two packs both reporting tokens need not be counting the same tokenizer's tokens.

items_costed is how many rows carried a cost. Where it falls short of items, the totals undercount by an amount nobody knows, and touchstone report marks the cost item not met. example_pack calls no real system and reports no cost, so a tutorial run records how long it took and nothing about what it spent.