BENCHMARKING
The testing bundle
Submissions to a specification are evaluated automatically. For that the platform needs three capabilities: produce input data, compute the expected output, and compare a submitted output against it. These capabilities are provided by a testing bundle — a directory of components attached to the specification.
Each specification has its own bundle. Nothing is shared between specifications: a kernel can accept submissions under one specification and not under another.
Components
| component | kind | required | provides |
|---|---|---|---|
| generated module | code, generated | yes | Point, Inputs, Outputs, Verdict, Tensor, the Stream PRNG, the codec — all derived from the signature |
harness (main.py) | code, generated | yes | the bundle's command-line interface: make, verify, info |
| generator | code, authored | yes | input data as a function of (point, seed) |
| oracle | code, authored | yes | the expected output for given inputs |
| verifier | code, authored | no | the comparison rule, when it is not exact equality |
| named sources | code, authored | no | constructed inputs that random seeds cannot produce (identity, zero, extreme values) |
| test vectors | data | no | worked examples the oracle is validated against |
| assets | data | no | files the authored functions read |
The set is not fixed. Generated components are rewritten when the signature changes; authored components are never touched by regeneration. The directory layout and the command-line contract are specified in the bundle protocol; the contracts of the authored functions are specified in component reference.
Absence semantics
A missing verifier is not an error. Without one, outputs are compared to
the oracle's byte for byte, and the verdict metrics record
{"rule": "exact"}. A verifier is written only when the comparison rule is
something else: a tolerance, an accuracy threshold, or a property check
where more than one output is correct.
A missing generator or oracle makes the specification unable to accept submissions: there is no input data, or no expected output.
Who judges a point
The vocabulary here — point, case, sample, verdicts — is defined formally in Points, cases, verdicts.
The verifier judges one case — one seed of one point. Who judges the point, over however many seeds ran, is declared in the bundle's marker:
[accept] seeds = 100 # the sample a point's verdict is defined over passed = 80 # how many of it must pass
Without the section the default rule stands: every case of however many ran
must pass. With it, a run on fewer seeds than seeds is measured but not
judged — its times are shown, no verdict is claimed, and standings do not
rank it; a run on the declared sample or more is held to the same
proportion (passed / seeds). The Benchmark button defaults to the declared
sample, so an ordinary press produces a judgeable run.
The bar is data, not code: the platform applies it, the pages print it, and it is part of the bundle — a run's verdict is computed under the bar of the bundle that judged it, so changing the bar never rewrites an old verdict.
Component states
Each code component is in one of these states, derived from checks — not set by hand:
| state | meaning |
|---|---|
missing | no file with the expected entry point |
unparsed | the file does not parse |
stub | the scaffolded placeholder is still in place |
written | parses, defines the entry point, body replaced |
accepted | all checks pass |
failing | at least one check fails |
The checks per component are listed in component reference.
Readiness
A specification accepts submissions when its generator and oracle are
accepted. The Testing tab of a kernel page shows this state per
specification, per component. Verifier state does not affect readiness.
Isolation properties
The authored components are deliberately isolated from everything they must not influence:
| they never see | consequence |
|---|---|
| the implementation's language | the only interface is bytes on disk, per the protocol |
| the mechanism (plain, FHE, MPC, …) | encryption and decryption happen outside the bundle; generator and oracle operate on cleartext |
| the machine, repeat counts, scoring | run planning belongs to the platform, execution to the runner |
| the filesystem | the functions receive and return values; serialisation is the harness's |
Adding an implementation language changes no bundle. Adding a kernel changes no runner.