BENCHMARKING

The bundle protocol

This page specifies the contract a testing bundle implements. It is a protocol, not a description of the Python implementation: a bundle in another language is equivalent if the commands, arguments, files and bytes match. Compliance is testable — generate one point with two implementations and compare cases/ byte for byte.

Directory contents

The scaffolded Python bundle:

fherma.py      generated: types, Stream, codec
main.py        generated: the command-line interface specified below
generate.py    authored: generator
oracle.py      authored: oracle
verify.py      authored: verifier (optional)
fherma.toml    marker: specification reference, signature fingerprint, entry points, the [accept] bar
vectors/       test vectors
assets/        data files the authored components read

File names are implementation details; the contract is the commands and the work directory.

Commands

<bundle> make   <dir> --point '<json>' --seeds <spec>
<bundle> verify <dir>
<bundle> info

<bundle> is whatever starts the program (python main.py, ./bundle, go run .); the arguments after it are the contract. --help must exist on the command and on every subcommand.

make

Fills <dir> for one point.

argument	format
`--point`	the point as a JSON object: `'{"N": 1024, "W": 2}'`
`--seeds`	a range `0-7`, a list `0,3,5`, or a mix `0-2,7,9`

For each seed in order: call generate, write cases/NNNNNN/; call oracle, write expected/NNNNNN/. Case numbering is the seed's position in the list, six digits, zero-based. The seed itself is not written to disk.

manifest.json is written last. An interrupted make therefore leaves a directory without a manifest — detectably incomplete — rather than a directory with fewer cases than requested.

Exit code: 0 on success; non-zero if an authored function raised.

verify

Reads manifest.json; for each case index 0 … cases-1 reads the solution's output from out/NNNNNN/, the expected output from expected/NNNNNN/, the inputs from cases/NNNNNN/, calls the verifier (or applies exact comparison), and writes verdicts.json:

{ "cases": [
    { "i": 0, "passed": true,  "metrics": { "worst": 3.1e-7 } },
    { "i": 1, "passed": false, "metrics": { "wrong": 3, "of": 2048 } },
    { "i": 2, "passed": false, "note": "no answer" },
    { "i": 3, "passed": false, "error": "ValueError: …" } ] }

field	present
`i`	always
`passed`	always
`metrics`	when a verdict was reached
`note`	output missing, or of the wrong length
`error`	the verifier raised — not the solution's fault

The list has exactly one record per case, including cases with no output. When no verifier is written, the rule is byte equality and the metrics record {"rule": "exact"}.

Exit code: 0 regardless of verdicts. A failing case is a result. A non-zero exit is reserved for the bundle itself failing to run.

info

Prints one JSON object to stdout: the specification reference, the point parameters, the arguments and results with their dimension names and element types, and whether a verifier is present. Reads and writes nothing else. Purpose: a runner or a person can size every file from a point without importing the bundle.

Work directory layout

One directory per point; any number of cases in it.

<dir>/
├── manifest.json          written by make, last
├── cases/
│   ├── 000000/            one file per argument: a.bin, b.bin, q.bin
│   └── 000001/ …
├── expected/
│   ├── 000000/            one file per result: c.bin
│   └── …
├── out/                   written by the solution
└── verdicts.json          written by verify

manifest.json contains the point and the case count, nothing else:

{ "point": { "N": 1024, "W": 2 }, "cases": 8 }

Constraints and their reasons:

expected/ is a sibling of cases/, so a runner can mount cases/ and out/ into the solution's container and not mount expected/.
No shapes are stored. Both sides derive every buffer size from the point and the signature, which they share. This also sets the read cost: one JSON parse per point, then two open + two read per case, no parser required in any language.

Data file format

One file per binding, named after it: %a → a.bin.

property	value
header	none
byte order	little-endian
element order	row-major
integers	two's-complement for `iN`, unsigned for `uN`
floats	IEEE 754, `f32` = 4 bytes, `f64` = 8 bytes
scalars	written as a tensor with no dimensions: one value
file length	exactly `product(dimensions) × element_width / 8`

A file of any other length is reported by verify as a failing case with a note, not as a crash.

Out of scope for a bundle

a bundle does not	because
choose points or seeds	comparability requires two solutions to run identical data; the plan is the platform's
execute the solution	orchestration is the runner's; the bundle is invoked, it does not invoke
aggregate verdicts	`verdicts.json` is a list; scoring is the platform's
access the network	it is handed a directory

Compatibility checklist

A bundle in another language must match the reference implementation on:

bytes — make at the same point and seeds produces byte-identical cases/;
arguments — same commands, flags, --seeds syntax;
layout — same file and directory names, same numbering;
JSON — same fields in manifest.json and verdicts.json;
exit codes — including 0 from verify on failing verdicts.