Writing a testing bundle

The authored components of a testing bundle are written in a scaffolded directory. The cycle is four commands.

Scaffold

fherma testing init polynomial-multiplication/negacyclic@1.0.0
fherma testing init polynomial-multiplication/negacyclic          # latest published
fherma testing init --signature ./sig.fhk                         # no platform involved

The platform is asked for one thing — the signature — and the scaffold is built locally by fherma-lang; with --signature no network is touched at all. What lands:

negacyclic-1.0.0/
├── fherma.toml      generated — which specification this belongs to
├── fherma.py        generated — Point, Inputs, Outputs, Stream, Tensor
├── main.py          generated — the bundle's command line: make · verify · info
├── README.md        generated — the signature, and what to write
├── generate.py      yours
├── oracle.py        yours
├── verify.py        yours, and usually unnecessary
├── vectors/         worked examples, for accepting the oracle
└── assets/          data files the three functions may read

The types in fherma.py are derived from the signature; the author declares no types. Write generate.py and oracle.py; verify.py is required only when the comparison rule is not exact equality.

Name parts to scaffold one or two: --generator, --oracle, --verifier. When the specification's signature moves, fherma testing init --update rewrites the generated files and never touches yours.

Check

fherma testing inspect                # read the three without running them
fherma testing inspect oracle

Static checks: the file parses, the entry point exists with the right arity, the scaffolded stub has been replaced. Checks that require execution are reported as not run, not omitted.

Run

fherma testing run generator --point '{"N": 8, "W": 2}' --seed 7
fherma testing run generator --point '{"N": 8, "W": 2}' --twice     # determinism
fherma testing run oracle    --point '{"N": 8, "W": 2}' --seed 7

Each command runs the component in a child process with a time and memory limit. The output is a description — shapes, sizes, the first few values — not the full data. --twice runs the generator twice at the same seed and compares the results byte for byte: the determinism check.

The bundle is also an ordinary program — the same layout a runner uses:

python main.py make   /tmp/p --point '{"N": 8, "W": 2}' --seeds 0-7
python main.py verify /tmp/p

Submit

fherma testing submit
fherma testing submit --dry-run

The upload includes both authored and generated files, so the directory stored on the platform is identical to the one tested locally. The response reports each component's state and checks. fherma testing status reports the same later, including whether the signature has changed since the bundle was generated.