The execution layer

The execution layer takes a submission — an implementation frozen at a commit — and turns it into measured, judged numbers. It exists to pin down the four things a time belongs to: the code, the machine, the environment, and the point the time was taken at. Change any of the four and it is a different number.

The vocabulary — point, case, sample, the three verdicts — is defined in Points, cases, verdicts. This page is the pipeline.

The chain

implementation      the author's answer, versioned by commits
   │  press Benchmark, or submit
   ▼
submission          one commit, frozen: repo, commit, image, requirements
   │  the platform plans: points, seeds, bundle, limits
   ▼
job                 a few kilobytes of recipe — no data travels
   │  a matching runner takes it
   ▼
run                 what one machine actually did: times, verdicts, log

A submission freezes what is measured. The artifact is a commit, never a branch — main today and main tomorrow are different code.

A job is the platform's plan: which points, which seeds, which testing bundle (by id and content hash), which image (by digest), what limits. No test data is in it; data is derived deterministically from (point, seed) by the bundle's generator, so the recipe is the data.

A run is one runner's execution of one job. It records everything needed to read the numbers later: the machine as it described itself at assignment, the image reference, per-case times and statuses, per-stage clocks, the memory profile, the build log. Runs are never rewritten; a newer run replaces an older one only in the current state of a point.

Who chooses what

A runner chooses nothing. It measures.

What a runner does with a job

clone the solution at its commit
build it in the declared image                 once per job

for each point in the plan:
    bundle make   <dir> --point … --seeds …    the bundle's container
    ./solution    <dir>                        the solution's container
    bundle verify <dir>                        the bundle's container
    collect results.json and verdicts.json
    delete <dir>

report the run: times, statuses, verdicts, stages, memory, log

The bundle and the solution run in different containers. They have different runtimes, and an isolation failure in the solution must not land next to the oracle. expected/ is never mounted into the solution's container — the right answers are absent from its filesystem, not hidden in it.

One process per point, because init builds a context for one set of parameters. A hundred points are a hundred launches; process start and init are not in the timing window, so this costs only calendar time.

The timing window

For a plain (cleartext) solution:

init(p)                context, tables, keys      clocked, reported, never scored
run(state, inputs)     the answer                 ← the metric, whole
free(state)                                       optional

Only run is inside the window — a monotonic clock around the call and nothing else. Reading a case from disk and writing the answer happen in generated code outside the window, so the solution's code is not executing while data is visible to it.

An encrypted solution has a longer perimeter — key generation, encoding, encryption, decryption, decoding — every stage clocked and none of it scored except the measured call. See Encrypted solutions.

What comes back

Per case: a time and a status — ok · wrong · timeout · crashed · oom — and the bundle's verdict with its metrics. A failing case is a result, not an error: a run that died on the seventh point of ten still reports the six before it, and a seed that produced no answer is reported as lost, in its place, not silently dropped.

Per point, the platform folds seed timings on the way out — min, median, mean, max — and keeps every individual seed result, so the spread itself can be shown. Nothing is stored folded.

Beside the times, a run carries:

• per-stage clocks — for encrypted solutions: keygen, per-case encoding, encryption, decryption, decoding;
• the memory profile — the runner samples the solution container's resident memory five times a second, per point, so the pages can draw where the memory went, not just its peak;
• the build log — the compiler's own words, kept with the run.

What keeps it honest

Determinism. (point, seed) gives the same bytes on any machine. Any verdict can be re-derived at home from (bundle, point, seed) without the runner's help.

Everything pinned. The job pins the commit, the bundle and the image by content, and the run records the machine. A number on a page can always answer "measured what, where, in what, judged by what".

The platform judges. The harness writes an answer and a time; the bundle's verifier judges each case; the platform judges the point under the bundle's declared bar. A run on a smaller sample than the bar is defined over is measured, honestly not judged, and never ranked — see Comparability and standings.