Runners and images

The runner

A runner is a machine that measures. It holds no state between jobs — reinstall the server and nothing is lost — and it makes no decisions: what to measure, at which points, in which image all arrive in the job.

What comes from where:

Admission and liveness

A runner joins the fleet with a join token and starts pending; an administrator approves it before it is offered work. From then on it speaks a heartbeat: a runner that goes silent is marked offline and receives nothing; the moment it speaks again it is idle — coming back is done by speaking, not by re-registering.

The run records the machine as it described itself when the job was assigned. A runner re-provisioned since must not silently re-describe a number it produced last month.

The reference machine

One runner may be marked as the reference: the machine whose times the catalogue quotes. Every other runner's numbers are recorded and compared only with themselves; what those runs add is confirmation that the answers are right in more places than one.

Targeting

A benchmark request may name the runners it wants. That is how two implementations end up with numbers on the same machine — the only place a ranking between them exists. See Comparability and standings.

Hardware and environment

The rule: what can be installed is environment, and it lives in the image; what cannot is hardware, and it belongs to the machine.

machine:  cores · memory · architecture · avx512f · GPU · driver
image:    openfhe · poulpy · cuda toolkit · compiler · python

"Needs a GPU" decomposes across all four parties: the card and the driver are the machine's; the toolkit is the image's; the --gpus flag is the runner's. The platform checks the pieces fit — a CUDA 12 toolkit does not run on an older driver — before a job is offered, not after it fails.

Requirements an implementation declares are matched in one closed vocabulary of capability names, so openfhe, OpenFHE and open-fhe cannot silently fail to match; an unknown name is refused at submission with the name spelled out. Requirements also set the class an implementation competes in: ask for a GPU and you are compared with those who use one.

Images

An image is identified by digest, never by tag — a tag is moved, and fherma/openfhe:1.5.1 tomorrow may be different bytes. The digest is to an image what a commit is to a repository.

The catalogue records, per image:

provides is what an implementation's software requirements are checked against at submission: declare openfhe and pick an image that does not carry it, and the submission is refused there — not discovered by a failing build on a machine an hour later.

The image is a property of the job

submission   the image the author chose
job          the image this measurement ran in, by digest
run          the reference, recorded in the result

The same code in a different image is a different measurement, and the pages say so: every run names the image it happened in, digest included.

Two containers per point

The bundle runs in its own image — the platform's Python environment — and the solution in the image its author chose. Different runtimes, different trust: nothing the solution does can reach the oracle's process, and the directory with expected answers is not mounted into the solution's container at all.