Points, cases, verdicts

For a specification to be measurable, a benchmarking environment has to be set up for it: a testing bundle that makes the inputs, knows the right answer and judges each case — plus the bar that says when a point counts as passed. This page describes how that judging works and what every word in it means precisely.

Everything here is defined over the specification's signature. A signature declares three things: parameters, arguments and results.

spec poly_mult 1.0.0 "64-bit coefficients" {
    kernel poly_mult<type T = i64, N: u32, L: u32>(
        %a: tensor<N x L x T>,      ← argument
        %b: tensor<N x L x T>,      ← argument
    ) -> %c: tensor<N x L x T>      ← result
}

Here N: u32 and L: u32 are parameters — strict scalars that set the size of the problem. %a and %b are arguments, the solution's inputs; %c is a result, its output. type T = i64 is a type parameter, already settled by the specification.

Point

A point is a complete assignment of values to every parameter of the signature: {"N": 1024, "L": 2}. Type parameters are not part of a point. A point fully determines the shape of every buffer; its name is written N1024-L2.

Case

A case is a pair (point, seed); the seed is an unsigned integer. A case fully determines the input bytes, because generate(p, seed) is deterministic.

Sample

A sample is the set of seeds S run at one point within one run.

The minimal sample is how many cases must run at a point for a verdict about it to carry any weight. It is declared by the bundle's bar ([accept] seeds = n, passed = k — see the testing bundle): n is the minimal sample, k is how many of it must pass. Without a bar the minimal sample is one case, and every case must pass.

Case verdict

verify(p, inp, exp, got).passed ∈ {passed, failed}. A case the solution gave no answer for — it crashed, or ran out of time — is lost; lost is not passed.

Point verdict

A point is judged within a run, under the bar of the bundle that judged it:

• no bar: the point is passed ⇔ every case in S passed;
• bar (seeds = n, passed = k): with |S| < n the verdict is undefined — the run is measured, not judged; with |S| ≥ n the point is passed ⇔ |passed cases| / |S| ≥ k / n.

Coverage

An implementation declares the region of points it answers for: a list of blocks, each block a set of constraints on parameters — an interval, a set of values, or a predicate (power of two, even, odd, divisible by). An implementation claims a point if the point satisfies every constraint of at least one block. Outside the claimed region the platform never measures it, and the absence of a number there is not a failure — pages show it as NOT CLAIMED.

A claim is a promise, not a fact. Only a run confirms coverage: for an implementation to say it covers the point {"N": X, "L": Y}, it needs a run where at least the minimal sample was measured at that point and at least k cases passed — that is, a run where the point's verdict is passed.

Run verdict

A run is one execution of one submission — the frozen code of an implementation — on one machine, under the platform's plan: points, samples, bundle, image. Its verdict: passed ⇔ every point in it is passed; any point failed makes it failed; no failures but a point without a verdict leaves the run without one.

Passed at a point, on a machine

The current answer is the point's verdict in the newest judged run of the implementation on that machine that contains the point. A new run replaces the old one in the current state; the runs themselves are never rewritten and never disappear.

Standings

At a fixed point on a fixed machine, standings rank only implementations whose verdict there is passed with no lost cases. A run below the minimal sample is listed apart — measured, not ranked — and says why.