THE PLATFORM
The performance map
Every successful evaluation adds one measured observation: an implementation, valid for a point, produced a result on a known machine in a known environment. Over time these observations form a performance map.
Instead of answering
What is the fastest implementation?
the platform answers a more precise question:
Which valid implementation performs best for this point, on this hardware, under these constraints?
There need not be a single winner
The frontier can change across parameter ranges and across hardware. A table of observations makes this concrete:
CPU H100 B200 Point A I1 I3 I3 Point B I2 I4 I4 Point C I2 I5 I6
No implementation wins everywhere. The useful knowledge is the mapping between context and implementation: where each one is valid, where it wins, and how it compares with the alternatives. Each new evaluation adds evidence to that map.
Why local benchmarking is not enough
An engineer or an AI agent can generate an implementation and benchmark it locally. That answers one question — how it performed in that environment — and leaves the ones that make a comparison meaningful unanswered:
- Is it solving exactly the same problem as the alternatives?
- Is it valid under the same constraints?
- Was it measured using the same methodology?
- How does it compare with implementations evaluated before?
A number copied from a README is not equivalent to a controlled platform result. FHERMA supplies the shared specification, the common evaluation methodology, the controlled runner context and the accumulated result history that make those comparisons meaningful. As implementation generation gets cheaper, this comparative evidence becomes more important, not less.
What accumulates
FHERMA builds three connected layers:
| Layer | What it records |
|---|---|
| Computational knowledge | what computations exist, how they are defined, how they depend on one another, and where each implementation is valid |
| Evaluation evidence | which implementations were tested, under which conditions, against which requirements, and how they performed |
| Execution infrastructure | the environments and interfaces to build, validate, benchmark and run those implementations |
Together they move the answer from
I found some code that appears to implement this.
to
This implementation solves the computation I need, is valid for my parameters and constraints, has been evaluated under known conditions, and is the best measured option for my case.