Evaluation Cards: Evidence Signals, Coverage, Limits, and Evaluation

This older record retains its original scope. Current buying qualifications above take precedence where terms have changed.

Kingy verdict: Evaluation Cards is useful because it treats a benchmark result as evidence that needs context, not as a self-explanatory score. Its public beta brings scattered evaluation reports into a common card interface and adds signals for reproducibility, completeness, provenance and comparability. That can improve due diligence, but a well-filled card does not validate the underlying benchmark or make unlike results comparable.

What the product actually is

The EvalEval Coalition launched Evaluation Cards on June 11, 2026 as an open-source interpretive layer over evaluation reporting. The launch article described a corpus of 101,955 reported results across 638 benchmarks, 31 organizations and 5,816 models as of June 9. Those counts are a dated provider snapshot, not a live Kingy audit, and they should not be repeated without the date.

The current official Space README clarifies the architecture. The web application is a read-only consumer of a materialized warehouse view produced upstream by the Eval Cards backend. It reads typed Parquet data and JSON sidecars through DuckDB; it does not author evaluation data, resolve identities or aggregate results itself. That distinction matters when a card looks incomplete or two records appear inconsistent: the root cause may sit in upstream normalization rather than the reader interface.

October 5, 2026 corpus snapshot: The public application reports 9,215 models, 314,216 result triples, 51 reporting organizations and 1,734 model developers. These are app-reported counts, not a Kingy warehouse audit. Its four percentages describe different measures: 7% of reported scores have a complete setup, completeness averages 48% across 107,303 triples, 11% of unique triples have reports from more than one party, and 20% of 2,896 setup-eligible groups diverge across variants. The last figure measures divergence, not a 20% model-quality or comparability success rate.

What the four signals can and cannot tell you

Reproducibility asks whether enough configuration detail exists to repeat an evaluation. Completeness surfaces missing fields. Provenance helps trace a reported result back to its source. Comparability asks whether two results were produced under sufficiently aligned conditions. These are interpretive aids, not quality scores. A reproducible evaluation can still test the wrong thing; complete metadata can document a weak dataset; provenance can lead to a marketing report; and a comparability warning does not establish which result is more trustworthy.

Use a card to identify the next document to inspect. Follow the provenance chain, read the benchmark and dataset documentation, check model and prompt versions, inspect sampling settings and confirm whether the displayed metric has the same definition across reports. For safety or procurement decisions, record unresolved discrepancies instead of averaging them away.

Pricing, openness and operating boundary

The coalition describes Evaluation Cards as an open-source beta and provides a public application and source-backed Hugging Face Space. Kingy found no separate paid plan for the card reader in the reviewed official material on July 28, 2026. That does not guarantee permanent free access, an SLA, bulk-export rights or a stable public API. The README documents a frontend and data pipeline, not a general-purpose supported API contract.

Teams considering reuse should verify the licenses for the application, warehouse data, benchmark descriptions and any upstream evaluation artifacts separately. A public result can still carry restrictions, attribution requirements or personal and confidential information that should not be copied into an internal system without review.

How Kingy would evaluate it

Select twenty results across open and closed models, old and recent benchmarks, repeated runs and known reporting disputes. For each card, trace the source, reproduce the displayed configuration from the original artifact, and compare the card’s four signals with an independent reviewer’s judgment. Log missing fields, duplicate identities, version ambiguity, broken provenance links and cases where the interface makes incompatible runs look adjacent.

Then test update behavior: note a source correction, measure when it appears upstream and confirm how the card changes. Export or capture the evidence needed for an internal decision and verify that citations remain stable. Kingy reviewed the public app, launch article, current Space README, Space and coalition surfaces; we did not reproduce an evaluation, audit the full warehouse or test contribution workflows.

Primary sources

• Evaluation Cards public application
• Official Evaluation Cards launch article
• Current official Space README
• Official Evaluation Cards Space
• EvalEval Hugging Face organization
• EvalEval Coalition site