Building eval leaderboards with compere

The problem

Scalar eval scores are hard to calibrate across raters and tasks, and absolute grades drift. Head-to-head “which output is better?” is a far more reliable judgement — but you need a principled way to turn a pile of pairwise verdicts into a single ranking.

How compere approaches it

1. Register each model / prompt / configuration as an entity.
2. Serve UCB-selected pairs to human judges or an LLM-as-judge.
3. Record verdicts; Elo updates the ratings after every comparison.
4. GET /ratings for a sortable leaderboard that is a pure function of the votes.

Where it fits (and where it doesn't)

compere fits single-dimension “which is better overall?” leaderboards. If you need per-criterion weighted scoring, an AHP-style tool is a better match — see the comparison.