How it works
Two algorithms, one honest answer
compere separates two questions most tools conflate: which pair should I ask about next, and how do I score the answer? UCB1 handles the first, Elo the second.
The request flow
picks next pair
updates ratings
Both steps read and write through SQLAlchemy — SQLite by default, PostgreSQL when configured.
Step 1 — UCB1 pair selection
The bottleneck in a comparison study is not the maths of scoring; it is deciding which
pair a rater should look at next. Comparing every pair is quadratic and wasteful — most matchups
are between an obvious winner and an obvious loser. compere frames pair selection as a
multi-armed bandit and applies the Upper Confidence Bound (UCB1) rule (implemented in
compere/modules/mab.py):
UCB(i) = win_rate(i) + c · sqrt(ln(N) / n_i)- win_rate(i) — the observed win rate of entity i (exploitation).
- c — the exploration constant, default
1.414(setUCB_EXPLORATION_CONSTANT). - N — total comparisons so far; n_i — comparisons involving entity i.
New entities get a large initial weight (UCB_UNEXPLORED_WEIGHT, default
1000.0) so they are surveyed before they are scored. The two highest-UCB entities
are paired for the next comparison. The effect: votes concentrate on genuinely uncertain
matchups, and you reach a stable ranking sub-quadratically.
Step 2 — Elo rating updates
Once a verdict comes in, compere updates ratings with standard Elo (implemented in
compere/modules/rating.py) — the same formulation used in chess:
expected_a = 1 / (1 + 10 ** ((rating_b - rating_a) / 400))
new_rating_a = rating_a + K · (actual_a - expected_a)- K — update step size, default
32(setELO_K_FACTOR). - Initial rating —
1500by default (setELO_INITIAL_RATING). - actual_a — 1 if a won, 0 if it lost (draws split the difference).
Nothing fancier is claimed. Because the update is transparent, you can explain any rating change to a stakeholder in two sentences — which is exactly why Elo was chosen over a latent-variable model.
An alternative: similarity pairing
A second selection strategy lives in compere/modules/similarity.py and is exposed
at /comparisons/next. Instead of maximising information gain, it pairs entities
that are similar — useful when you specifically want to resolve close calls. It is a
drop-in replacement for the UCB endpoint; the Elo layer does not change.
What compere deliberately leaves out
compere does not implement Bradley-Terry maximum-likelihood estimation, Thurstone Case V, TrueSkill, Glicko, or any neural ranker. Those models can be more statistically efficient, but they trade away interpretability. compere's bet is that for most applied ranking work — evals, RLHF, A/B, taste graphs — an interpretable UCB + Elo pipeline is the better tool.