compere vs. AHP-style ranking SaaS
AHP (Analytic Hierarchy Process) tools — 1000minds, TransparentChoice, SuperDecisions and similar — also use pairwise comparisons. They are aimed at a different problem: multi-criteria decision analysis where every option must be scored across several criteria with explicit weights. Compere is a single-dimension ranking tool. Here is when each fits.
| Dimension | compere | AHP ranking SaaS |
|---|---|---|
| Mental model | One dimension: “which is better?” | Multiple criteria with weights, e.g. cost / risk / time. |
| Comparison granularity | Binary outcome per pair (win, loss, draw). | Graded preference scale per criterion (typically Saaty's 1-9 scale). |
| Underlying math | UCB1 pair selection + Elo rating updates. | Eigenvector computation on a pairwise preference matrix per criterion, then weighted aggregation. |
| Pair selection strategy | UCB — the system picks the next pair adaptively. | Exhaustive — reviewers compare all pairs (or all in a hierarchy node). |
| Number of comparisons required | Sub-quadratic: usually 5x-15x the number of entities. | Quadratic per criterion: n(n-1)/2 comparisons per criterion node. |
| Audience | Engineers and researchers running comparison studies (eval, RLHF, A/B). | Decision-makers and analysts running structured group-decision exercises. |
| Consistency check | None built in — we do not run an inconsistency metric. | Saaty's Consistency Ratio is standard. |
| Delivery | Open-source (MIT). Self-host via pip or Docker. | Typically commercial SaaS with seat-based pricing. |
| API surface | HTTP / FastAPI, library import, OpenAPI docs at /docs. | Web UI primarily; some offer export to Excel; APIs vary. |
When AHP-style tools are the right answer
You are running a structured decision exercise where the decision criteria are themselves part of the deliverable — for example, a procurement evaluation where the weighted sub-scores need to appear in a board report, or a public-sector exercise where the methodology must be auditable end-to-end. AHP was built for that, and the SaaS tools that implement it are mature.
When compere is the right answer
You have a one-dimensional ranking problem — “which model output is better?”, “which headline performs best?”, “which design wins?” — and you do not want to spend the reviewer-time required for an exhaustive pairwise matrix. Compere’s UCB pair selection means you collect verdicts on the pairs that move the ranking, and skip the obvious ones.
What we are not claiming
We are not claiming to replace AHP for multi-criteria decisions. The mental model is genuinely different; if you have weighted criteria, you want AHP’s explicit framework, not a single Elo number. We are claiming that for single-dimension preference ranking, AHP’s exhaustive-pairwise step is overkill, and compere lets you skip it.