compere vs AHP-style ranking SaaS

Dimension	compere	AHP ranking SaaS
Mental model	One dimension: “which is better?”	Multiple criteria with weights, e.g. cost / risk / time.
Comparison granularity	Binary outcome per pair (win, loss, draw).	Graded preference scale per criterion (typically Saaty's 1–9 scale).
Underlying math	UCB1 pair selection + Elo rating updates.	Eigenvector computation on a pairwise matrix per criterion, then weighted aggregation.
Pair selection	UCB — the system picks the next pair adaptively.	Exhaustive — reviewers compare all pairs in a hierarchy node.
Comparisons required	Sub-quadratic in practice, thanks to adaptive selection.	Quadratic per criterion: n(n-1)/2 comparisons per criterion node.
Audience	Engineers & researchers running comparison studies (eval, RLHF, A/B).	Decision-makers & analysts running structured group-decision exercises.
Consistency check	None built in — we do not run an inconsistency metric.	Saaty's Consistency Ratio is standard.
Delivery	Open-source (MIT). Self-host via pip or Docker.	Typically commercial SaaS with seat-based pricing.
API surface	HTTP / FastAPI, library import, OpenAPI docs at `/docs`.	Web UI primarily; some export to Excel; APIs vary.

Dimension

compere

AHP ranking SaaS

Mental model

One dimension: “which is better?”

Multiple criteria with weights, e.g. cost / risk / time.

Comparison granularity

Binary outcome per pair (win, loss, draw).

Graded preference scale per criterion (typically Saaty's 1–9 scale).

Underlying math

UCB1 pair selection + Elo rating updates.

Eigenvector computation on a pairwise matrix per criterion, then weighted aggregation.

Pair selection

UCB — the system picks the next pair adaptively.

Exhaustive — reviewers compare all pairs in a hierarchy node.

Comparisons required

Sub-quadratic in practice, thanks to adaptive selection.

Quadratic per criterion: n(n-1)/2 comparisons per criterion node.

Audience

Engineers & researchers running comparison studies (eval, RLHF, A/B).

Decision-makers & analysts running structured group-decision exercises.

Consistency check

None built in — we do not run an inconsistency metric.

Saaty's Consistency Ratio is standard.

Delivery

Open-source (MIT). Self-host via pip or Docker.

Typically commercial SaaS with seat-based pricing.

API surface

HTTP / FastAPI, library import, OpenAPI docs at /docs.

Web UI primarily; some export to Excel; APIs vary.

When AHP-style tools are the right answer

You are running a structured decision exercise where the decision criteria are themselves part of the deliverable — a procurement evaluation where weighted sub-scores must appear in a board report, or a public-sector exercise where the methodology must be auditable end to end. AHP was built for that, and the SaaS tools that implement it are mature.

When compere is the right answer

You have a one-dimensional ranking problem — “which model output is better?”, “which headline performs best?”, “which design wins?” — and you do not want to spend the reviewer time an exhaustive pairwise matrix requires. compere's UCB pair selection means you collect verdicts on the pairs that move the ranking and skip the obvious ones.

What we are not claiming

We are not claiming to replace AHP for multi-criteria decisions. The mental model is genuinely different; if you have weighted criteria, you want AHP's explicit framework, not a single Elo number. We are claiming that for single-dimension preference ranking, AHP's exhaustive-pairwise step is overkill, and compere lets you skip it.