Collecting RLHF preference data with compere

The problem

Reward-model training needs preference labels: given a prompt and two responses, which is better? Labelling every response pair is expensive, and most pairs are easy calls that teach the reward model little. You want to spend annotator budget where the signal is.

How compere approaches it

1. Register each candidate response as an entity (optionally scoped per prompt).
2. Ask /mab/next_comparison for the response pair whose outcome is most uncertain.
3. Annotators (or a judge model) record the winner via /comparisons/.
4. The Elo leaderboard gives a graded preference ordering you can export as reward signal.

Where it fits (and where it doesn't)

compere fits when the bottleneck is deciding which pairs to label, not the labelling itself. It is a data-collection and ranking layer, not a reward-model trainer — you export its ordering into your training pipeline.