BetterRankings View on GitHub

Off-policy evaluation, in your browser

Test a new ranking policy before it meets real traffic

BetterRankings estimates a ranking policy's clicks from logged data alone, then checks every estimate against a simulated online test where the true answer is known.

Estimates against the truth

Running the first evaluation…

Clip at 20

Applies to clipped IPS and doubly robust. Lower clips cut noise from rare, heavily weighted queries but pull the estimate down.

Show these numbers as a table

How it works

Evaluate a policy from logs you already have

If the current policy shows lists with some randomness and records how likely each list was, its old logs can answer a new question: how many clicks would a different policy have earned?

Evaluating from logs

A ranking system already shows lists and records clicks. Reweighting those logs estimates a new policy's clicks without showing it to anyone.

Propensities

The propensity is the probability the old policy gave the exact list it showed. When a new policy would show a list more often, the clicks on it count for more. That ratio is the importance weight.

Why naive comparison misleads

Averaging clicks over logged lists the new policy agrees with sounds fair, but the old policy showed its favourite lists most often and where it liked them best. Weighting by propensities undoes that choice.

The estimators

Inverse propensity (IPS)

Reweights every logged query by its importance weight. Unbiased, but noisy when weights are large.

Self-normalised IPS

Divides by the total weight instead of the row count. A little bias, much less noise.

Clipped IPS

Caps each weight at the clip level. Lower noise, but it pulls the estimate down.

Reward model

Learns a click model from the log and asks it about the new policy. Steady, but only as good as the model.

Doubly robust

Starts from the reward model and corrects it with weighted errors. Holds up if either part is right.

The check

The simulator knows every true click probability, so it runs the online test real teams cannot run for free. The coral line is that answer.