On why a great-looking pooled AUC can hide a portfolio of coin-flips
Give me a portfolio of actions with different success rates and I’ll hand you an NBA model set where every single model is pure noise — zero discriminative power — that still reports a pooled AUC of 0.85. Measure discrimination as a single AUC across all the actions at once — pool every interaction into one ROC curve — and that’s the number that falls out: 0.85, or 85 in Pega’s metrics. It would look like one of the healthiest deployments you’ve ever seen. It would also be worthless. This article is about why we report AUC as a per-action weighted average, why that number comes in lower than the pooled one you might be used to, and why the pooled one is the wrong measure of a model’s discriminative power to begin with.
Take three actions with base rates of 2%, 10%, and 60%, each served by a coin-flip model (AUC = 0.50). Pool the scores and outcomes, draw one ROC curve, and you land on 0.846. Compute AUC within each action and weight by response count, and you get 0.504 — exactly where three random models belong.
Figure 1. Same 100,000 interactions, two stories. The orange pooled curve reaches 0.846; the teal per-action curves sit on the chance diagonal. Reproduces the AI Chapter white paper (SEED = 18).
In one sentence: pooling scores across actions with heterogeneous base rates conflates two distinct signals in a single ROC statistic — the trivial between-action separation (high-base-rate actions simply score higher) and the operationally meaningful within-action discrimination — so the resulting AUC is inflated by action-mix rather than by any model’s ability to rank responders above non-responders.
Why the curve lies
Look at the orange curve. It isn’t separating likely responders from unlikely ones — it’s separating Action C from Action A. Positives pile up in the 60% action, negatives spread across the 2% and 10% actions, and the ROC curve happily cleaves them apart. The models did none of this work; the base rates did. This is a textbook case-mix confound, structurally the same mechanism as Simpson’s Paradox: an omitted variable — here, action identity — manufactures an association that exists in none of the sub-populations.
Driven by both, dominated by one
Here’s the entire mechanism in four numbers. Same three-action setup, but now turn two dials independently: whether the models actually work (perfect vs. pure noise), and whether the base rates are spread apart (2/10/60) or flat (all 30%).
| scenario | pooled AUC | weighted AUC |
|---|---|---|
| perfect models · spread base rates | 0.889 | 0.749 |
| perfect models · flat base rates | 0.742 | 0.742 |
| pure-noise models · spread base rates | 0.846 | 0.504 |
| pure-noise models · flat base rates | 0.506 | 0.506 |
Same simulation, SEED = 18. The noise · spread row is the headline case above.
Read the weighted column first. It tracks one thing only — whether the models work: ~0.75 when they’re perfect, ~0.50 when they’re coin-flips — and it doesn’t budge when you change the base-rate spread. That invariance isn’t luck. AUC is a ranking metric, and ranking metrics are blind to class prevalence; computing it within each action’s own population immunizes it against the action mix by construction.
Now read the pooled column, under spread specifically. A flawless model set scores 0.889. Replace every model with a coin flip and you still score 0.846. Going from perfect to pure noise costs you four hundredths of an AUC point. Once the base rates are spread, the pooled number is very nearly blind to whether the models do anything at all — it’s a base-rate-spread meter with a faint quality signal riding on top.
And notice the two flat-rate rows, where pooled and weighted agree to the decimal (0.742/0.742 and 0.506/0.506). That’s the tell: the entire pooled–weighted gap is the base-rate spread. Nothing else.
So the honest framing is this. Pooled AUC is driven by both model quality and base-rate spread — and dominated by the spread to the point where it can barely see the quality, which is the one thing you actually wanted to measure. Weighted per-action AUC ignores the spread by construction and reports the quality straight. It’s not “proportional” to either; the relationship is monotone with an interaction, spread compressing what’s left of the quality signal. But the direction is unambiguous: spread inflates the pooled number, and it can’t touch the weighted one.
If the cannibalization post rang a bell, it should
There, the super-sub fallacy came from treating a contextual bandit’s arm rates as if they transferred across audiences, when they’re audience-conditional. Same disease, different organ: a number that is only meaningful within a conditioning set gets ripped out of that set and read as if it were global. Cannibalization was about aggregating actions you shouldn’t compare; this is about aggregating populations you shouldn’t pool. Both times the aggregate flatters you, and both times the fix is the same — respect the strata.
And yes, we did it too
Before this reads as us lecturing from the cheap seats: we made this exact mistake in our own product. For a long stretch, prediction-level AUC in Prediction Studio was reported by pooling across the actions underneath — the very sin described above. Fortunately, the Adaptive Model screens have always shown the right thing: AUC computed per model, within each action’s own population, presented as an overall weighted average.
One honest exception survives to this day: the model download for AGB models carries only the pooled AUC, not the weighted average. It’s an internal artifact, not a reporting surface — nothing customer-facing leans on it — but if you ever find yourself reading an AGB download, know that the figure in front of you is the pooled one. Read it with everything above in mind.
The honest number
The honest number is boring by comparison. Response-count-weighted per-action AUC asks the only question that survives contact with production: among customers shown the same action, does the model rank a responder above a non-responder? It lands around 0.60–0.70 for genuinely strong NBA — lower than the mirage, but real, and it stays put when your action mix shifts — new actions, policy changes, seasonality — instead of drifting with it.
One precision worth stating, since a sharp reader will ask which personalization: weighted per-action AUC measures within-action personalization — can the model tell customers apart for a fixed action. It is silent on whether arbitration then picks the right action for a given customer; that’s a separate question, measured within-customer, and no pooled curve answers it either. Two different jobs, two different numbers — just don’t let the pooled one impersonate either.
So the next time someone waves a single 0.80+ at you as proof of model health, don’t ask “how good is the model?” Ask “how different are your base rates?” The more heterogeneous the portfolio, the bigger the lie. Weight it, or don’t trust it.
Reproduce it yourself
Fifteen lines, numpy + scikit-learn, SEED = 18. Prints pooled AUC = 0.846 and weighted AUC = 0.504.
import numpy as np
from sklearn.metrics import roc_auc_score
SEED, N = 18, 100_000
BASE_RATES = np.array([0.02, 0.10, 0.60]) # three actions: A, B, C
rng = np.random.default_rng(SEED)
action = rng.integers(0, 3, N) # each action shown ~1/3 of the time
br = BASE_RATES[action]
y = (rng.random(N) < br).astype(int) # outcome ~ that action's base rate
score = np.clip(rng.normal(br, 0.05), 0, 1) # random model: score tracks base rate only
pooled = roc_auc_score(y, score) # one ROC curve over the pooled data
weighted = sum((action == a).sum() * roc_auc_score(y[action == a], score[action == a])
for a in range(3)) / N # per-action AUC, weighted by response count
print(f"pooled AUC = {pooled:.3f}") # -> 0.846
print(f"weighted AUC = {weighted:.3f}") # -> 0.504
The models are pure noise — score carries no within-action signal, it only tracks each action’s base rate. That’s enough to push the pooled curve to 0.846. To reproduce the four-cell table above, run the same generator twice more: swap BASE_RATES for a flat [0.30, 0.30, 0.30], and swap the noise score for a genuinely discriminative one (score = P_true) — pooled tracks both dials, weighted tracks only the model.