When Better Targeting Makes an Action Look Worse
There is a very natural question in NBA measurement:
“This action had a 5% success rate in the random group and 4% when the model was active. Did the model make this action worse?”
Usually, no.
That answer sounds evasive until you look at what the number is actually measuring. The success rate of an individual action is not just a property of the action. It is a property of the action and the customers the policy decided to show it to. Once a model starts routing customers, that audience changes. Sometimes it changes exactly because the model is doing the right thing.
So here is the serious version of the cannibalization story:
Action-level lift is not a clean model-quality metric. Portfolio lift is.
If you want to know whether NBA is creating more value, compare the overall success rate of model-driven decisions against the randomized control. If you want to know whether an individual model can rank customers for a fixed action, use within-action AUC. If you want to know whether an action caused incremental value — conversion, retention, margin, or whatever downstream outcome matters — compare similar eligible customers where the action was randomly available for some and randomly withheld from others.
But do not ask one action’s observed lift to answer all three questions. It cannot.
This is the fourth, and for the time being probably last, article in a small tetralogy about measurement traps in NBA: why an action that is great for a small sub-population should not necessarily be rolled out to the wider audience, why globally measured AUC can lie, and why optionality needs to be much greater than seven. The common thread is simple: a number can be perfectly calculated and still answer the wrong question.
The small example, because the arithmetic matters
Take two customers and two actions.
| Offer A | Offer B | |
|---|---|---|
| Brian | 6% | 10% |
| Sammy | 4% | 2% |
| Random base rate | 5% | 6% |
Brian is generally more responsive, and likes B more than A. Sammy is less responsive, but A is still his better offer.
Under random allocation, both customers see both actions. Offer A gets a blended 5% success rate: Brian’s 6% mixed with Sammy’s 4%.
A good model does something simple. It sends Brian to B and Sammy to A.
Now Offer A only sees Sammy. Its measured rate drops from 5% to 4%.
That gives Offer A a -20% action-level lift.
And the model is perfect.
Figure 1. Reconstructed from the Brian/Sammy example in the internal lift-measurement paper. Action A looks worse, but total accepts rise from 110 to 140 per 2,000 decisions: +27% portfolio lift.
The action-level number says “worse” because the population changed. The portfolio number says “better” because customers got better-matched offers.
Both numbers are arithmetically correct. Only one answers the NBA question.
The mechanism: audience shift
Under random targeting, each action sees a broad sample of the eligible population. Its success rate is a mixture: high responders, low responders, people for whom this is the best offer, and people for whom it is merely acceptable.
Under model-driven targeting, that mixture is no longer random. The action increasingly sees the customers for whom it is the best available option. If the high responders have an even better action available, the model removes them from this action’s audience.
That is not a bug. That is personalization.
This is why the phrase “this action’s lift” is slippery. Lift relative to what population? The random population? The model-selected population? The customers who were eligible for the action? The customers for whom it was top-ranked? Those are not interchangeable sets.
The more contextual the policy becomes, the more those sets diverge.
Three questions people accidentally collapse into one
When someone asks for “lift by action,” they may mean at least three different things.
1. Did NBA create more value overall?
That is a policy question.
Measure it at the policy level:
Portfolio lift = (Success rate model-driven - Success rate random control)
/ Success rate random control
This is the cleanest production scoreboard because the unit of optimization is the decision, not the action. NBA does not try to maximize every action’s individual CTR. It tries to choose the best eligible action for each customer.
2. Is the model for this action discriminating well?
That is a model-quality question.
Use within-action AUC, computed inside the action’s own prediction scope. This asks: among customers evaluated for the same action, does the model rank responders above non-responders?
This is exactly the reason pooled AUC is dangerous. Pooling across actions lets action mix masquerade as discrimination. Action-level lift has the mirror-image problem: model allocation changes the audience, and the audience masquerades as action performance.
Same disease, different metric.
3. Did this action cause incremental value?
That is a causal question.
The practical answer is an action-level holdout. For customers where Action A is eligible and would normally be a serious candidate, randomly suppress A for a small control group and let arbitration choose the next-best alternative. Compare outcomes for A available versus A withheld within that same eligible population.
The outcome does not have to be the immediate click. It can be conversion, retained value, product take-up, revenue, margin, complaints avoided, or any longer-term business measure you can attribute reliably. The important part is the random withholding of the action among comparable customers; without that, the measured difference is still tangled up with who the policy chose to show the action to.
A randomized model-control split can tell you whether the policy works. It does not automatically tell you the causal lift of every individual action, because the policy also changes the composition of customers receiving that action.
A practical diagnostic workflow
So should we remove action-level lift from dashboards? No. But we should demote it.
A useful reporting hierarchy is:
- Lead with portfolio lift. Did the model-driven policy beat randomized control overall?
- Monitor within-action model quality. Is AUC healthy for the models attached to each action?
- Use action-level lift as a diagnostic, not a verdict. If an action is persistently negative, inspect what it is competing against and who is still receiving it.
- Check whether the action is reaching the same audience. For the action whose lift changed, compare the customers who received it in test and control: segment, propensity band, eligibility reason, issue/group, channel, and which other action would have won if this one was not shown.
- Only make action-level causal claims from causal designs. Otherwise say “observed action-level lift,” not “the action caused lift.”
The fourth step is the one that usually settles the argument. If the action’s test audience is materially different from its control audience, the lift number is contaminated by allocation. That does not make it useless. It makes it descriptive.
In CDH, the practical starting points are already there. Use Action Performance for the “what happened” view: impressions, clicks, accepts, success rates, and trends by issue, group, action, treatment, channel, and time window. Use Action Analysis for the “why did it happen” view: funnel, optionality, engagement policy, prioritization, sensitivity, and win/loss analysis. The win/loss view is especially useful here because it tells you what an action tends to beat, what it tends to lose to, and therefore whether a falling action rate is really an audience-shift or competition story. Use Prediction Studio / Adaptive Models for model-health checks such as AUC, success rate, lift, and response counts at model level. And if the dashboard view is not enough, export or query Interaction History / Actuals data and compare the served populations directly; for analysis outside Infinity, pdstools can be a useful helper.
And descriptive numbers are allowed to be useful, as long as we do not promote them to causal truth.
The uncomfortable implication
A well-functioning NBA system will sometimes make an action’s observed rate go down.
That sentence is worth sitting with.
It can happen because the action is bad. It can happen because the model is bad. But it can also happen because the model found better homes for the customers who previously made that action look good.
Those are three very different stories. A single action-level lift number cannot tell them apart.
So the next time someone points to a negative lift for one action and says “the model hurt this offer,” the right response is not “ignore it.” The right response is:
“Maybe. But first show me the audience shift, the competing actions, the within-action AUC, and the portfolio lift.”
If the portfolio is up, the model quality is healthy, and the action’s audience moved toward the customers for whom it is genuinely the best remaining option, then the negative action lift is not a failure signal. It is the footprint of personalization.
Measure the policy globally. Diagnose the action locally. Do not confuse one for the other.