Why optionality — not AUC — is usually the binding constraint on NBA performance
Takeaway…
Optionality is not a box you tick once you clear some threshold. It is the ceiling on everything your model can do. A model with a brilliant AUC and a handful of actions to choose from will quietly underperform a mediocre model with a rich set of actions, because no amount of ranking skill can select an action that was never in the eligible set.
Most of the time, the actions are not missing because you ran out of ideas. They are missing because targeting logic misfiled as eligibility, and suppression thresholds no one ever measured, remove them before arbitration runs. The single highest-return move in most deployments is not a better model — nor even a larger action library. It is a wider eligible set: keeping the strong actions you already have from being filtered out before the model ever runs.
A propensity model can only ever choose from the actions it is given. That sentence looks trivial, and it is routinely ignored. We invest heavily in the quality of the ranking — features, retraining, AUC — and almost nothing in the size of the set being ranked. Yet the second quantity sets a hard ceiling on what the first can deliver: a model with an excellent AUC and a handful of eligible actions will underperform a mediocre model with a rich set, because ranking skill cannot select an action that was filtered out before the model ran.
Optionality is the name for that ceiling — the breadth of the eligible set the model is allowed to choose from. This paper argues three things. First, that optionality, not AUC, is usually the binding constraint when strong models produce weak lift. Second, that under the skewed propensity distributions real customers actually exhibit, the value of an additional action keeps rising well past the point most deployments stop adding them. Third, that the missing actions are rarely missing by design — they are stripped out before arbitration by targeting logic that has been misfiled as eligibility, and by fatigue rules set to convenient round numbers.
This is the fifth in a running series on the metrics that mislead in Next-Best-Action. The cannibalisation piece showed that per-action CTR misleads once the model changes who sees each action. The pooled-AUC piece showed that unweighted AUC flatters itself on base-rate differences. Ivar’s Beyond AUC argued that model discrimination is not the same as NBA success and that lift in success rate is closer to the outcome that matters. Andy’s Beyond PxV showed that a value function optimal for one interaction is wrong for a relationship. Same shape every time: the convenient number is not the number that matters. This one is about optionality.
Where optionality came from
Optionality was not invented in a whiteboard session. It was reverse-engineered from a frustration.
We kept seeing the same pattern in deployments: strong models, disappointing lift. The AUC was healthy — 65+ — and yet the decisions coming out the other end were barely better than a static rule. The instinct in that situation is to blame the model. Retrain it, add features, tune the hyperparameters. And it almost never helped, because the model was not the problem.
The problem was that the model had nothing to choose from. It was ranking a candidate set of three or four actions per customer with surgical precision, and three or four was the whole game. A perfect ranking of a poor menu is a poor decision.
So optionality entered the vocabulary as the thing to check first when lift is low and the model looks fine. It captures a simple idea: model quality determines how close you get to the ceiling; optionality sets the ceiling. When the two disagree — great model, weak lift — optionality is almost always the reason.
There is a heuristic that has travelled a long way through the organisation: an optionality of around seven is “good enough.” As a floor, that is not a bad instinct. Below it, the search space is visibly starved. The mistake is treating a floor as a finish line. Optionality is not a bar you clear and forget. It is a quantity you keep raising for as long as raising it pays — and, as we’ll see, it pays for a lot longer than seven.
The best action is rarely the seventh one
Here is the part that the “we have enough actions” reflex misses.
NBA does not care about your average action. It selects the single best one for each customer and shows that. So the performance of the system is driven by the maximum propensity across the eligible set — not the mean, not the median, the max. And the expected maximum of a set of draws never falls as you add draws — and under any realistic distribution, it rises. This is a plain order-statistics result, and it is the whole argument in one line: the larger the eligible set, the higher the expected value of the best action in it.
The interesting question is not whether it rises — adding an action never lowers the expected best, and normally raises it — but how fast, and that depends entirely on how the good fits are distributed.
Real relevance is not uniform; it is skewed and sparse: for any given customer, most actions are a poor fit, and a few are genuinely strong. The genuinely strong one is the whole point of NBA — and it might be the third action in your catalogue, or the fortieth. You do not know in advance which. The only way to guarantee the best action is available to be chosen is to make sure it is in the eligible set — and the only way to do that reliably is to keep the set large.
One customer’s offers by success rate. The shaded band is a seven-action eligible set — every action in it is low-propensity, while the genuinely strong offers sit outside it.
To see how much this matters, plot the share of the achievable value a deployment captures as a function of optionality — the expected best action, as a fraction of the best the portfolio could ever offer that customer — under both distributions.
The same measure under two distributions. Blue: every offer roughly equal. Amber: a few strong, most weak — the case real customers exhibit.
Under uniform relevance (blue), returns saturate fast: seven eligible actions already capture 87.5% of everything available, and adding more barely moves the line. That is almost certainly where “seven is good enough” came from — and in that world it is a fair call. Under skew (amber) — where a given offer has, say, a one-in-ten chance of being a genuine strong fit — the same seven actions capture only about 45% of the achievable value, and the curve keeps climbing long after the uniform one has flattened: 74% at twenty actions, 90% at fifty. The shaded gap between the curves is the difference between what the heuristic assumes you have and what you actually have.
For the customers whose strong-fit offer is simply not in the eligible set, the model’s AUC is beside the point — it is ranking a shortlist that never contained the right answer, and no amount of ranking skill repairs that. That is why the best action might be number 20, or number 100, and why “we have seven, we’re fine” leaves most of the lift on the floor. The ceiling did not stop mattering at seven; under the only distribution that describes real customers, it had barely started.
Where the actions actually go
If more actions are so obviously better, why does anyone run with seven?
Not because they only built seven. Most mature CDH deployments have dozens or hundreds of actions in the portfolio — enormous marketing effort goes into adding them, and comparatively little into weeding out the poor performers or, more to the point, into not silently discarding the strong ones. The customer arrives eligible for a rich set — and then, before arbitration ever runs, that set is quietly stripped down to a handful by two mechanisms that were never meant to be doing this much work.
Targeting smuggled into engagement policies. Eligibility is supposed to answer one question: is this action permissible for this customer? That is a hard constraint — regulatory, contractual, a genuine product prerequisite. But walk through the suitability rules in a typical deployment and you will find something else entirely: conditions like “only offer the premium card to customers with income above €80,000.” That is not permissibility. That is a prediction about who will respond — and predicting who will respond is the model’s entire job. Every rule like that is a hypothesis the model could test empirically, frozen instead into a hard filter that removes the action from the set before the model ever sees it — a static assumption the model is never allowed to challenge, however much evidence would contradict it.
Suppression set to round numbers. Fatigue and cooldown rules — suppress an action after it has been shown so many times, or for a fixed window after a decline — remove actions from the eligible set based on interaction history, before arbitration runs. Their thresholds are almost always conventions (“suppress for thirty days after a decline,” “no more than three impressions”) chosen because they are tidy, not because anyone measured the response curve behind them. Each one removes actions the model would otherwise have ranked, and stacked together they can take a portfolio of eighty actions down to six by the time arbitration sees it.
It is worth being precise about the mechanism, because it is easy to overstate. In the NBA Strategy Framework, this history-based suppression is evaluated in the constraints step, before arbitration — so a suppressed action genuinely never reaches the ranker, and the optionality cost is real. Outbound channel limits (“at most N sends per channel per period”) are a related but separate control: they are enforced partly there and partly again after the framework, in FinalActionLimitsAndBundling, on the already-ranked winners. That post-arbitration pass trims the send list; it does not shrink the search space the model chose from. The argument in this paper is about the pre-arbitration suppression, not the bundling-stage limits — treating “contact policy” as a single monolithic pre-arbitration gate would overclaim, and it isn’t one.
Neither mechanism announces itself. Nobody logs “the best action for this customer was removed by a suitability rule.” The action simply isn’t there, the model picks the best of what remains, the decision looks reasonable, and the lost lift is invisible. This is the quiet part: your optionality problem does not look like a problem. It looks like a normal decision.
The eligible set is stripped down before arbitration ever runs. By the time the model ranks anything, most of the portfolio — and often the best action for this customer — is already gone.
The errors are not symmetric
There’s a reason to be aggressive about this, and it comes down to which mistake is expensive.
Wrongly including a weak action in the eligible set costs you almost nothing. The model scores it, gives it a low propensity, and doesn’t pick it. The cost is a few floating-point operations.
Wrongly excluding a strong action is a different order of harm. If that action was the best option for some segment of customers, those customers now receive an inferior alternative — not once, but every single time, for as long as the rule stands. The cost of false inclusion is computational. The cost of false exclusion is commercial, and it compounds. When the exclusion isn’t a genuine hard constraint, that asymmetry argues for leaving the action in and letting the model decide. Let the model be wrong cheaply, rather than letting a rule be wrong expensively forever.
So the response is not a new dashboard — it is a review. Walk the suitability rules and sort each one: is this permissibility, or a guess about who will respond? Permissibility — regulatory, contractual, a genuine prerequisite — stays a hard filter. Everything else is a hypothesis the model can test — so it belongs in the model, not in the eligibility layer, where it can be corrected when it turns out to be wrong. Then walk the suppression thresholds and ask, for each, whether the number was measured or merely chosen. Most were merely chosen. Neither task needs data you do not already have, and both directly widen the set the model gets to see.
A deeper cut, for another day
There is a subtler layer underneath all of this, and honesty requires flagging it even though it’s not today’s fight.
Actions are not actually the finest grain of the decision. The model ultimately selects a treatment — the specific creative shown to the customer — and a single action can carry many treatments. Two customers with the same seven eligible actions can face very different real search spaces if one action carries five treatments and another carries one. By that logic, the true ceiling is treatment optionality, not action optionality.
That may be worth its own paper. But it’s a refinement, not the headline, for a practical reason: we don’t currently track treatment optionality anywhere — not in Action Analysis, not in PDC, not in DSI — whereas action optionality is sitting in dashboards you already have open. The first-order win is to use the metric you’ve got, on the problem that’s crushing most deployments right now: portfolios of eighty actions arriving at arbitration as six. The treatment-level refinement is the next turn of the screw once the obvious door is open.
What this means for a deployment
Optionality is not a threshold to be cleared and forgotten; it is the ceiling on model ROI, and in most deployments it binds long before model quality does. The order-statistics argument is why: because NBA is rewarded for the maximum and not the average, every action added to the eligible set carries a non-trivial chance of being the one that matters, and under skew that chance decays slowly. Going from seven eligible actions to twenty raises the probability that a customer’s genuine best-fit action is even available from a coin-flip to seven-in-eight. That is not a rounding correction; it is the difference between a system that personalises and one that selects the least-bad survivor of its own filters.
The practical work follows directly, and none of it requires a better model or new data. Treat the eligible set as something to widen rather than a box that has been ticked. Separate genuine eligibility — regulatory, contractual, prerequisite — from targeting assumptions that have been misfiled as eligibility, and move the latter into the model, which can test them and be corrected when they are wrong. Scope suppression to real fatigue, and replace chosen thresholds with measured ones. The goal is not to assemble a nicer set of candidates to choose from; it is to stop eliminating the action that would have scored highest. For most deployments that is the single most reliable way to improve the decisions Pega NBA makes — and it needs neither a better model nor even a larger action library.
