Adding a channel that does not exist made the model better

On 21 August, an MMM for a national retailer improved when we added a media channel made of shuffled numbers. Its out-of-sample score rose by 0.018. Four real channels together had improved the same score by only 0.007. The fake channel was more useful to prediction than the media we were trying to measure.

That was not an entertaining edge case. It was an ablation result. We had fitted the full model, then removed or replaced one part at a time while holding the rest of the design fixed. If a claimed signal is real, its removal should damage the model and a nonsense replacement should not repair it. Here the opposite happened.

The usual response is to tune the full model harder. That is precisely the wrong response. A flexible seasonal baseline can predict revenue while distributing credit almost arbitrarily among correlated channels. One of our earlier fits achieved a media-out score of 0.4057, almost identical to 0.4028 for the champion and 0.4027 for controls only. It nevertheless assigned $3,351 of total contribution across 693 days, including $250 to two channels with zero spend. The attribution layer was busy; the predictive layer did not need it.

Ablation turns one plausible model into a contest

We now use four ablations. First, remove media entirely. This asks whether media adds predictive information beyond trend, seasonality and controls. Second, insert a placebo channel by rolling or shuffling spend. This asks whether the model credits structure it knows cannot be causal. Third, invert the prior. This asks whether the posterior learned from the outcome or repeated what we told it. Fourth, change the channel grouping. This asks whether individual coefficients survive the collinearity that bundling was meant to reduce.

Bundling matters because paid channels often move together. In our engine, candidates above |r| = 0.90 can be merged, with a cap of three merges and two consecutive bundling iterations. That can reduce variance. It can also erase the distinction the business cares about. We saw pairs at 0.954 and 0.947 merge cleanly, while another pair that looked above 0.90 within one year was -0.190 over the full panel. The window decided the bundle.

Pruning has the same dual character. A variance inflation factor above 10 triggers removal, but protection is an exact-name allowlist checked before that test. A shipped unit test keeps a protected channel at VIF 99.0 and removes an unprotected one at 50.0. In a real run, a flagship paid-search series at 10.04 was deleted when the protection rule failed to match its name. The model did what the code asked. The code did not represent the business rule.

The hard part is keeping the perturbation symmetric. We once withdrew two placebo results at 32% and 41% contribution because the placebo was exempt from an allocation prior imposed on the real channels, and both fits also failed sampler health. Those numbers looked damaging to the model, but the comparison was not fair. An ablation earns a decision only when the candidate and the control pass the same priors, sampling gates and evaluation window.

That discipline also prevents a common executive mistake: treating each failed ablation as an independent scandal. The tests are a sequence. If media-out says the baseline already predicts the outcome, a placebo win is expected evidence about weak separation, not a second discovery. If prior inversion then changes allocation while fit barely moves, the three results form one mechanism: the outcome identifies the baseline, while assumptions allocate the residual. The report should explain that mechanism once.

The useful output is often a refusal

Ablation does not prove causality. It separates claims that survive reasonable perturbations from claims that exist only inside one specification. On a portrait-studio account, placebo channels rolled 183 and 274 days ranked last among eleven channels, while a 91-day roll still captured 8.0% of contribution. The first two were reassuring. The third was the finding.

We then ran sixteen specification cells. One channel, representing 58.7% of spend, stayed inside the predeclared efficiency band in all sixteen. The remaining 41.3% of spend did not earn a channel-level claim. This is what enterprise triage looks like: a short trust list and an explicit refusal list, not a single model painted green.

Google's Meridian documentation says that a model with 99% out-of-sample R-squared can still be poor for causal inference. I agree, and would add a more operational point: a model that predicts well after you remove the thing it claims to measure has failed a simpler test. The right next action is not another colour on the dashboard. It is the ablation that can make the dashboard unnecessary.