Your causal claim is a promise, not a proof

On 13 August, we pointed a cross-fitted DoubleML battery at an MMM before a budget conversation. The panel had 89 national weeks, 16 controls, five folds and ten repetitions. Three standard nuisance learners, with everything else held fixed, returned three different answers for the same channel.

Google Ads was 11.1 under a lasso, -7.1 under a random forest and 107.6 under gradient boosting. The first two were indistinguishable from zero, with p-values of 0.81 and 0.86. The third was precise at p = 0.00072. TikTok ranged from 21.1 to 306.2, a factor of 14.5. Facebook was the only channel whose sign and significance survived all three learners.

This was useful. It was not a causal estimate becoming true. Double machine learning can reduce regularisation bias when observed controls are numerous and spend is collinear with them. It cannot adjust for a promotion, competitor move or demand shock that was never measured.

Orthogonalisation is an estimator, not an instrument

DML fits two helper models on folds that exclude the week being scored. One predicts the outcome from controls. The other predicts channel spend from the same controls. It then relates what is left over. Cross-fitting limits the tendency of a flexible helper to memorise its own training row, while the orthogonal score makes small helper-model errors matter less to the final coefficient.

What you would say to a board is simpler: compare the part of demand we could not predict from known context with the part of spend we could not predict from that same context. The word known carries the identification assumption.

Our fold choice mattered as much as the learner. Switching from random to contiguous blocked folds moved the same channel by up to 2.49 times, and the direction repeated on a second exposure definition. Random folds let a weekly model train on neighbouring future observations. Blocked folds respect the time boundary. Nothing in the library warned us.

To attack an unmeasured confounder, we need variation that moves spend for reasons unrelated to demand: an experiment or a valid instrument. We screened five instrument candidates against strength, leave-one-out stability and negative controls. None survived.

The screen separated relevance from validity because teams routinely collapse them. An F-statistic above 10 says that the candidate predicts treatment strongly enough for conventional first-stage work. It does not say why the candidate predicts treatment, or whether it reaches the outcome through another path. We therefore required leave-one-out stability and a negative-control outcome that the proposed instrument could not plausibly affect. A strong candidate that predicts the negative control fails.

One candidate was derived from CPM. But spend equals impressions times CPM divided by one thousand. Instrumenting spend with a component of that identity guarantees strength without guaranteeing validity. Facebook's version had an F-statistic of 470.04 and remained 348.69 after dropping its most influential week. It produced an effect of -294.11 with a 95% interval from -788.40 to 200.21, contradicting all three DML estimates. We discarded it because the exclusion restriction failed by construction, not because the result was inconvenient.

Meridian states the assumption more clearly than most decks

Google's Meridian documentation calls conditional exchangeability the main untestable assumption required for causal interpretation from an MMM regression. It also says the condition is never perfectly met. That is fair disclosure, not a vendor defect. Meta's Robyn documentation recommends experiments as a route for introducing causality into an MMM. Both acknowledge the boundary.

This distinction is identification before estimation. Identification asks whether the causal quantity can be recovered from the information and assumptions available. Estimation asks how to calculate it once that question is answered. Cross-fitting, regularisation and Bayesian priors can improve estimation under the chosen model. None supplies the missing counterfactual variation needed to establish identification. Better estimation of an unidentified quantity is still an unidentified quantity.

The category error happens later, when an identified-under-assumptions estimate becomes a measurement in a board slide. A back-door adjustment in a Bayesian model and a cross-fitted DML score are different ways to estimate after adjustment. Neither tests whether the adjustment set was complete.

We still use DML. In this case it gave us a learner-sensitivity map, a fold-leakage diagnosis and a robustness value: 39.87% for Facebook and roughly 25% for the other channels under the one learner where all three were significant. The last qualification matters. Under the lasso, Google Ads was already at p = 0.81, so its robustness to an omitted confounder was effectively zero. Sensitivity analysis inherited the learner choice.

The correct sentence is therefore conditional: given this control set, time split and nuisance learner, this is the remaining association after orthogonalisation. If the decision needs a causal number, the next dollar belongs to experimental design, not another observational estimator.