Your MMM passed cross-validation. So would a calendar.

On 27 August, 886 fits produced a champion with an out-of-sample R-squared of 0.5243 on a Christmas holdout. Every one of seventy scored folds had converged. The campaign still stopped, because the apparent winner lost across the full fourteen-fold sequence: pooled R-squared ranged from -0.31 to -0.57 across the five finalists.

Two holiday folds created 53% of the squared error. Excluding them moved the pooled score to +0.13. That was useful diagnosis, not permission to delete December. A ski resort does not get to declare Christmas out of distribution after seeing the result.

Cross-validation is often described as a way to choose models. In MMM it has a prior job: make the business claim travel through time. A channel effect learned from January to October must survive November without borrowing November's outcome, scale or neighbouring weeks. If the procedure leaks any of those, the score is about the harness, not the model.

Random folds answer the wrong time-series question

Random K-fold validation lets the nuisance or outcome model train on weeks either side of the held-out week. Weekly media, demand and promotions are autocorrelated. The model partly recognises the season it is supposed to predict. A blocked or rolling-origin split is stricter: train on the past, leave an embargo at least as long as the maximum adstock, then score a contiguous future block.

We saw the cost directly in a DoubleML audit. With the data and learner fixed, changing shuffled folds to blocked folds moved channel estimates from 1,413 to 2,690, from 114 to 285 and from 108 to 163. The ratios were 1.90, 2.49 and 1.50. A second exposure specification moved in the same direction, with ratios from 1.79 to 1.94. We do not know which set is nearer the truth because the series has no known causal effect. We do know that the random split had future neighbours available.

Scaling can leak too. One production path scaled the full panel before separating train and test. A future spike then changed the unit in which past weeks were fitted. The result looks modest because no row crosses the split, but information does. Every transformation that learns a parameter, scaling, imputation, feature selection and residualisation, belongs inside the training fold.

We found a more direct overlap failure in another engagement: two nominally separate test windows shared zero days, yet the training set for the second fold contained the whole first test window. That design can still be legitimate for rolling-origin forecasting, where yesterday becomes available tomorrow. It is not legitimate when the metric is later described as independent repeated validation. The fold manifest must therefore print every train and test date, not merely the number of folds.

A second trap is selection after inspection. Trying 886 configurations and quoting the best cell makes the maximum the product, even when no explicit p-value appears. Nested validation is the clean fix: an inner window chooses hyperparameters, and an untouched outer window estimates performance. When the panel is too short to afford that split, we keep the search narrow and report the full finalist distribution. Scarcity of data does not turn selection bias into evidence.

Fold geometry changes the metric even without leakage. The same 32 held-out rows scored +0.1751 with eight four-week folds and +0.1286 with four eight-week folds; RMSE moved from 857.5 to 881.4. The held-out standard deviation was about 944 in both runs, so this was forecast error, not a denominator trick. Short folds, however, can make R-squared almost uninterpretable when the outcome barely varies within them.

A score needs an adversary and an oracle

Our current battery reports rolling-origin and blocked results, MASE against a seasonal naive forecast, RMSE in business units, and fold-level error concentration. Each answers a different failure. R-squared says whether the model beats a fold mean, which is often a weak baseline. MASE asks whether it beats repeating the seasonal pattern. RMSE keeps the miss in the unit a budget owner understands. Fold concentration shows whether one event owns the conclusion.

The ski-resort campaign also ran a last-week-actual oracle. It scored -0.058, while a constant-accuracy forecaster averaged -5.24 at the eight-week horizon. Those numbers did not rescue the MMM. They told us that the series itself was hard and that a contractual R-squared above 0.5 was unattainable without changing the design. At RMSE 1,186, the test-set standard deviation would need to exceed 1,677; the whole series was 1,236. Clearing the target on that spread required RMSE near 874, a 26% improvement. We cancelled after roughly 145 of 500 planned fits.

A champion is the model that survives the future boundary, the naive adversary and the event-level audit. Everything else is a candidate with a flattering fold.