I once worked with a fashion retailer that launched a new season with its biggest campaign of the year. The media plan was sound: the right channels, the right weight, the right timing. Sales barely moved. The model measured that clearly enough, and it was tempting to go looking for what was wrong with the plan. But nothing much was. The range, and the advertising for it, had simply missed what customers wanted that season. No amount of media was going to make up for that.

It’s an obvious point when you see it written down, yet it’s easy to lose sight of in a results meeting. Planning alone doesn’t decide whether a campaign succeeds. The ad matters, the product matters, and a good deal else matters that a Marketing Mix Model (MMM) will never see.

What the model can see

A model is good at measuring what a campaign did – how much it added, over what period, and at what return. Where there’s enough variation in the data, it can go further. If campaigns or creative executions ran at different times, with enough weight behind each, the model can estimate the difference in their effect. That is genuinely useful, and a good modeller will look for the chance to split them out.

Two campaigns, identical spend and channel plan (illustrative)
Incremental sales over the weeks after launch for two campaigns with identical spend and channel plans. Campaign A rises far higher than campaign B. The model can measure the gap between the two curves, but not the reason for it. the gap: measurable the reason: not campaign A campaign B identical spend, identical channel plan – weeks since launch

What it can’t see

What the model can’t do is tell you why. Why one ad landed and the other didn’t comes down to how people responded to it: what they felt, what they understood, and whether it gave them a reason to buy. That human response – emotional and rational – is not in the data. The model sees that campaign A outperformed campaign B. It has no way of knowing whether that was the idea, the casting, the music, the offer or the moment.

And it isn’t only the ad. Every model has a residual: the part of sales it couldn’t explain. A large model might include a hundred different factors. But at the point someone decides to buy, there are countless influences at work – their mood, their circumstances, what a friend said, what’s happening in the world that week. No model can capture all of that, and it’s wise to remember the residual is there whenever a model’s numbers start to feel complete.

Why aggregates work

MMM works because it looks at aggregates: whole markets and populations, where individual differences tend to cancel each other out. That’s its strength. The more finely you split the data – by region, by segment, by audience – the closer you get to looking at individuals, and the more noise you bring in. Some disaggregation is worth having, but there is always a balance to strike between detail and reliability.

Two easy conclusions

That leaves two easy conclusions to be wary of.

  1. A strong result isn’t proof the media plan was right. The creative may have done the work. A channel ends up with the credit an idea deserves, and next year’s plan is built on the wrong lesson.
  2. A weak result isn’t proof the plan was wrong. Sometimes the ad, or the product behind it, simply missed – as it did for the fashion retailer. Cutting a channel that was doing its job won’t fix that.

Other lenses

To understand the why, you need other lenses. Qualitative research is the one I rate most highly: it tells you how people actually responded to the advertising, in their own words. Pre-testing has its fans too. Use them alongside the model, not instead of it, and expect each to see things the others can’t.

MMM isn’t a panacea. Used well, it’s one lens among several, and a good practitioner will tell you where its view ends. So when a result lands, good or bad, it’s worth asking your team: how much of this was the plan, and how much was the ad?

Knowing what a model can and can’t see is part of a wider readiness question – see The four disciplines of Marketing Mix Modelling.

If you’d like a second opinion on how your results are being read, drop a line to [email protected].