DMA 7040 Module 4 Final Project: Trend Report on Marketing Mix Modeling and Incrementality Example

Measuring What Advertising Causes: A Trend Report on Marketing Mix Modeling and Incrementality Testing

Student Name

MBA in Marketing, Thomas Edison State University

DMA-7040: Digital Marketing Analytics

Mentor Name

Month Day, Year

Measuring What Advertising Causes: A Trend Report on Marketing Mix Modeling and Incrementality Testing

1. Trend Overview

Marketing mix modeling (MMM) estimates how much each marketing input contributed to sales using aggregate weekly data rather than individual tracking. After years in which digital marketers relied on click-based attribution, MMM is returning in an open-source, Bayesian form and is being paired with incrementality tests, experiments that withhold advertising from a random group to measure what it truly causes. The return has three causes: privacy changes that removed much user-level tracking after Apple's App Tracking Transparency prompt in 2021, research showing attribution often overstated advertising's effect and new free tools that let in-house analysts fit models once built by consultants (Jin et al., 2017).

2. Reach and Scope

The trend is standard among large advertisers, growing quickly among direct-to-consumer brands and mid-sized retailers and still thin among small firms and long B2B sales cycles. It covers online and offline media on one scale, which no click-based method does, and it suits budget allocation better than tactical choices such as creative or keywords.

The scope has limits worth stating plainly. MMM works on weekly or monthly totals, so it cannot say which ad a particular customer saw or which keyword to bid on, and it needs enough variation in spending, usually two or more years of it, to separate one channel from another. Incrementality tests answer narrower questions with more confidence but cost money, since the held-out group goes without advertising for the test period. The two methods are complements: the test supplies a trusted number for one channel, and the model spreads that knowledge across the whole budget.

3. Reported Impacts

The strongest evidence for the trend comes from experiments that exposed how far attribution can drift from causation. Gordon et al. (2019) lined up randomized Facebook ad tests against what standard observational methods would have reported and found those methods often failed to recover the true lift even with rich data on each user. Lewis and Rao (2015) showed that because sales vary so much on their own, an advertiser needs very large experiments to measure returns precisely; the implication is that measurement must be designed, not read off a report. Firms that adopt calibrated MMM report moving budget away from channels that captured existing demand toward those that create it, though such reports come mostly from vendors and should be read with that in mind.

Public changes added pressure from another direction. Once Apple made apps ask before tracking people across other firms' apps and sites, most people said no, and attribution lost much of its user-level data. Browser limits on third-party cookies pushed in the same direction. For many firms the question was no longer whether to adopt aggregate measurement but how quickly, which is why open-source tools from Google and Meta spread so fast after 2021.

4. Two Illustrative Examples

eBay's paid search experiment. Blake et al. (2015) describe how eBay halted its brand-keyword ads on two search engines and switched off non-brand ads in a randomly chosen set of regional markets, using the untouched markets as a control. Brand-keyword ads produced almost no additional traffic or sales: users who searched for eBay simply clicked the free listing below. Non-brand ads helped mainly new and infrequent buyers, while frequent eBay customers would have bought regardless. Click-based attribution had credited all of those sales to search ads. The example shows the core of the trend: measure what advertising adds, not what it touches.

Facebook's field experiments. In the experiments analyzed by Gordon et al. (2019), the platform randomly withheld campaign ads from a control group, allowing the true lift to be compared with what observational methods would have estimated. In many of the studies, the observational estimate landed far from the experimental one, most often above it. The example shows why MMM is being paired with experiments rather than trusted alone: any model built on observational data inherits the risk.

5. Two Direct Applications

Application 1: A geographic holdout for paid social. Riverbend Stitch Works, the composite vintage sewing machine seller analyzed earlier in the course, spends $6,900 a quarter on paid social, which last-click reporting shows returning $1.90 per dollar against a break-even of $2.22. For six weeks the seller can pause social ads in a random half of its states and compare all orders between the halves. Before the test starts, the seller commits to a rule: a drop of under roughly 32 monthly orders where ads are paused means the budget moves to search.

Application 2: A starter MMM. With 104 weeks of spending and sales by channel, Riverbend can fit Google's open-source Meridian or Meta's Robyn to estimate each channel's contribution, carryover and the point of diminishing returns. The holdout result from Application 1 is used to calibrate the model's estimate for social, and the model then guides the next year's budget split.

6. Evaluation of the Cases and Applications

The two examples are strong evidence because both used randomized controls at enormous scale, but both come from platforms with millions of users; a small firm cannot replicate their precision. Lewis and Rao's (2015) arithmetic applies: Riverbend's six-week test will detect only a large effect, so a "no difference" result means "no large difference," not "no effect." The MMM application faces a data constraint, since 104 weeks is near the minimum and Riverbend's spending has varied little, which limits what the model can separate. Chan and Perry (2017) identify exactly this problem, correlated and low-variance inputs, as the most common failure of media mix models. The applications are still worth running: the holdout answers the urgent question cheaply, and the model, even if uncertain, forces the firm to think in terms of incremental return.

7. Expert Interview

The interview was with a marketing analytics manager at a mid-sized home goods retailer in Pennsylvania; the interviewee and quotations are a composite written for this sample. Asked what changed after 2021, she said, "Our attribution reports kept showing great results for retargeting, and our sales didn't move when we doubled it. That was the moment we stopped trusting clicks." Her team now runs a lift test each quarter on one channel and refits its MMM twice a year. Her advice for small firms was to start with tests rather than models: "A clean holdout on your biggest channel will teach you more than a model you can't feed."

8. Business Impacts

The trend shifts budgets, roles and relationships. Budgets move from channels that harvest demand, such as brand search and retargeting, toward those that create it, which can raise short-term cost per order while improving total growth. Analysts need statistical skills alongside platform skills, and marketing teams must accept wider uncertainty ranges in place of the false precision of attribution dashboards. Relationships with ad platforms change too: a firm that measures incrementality itself negotiates from evidence rather than from the platform's own reporting.

The change also affects how marketing results are reported to finance. An attribution dashboard gives a single return figure for each channel; a calibrated model gives a range, and a test gives one channel's lift with a margin of error. Finance teams used to precise-looking numbers may at first read the wider ranges as weaker evidence, when in fact they are more honest. Marketing leaders who explain this shift early, and show how decisions will be made inside the ranges, keep the budget conversation grounded in evidence.

9. Relationship to Course Content

The trend extends an argument made by Kaushik (2009), that clickstream data describe what happened but not why: incrementality tests are a form of the experimentation and testing source in his Web Analytics 2.0 model. It connects to the course's work on owned, earned and paid media, since MMM is one of the few methods that values all three together, and to the dashboard module's lesson that return on ad spend must be read against margin and questioned when it comes from last-click credit.

10. Keeping a Competitive Advantage

Because the tools are now free, the advantage lies in how a firm uses them. Three capabilities matter. First, an experimentation habit, one controlled test a quarter, keeps models calibrated. Second, clean weekly data across every channel, including offline sales, are the raw material a model needs and the hardest part for small firms to assemble. Third, people who can read uncertainty and explain it to managers turn a model into a budget decision. Firms that build these capabilities will allocate spending more accurately than rivals still reading platform dashboards.

For Riverbend specifically, the advantage would come from sequence rather than scale. Running the paid social holdout first gives the firm one reliable number within two months; feeding that number into a starter model gives a budget split for the following year; and repeating a test on a different channel each quarter steadily improves the model. A small firm that follows that cycle can make better budget decisions than a larger rival that relies on platform reports alone.

References

Blake, T., Nosko, C., & Tadelis, S. (2015). Consumer heterogeneity and paid search effectiveness: A large-scale field experiment. Econometrica, 83(1), 155-174. https://doi.org/10.3982/ECTA12423

Chan, D., & Perry, M. (2017). Challenges and opportunities in media mix modeling. Google.

Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193-225. https://doi.org/10.1287/mksc.2018.1135

Jin, Y., Wang, Y., Sun, Y., Chan, D., & Koehler, J. (2017). Bayesian methods for media mix modeling with carryover and shape effects. Google.

Kaushik, A. (2009). Web analytics 2.0: The art of online accountability and science of customer centricity. Wiley.

Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4), 1941-1973. https://doi.org/10.1093/qje/qjv023