Demystifying incrementality: An essential guide for marketers

Measuring the impact of your marketing campaigns takes time and effort. And it can be difficult to tell which marketing methods meaningfully contribute to success and which might be redundant. That’s where incrementality analysis can help.

Incrementality measures how effective your strategy is and surfaces which marketing method deserves credit for conversions. This crucial information helps you optimize your campaigns to deliver the biggest returns for the tightest investment. Learn how to leverage the incrementality metric for maximum business success.

What is incrementality?

Incrementality is a sophisticated testing method that measures which marketing campaigns impact your audience most. It helps you determine which efforts have caused measurable change and which were more a matter of luck.

Say someone decides to buy a new sound system. They peruse product reviews and company websites and eventually choose a Bose system. When they’re ready to buy, they visit an electronics store, where they walk past all the displays and pick up their chosen model.

So, which marketing channel drew this customer to Bose? It may have been the demo units Bose sent to product reviewers, the website the marketing team carefully designed, or the endcaps and signage in the store. Typical last-touch attribution would credit the conversion to the last marketing touchpoint the customer encountered. In this scenario, the in-store displays would get the credit, even though they had a negligible impact on the decision.

Bose could use incrementality analysis to determine the efficacy of their multiple marketing efforts for that specific sound system. This would involve A/B iteration using test and control groups. This process helps isolate a campaign to determine whether it deserves credit for conversions, clicks, or shares.

How to design incrementality tests

Effective incrementality marketing tests must factor in the following elements to generate the most precise data: 

• Size: Your test and control groups must include enough potential customers to create valuable results. Ensure your control group is 10% larger than your test group to account for the naturally higher attrition rate.
• Specificity: Testing works best when using a single key performance indicator (KPI) to contextualize results, such as conversions, click-through rate (CTR), or net promoter score (NPS).
• Similarity: Your two groups must perform similarly in the metric you’re measuring. For instance, if you’re measuring CTR, both test groups must already have a similar result.
• Patience: Your tests must take place over periods that are long enough to account for seasonality and other natural market shifts.
• Neutrality: Select groups who’ve never seen the marketing campaign you’re evaluating to ensure a neutral testing environment.

How do you calculate incrementality?

Calculating incrementality requires a simple formula that measures a marketing campaign’s impact on a test group compared to a control group. If the resulting percentage is positive, you have an “incremental lift.” If not, you’ve discovered that the campaign has a negligible — or even negative — impact. The formula looks like this:

((Test Result – Control Result) / Control Result) x 100 = Incrementality

2 types of incrementality testing

There are two primary ways to approach incrementality testing. Determining which method suits your situation depends on your marketing goals and the variables you’re testing for.

1. Holdout testing

Holdout testing follows a standard A/B testing procedure. But rather than conducting two competing experiments, you’ll evaluate one experiment against a control. Here’s how it might work if Bose marketers wanted to determine how their in-store displays impacted a customer’s purchasing decision.

First, the Bose marketing team identifies two stores where they’ve never used physical marketing. The stores have matching sales figures, and they’re in demographically similar locations. So, the team will label the two stores this way:

• Test store: In this store, the company will install demos, endcaps, and signage. 
• Control store: In this store, which has a 10% larger customer base, they’ll add nothing.

After running the test for two months, the Bose team checks the sales figures and finds these results:

• Test store: 80 sales
• Control store: 74 sales

 Then, they plug those numbers into the formula:

((80-74) / 74) x 100 = 8.1%

Bose concludes that in-store marketing causes an 8.1% increase in sales. That might seem like a lot or a little, depending on their sales goals and how much the marketing costs. Regardless, it’s vital information they can use to optimize their marketing strategies.

2. Multivariate testing

Multivariate tests take incrementality a step further. With this approach, you can test multiple variables, such as copy variants or audience focus, against one another. Here’s how it might work if the Bose team wanted to test which product reviews most significantly impact ad CTR.

In a multivariate test, the Bose team would create three nearly identical ads with the following titles, such as:

• “Check out our award-winning sound system!”
• “This sound system won CNET’s Editor’s Choice award!”
• “This sound system won TechRadar’s Editor’s Choice award!”

They run these ads on a major platform, such as Meta, which allows them to segment their audience. Then, the team splits their audience into three demographically similar groups:

• Group A: This group, which is 10% larger, gets ad #1.
• Group B: This group gets ad #2 with the CNET reference.
• Group C: This group gets ad #3 with the TechRadar reference.

After running the test for two months, the Bose team gathers the following results:

• Group A: 4 clicks per 100 impressions
• Group B: 9 clicks per 100 impressions
• Group C: 7 clicks per 100 impressions 

Then, they plug those numbers into the formula for Group B:

((9-4) / 4) x 100 = 125%

 Lastly, they repeat the formula for Group C:

((7-4) / 4) x 100 = 75%

The team concludes that referencing the CNET award is 50% more effective at generating CTR than referencing the TechRadar award. As a result, they green-light the best-performing ad and get more web traffic.

Attribution and modeling

Attribution is the act of crediting a successful outcome to a specific marketing tactic. In our earlier example, the customer’s purchase was initially attributed to in-store marketing. But after a few incrementality tests, we determined that a product review from CNET deserved the credit. That’s the problem with the LTA model, which doesn’t consider the customer’s purchasing journey.

The Bose customer probably encountered many things that impacted their decision, and that’s where multi-touch attribution (MTA) steps in. MTA uses many incrementality tests to measure every marketing tactic’s relative impact on a customer’s decision. With that data, marketers can determine which strategy deserves credit for a purchase.

With all this data built up, marketers can begin mix modeling, which uses historical marketing data to estimate campaign outcomes. At this point, the incrementality formula won’t help. It takes complex statistical analysis, but luckily, tools like Segment and Qualtrics can digest all the data for you.

The incrementality process can become a powerful loop: You select a marketing tactic, perform incrementality tests to determine its impact, and then update your marketing mix model. After you complete that loop enough times, your model should accurately prescribe a unique blend of marketing tactics that result in the best ROI for your company.

Power campaigns with evidence-based marketing

Like most consumers these days, our example customer based their decision on research that they performed well before making a purchase, and the numbers backed this up. But what about less intuitive marketing factors, such as website color schemes, button placement, and font choices? 

Incrementality testing measures those, too — and Webflow Enterprise can help. Webflow provides a modern tech stack and app integrations, so you can design and launch your own incrementality tests. Combined with a visual web development environment and CMS, Webflow’s platform offers everything you need to test endless design possibilities.