Loom video
Mindmap
A Simplified Overview of the Results from the Incrementality Testing Process
The table below presents mockup data illustrating how different channels perform when traffic types are compared after conducting incrementality tests. The green columns represent data from ad platforms, while the purple columns display internal data post-incrementality testing. Although the results of incrementality tests are typically estimations, they help guide smarter spending decisions throughout the fiscal year. The most critical purple columns are the Multiplier and Conversion (in this example, Customer Acquisition Cost), which allow for applying filters to see how far off ad platform reporting is in each instance. Keep in mind that advanced techniques like causal inference can further enhance accuracy.
Introduction
Hey everyone! Today, we're diving into something that sounds a bit technical but is super important if you're serious about getting the most out of your marketing budget: incrementality testing. Now, I know that might sound like something only data scientists or analytics nerds would care about, but stick with me. I’m going to break it down in a way that makes sense, and by the end, you’ll see why this could be a game-changer for your business.
What Is Incrementality Testing, and Why Should You Care?
Incrementality testing is all about figuring out how much extra impact your marketing efforts are having—over and above what would happen naturally. Think of it like this: Imagine you’re selling ice creams on a hot day. Without any advertising, you sell 50. But with ads, you sell 70. Those extra 20 ice creams? That’s your incrementality—sales that happened because of your marketing.
So why does this matter? Well, it’s the key to making sure you’re spending your marketing budget where it actually makes a difference. Just like you wouldn’t keep using a fertilizer that doesn’t help your plants grow, you shouldn’t keep pumping money into marketing channels that aren’t really adding value.
The New Marketing Landscape: What’s Changed?
In case you haven’t noticed, the world of digital marketing is changing—fast. New privacy regulations like GDPR, CCPA, and iOS 14.5 have made it harder to track users and deliver those personalized ads we all love (and sometimes hate). Gone are the days of easy access to heaps of user data. Now, everything’s about user consent and privacy.
This shift has made it tougher to deliver targeted ads, driving up customer acquisition costs and making our campaigns less effective. It’s like trying to hit a target while blindfolded. The precision we used to rely on is slipping away, and we’ve got to adapt.
Adapting to the New Reality
So, how do you adapt to this new, privacy-focused world? You’ve got to rethink your strategies and the tools you use. The old methods, like multi-touch attribution (where you credit all the touchpoints a customer interacted with before converting), are becoming less reliable. Instead, you need to start leveraging your own first-party data (the data you collect directly from your customers) and building strategies that can weather the storm of changing regulations. It’s like swapping out an old, unreliable map for a shiny new compass.
Why Incrementality Testing Matters
This is where incrementality testing comes into play. It helps you figure out which of your marketing efforts are actually driving results. Think of it as the ultimate tool for cutting through the noise and making sure your money is well spent. Want to know which ads are really pushing your sales? Incrementality testing is your answer.
Breaking Down the Types of Incrementality Tests
There are a few different ways you can approach incrementality testing, depending on what you’re trying to measure. Let’s break down the three main types:
1. GeoMatch Testing
This method involves comparing the impact of your marketing across different geographic areas. For example, you might run ads in one city but not in another similar city to see what difference it makes. It’s great for location-based targeting but can be tricky to set up, especially if your product isn’t tied to a specific location.
- Pros: Precise comparison between geographic areas.
- Cons: Complex setup, not ideal for non-location-based products.
2. Audience Split Testing
Here, you split your audience into groups and show ads to only one group. For instance, show ads to half your website visitors and compare their behavior to the half that didn’t see any ads. This method gives you detailed insights into how your audience behaves.
- Pros: Great for detailed insights, especially in digital campaigns.
- Cons: Needs a large audience, and you might lose revenue from the non-targeted group.
3. Scale-Up Testing
Want to know what happens if you double your ad spend? Scale-Up Testing is your go-to. This test measures the impact of increasing your marketing budget to see if you get more bang for your buck.
- Pros: Helps you understand the impact of budget increases.
- Cons: It can be hard to isolate what’s really driving the results, and there’s a risk of overspending.
Common Pitfalls to Avoid
As with anything, there are a few traps you want to avoid when doing incrementality testing:
- Dark Testing: This involves shutting off all your ads to measure their impact. Sounds logical, right? But it can actually lead to misleading results—it’s like trying to measure how much light a lamp gives by turning off all the lights in the room—not exactly helpful.
- Poor Experimental Design: If your test sample size is too small, your results might not be reliable. Imagine trying to figure out the average height of a population by measuring just five people—your results are going to be all over the place.
- Overcomplicating Tests: Adding too many variables can muddy your results. Sometimes, simple is better. Stick to the basics and get clear, actionable insights.
Setting Up Your Own Incrementality Test
So, how do you actually go about setting up an incrementality test? There are a few different routes you can take:
1. In-House Setup
If you’ve got the expertise, doing it in-house gives you full control. You’ll need a data scientist and a data-driven marketer to pull this off, but it means you can tailor the test to your specific needs.
2. Using Third-Party Tools
Don’t have the resources to do it all yourself? No problem. There are plenty of third-party tools out there, like Measured, Haus, or incrmntal, that can help you run these tests. It’s kind of like using a cake mix—faster, more reliable, but not free.
Example: How Haus Picks Regions for a Geo Holdout Test
When running a geo holdout test, it's crucial to pick control and treatment regions that are as similar as possible. Haus, for example, uses advanced algorithms to look at things like demographics, historical sales, and even local events. So if you’re testing an ad campaign in New York, they might choose Chicago as the control city because it’s got similar characteristics. By turning off ads in Chicago and running them in New York, you get a clearer picture of how well your ads are working.
3. Leveraging Platform Tools
Some marketing platforms offer their own tools for incrementality testing. These are usually free but come with limitations. It’s like buying a ready-made cake—it’s convenient, but you don’t have much control over the ingredients.
Analyzing and Optimizing Based on Results
Once you’ve run your test, the next step is analyzing the results. You need to compare what the platform reports with what your internal data says—this is key for making smart decisions. Understanding the data means recognizing where the differences are between what the platform is telling you and what your own data shows as the true incremental impact.
Based on what you find, you’ll need to decide whether to scale up, cut back, or change direction. The insights from incrementality testing help you put your budget where it’s going to do the most good.
Continuous Testing and Optimization
Here’s the thing—testing isn’t a one-time deal. You’ve got to keep testing and optimizing regularly to stay on top of changes. Think of it like regularly pruning a garden to make sure everything grows just right. Keep testing, keep learning, and keep refining your strategy based on what the data tells you.
Conclusion
So, there you have it—a simple, straightforward guide to incrementality testing. I hope this has given you a solid foundation to start experimenting with your marketing strategy. Remember, you don’t need a huge budget or a degree in data science to start seeing results. Just keep testing, keep optimizing, and most importantly, keep growing. Thanks for reading!
