All you need to know about geo holdout testing

Understanding the effectiveness of campaigns gives marketers several advantages. From adjusting spend for maximum results to advising stakeholders where the results are coming from, precise analysis is key to success. But how do you achieve these sorts of perspectives on your campaign? 

One way is through geo testing.

Geo testing, and its counterpart geo-holdout testing, involve conducting experiments in specific geographical areas to measure the impact of various marketing activities. Today, you’ll learn how to use these techniques through a fictitious sports apparel brand called RunFast. So, put on your trainers and let’s discover the world of geo testing.

What is geo testing in advertising?

Geo testing, or geographical testing, is a form of A/B testing where a marketing campaign is implemented in specific geographical areas while other areas (without the campaign running) serve as control groups. Data collected from these regions allows marketers to measure the impact of their activities on key metrics, such as website visits, brand recognition and sales. The goal of geo tests is to understand the incremental gains a business can realize from additional marketing activities or campaigns.

Example of a geo test

Ok. Let’s imagine our imaginary sports apparel company, RunFast. It’s an emerging brand that sells running shoes, clothes and accessories across the US. RunFast has been exploring various marketing channels to reach their audience and drive sales. So far, RunFast’s marketing department has relied on paid search and social media advertising to reach their audience and drive sales. With a new CMO in place, though, the company is thinking about expanding its marketing efforts to include YouTube ads. To evaluate the effectiveness of this new channel, the team decides to conduct a geo test.

Geo test setup

Let’s examine one way you could set up a geo test in order to achieve meaningful results. 

Objective: Assess the impact of YouTube advertising on website sessions, sales and revenue.

Method:

1. Selection of test markets: RunFast selects four states to run YouTube ads over six weeks. These states are representative of the broader US market.
2. Control Group: The rest of the US serves as the control group, where no YouTube ads are run during the same period.
3. Experiment run time: The experiment period for this geo test is six weeks.

Metrics to measure: 

• Website sessions
• Number of sales
• Revenue

Results:

After RunFast runs the geo test for six weeks, the team gets together and reviews the results. They look at some key metrics for both the control group and the test group. Here’s what they find:

Control group (no YouTube ads):

• 4% increase in website sessions (compared to the previous period)
• 3% increase in number of sales
• 3% increase in revenue

Test group (with YouTube ads):

The states where the YouTube ads were delivered to shoppers saw:

• 10% increase in website sessions
• 6% increase in number of sales
• 12% increase in revenue

Conclusion:

Assuming all other variables remained constant, the marketing department can conclude that the introduction of YouTube advertising significantly boosted website sessions, sales and revenue during the six-week trial period. This suggests that investing in YouTube ads is a beneficial strategy for driving traffic and sales for RunFast.

Sure, this is a simplified fictional case study. When analyzing the results from a real geo test, you may need to collect a lot more data in order to draw a conclusion. For example, was the difference in increase in revenue, 12% against 3%, worth the investment made in additional advertising dollars? 

 After all, this test was not only a change in marketing mix, it was also an increase in ad spend. The marketing team will need to calculate the cost versus the investment. Then, they will understand if it makes sense to add YouTube to their standard mix of advertising channels. 

Now, let’s also look at the lesser known cousin of regular geo tests: geo-holdout tests.

What is a geo-holdout test?

Geo-holdout testing involves pausing a specific marketing activity or advertising channel in a certain location to evaluate its impact on performance metrics. This is done by comparing the holdout group (the group where the marketing activity is paused) to a control group (where advertising is running as usual). 

There can be different reasons to perform this sort of test. Some holdout tests involve pausing all ad spend in a certain region to understand the baseline sales — the amount of sales generated without any advertising. Other geo holdout tests pause one or more specific campaigns or channels to determine the incremental sales those channels bring in.

The idea to remember is that geo-holdout testing is used when you’re already running advertising in a location, and you want to see what happens if you stop.  

Example of a geo holdout test

Let’s see what happens when RunFast wants to implement geo-holdout testing. Imagine they have been investing in brand search campaigns on Google and Microsoft (Bing). Before initiating these campaigns, though, they didn’t perform a geo test as described above. That means they aren’t too sure how much these campaigns affect overall sales. 

To understand the impact of these campaigns, they decide to conduct a geo-holdout test.

Geo-holdout test setup:

• Objective: Evaluate the impact of pausing brand search campaigns on key performance metrics.
• Method:
  • Selection of holdout markets: RunFast selects four states where they will pause the brand search campaigns on Google and Microsoft (Bing) for six weeks.
  • Control group: The rest of the US continues running the campaigns during the same period.
  • Metrics to measure: Website sessions, number of sales and revenue.

Results:

After the testing period, the entire team came together to take a look at the test result. Here is what they found:

• Control group (with brand search campaigns):
  • 6% increase in website sessions compared to the previous period.
  • 5% increase in the number of sales.
  • 5% increase in revenue.
    
    
• Holdout group (without brand search campaigns):
  • 5% increase in website sessions.
  • 4% increase in the number of sales.
  • 4% increase in revenue.

Conclusion: Assuming all other variables remained constant, the marketing department can conclude that pausing brand search campaigns had a small negative impact on website sessions, sales and revenue during the trial period. This could suggest that most of the clicks from paid brand search campaigns were effectively captured by organic search results instead. Therefore, RunFast can safely pause their brand search campaigns without significantly affecting their overall performance. 

Methodology of geo tests and geo holdout tests

Both geo tests and geo holdout tests utilize the same fundamental methodology: exposing a test group to a specific marketing activity while maintaining a control or holdout group that is not exposed. The primary difference lies in the objective of each test.

Geo tests typically introduce a new marketing activity or advertising channel in selected geographic areas to measure its impact, aiming to assess the potential benefits of the new initiative. Conversely, geo holdout tests pause or remove an existing marketing activity in certain geographic regions to understand its contribution, evaluating the necessity and effectiveness of current strategies by observing how performance metrics change when those activities are absent.

Planning your holdout tests

Setting up a geo test involves several important steps to ensure that the results are meaningful and actionable. Here’s a guide to help you plan your geo test effectively:

Decide whether you want to conduct a regular geo test (introducing a new marketing activity) or a geo-holdout test (pausing an existing marketing activity). This decision will shape the overall approach and objectives of your experiment.

Identify the key performance metrics you will monitor, such as website sessions, number of sales and revenue. Ensure you have the necessary data available, using tools like Funnel to help collect and analyze this data.

Decide how long your experiment will run. Shorter experiments (one or two weeks) can provide quick insights, especially if you spend substantial amounts on advertising. Longer experiments (up to three months) allow for more comprehensive data collection, which can lead to more robust conclusions.

Keep in mind that it may not be wise to run a geo test during a seasonal or special event, such as Black Friday/Cyber Monday, since additional variables will affect your experiment and results. If your company decides to run a geo test during such a period, you’ll need to account for the seasonal effects across your test and control groups. For example, it wouldn't be smart to run a geo test where the control region offers many discounts while the test region has none.

Select the regions for your control group (where no changes are made) and your experiment group (where the test will be conducted). These regions should be large enough to provide relevant information. Test and control markets can be regions, states or even whole countries, and they don’t need to be of the same size. However, they must be comparable in terms of demographic and market conditions.

Inform all relevant colleagues and teams about the experiment. Clear communication ensures everyone is aware of the test, understands their roles and knows what to expect.

Implement the planned changes in your test regions and monitor the results over the predetermined period.

Gather data throughout the experiment and analyze the results. Discuss the findings with all stakeholders and ensure that the learnings are shared with all involved parties. This collaborative review helps refine future marketing strategies and improve overall performance.

By following these steps, you can set up a geo test that provides valuable insights into the effectiveness of your marketing activities, enabling you to make data-driven decisions and optimize your marketing efforts.

Analyzing the results

When analyzing the results of your geo test, it’s crucial to follow best practices to ensure accurate and actionable insights.

Firstly, ensure data accuracy by verifying and cleaning the data collected during the experiment period. Remove any anomalies or outliers that might skew the results. Applying appropriate statistical methods helps determine if the observed differences are significant and not due to random variation.

Consider contextual factors such as seasonality, market trends and demographic differences between test and control regions. This analysis provides a deeper understanding of the broader context and ensures more accurate interpretations of the results.

Comparing the test results against historical data helps assess the impact of the tested changes. This comparison can validate whether the results are genuinely due to the experiment. Engaging relevant teams and stakeholders in the analysis process allows for multiple perspectives and insights, leading to a comprehensive understanding of the findings.

Don’t draw conclusions too soon

Avoid making decisions based on preliminary data. Wait until the test period is over before drawing your conclusions. Be cautious with results from small sample sizes, since they may not be representative or statistically significant.

Consider both short-term and long-term impacts to understand the full effect of the tested changes. Analyzing the control group’s performance is essential to isolate the effect of the tested changes from other variables. Recognize natural variability in data and distinguish it from real changes caused by the experiment.

By focusing on these essential practices, you can ensure a thorough and accurate analysis of your geo test results, leading to better-informed marketing decisions and optimized strategies. Taking a holistic view, assessing incremental impacts and continuously improving based on insights will help drive future marketing success.

Benefits of geo experiments

Geo testing and geo-holdout testing provide valuable insights that help marketers make informed decisions about their strategies. 

Overall key benefits: 

1. Accurate measurement: By isolating specific geographic areas, marketers can accurately measure the impact of their campaigns without external influences.
2. Cost efficiency: Testing in smaller, controlled areas can save costs compared to a full-scale national campaign.
3. Scalability: Successful strategies identified through geo testing can be scaled up to larger markets with confidence.

Benefits of geo testing:

• Risk mitigation: By testing new marketing activities or advertising channels in selected geographic areas, companies can minimize risks before a nationwide or global rollout.
• Data-driven decisions: Geo testing allows marketers to gather concrete data on the effectiveness of new initiatives, ensuring that resources are allocated to strategies that deliver positive returns.
• Performance optimization: This method helps identify successful strategies, enabling marketers to fine-tune their campaigns for better results when scaling up.

Benefits of geo-holdout testing

• Channel evaluation: Pausing existing marketing activities in specific regions helps marketers understand the true impact of these channels on key performance metrics.
• Budget optimization: By identifying which channels are essential and which may not be as critical, geo-holdout testing allows companies to optimize their marketing mix and budget allocation.
• Strategic refinement: This approach provides insights into the contribution of individual channels, enabling marketers to make data-driven adjustments to their overall strategy.

Both geo testing and geo-holdout testing offer a structured, data-driven way to refine marketing strategies, ultimately leading to improved performance and more efficient use of marketing resources.

Other forms of marketing measurement

In addition to geo testing and geo-holdout testing, there are several other forms of marketing measurement that provide valuable insights into the effectiveness of marketing strategies.

Marketing mix modeling (MMM) is a statistical analysis technique that evaluates the impact of various marketing activities on sales over time. This technique helps marketers understand the effectiveness of each component in their marketing mix, allowing them to allocate resources more efficiently and optimize their overall strategy.

Multi-touch attribution (MTA) assigns credit to multiple marketing touchpoints along the customer journey. This offers a granular view of how different channels and interactions contribute to conversions, providing a detailed understanding of the customer path to purchase. This method helps marketers identify the most influential touchpoints and adjust their tactics accordingly.

Time series comparisons analyze performance metrics over different time periods to identify trends, seasonal patterns and the impact of specific marketing activities. By examining data in a temporal context, marketers can discern long-term trends and evaluate the success of their campaigns relative to past performance.

These methods complement geo testing by providing a comprehensive view of marketing performance, enabling marketers to make more informed data-driven decisions.

What our experts say

We asked our Performance marketing expert Lee Riley about his experience with geo experiments.

About geo testing and incrementality:

“In Marketing science, geo testing is about determining the incrementality of ad spend, aka, how many conversions occurred BECAUSE of the media strategy and how many would have occurred anyway due to the brands existing customer base.”

About his own experience with geo tests:

“In my experience within various agencies, my primary goal has been to demonstrate the true value of marketing channels or strategies that clients either overestimate or underestimate. For instance, Google Search may appear highly effective in GA4, while META performs poorly. This discrepancy is mainly due to GA4’s problems with cross-device and cross-browser tracking, as well as its reliance on last-click attribution. By conducting a geo test, we can bypass these industry challenges and gain a more objective insight into META’s actual value compared to other channels.”