Control Groups in Marketing

What are Control Groups?

A control group, or holdout group, is a subset of the total group of customers being exposed to a test. Many clinical trials and social experiments use control groups to illustrate what would have happened to patients or participants if they had not received the test treatment or if they had received a different treatment known to be effective.

In marketing, control groups are used to measure the impact of a specific campaign or customer journey. Specifically, control groups are the customers you are targeting with a particular campaign who will not receive that campaign. The counterpart of control groups is test groups which are the customers you are targeting that will receive that specific campaign.

For example, if you are sending a particular offer to a segment consisting of 1,000 customers and using a control group size of 10%. Your campaign will only be sent to 900 members, the test group, while 100 customers will be set aside to be your control group. Those 100 customers will not receive the campaign at all.

Why are Control Groups Important in Marketing?

It is crucial to understand why a control group is important. Using a control group in marketing campaigns is the only way to measure the incremental impact, AKA uplift, of a campaign or series of campaigns on a measured KPI. Without control groups, you would never know if your campaign was effective, ineffective, or even detrimental. By measuring the difference between the test and control groups, both in response rate and the chosen KPI, you can learn the true impact of each marketing activity.

Consistently using control groups in marketing campaigns reduces the possibility that you are cannibalizing sales that would have happened anyway. Cannibalization occurs when customers who did not receive a communication or promotion end up purchasing your products, implying that whatever was offered to the test group customers was unnecessary and costly for your business.

This short video explains the basics of it all and the benefits that control groups offer you as a CRM marketer. Watch the video or read the transcript here.

Why do some marketers not use Control Groups?

Many marketers believe that using control groups is equivalent to leaving money on the table. Those marketers must believe that everything they create has positive results. However, we all know, not every campaign works and not every promotion is necessary.

Other marketers believe that if a campaign has proven to be successful, they shouldn’t miss out on the potential revenue that would otherwise be generated through sending the campaign to the entire segment. Though understandable, this assumption fails to consider possible changes in circumstances that may affect customers’ behavioral patterns and preferences, as well as the need to constantly track campaign results to ensure that they develop positively.

Marketers looking to get started with using control groups, can a/b test control groups as part of their larger strategy in order to determine their value.

When are Control Groups beneficial for Marketing Campaigns?

Control groups can not only help marketers learn the incremental impact of their campaigns, but also quickly identify changes in customer behaviors that require adjustments to current marketing campaigns and journeys. By identifying the changes in customer response and incremental impact to KPIs, marketers can adjust or completely stop their current efforts

Below are a few common situations in which control groups can help marketers discover a need for such changes:

• Unforeseen Market Events or Changes in Customer Preferences – A new campaign by a competitor or a sudden change in customer behaviors can quickly reduce your customers’ response rate and the order value of your existing customer base. This may require a stronger incentive or an alternative type of offer that the competitor is not offering.
• Repeating Offers – Offers may not be as intriguing to customers who have seen them repeatedly. This may require a refresh of the campaign template to or a need to sub-segment the entire audience into smaller segments to separate those that are still responding to the campaign from those who are not. Targeting the latter with alternative offers.
• Overlapping Offers – Overlapping campaigns can cause a decrease in response rate due to customer fatigue. In this case, the use of control groups may help first uncover the existence of overlapping campaigns and then determine which ones are unsuccessful.
• Irrelevant Offers – Targeting customers with irrelevant offers, can lead to them becoming indifferent to your campaigns, significantly reducing engagement.  Irrelevant offers are mostly the result of human errors that should be prevented. However, when they do happen, control groups can surface the problem quicker as they don’t only focus on vanity metrics such as Open Rates but also report on the direct financial impact of campaigns.

When are Control Groups not Necessary in Marketing?

There are a few situations in which using a control group is recommended but not completely necessary. These are situations in which either marketers aren’t interested in seeing the ROI of a certain campaign or there is no risk of a change in circumstances that may lead to a change in customers’ responses to the campaign. Some examples include:

• Reminder campaigns – in which only the customers who were part of the initial test group receive a reminder of the campaign.
• Non-trackable campaigns – if you cannot track the responses of your customers to a marketing campaign, for example when targeted customers are used only as a reference point (e.g., Facebook Lookalike Audiences). In these cases, there is really no need for a control group since the responses to the campaign will come from customers outside your initial segment.
• Educational campaigns – some marketers prefer that all customers receive educational campaigns. While this is understandable in situations such as additional purchasing platforms “Sign up to our new App”, it is important to remember that even educational campaigns or newsletters can show uplifts between the test and control groups since they are bringing awareness to the brand, causing customers to make purchases.

How to Select Control Groups?

It is very important that control groups are a representative sample of the overall population receiving each campaign. To ensure this, you must take into consideration the control group size and its randomness.

The first step to ensure the control group size is accurate. As a rule of thumb, campaigns targeting fewer customers require a larger percentage of the recipient population to generate a valid control group, whereas campaigns targeting more customers can set aside a smaller percentage. For 10,000 customers, a control groups of 5% is sufficient, while for campaigns targeting less than 2,000 customers, it’s a good idea to use 10%-20% instead.

There is an additional factor to take into consideration when deciding upon the size of your control group: your expected response rate. When you expect a particularly low response rate for a particular campaign, such as when sending an offer to long-dormant churned customers, you will need a larger control group in order to obtain statistically significant results. On the other hand, if you expect a particularly high response rate, such as when sending a special promotion to your best customers, a smaller control group will be sufficient.

The second factor is the control group’s randomness. For you to accurately report on the incremental impact of your campaigns or journeys, the control group must be a representation of your target segment. As a result, it is recommended you divide your segment into various micro-segments and choose customers at random from each one. How many customers you choose from each micro-segment should be proportionate to the size of the micro-segment compared to the size of the entire segment.

As already mentioned, the idea is for the control group’s customers to be similar to the members of the entire group, and thus represent what the entire group of customers would likely have done had they not received the particular marketing campaign being tested. The approach of randomly selecting a proportional number of customers from each micro-segment is an ideal way to accomplish this.

How to Minimize Control Group Sizes

Let’s face it, no matter how convinced you are of the value of control groups you will always want to minimize its size. However, manually reiterating on the control group selection process is too resource intensive to be a viable option.

Since with each campaign iteration, additional data regarding the segment’s performance is collected, it is possible to algorithmically determine the ideal size of the control group for each new iteration of your campaigns. If the campaign is successful, this will reduce the control group size each time, minimizing it while still allowing you to measure your campaign’s impact.

The logic is as follows. A campaign that proves itself will gradually reduce its control group to allow more and more customers to receive the campaign. On the other hand, a campaign that shows signs that it is no longer working will increase the control group to better understand whether it really is not working any more or if the apparent decline is a false alarm. This approach helps you achieve statistical significance more quickly, reducing the time and number of campaign occurrences required to draw the solid conclusion that the campaign should be stopped.

Optimove’s Productized Control Groups

Every campaign within Optimove is treated as an experiment, with a control group allowing marketers to measure the incremental uplift they have on any custom business KPI. Optimove provides AI-based optimization of control groups, minimize size while still accurately measuring results.

To learn more about Optimove’s marketing platform request a demo.

Related Articles

Marketing Incrementality  Statistical Significance and Credibility in Marketing Customer Segmentation