Even though marketing has become way more sophisticated and data-driven in recent years, we still see too many organizations that measure their marketing efforts using soft, “proxy” metrics.

Measuring campaign efficiency through metrics such as open rates, click-throughs, impressions, and conversions, isn’t truly indicative of performance. This entire incentive structure is wrong. And it’s evident in the numbers.

The goal of marketing is not to generate higher email open rates and conversions. It’s to create incremental engagement and/or sales for the business.

And especially in times when almost everybody’s cutting ad spend, and striving to be as smart and direct with their budgets, who can afford NOT measuring the right things?

Keep Your Eyes on the Prize (and your existing customers)

The cost of acquisition was always higher than that of retention. And while in this reality of a global pandemic, on the cusp of a worldwide recession, focusing investment on your existing customers is the “safer” bet – that doesn’t mean you should be doing CRM without the courage to measure the right things.

The overbearing focus put on those “golden KPIs” has been counterproductive to measuring and understanding impact. Real performance means the clear scientific understanding that a campaign generated an incremental difference, such as a positive benefit in terms of revenue or engagement, that would not have otherwise happened.

For example, a 6% conversion on an email send might be considered very high, but what if we can show that without that email, the same customers would have converted with a 5% rate? Then, the actual impact here is +1%. In other words: maybe not so good.

Control Groups. This is The Answer

The use of control groups in marketing is not a new idea. So why is it still so rarely used or difficult to apply across the board?

The only way to measure true campaign effectiveness is through the systematic use of control groups. A control group, or a group identical to the test group, which is excluded from any action, creates a baseline to compare your performance against and see if in fact there was any impact at all.

Running control groups means that for each campaign, a portion of the eligible audience is excluded from the campaign. However, many marketers who are incentivized or evaluated based on sales (not incremental sales) are very reluctant to risk losing out on potential revenue by excluding customers from a campaign. This is how their employers assess performance, so why would they want to endanger themselves?

Ignorance Is Bliss But Knowledge Is Power

In short, ignorance sometimes can be bliss, but the knowledge of what truly works is immense power. The temporary downside of excluding a few customers from any campaign is entirely worth the upside in future revenue generation. But some marketers still beg to differ:

1. I don’t want to sacrifice topline revenue

This is an issue that starts at the top. As long as CMOs continue to stare at weekly sales numbers as the golden metric, without any calculation for incrementality, there will be entire marketing departments shooting in the dark without any understanding of whether they are actually impacting behavior. Sales should certainly be an evaluation factor, but not just the only one.

Marketing teams who made the courageous change towards the systematical use of control groups (and incremental uplift as the guiding performance metric) significantly increased their overall team performance. That is because they gained visibility into what generated enough of an impact to justify the spend. This allowed them to make thoughtful adjustments over time.

As to the fear of having too many people, or the wrong people stuck in a hold-out, there are several marketing orchestration tools that solves for this through AI. They will for instance use the smallest possible control group size, ensure representation of customer sub-segments, and reduce the size of the control (as confidence in results grow) in an effort to mitigate any loss.

2. I use control groups/want to use control groups but cannot scale or automate

The question of how to best scale, test, and use control groups is a recurring one. Regardless of the size of your business, the sheer logistics of implementing Test vs. Control at scale is overwhelming. It entails isolating a control group, waiting to see the significance, making a decision, implementing the decision (and doing so several times.)

It is a real issue and this exactly where AI-powered technology can serve as an accelerator. It’s not about replacing the marketer but providing a framework for testing repeatedly, at scale. It lets technology provide a framework for execution so that marketers can focus on designing experiments.

In that sense technology-productized test vs. control modules allows you to:

1. Systematize and automate the use of control groups for every campaign. Get rid of the manual labor.
2. Ensure minimal top-line disruption by reducing the hold-out group to its smallest possible size and guaranteeing optimal representation (hint – all your VIPs won’t be in the control group.)
3. Manage controls across channels and ensure that customers who are excluded not only from this campaign won’t be targeted by other channels during the chosen timeline to keep the experiment accurate. (This is exactly what doesn’t happen when you test from your ESP and is only possible because Optimove is connected to all your execution channels.)
4. Automate the reporting process and surface incremental lift analysis based on chosen KPIs.

3. It doesn’t cost me much to simply email all of my customers

Well, that’s just not true. Bombarding customers because “email is cheap” is a marketer-centric approach, not a customer-centric approach. The opportunity cost of sending communications with no real impact (just noise to the customer) is massive. Here are just three reasons why:

1. It’s about the money you’re losing by not testing, not about how cheap emailing is. Every time you send a campaign without a control group, you are missing out on the opportunity to understand whether a customer purchased because of your campaign, or without any connection to it. This is especially true if your campaign included a discount. What if your customer would have purchased without that discount? Then this is a miscategorization of revenue that you will forever be unaware of. The small cost of running a control group on those campaigns could have saved you thousands of dollars in additional margin.

• By not testing, you’re blind to the true effect that your campaign is having on customers. You’re ignoring the fact that campaigns that are just noise to customers will contribute to alienating them. The biggest risk is for your customers to unsubscribe as a result of too many communications that have too little impact. You must understand impact to avoid email fatigue, which may ultimately send you off to the Spam folder.

• You’re losing “Share of Attention”. Every email that you send that has no incremental impact is taking attention away from another campaign you send that might move the needle better.

We hope that after reading this you understand why control groups keep you in control as a modern marketer. And why it’s all the more critical to use them when budgets are tight.

Keeping up to date with the most effective and efficient email marketing practices will make sure that both you and your customers remain happy. And your boss, too.

Published on March 26th, 2020

Posted in

• Customer Retention

Agathe Westad

Agathe Westad is Director of New Business for Optimove in North America. Agathe specializes in advising large corporations and growing startups alike on the best ways to deploy more personalized and scientific marketing strategies, with the aim of achieving measurable growth in customer engagement, loyalty and LTV. Agathe holds an LLB in International Law from King’s College London and a Masters in Commercial Law from the Sorbonne.