Video Transcript

– [Matan] Thank you for joining us. My name is Matan Block-Temin. I’m a marketing data scientist in Optimove as part of the Strategic Services team. – [Jonathan]

Hi. My name is Jonathan Inbar, also a marketing data scientist same as Matan.

– So, like I said, we’re going to talk about the scientific way to compare CRM tactics. I’m going to start out with talking about the motivation for comparison, why do we even have this as a topic here at the workshops, and what’s the motivation behind it. What is the journey within the CRM world? We’re going to describe two main comparison methodologies that we use in order to compare between tactics.

The first one is super controller and the second is streams which is basically comparing two journeys to each other. And we’re going to finish out with takeaways and some useful advice you can take with you afterwards with your day-to- day work. So, before we start, we’ll just start with defining the difference between tactic, the strategy. Sometimes it can be kind of misleading or you’re not sure what is what or they can sound as the same word.

But when we talk about strategy, so, strategy refers to the direction towards our business goal, and tactics are the actions that are taken to support the strategy. So, basically, when we’re talking about tactics we’re talking about the actual campaigns, offers, communication to our clients. So, what’s the motivation for comparing tactics? First of all, there are many different reasons to using different tactics within our strategy.

But first of all, optimizing tactics for each customer persona. We have many different clients with different behaviors and we need to use the right tactic for the right person and right customer. Second of all, different campaigns have different objectives. Obviously, we need to use the right tactic for the right objective per campaign.

If we’re trying to cross customers between platforms, between desktop to mobile, for example, we’ll have a different tactic from a campaign that’s the purpose is to retain [inaudible] customers. So, we need to work on what is the right tactic for the right objective. Objectives may change over time. We’re all working in a dynamic environment. It could be new competitors in the market, it could be different regulations that forced us to change our tactic, and it could be that we’re an online betting company which during the offseason, obviously, it’s not the same as the main season, and we need to change our tactic accordingly.

And last but not least, testing and validating. Us as scientists in the CRM world and the marketing world, our main goal is to test and validate that what we’re doing and what other team members are doing. works and actually really is effective. And sometimes based on these tests that we do, we need to tweak and change our tactics according to the results.

So, what is a journey? A journey is a sequence of campaigns put together to implement a CRM tactic. So, as we said before that the tactic is the action, so in this case, the actions are three different campaigns, campaign A, B, and C. Each one of them has their own purpose, own goal. When we look at them together as one, basically, look at them together, we create a journey.

– Thank you, Matan, for explaining a bit about the motivation and start talking about the journey, but we can’t really start talking about what is a journey without talking about the moving parts inside of it. Okay? So, I’m going to explain about a few variables that are in the campaign level which eventually help us in building the whole journey itself. So, I’ll start with the content, all right?

And by content, basically, I’m talking about testing may be different subject lines or different images inside the templates or even the whole template itself. Right? We’ll measure it probably with click rates, open rates and so on. In terms of promotion, so we can do either an A/B test or regular test versus control campaign.

And there we will measure the response rate of our customers or maybe the average KPI that we set up before, it could be number of orders, average purchase amount, and so on. Also, channel is an important variable as well because basically it gives us the option to like, see the effect of the same offer on our customers using different channels, right?

So, sending the same discount maybe via email and SMS and push. And over there we’ll check the channel deliverability metrics if it’s an email. So, also clicks and opens, Facebook and be like conversions and so on. Just before I move forward, it’s also important to say that all of the variables are connected to each other, right?

So, different… You can send different offers with different subject clients inside the template, you can change it. You can say change stories, the image for each template for each offer and send different offers through various channels. So, everything is connected to each other. And also we have something in the journey level, right? So, we talked about the campaign level.

In the journey level, we can test using either super-control group or a journey versus journey method. We will talk about the two more thoroughly later in the presentation. But just the touch about the KPIs here. So, here we’re talking about the more high-level KPIs that we want to measure, right? We’re looking at our population and want to see the change in the total retention rate or the increase in total revenue, right?

All the things that are coming after the journey and basically the long-term impact from the journey. Sorry. So, let’s go quickly over some campaign analysis parameters. I’m sure most of you know all of them, but just for the sake of the presentation, let’s have a quick reminder. Okay? So, starting with the control percentage, an important business decision basically, deciding how many customers am I going to exclude from my campaign in order to be able to do the tests.

So, it’s sometimes not the easiest thing to do, but for us in order to test something it’s a must. Now also in terms of multiple offers, right? Instead of doing just the regular test versus control campaign we can take our test group, split it into several small groups and send different offers to each one of them, and then measure basically the impact of each offer compared to the control group itself.

Now we talked about the control group size, but what about the entire group size, right, that we want to send the campaign to? In order for us to be able to measure or get statistical significant results in a decent time, we need to make sure that our entire group size is large enough. Sometimes over-granularity can hurt this and we found ourselves run a campaign for three, four months without being able to actually know if it’s working or not.

Duration. So, basically, what’s the duration the campaign is going to be scheduled around of each occurrence? Basically, it can be derived from two things. First one, most simple one, for how long the offer is going to be valid, right?

Am I going to give you a discount for only today, for the whole week and so on? Or the second thing, in terms of analysis view, so how long am I going to measure the effect of the campaign or the impact of the campaign? Is it for one day, two days, three days? And from personal experience, we can say that it really depends on where what the stage your customer at right now.

For example, if we’re talking about new customers, so we usually suggest to make the duration between two, three days because we will probably want to target them more frequently during the week, right? So, to be able to make more cleaner tests and not have like overlapping campaigns, we need to do the duration a little bit shorter.

KPI. So, we talked about it a little bit about in previous slide. But basically, it’s what we decide to measure the success of the campaign, right? It can be the number of orders, it can be the average order amount, it could be the number of customers that for the first time have used our mobile app if this is the purpose of our campaign.

But it’s something that we need to consider and define before we schedule it. And last thing is timing. Basically, when do I want to send my campaign to my customers? So, it’s something that can simply be decided by doing like a time-of-day analysis or something like that, but based on the purpose of the campaign to decide if I want to send it to my customer during evening hours where they’re at home and probably have more spare time to open emails or during the daytime, maybe during lunch time at work.

So, another important thing to consider. Now, how do we measure the campaign uplift in Optimove, right? For those who doesn’t know the way that we do it, we can basically calculate the campaign uplift by taking the actual value that we get, right?

So, taking, like, for example, the total order that our test group and our control group generated and subtracting the estimated organic value meaning what would have happened if we didn’t send any campaign at all to those customers, right? So, we can calculate this part by taking our entire group size, multiplying it by the control response rate and the control average KPI.

– Thank you, Jonathan. So, now that we covered the basics and remembered what are the different parameters and how we measure a single campaign, let’s move forward to talking about the main purpose of this workshop and it’s basically talking about the transition from a single campaign to a full journey.

Basically how we move from looking at a single touch point to a multi-touchpoint within the customer’s life, within the customer’s journey. So, as we talked about in the beginning, we’re going to cover two different analysis methods that we use in order to evaluate if our tactic is working or not. The first one is a super-control group which you can see on the left side, the visualization of it.

And on the right side is a streams. Basically, we compare between two different journeys, compare between A to B. So, to start, we’re going to start with the super-control group. So, what is a super-control group? Basically, it’s excluding a percent of the total population from all the journey communication. So, if we’re looking at this example, we have eight different campaigns throughout the journey of the customer and we’re going to exclude 5% of the population from receiving all the aid campaigns within the journey.

Now, it’s important to take a representative sample of the population in order to be able to run statistical tests. Now, the 5% is only an example. It could be 2%, it could be 4%. It all depends on the size of the group, of the total group. If we’re running a campaign or a journey, for example, on the lapse cycle, on churn customers and we have hundreds of thousands of customers, we don’t necessarily need to exclude 5%, but we can only use one or two.

So, now that we understood how it works, let’s talk about the pros and cons. So, the main advantage of using this method is basically that we have the ability to measure the impact of the whole journey. This is something that is most important to know if our tactic is working or not. We’re able to look at the big picture and get a financial impact of the full journey.

Second of all, we’re able to run statistical significant results by using different tests. And third is we’re able to see the correlation between campaigns. Sometimes when we look at a single campaign and we say, “Oh, it’s very successful or it’s unsuccessful,” but we’re not aware that the campaign before is reflecting the results of the campaign.

For example, if we look at the onboarding journey of a customer and the first email the customer gets on the first day, it generates lots of revenue, but the three campaigns we’re sending later have no responders and are not working because we’re cannibalizing them with the first campaign. So, here we’re able to see the big picture and how one campaign reflects the other.

Now, when we look at the disadvantages, we need to remember that we’re excluding a percent of customers from receiving all communications, meaning, like an example we saw before, they’re eight different campaigns throughout the journey. We’re excluding them from all eight campaigns in order to run statistical tests. And we need to make sure that, for example, if it’s a strong group of customers, the VIPs, or any strong tiers we’re excluding them from receiving communications.

So, we need to take that in mind. And last is a long measurement timeframe. As Jonathan said, we’re looking at the long-term effect. And to get the long-term effect results, we need to exclude them for a long period of time, usually at least three months to get significant results. So, that’s something we need to take into consideration as well. So, how do we actually measure the tactic using the super-control group?

So, what we do is look at the different timeframes that we’re running the test and compare the results between the test group to the super-control group. In the example here we see six different months where we were testing the retention rate of the tests and we see that basically, the test group outperformed the super-control group in each one of the periods that we were measuring.

Now, as I said, we’re able to come out with a financial uplift of the whole journey by using the same logic that Jonathan explained before on a campaign level, by applying it on the whole journey level. How do we do that? We basically take the results of the test group, in this case, 10,000 customers in the test group, 7,500 of them responded with a certain response rate and an order amount compared to the super-control group.

And if we apply the same logic, if we multiply the 7,500 by 60 plus the 700 by 50, that’s the whole group results and we subtract what would have happened either we will not send the journey at all meaning we take 11,000 multiplied by 70% which is the response rate of the control group and multiply by 50.

If you do the math, you come up with 100,000 which is the uplift of the total journey.

– Right. Thank you, Matan, for explaining what is a super-control group. Sorry. Now, I’m going to discuss about the second method of comparing tactics which is streams. Okay. So, let’s start with what is it. Okay.

So, basically, streams meaning taking our entire population that we’re going to send the journey to. Let’s say, for example, a new customer cycle. Okay? And we’re dividing it to two different communication journeys in order to define what will be the optimal journey to send to our customers, let’s say, testing to onboarding journey on the same time.

Okay? So, pros and cons for this, as I said, which I think two journeys in parallel. So, it saves us a lot of time. This is something that is good for us, right? We don’t need to run a journey after journey, see what are the results, fix and then try again. And also here, no customer is left behind. Right?

So, as opposed to the super-control group where we said that we need to exclude a portion of our customers, here every customer is falling into one of the two journeys and never miss a communication. Now, the downside here is first, it’s more resource-heavy, right? We need to create more templates, we need to create more bonuses, it takes more time to manage two journeys in parallel.

And also as opposed to the super-control group, here we can calculate a financial uplift because we don’t have a control. But let’s go to the next slide and see what results we can get and why is it good for our business. Okay? So, starting on the left-hand side, same chart that we saw in the super- control group, we measure the monthly retention rate for the stream A and stream B.

We can see that both of them start at the same point, 74%. Just for this example, we can call it like the pre-test time. And then we can see basically the progress in each month for each of the journeys, we can see that obviously stream A perform better, so we know what journey is preferred to our customers. Also on the right-hand side, we can basically calculate the total difference pre- and post-test in terms of retention rate and using our statistical tests that we discussed before, basically see if the difference is really statistically significant, which can be a really powerful insight to our business.

Okay. So, guys, what do we want you to take from this session. Okay? So, first, focus on the macro point of view. Hey, we usually tend to focus on the single campaign and not on the entire journey which is like focusing on the single tree and not seeing the entire forest surrounding it.

Okay? Also, it’s not an alternative, just rather another perspective. Okay? It’s still important to analyze our single campaigns, but going on the macro level help us to understand the connection between the series of campaigns and the results of the entire series itself. Also in terms of multi-touch, we talked about understanding the correlations between campaigns, right?

So, we can maybe sometimes cannibalize other campaigns using higher bonuses earlier in the journey. Okay. We won’t know for sure if we won’t go macro. We also care about the long-term effect here, right? We said we want to maybe measure the change in retention rate or the increase in lifetime value of customers or maybe changing survivability, right?

So, basically, all the things that are coming after the journey ends and not during the journey itself. And last thing which I think is the most important here, is don’t be afraid to experiment, right? You won’t know for sure if you won’t test it. And believe me that it will help to optimize and understand our campaigns better and get more targeted results.

Right? Thank you very much.