Video Transcript

Well, I hope you’re enjoying the session so far. In our session, we’re going to be talking about the CRM contribution, knowing what your CRM is worth and how you can improve it. So, I’m Roni, I’m the director of data science, and I sit in our London office, and this is Chen, who is our data science team lead here in our Tel Aviv office. We’ll start off today’s session, talking about the motivation of why we need to have a metric that helps us understand what is our CRM contribution.

Then we’ll talk about how we define, that’s CRM contribution metric, the calculation of that metric and the key components. And then Chen will actually go over a very interesting research that he has conducted across all of our clients to understand what does CRM contribution like across the industry? And then what are the key factors that we’ve noticed for clients that have a successful CRM contribution?

And, of course, we’ll finish off with a few takeaways. So, as smart marketer tiers, it’s very important for us to be able to understand what is the value that we generate for our company? But being able to know what is the right way to measure and engage your contribution can actually be very challenging. Now, even though this might not be the most straightforward task, it is still one that we very much need to focus on.

So, we see the CRM contribution as the North Star metric of the whole CRM organization is going to be one long-term metric for the full CRM team to get behind, continuously monitor and see how they can increase. And even though individuals might have their own performance KPIs, they should still all be linked to the common goal of increasing our CRM value. Now, the CRM contribution metric shouldn’t just be used internally within the marketing team.

It just still be one that is publicized across the company because having a figure, having a value that shows what is the revenue that we generate, puts us market tiers in a very, very good place and very much helps to establish the business case of our strategic CRM. So, before just diving into how we’re going to look at the CRM contribution metric, I want to quickly go over what are the common approaches we see marketing teams take today when they want to understand the value of their marketing efforts.

Usually, we see different attribution models that are very much focused on acquisition. This can be last click, first, click, or a fractional attribution model. But we’re noticing though with ties that marketers are now slowly growing their investment in actual relationship marketing. Now, main reason for that is that consumers have just become much more sophisticated.

They expect much more from our brand. They’re not afraid to try new things and continuously look for that brand for that company that very much resonates with their value. And we see this. We all see competition continuously arise, and then because of this, acquisition costs continue to increase, and we understand that it’s becoming much more cost-effective for us to see how can we retain and actually grow the value of our existing customer database?

Which is why we need to have a metric that helps us understand how well we’re doing that. That’s going to be our CRM contribution. So, we define the CRM contribution as the incremental uplift generated from our marketing efforts. And notice that I have the word incremental highlighted over here. One of the biggest flaws we actually see in those attribution models that I’ve talked about is they tend to take all of the credit for our customer’s transactions without even asking the question, would a customer make another purchase or another deposit regardless if I communicated with that customer or not?

So, the question of incrementality is a very important one because that incrementality is what’s going to help us accurately measure how our marketing efforts are changing our customer behavior. So, like I said, it’s going to guarantee that we’re going to have accurate calculations and give us the assurance that our marketing efforts really do change our consumer behavior.

So, for us to be able to measure that incrementality, we’re going to have to adopt a scientific approach. We’ll do that by looking at every single interaction that we have with our customers as a small scientific experiment. That means that if we take the whole customer group that is eligible to receive a specific campaign, the majority of those customers will receive that campaign and they will be our test group and they’re a major group of customers even though they’re eligible for that interaction they won’t receive it and act as our control.

Then we’ll also decide what is the business KPI of this interaction? What would we like this interaction to achieve? Again, as in making other deposit or making other order. And then, we’ll look at the difference in behavior of the test versus control and looking at that change in the behavior. That’s what’s going to allow us to measure the uplift against those actual business KPIs. So, I’ll go through one example of how we can calculate that incremental uplift.

I know this is going to be familiar to a lot of you, especially any Optimove clients, but it is important that we just quickly go over it because it is the basis of everything else that we are going to cover. So, let’s say we have a campaign that there are 160 customers that are eligible to receive this campaign. Every little figure here represents 10 customers. So, 120 customers will receive this campaign.

They will act as our test group, and the remaining 40 customers will receive this campaign, and they will act as our control. And maybe in this example, you want to get this campaign to get customers to make another purchase. So, we’re going to look at the response rate. We’re going to look at what percent of customers made another purchase and the average revenue of that purchase. So, if we focus on the test group, we see that 25% of them responded, meaning 25% of them made another purchase, and the average revenue of that purchase was $60.

So, the total revenue from the test group is 120 times 25% times $60, which equals $1,800. Very similarly, we’re going to look at the control group. Here, we see that the response rate was only 10%, and the average revenue was $50. The total revenue from the control group is then $200. The total revenue from this campaign is $1,800 plus $200, which equals $2,000. So, for us to be able to understand what is the change in behavior?

We have to understand what did we expect these 160 customers to do if we did not send them this interaction? So, what we’re going to do is we’re going to take the control group’s behavior and apply it to the test group because the only difference between the test and control is that the control group did not receive this campaign and the test group did, and you would have expected the test group to behave like the control group if they did not receive this interaction.

So, we have the $200, what we saw from the control group, and then we take the 120 customers in the test group multiplied by the control group’s response rate, which is 10%. Multiply that by the control group’s average revenue, which is $50. So, the total expected income from these 160 customers in absence of this campaign would have been $800.

Now, the difference between these two figures, this is the incremental uplift. This is the change in behavior that this campaign was able to create when we send out this campaign. So, before we just get excited about the fact that we were able to generate a bit more than $1,000, we have another question to ask, which is how can we actually be sure that the uplift that we see here should be attributed to this campaign?

So, this is where the idea of statistical credibility comes in, and basically, when we look at the calculations, we want to differentiate between these two cases. One, then maybe the difference in test and control should have been expected anyways regardless of this interaction or that the difference in the test of controlled group’s behavior differs only because the test group received this interaction while the control group did not.

So, I’m not going to get into how we understand statistical credibility or statistical significance, but the main takeaway over here is for us to understand that we do need to be able to back up our incremental uplift calculations in some sort of statistical way. So, going back to our example, you can see that we have both the response rate and average revenue highlighted over here.

So, if we did see that the difference in behavior based on both of these metrics was statistically significant, then we can accurately say that this campaign was able to generate an additional $1,200 that we wouldn’t have seen if this interaction was not sent out. But there can be cases where maybe only one of these metrics are statistically credible.

Over here, you can see we only highlighted the response rate and the average revenue. The difference was not statistically credible. So, you can notice over here that we change how we calculate the expected income if this campaign was not sent out, and we only apply the control group’s response rate to the test group. And also, the other way around, there are cases where the average revenue can be statistically credible while the difference in response rate was not.

And again, you can see over here we change the expected income, and we only apply the control group’s average revenue to the test group, and the campaign uplift obviously changes as well. So, we covered how we can understand the incremental uplift of one specific interaction, but at the end of the day, we want to know what is the overall contribution all of our CRM activity?

So, the way we’re going to do that is, first, we’re just going to simply add up all of those uplifts that we see for every single campaign. In this example, that’ll be $100,000. But we can look at the uplift only for campaigns that have a control group. But we do know that some campaigns don’t have a control group. But if we see that the campaigns that do do generate a change in behavior and do generate some uplift, then we do believe that the campaigns without a control group did change our customer behavior in some sort of way, and we want to take that into account when looking at our CRM contribution.

In this example, we said that the percent of campaigns and control groups was only 80%. So, what we’re going to do over here is we’re just basically going to assume linearity. We’re going to assume that the change in behavior we saw in campaigns with control would have expected to be seen in campaigns without control. So, we’re going to divide $100,000 by 80%, which equals $125,000.

And this $125,000, this is the actual monetary incremental uplift that our CRM activity was able to generate. But we’re interested in understanding the proportion of this out of the overall revenue. So, if, for example, the overall revenue is $1.5 million, we’re going to divide between these two figures.

So, $125,000 divided by $1.5 million equals 8.3%, and this 8.3%, this is the CRM contribution metric. In this example, this is the part of the overall revenue that our CRM activity can actually take credit for. –

So, as we previously proposed, every marketing team should really look for the North Star and an Optimove one to do the same and find ours. And in this case, the CRM contribution was a perfect candidate for two reasons that I will describe. Eventually, we wanted to get to our industry benchmark and understanding where do our clients stand in terms of CRM contribution? Now, why is it such a good metric to track a long time?

So, the first reason is that we’re looking not just at the monetary value that was attributed to CRM activities but also looking at the old revenue of the business as well. Therefore, the results will not be skewed due to natural expansion of the business itself. Now, this is the first reason. The second reason we can really use it as an industry benchmark metric is that this metric is comparable among clients in different sizes.

Now, if we just have a look at the uplift and trying to compare two clients in two different sizes, it wouldn’t be a fair fight. For example, if a client has a natural revenue of $100 million and we’ll try to compare the uplift of this client to another one that has just $5 million of natural revenues. It will just not be a fair fight, and, of course, the big client will have a big advantage here.

But when looking at in terms of percentages and this is what we are doing, the same activity and the same contribution, basically it gives us the ability to compare clients with different sizes and also get to an industry benchmark that will not be skewed due to those different sizes. Now, we couldn’t really include all of our clients in the same sample of the research.

We had to exclude some in order to have valid and solid and also stable results that can be tracked a long time. So, there are four categories in which we excluded clients based on, and in order to be a part of the sample, a client must apply to all. So, the first one is seniority. We wanted to add clients that launched with Optimove at least six months ago in order to avoid having clients that are not in sort of a steady-state.

We also excluded based on annual income. So, we wanted eventually to avoid volatility. Now, when we have very small clients in December, their overall revenue is very small, and it also means that it’s very easy to change the same contribution, and it will not be stable along time, so we also excluded small clients. Now, coverage ratio is basically meaning what is the percentage of customers you’re covering on a weekly basis?

So, if the coverage ratio is very, very small, it means that eventually, you are just not active. You’re just not using Optimove in order to target to clients. That’s why we also exclude the clients that are just not covering at least a small portion of the clients. And maybe the most important category is test versus control. As Roni mentioned, CRM contribution is also based on the percentage of campaigns that were executed with test versus control.

And also, without having the test versus control, you can’t really get to the actual results the campaigns had. Therefore, if less than one-third of their campaigns were executed with test versus control method, client was also excluded from the sample. Now, we conducted this research in order to understand the current state, and the bottom line is basically that the average CRM contribution is 10%, while the 10th percentile is 33%.

Or, in other words, on average, our clients can attribute 10% of the revenue into CRM activities. And even more interesting to see is that our top-performing clients, the top 10% clients of ours, can attribute third of the actual overall revenue to CRM activities. Now, when breaking it down, we can see this chart over here, so we can see the first range that I just talked about, the top 10%.

They have, on average, 33% of CRM contribution on average. When moving to the second range, 10 to 20th percentiles, we can see another pretty good CRM contribution, which is 18% on average. And of course, as we’ll go down, we’ll see it decreases. Now, this trend also got us thinking, “Okay, we got the understanding of the current state.We know what is the average CRM contribution.We can track it a long time, and it really is a great North Star.”

But we also wanted to conclude from this research and try to understand what those clients on the left do better than the clients on the right. Or, in other words, how to increase your CRM contribution. Now, we used feature selection methods in order to understand what are those explanatory attributes that has high correlation to high CRM contribution. We were trying to understand what our best-performing clients in terms of CRM contribution do differently than the others.

Now, we took a sample of 100 clients that started working with Optimove after 2016, meaning this is a pretty relevant set of data. And also, we took only those clients that applied to different other categories as well.

They are big clients. They are covering the clients, and they use test versus control. These are the same rules that we used in the previous one, just that the timeframe now differ. We’re looking at the first year of performance in order to exclude maturity as a factor. We want to have a look at the first-year performance in Optimove of our clients and compare them. How do we compare them? We want to compare the top 20% clients in terms of CRM contribution and compare them to the remaining 80%.

So, we found the three different KPIs that has very high correlation to high CRM contribution, and then we used this comparison of top 20% to remaining 80% and trying to understand what is the difference? So, the first one is customer base coverage. This is the first metric that has a high correlation to high CRM contribution. Now, what does it mean customer base coverage?

As I just mentioned earlier, this is the percentage of customers we were covering on a weekly basis out of all of our customer base. Now, this X-axis here represents monthly go-live 1 to 12. It’s the first year of performance of these clients, and the Y-axis represent here the percentage of coverage on a weekly basis. Greenline will be top 20% customers in terms of CRM contribution, and pink line will be bottom 80%.

Now, what we can see here, for example, in the first month, we can see that both groups kind of struggling to get to a better coverage of their customer base. These are the first steps, but the big gap is trying to…that we can see is on the second month. This is where the top 20% clients in terms of CRM contribution also reach almost 50%customer base coverage.

And this is the point where the bottom 80% are just reaching around 12%. Now, as time goes by, we can see that the trend is pretty clear. The gaps remain sort of the same. So, this is in terms of customer base coverage, but we also set a very small fine-tuning to this metric and added the live coverage. What is live coverage?

Basically, live coverage is looking at what is the percentage of clients we are covering on a weekly basis but not out of all of our customer base but out of our live customer base? Meaning clients that also made an activity and haven’t shown yet. So, when looking at this trend, we see a very similar trend as well. We considered the top 20% customers are overcoming the bottom 80% in this metric as well of course, and just a few nuggets we considered after three months, the top 20% are getting to almost 70% of live customer base coverage and around six months from go-live they’re stabilizing around 80%.

Now, main takeaways from this one. We will want to maximize our live customers coverage, so it’s clear that we will want to cover as many as possible, but it’s not really possible to cover 100%of our customer base, and we found out that in order to get a better CRM contribution you will also want to maximize your live customers coverage. This means also focus on proactively retaining your customers rather than afterwards trying to reactivate them.

We noticed reactivating tuned customers is more expensive, and it’s more difficult. So also try to focus on covering your live customers on a weekly basis. So, this is the first KPI. The second one is pretty straightforward. We are talking here about the number of channels. So, we know that having many channels and communicating with our clients on a multi-channel and finding the preferred one has a lot of impact, but you also see it in terms of CRM contribution.

So, here again, we see demand for go-live, and we see the average number of channels campaigns were executed with on a weekly basis. So, at the beginning, we see that both groups struggling to get to the second…so the second channel. But we do see also that from the fourth months, the top 20% clients in terms of CRM contribution are also over-performing in terms of executing with multi-channels and they reach to the second channel in just four or five months.

And I can tell you this, in one year they get to 2.6 channels on average, while the bottom 80% are just scratching the second channel. Now, it’s not in your thing. We know that having more channels improve the ability to engage customers, and we also understand from this that having many channels will increase the probability of us to use the preferred channel when talking with our clients. So, it’s not easy to integrate many clients, but we know that it has a great impact.

Less KPI is number of target groups, so, in other words, number of different audiences we are talking to, we are communicating with on a weekly basis. And here, we can see that the top 20% customers in terms of CRM contribution are reaching just two months around 75 different target groups. It means that they have 75 different audiences they’re talking with. They have different communications to different groups of customers.

We will want to be as granular as possible, and having more target groups will lead to us being more granular. In this period of time, when the top 20% has around 75 target groups, the bottom 80 are just scratching the 20 or 25 target groups, and as time goes by, we can see the gap just keep increasing eventually. When going granular, the sky is the limit.

We don’t have any limit. We can just get and more target groups and more and more granular. So we would want to be as granular as possible. The more granular, the better results. Getting granular is not something that can be done in just one day. We don’t assume that you guys would just go back home and create 100 target groups in just one day.

It takes time, and we also saw it in the trends of our best-performing clients, but in just one year, they got to over 200 target groups, which is a great goal to have. And we want to leverage our campaigns. Eventually, we were using ABC testing, where you use test versus control. We can further drill down existing target groups and get more and more target goals based on the ones we already have and campaigns we are conducting.

Takeaways. So, find your North Star. We know that every marketing team should have a solid metric they can track along time that represents marketing organization. Value incrementality. Put efforts in examining your campaigns using test versus control in order to get to accurate results. Treat every interaction with your clients as an experiment, and eventually use incrementality in order to measure your results.

The last thing is that really all small wins truly adds up. We don’t assume that you will be able to double your CRM contribution just one day. Continuously move the needle by building more and more granular segmentations while increasing your customer base coverage and concentrating on live ones. And in the meanwhile, keep increasing your channels of communications.

Thank you.