Video Transcript

– [Yalon] Hi, and thanks for joining us for our pricing session. I hope you are all excited to learn how to test and to measure your pricing effectiveness. My name is Yalon Pauker. I am a marketing data scientist at Optimove as part of the Strategic Services department. And together with me today is Tom Marcus, a retention team leader in theLotter, which is one of the leading lottery brands in the world.

So, before we start, I would like to ask you a question. Please raise your hand if you came from abroad. Okay, So, even if you didn’t come from abroad, how many of you shop around on multiple sites before buying a flying ticket?

Okay, so if you shop around, you probably know that for each flight you can find at different prices and the airline industry is notorious for pricing the changes based on the profile of the search, but it can also change based on the demand.

And we have a lot of examples. Amazon reportedly changes its prices more than a million times a day. If you buy a dress in ASOS Spain versus ASOS UK, you will probably find different prices. Uber is changing its prices based on the demand and so on. And this is the concept of how the pricing world has changed and the introduction for our conversation today.

So, we have three main points to cover. The first two is what I’m going to discuss, which is the evolution of pricing and pricing tests, in particular, and how to run an effective AB pricing test. The third one will be covered by Tom. While I’ll going to focus more on theory and best practices, Tom will show you a real case study for pricing tests from theLotter.

So, setting the right price for your product is one of the most difficult tasks. And the pricing world has changed. We can see a real evolution from product-centric pricing strategies that becoming more and more customer-centric. The cost-based pricing is only one dimensional, it’s just a fixed margin on top of the production cost.

Then we add the second dimension, which is the demand. The relationship between the supply and the demand plays a key role in determining the price of the product. But then the technology has improved and this completely changed the game. In the past decade, the dynamic pricing has exploded mainly due to the improvement in technology because the stores are not physical anymore and the retailers are changing their prices on regular basis.

But actually it’s not only online. If I go to the market here in Tel Aviv for example, to buy foods, oranges, the price that I’m going to get speaking Hebrew will be definitely different from the price you are going to pay speaking English. So, sorry for that. But not only the pricing world has changed, but also the ways that we have today to test and to measure our pricing effectiveness.

Common research methods as surveys and focus groups can be really valuable in determining the pricing effectiveness and optimizing the pricing strategy, but they are resource-intensive time and money-consuming and also rely on small sample size. Then some market research companies were born to gather information and provide you insights to help you optimize your pricing strategy, Companies like Nielsen, for example.

But then the power has shifted into the company’s hands, which are collecting their data themselves and are able to harness this data to optimize their pricing effectiveness independently, in- house, without hiring external resources. In Optimove we take the science first approach.

So, we encourage you to use statistical methods to test and to optimize your pricing effectiveness. And today we are going to focus on how to design and implement an effective AB pricing test. We have three main steps to consider. The first one is setting the objectives.

Then we have to define an appropriate method. And then we have the actual analysis. So, let’s start with the objectives. And the first element as part of the objectives is setting the goals. Before you start a test, you have to be specific which goals you want to achieve. So, when it comes to pricing, it shouldn’t come as a surprise for you.

We either want to increase the revenue or the sales volume. Then we have to define KPIs. So, we recommend on a combination between a response rate and a monetary value. So, when talking about pricing, usually the response rate is the conversion rate, how you manage to convert your customers with the new price and the monetary value will be the revenue, profit, average income, and so on.

The last element as part of the objectives is to plan the test, how it will look like on the site. So, we differentiate between two types of pricing tests. The first one is just the perception, how the price looks on the site. It can be, for example, $10 compared to $9.99 or presenting a monthly rate compared to an annual rate.

The second type is an actual change of the price. It can be for a higher price or lower price based on your goals. So, we are done with the objectives, and now we have to plan an appropriate method. So, the first part is defining the test scope. Which customers do we want to analyze?

For example, do we want to test customers who haven’t seen our platform before or customers who have been there for a long time? Usually we recommend to test new customers because ideally, they are not familiar with the previous price that you had and also this is the first engagement with the brand, so, the test will be much more accurate.

In addition, you have to ask yourself, what are the demographics of the population you’re looking at? Do you want to analyze specific countries, specific gender? Do you have, for example, some low-value customers that you want to exclude from your test? And so on. But make sure not to filter too many attributes, so the target groups won’t be too small and we could reach the statistical significance.

So, how can we reach the statistical significance? We have to calculate the group size. We have the test group itself, now we have to calculate the size. So, we have to determine the group size. We have two main approaches. The first one, which is more simple and easier one is the predefined group size.

This is the method you will probably learn if you take an introduction to statistics course. It’s trusted and known method, but the easier one. It’s based on a statistical formula that uses four main statistical parameters, the conversion rate, the conversion increase, the confidence level, and the test power.

We have some best practices for all of them. You also have some free calculators available online and you can calculate the optimal group size without any statistical resources. However, we have a more advanced method, the dynamical group size, which is based on sequential analysis.

We can have a whole session about it, but I’ll only give you the main principles. In this case, you don’t start with predefined group size, you start with a certain amount of customers and then gradually add more and more customers to the test. The data is evaluated as it’s collected and you use a set of predefined stopping goals that tell you when to stop when you reach the statistical significance.

In this case, it potentially allows you to reach the statistical significance in a much earlier stage compared to the predefined method. So, this is the most sophisticated one, but both of them are valid. So, we have the test, the test group, and the group size, and now we want to start running the test.

But you also should ask yourself how long should the test be running? So, you have to define a timeframe for the test. Here again, we have two main approaches. The first one, the simple one is just to be patient. Wait until you see results. Wait until you see a change between group A and group B. It can be either few months or even more than one year.

The second approach, which is again based on sequential analysis, in this case, you start the test, you start tracking the gap between group A and group B, and, again, you use predefined rules, statistical rules that tells you whether to reject or accept the process and stop running the test.

So, we are done with the method and now to the fun part, at least for me as a data scientist, the actual analysis. The test was running, you have the results. Now, you have to analyze them. Three main points to consider while analyzing the results. First, you have to analyze subgroups.

A price change can affect differently different types of customers. For example, can you detect different effect on different countries, different effect on different value segment, VIPs versus low-value customers and so on? In addition, try also to detect some indirect effect. Don’t analyze only the initial KPIs defined but try to analyze also other business aspects.

Let’s take an example. If we reduce the price for a product, so maybe you can detect an increase in the sales volume for this product, but maybe you can also detect a decrease for another product. So, try to analyze all products and all KPIs related to the price. The last one is to analyze both short and long-term because usually we see a tradeoff between them.

If we continue with the same example, we reduced the price. So, probably in the short-term, you will see a decrease in the revenue, but because the conversion rate is higher and you have more customers, so hopefully in the long-term, we will see an increase in the revenue. But now let’s stop talking theory and see it in action.

I’ll hand it over to Tom Marcus, who will show you a real case study of a pricing test form theLotter. – [Marcus] Okay. So, thank you, Yalon. Hi, everybody. So, my name is Tom Marcus and I’m the head of retention at theLotter.

So, before dive through our pricing strategy test, just a bit of a background about our services. So, we operate thelottery.com which is one of the biggest lottery online services. We’re actually the biggest lottery messenger in the world, which means that we purchase the actual tickets on behalf of our customers.

We offer over 50 different types of lotteries on our website as well as scratch card games. We are the pioneers of the online industry, operating since 2002 and we have quite a lot of winners, which the biggest one among all of them is Aura from Panama who won two years ago, the Florida lottery, a jackpot prize of $30 million.

So, that’s about theLotter. So, about a year ago, we decided that we would like to test our pricing strategy, and the motivation for it was, well, duh, increasing our revenue. So, this was the main goal of this test. But additionally, motivation point of views was, as I mentioned, we are the pioneers of the lottery industry online.

So, once we set up the pricing back at 2002, we didn’t really had any benchmarks, so we didn’t know how to compare the pricing. So, we took a margin and we said, “All right.That’s the pricing, should be fine.” So, the pricing had never been tested before as well as along the time when competitors popped up along the time.

So, we came to understand that we are, maybe not the most, but one of the most expensive services of the lottery online. And for me, the most important point of view was the win-win situation, increasing our profits while make our services accessible to more and more customers, especially for the ones who has lower purchase power.

So, once we had the motivation for testing, we began asking ourselves some big questions. And the biggest one and most important was, “Is there a silver bullet price?” So, what is a silver bullet price? So, on our industry, or at least at theLotter, the pricing is a cost-based pricing, cost-based model which we purchased the tickets on a retailer price and then we sell it with a profit margin.

So, the question was, “What will be the price that will increase conversion, increase retention rates, increase the whole lifetime value of our customers as well as fight back with our competitors and the bottom line will set the best revenue line for the company?” So, after asking ourselves these questions, we began investigating the market and ourselves and we set ourselves an objective, not to harm our profit margin.

That was the main objective. Once taking a look around the market, we found out that we are pretty much the only company, which, let’s say, obligates our customers to purchase a minimum amount of lines, three as you can see in the screenshot, against competitors, which give the option of purchasing one line.

For those of you who doesn’t play the lottery every day, it’s pretty easy. You just take a ticket, we call it one line, and you pick your numbers and you send it. The more lines you purchase, the higher the odds, the winning odds. So, by looking at the industry, the market, and our objective, we came to understand that if we make a small business model change or a product change, reduce from minimum amount of lines from three to one, we cut the price by three, reduce it and we can test it.

So, based on that, we decided to see some hypothesis for the test. So, if we said we are cutting the price by three, reducing it, so most probably, the conversion rate will increase.

We thought it will increase tremendously as well as we estimated that the player value will be lower as we cut the price by

But along the time, the total income, that accumulative income will be higher than what we’ve seen until now. The overall lifespan will increase we thought, that’s what will happen, the retention rate and the bottom line of the mixture of everything together will set ourselves a higher revenue.

So, based on this hypothesis we defined ourselves very strict and structured KPIs and very clear ones. Conversion rate, we wanted to see what is the impact on conversion rate, the average player value, the income per player, and the total income from the whole batch of customers.

Additionally, we set our testing principles. So, the first one was timing, avoiding peaks. In the lottery industry, the peaks are mainly coming from jackpots. A year ago, exactly a year ago we had the $1.6 billion jackpot in the U.S.

Powerball. So, that’s a very high peak in the industry. So, we wanted to avoid these times as well as the testing group. We defined, the test group will be a clean, fresh, clean customers, ones were never engaging with us. Regarding the timeframe, we had an estimation of about four to five-months but we actually understood that this will take time and we need to be patient.

And the most critical, in my eyes, was the reporting, the monitoring. So, I set from day one a clear and clean report. That’s the actual report with not the action numbers, from day one for me to understand and monitor and track that behavior. So, once we had the motivation, the background, the hypothesis, and the testing principles, we just needed to implement the test.

So, what we did is actually duplicating one-by-one our website to a different domain. And the only change was as you figured it out, we reduced the minimum amount of lines from three to one. That was the only change in the website.

Additionally, we set the option to allocate or redirect the new customers’traffic from the main site, which is the control site, 20% to the new site, the tested site. And then once we had everything configured, we only needed to start the test.

So, that was a year ago. Whoever you think that the results are like the hypothesis? No one. Okay. That’s interesting. So, it actually was just like the hypothesis or it still is. So, this is an accumulative trend lines of the test and the base site.

The red line is the base site and the greenish line is a tested side. And as you can see, that’s accumulative revenue. So, at the beginning, the revenue of the base site was much higher but along the time it got straightened while the tested site keeps gaining the income.

The conversion rate is higher, we had more and more customers who along the time spent more money. And at that point, about eight months, that’s a break-even point, we can see the crossing and ever since then, the lower price is performing much better.

So, that’s one point of view. We wanted to test it from another point of view, the TROI, the time to return on investment, which is very important to understand how long will it take for us to achieve and return the investment of the acquired customers. Second would be the total revenue, to understand exactly per customer, how much revenue we will get from it, from the change and the player behavior that’s based mostly on cohort analysis, understanding the retention rate, the lifespan, and the total lifetime value.

Next steps, because we didn’t yet finish it. So, get to the point that the results are good or bad, but we can decide, that’s finalizing the test. Second would be analyzing the whole picture. So, as I mentioned, we only tested new customers, and most of the revenue, the running revenue, is coming from veteran customers.

So, we needed to take them all together and see what is the total impact on the business. So, this is something with which we still test now. Third point will be, check implementation options. So, based on that we should understand if we would like to implement only for new customers, both for new and veteran customers or not.

And last point would be making the decisions. So, that was our case study and…

– So, not yet. You probably understood that optimizing your pricing strategy can be one of the most powerful and impactful growth lever that you had in your company. But before you get too excited and pull out your laptops and start running pricing tests, so, just few precautions for you to be aware of.

First, you have to be patient, you have to wait for the long-term to see the real impact of the price change on your customers. In addition, you also have to align expectations. Different stakeholders in the company should be involved in this kind of process, the sales department, the customer support teams and so on.

So, you have to set expectations from start. And the last one, which is probably the most important one, is that these kinds of tests work well if no one knows what you are doing. If you charge different people different prices for the same product, it can have a terrible effect on the customer’s loyalty and also spread quickly in social networks.

So, this is the reason why we recommend to test new customers and also the reason why Tom in theLotter created two different sites with two different domains for the test. But you can overcome these precautions with a proper planning. And I think that the main take away from this session is to stop guessing. You have all the data, you have the tools, so starts testing your pricing effectiveness.

Thank you very much. I hope you enjoyed the session and we will be here after the workshop’s complete. If you have any question, any comment, we’ll be more than happy to answer. Thank you very much.