Kraken (/ˈkrɑːkən/): A legendary colossal sea monster in Scandinavian folklore that attacks ships and sailors, often portrayed as a giant octopus or squid.
You don’t have to be a Northern European mythology expert to know about the Kraken. The fictional creature, thought to have originated in Norse sagas, has been referenced in everything from classics such as Moby Dick to pop culture in Pirates of the Caribbean. It even has its own hockey team in the Seattle Kraken of the NHL. But have you heard about Optimove’s Kraken?
The eight arms beneath your orchestration engine
As any good ocean, Optimove has a Kraken in its depths. Optimove’s home-grown data processing tool carries the legendary name as it has a proverbial arm in everything that allows Optimove to automatically perform complex batch and realtime data processing for hundreds of brands. Specifically, Optimove’s Kraken is in charge of:
Extract/Transform/Load (ETL/ELT) – Obtaining data from any of the brands’ auxiliary systems. The data is validated, transformed, and loaded into a unified internal database.
Snapshots: Taking snapshots of data at various time frames, allowing brands to track their customers’ evolution and efficiently conduct cohort analyses.
Customer Profile: Aggregating customers’ data, in both batch and realtime calculations, to create and maintain an up-to-date ‘Customer 360’ view.
Campaign Orchestration: First, determining customers’ eligibility to receive campaigns, which optimizes the ideal path for each customer. Then, prioritize the campaigns they should receive next and trigger the actual campaign execution.
Realtime Campaigns Eligibility: Determining the eligibility of customers to receive realtime behavior-triggered campaigns.
Analytics: Conducting batch and realtime data aggregations and calculations needed by the various reports and dashboards in Optimove. It also includes leveraging AI algorithms to calculate predictive information such as future-value or risk-of-churn.
Reporting Data: Refreshing brands’ data through Optimove’s BI Studio for up-to-date standard and custom reporting.
Seven complex main tasks, but one arm is missing (cephalopods, such as squids, have eight). That last arm is one that stretches to tap Optimove’s Operations Support Group in the shoulder when something goes wrong. And this past summer, tapped it did.
When the beast comes up for air
2020 has been a challenging year for all of us. Some problems are more severe, and high impact, COVID-19 and the current California fires come to mind, and others are smaller in scale, like Optimove’s Kraken coming up for air (and not to just say “hi”).
The Kraken brought with it some glitches during the early summer. Mainly it began experiencing delays caused by tasks being “stacked” upon each other.
This prompted Optimove’s R&D and Operation Support Group to team up and discover what caused the beast to surface.
Taming the Kraken, for good
After a few months of work around the clock, divide and conquer, implement measured steps while keeping in perpetuum mobile and performing experiments and validations of tens of potential solutions – Optimove’s team rolled out a three-pronged solution that not only solves the problem but also ensures it doesn’t happen again:
Scale – Kraken’s agents (the processors doing the automated work) were doubled in number and organized better for horizontal scale, per type of load. Additionally, CPU availability has been significantly increased to compute more data faster.
Resiliency – each Kraken agent manages its message queues. It allows adding and removing agents gracefully, without a single action being affected by it.
Defense – if an agent begins to get “stacked,” it will automatically reduce additional load, diverting it to other more available agents. Once the agent becomes less heavily loaded, it will increase the level of additional tasks it executes.
Now, with this solution in place, the Kraken went back behind the currents and waves while Optimove hums again. And so, sooner rather than later, the Optimove Kraken will too become a myth. Something some marketers recall hearing about but that they have never seen in their own eyes.
Rachel joined Optimove in April 2019 as VP R&D. Before joining Optimove, Rachel held various positions at Hewlett-Packard, including Director of Engineering, SaaS HPE Software. Rachel was also an R&D Consultant for an Aerospace Solutions and Engineering company in Israel. She holds a B.Sc. in Computer Science, Maths and Physics from Tel Aviv University, and an Innovation and Entrepreneurship certificate from Stanford University.
