Fostering a company culture of employee empowerment and experimentation breeds evidence-based customer-centric product development. Discover lessons learned from Booking.com's Lukas Vermeer from his captivating talk at Digital Elite Camp 2018.
RockBoost on tour
I was honoured to be part of the RockBoost team who attended Digital Elite Camp at the stunning beach resort of Laulasmaa on the outskirts of Tallinn, Estonia in June 2018.
But before I get to the juicy bits... (read to the end to find out!) a serious tip-of-the-hat must go to Peep Laja and his hard-working team from CXL for organising a top-notch event!
From the high-quality speakers to the clockwork-like logistics; the idyllic venue to the sweltering-hot saunas that remedied my hangover… twice!; it was a truly fantastic experience which I strongly recommend to everyone in digital.
You would be mad to miss the 2019 edition!
If only they could’ve done something about those ravenous, pigeon-sized mosquitoes that gnawed their way through my poor little arms and legs, it would’ve been a 10/10!
But alas… a 9.5 will have to do.
You can’t have everything, I suppose.
There’s always room for optimisation. RockBoost on tour
Each RockBoost team member was tasked with creating a blog post on the topic of their choice. It was decided to first digest the conference content, then subsequently choose a topic or theme we wished to review.
In all honesty, halfway through the conference, I was still desperately struggling to nail down my topic. Although, this may have been due to my frenetic live-tweeting of speaker insights (sneak a peek at our Twitter feed here, if you don’t believe me!).
That is until I was utterly blown away by one particular presentation: “Democratizing Online Controlled Experiments at Booking.com” by Lukas Vermeer.
You would be forgiven for thinking the title would be better suited to an academic paper, rather than a conference talk. Actually, as it happens, you would be spot on. The talk was indeed inspired by a paper written alongside his esteemed colleagues at Booking.com, and if you’re that way inclined, you can read it in full here.
Perhaps, having an academic background in behavioural psychology myself is why I felt so drawn to this presentation over the rest.
Or… maybe, it’s just because I’m fond of hotel rooms!
||No time to read this now? Get the PDF sent to your inbox!
SEND IT TO ME!
Critical importance of culture
Reviewing the paper itself is far beyond the scope of this blog post, but luckily for you, it was also beyond the scope of the talk. So, to the delight of the audience, Lukas succinctly announced he would prioritise and focus!
Hence, his entire presentation was based on a single line in the abstract:
“Our methods and infrastructure were designed from their inception to reflect Booking.com culture, that is, with democratizing and decentralization of experimentation and decision-making in mind”
Sounds fancy, Lukas! But what does this even mean?
In essence, it’s about fostering a company culture of empowering employees to make their own evidence-based decisions.
But what does this company culture actually look like? And how can we even see or perceive this company culture?
It turns out Lukas didn’t have answers to these questions either! That is before methodical research was carried out to investigate. Methodology (Source: Lukas Vermeer @ Digital Elite Camp 2018)
The resulting employee responses led to a beautifully simple and empathetic definition of company culture:
“It’s all about the people we work with. Culture is what happens when two people interact”
Booking.com’s mission is to “Empower people to experience the world”, and they’re renowned in the industry for running thousands of experiments, in fact, I’m reliably informed everything is wrapped in a test!
Interestingly, this aligns with many employee responses to the question, “How do you use experiments in your job?”
Employees said they feel:
- Empowered to make decisions independently
- They have permission to try out their own ideas
- They have the creative freedom to validate their hypotheses
- Empowered as they are not bound by higher value opinions
- Value no centralized hub of decision-making
What does decentralizing decision-making look like?
The irony is it’s the centralization of the experimentation system that allows for the decentralized decision-making which leads to employee empowerment.
Let me break that down.
Every employee conducting experiments has access to a searchable repository of all experiment data from every test ever run; both successes and failures. Everybody can see what data was gathered and the methodology used. Everybody trusts the data is accurate, and everybody can view what subsequent product decisions were made.
Like what you're reading? Get regular growth hacking tips sent straight to your inbox!
Sign Me Up!
Because everybody running experiments is working from a central repository of data and previous experiment literature; employees can make subsequent product decisions independently, without the need for permission from higher management.
Effectively, everybody is ‘singing from the same hymn sheet’ and because there’s company-wide trust in what’s written on that ‘hymn sheet’; employees have the license to express their creativity through experimentation of their own product development ideas.
Lukas sums this up wonderfully with a simple quote, “You cannot have a democracy without empowering the people”.
Every company strives to be data-driven and to continually experiment their way to success. But how many are actually empowering their employees by providing the necessary environment to facilitate this?
Can data and experimentation be relied upon alone?
Before we go any further, perhaps it should be clarified what data actually is. An anecdote is data; a bunch of bits is data; customer feedback is data; any information you have gathered, quantitative or qualitative, is data; your experiment results is data, and even your quirky cat memes is data!
But as Lukas explains, “Data is just data.”
Data is just enough to support an idea, but to make product decisions, what we really need... is evidence.
Data vs. evidence (Source: Lukas Vermeer @ Digital Elite Camp 2018)
The experiment itself doesn’t help with formulating the idea; it’s only the vehicle that helps gather the data. But without sufficient context for that data, it can be twisted any which way that suits a particular narrative, or worse, satisfies an ego.
What’s wrong with most A/B test experiments?
It would seem there’s misinformation rife in the industry about how A/B testing should be approached. Here’s how many seem to tackle them, as Lukas candidly puts it:
“Let’s change something on the website and see if the magic number goes up?!”
This is based on the premise that we don’t know whether A or B variant is better, so in effect, we’re giving our best guess; randomly changing things to see what works, and then checking if the target metric increases!
There’s a critical element missing here that can give our data the context it so desperately desires, and that is… a hypothesis.
So, what is a hypothesis?
The Oxford Dictionary reliably informs us:
“A supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation.”
The key words here, for me, are “starting point for further investigation”. We need a reference point to give context to our data analysis, otherwise, we will be subject to our own internal biases and driven by our ego to celebrate “a winning test”.
This leads us to what Lukas advocates, instead of website optimisation, our industry should be conducting evidence-based customer-centric product development; achieved via user research and hypothesis testing.
In short, devise a theory on why you believe a certain change will have the desired impact. Next, conceive a way to validate that the change you have implemented has the predicted impact, and finish with an overall objective of what you’re trying to achieve. Hypothesis testing template (Source: Lukas Vermeer @ Digital Elite Camp 2018)
Explicitly defining in advance what the test is attempting to accomplish is crucial, as it allows for rational thought about why a certain implementation may work, and another might not.
This has got nothing to do the with the data, but everything to do with the methodology used, or as Lukas says, “Describing ideas in clear terms.”
But how do you begin to formulate a hypothesis?
First, talk to your customers to identify their problems and pain points! Don’t start with a test! Armed with this information, you can formulate your hypothesis with the goal of solving the user-defined problems.
Challenging core assumptions and the beauty of failure
Interestingly, one employee respondent turned the question of “How do you use experiments in your job?” on its head. Rather than approaching experimentation from the lens of validating ideas; he views it as invalidating ideas that were thought to be correct, or challenging core assumptions.
This brings us neatly into the final stages of Lukas’ presentation where he regaled us with a story of his own personal failure as a young and spritely product owner at Booking.com.
He was responsible for developing an algorithm for ranking and recommendations of hotels on the homepage. Essentially, which hotels to show for each destination and what order to rank them in. Ranking and recommendations (Source: Lukas Vermeer @ Digital Elite Camp 2018)
After testing a bunch of ideas like context, booking history, device type, country of residence with artificial intelligence and machine learning techniques; there was only moderate success.
Six months in, inspired by an academic paper, he tried a new ranking technique which required the collection of hover and click data of the hotels displayed.
Previously, this had not been measured, as the success metric was whether the user ends up booking a hotel.
So what were the results?
“Guess what? No one clicks on the damn hotels!”
This challenged Lukas’ core assumptions of how the feature works:
- People see the hotels
- People like the hotels
- People click on the hotels
- People book the hotels
It had never been confirmed whether or not this was the case, it was only an assumption, and you know what they say about assumptions?
When you assume, you make an ASS out of U and ME!
To further investigate, instead of showing the recommendations, they completely hid them; the algorithm was still computing the rankings and recommendations in the background, but they just weren’t displayed.
So what happened?
“Guess what? Nothing happened. No one cared!”
The feature that had been worked on for 6 months, whether displayed or not, had no effect whatsoever on people actually booking hotels!
So Lukas dropped it and moved on to another project.
But then, a backend developer suggested if the hotels weren’t going to be shown, why should the algorithm still compute the ranking in the backend?
So they removed it, and because everything is wrapped in a test, they found that by not having a complicated machine learning algorithm running in the background made the product faster!
It was a landing page.
They sold more hotel rooms!!!!!
So, what can be learned from this?
If nobody cares if a feature is removed, how can incremental changes ever improve it?
The key message here is whenever we are trying to improve a product, the first step should be to challenge your underlying core assumptions. Stack-ranking assumptions (Source: Lukas Vermeer @ Digital Elite Camp 2018)
Lukas concluded his presentation by telling us that if companies truly want to be customer-centric and create products that will help their customers, they must think about what they actually need.
To do this, assumptions should be stack-ranked of how customers interact with the product, and then challenged based on which are most risky for product development.
“Start by taking the biggest small step, so you can challenge your riskiest assumptions quickly.”
Only by following this methodology, can companies conduct evidence-based customer-centric product development.
As an aside, by far my favourite moment of the entire conference (other than the sweltering-hot sauna, of course!) was Lukas’ final and almost throw-away comment when he had already finished his presentation:
“By the way, A/B testing is not new. It’s been around for over a hundred years. We’ve just given it a new name. What it actually is, is the scientific method.”
Bosh! Boom! Bang!
So the next time you’re thinking about running an A/B test, start by challenging your core assumptions; identify the problem; formulate your hypothesis; design your experiment; implement the changes; and finally, analyse the data for evidence to support your hypothesis.
Using this methodology, you can empirically establish whether or not the changes you have made truly have the desired impact you initially proposed.
Otherwise, you will continue to be guessing; falling foul of confirmation bias, and being crippled by your own ego.
This negates all ego.
This is methodical.
This is evidence-based.
This is science.