K#15. A/B testing lessons you’ll thank yourself for learning later

How to deal with A/B/C Tests & Interrelated Features

Sep 29, 2024

Hey there 👋

Welcome back to K's DataLadder ✨! I regularly share a story from my life as a Tech Data Scientist to help you level up in your career (and sometimes in life too).

We’re now 2552 strong! Thank you for being part of this journey 💜

What I’ve been up to?

I had the chance to travel to Stockholm two weeks ago for a big onsite reunion with my department at Spotify, and it was a lot of fun!

After months of distributed work, it felt amazing to finally reconnect with the team in person. We also threw a big Dinosaurs-themed party! 🦕

It was one of those moments that reminded me how lucky I am to be part of such a fun and vibrant company (it’s easy to forget that sometimes).

And work?

The past two weeks have been extremely busy at work since we’re nearing the end of our current work cycle. This means a lot of deliverables, last-minute adjustments, and—of course—more A/B tests.

I’ve been stuck working on dozens of A/B tests for a few months now.

And while it may not be the most fun part of the job, it’s undeniably one of the most vital skills a data scientist needs to develop in tech.

Not everything about the job will be exciting, and I’ve been working on making peace with that. However, I find a lot of comfort in knowing that I’m developing one of the most sought-after skills for data scientists: experimentation.

If you haven’t checked out my intro on A/B testing yet (part 1, part 2), go back and read that first for the basics.

For now, let’s dive into this week’s lesson.

This week’s story

This week, I want to dive deep into one of the most complex A/B tests I’ve worked on recently. I learned a lot of important stuff you need to know about!

Without getting too much into the specifics (you know I need to keep those confidential 😇), I was testing two new features that form the foundations of a new user experience.

And I was faced with a new set of challenges – and crucial learnings!

Challenge #1: Interrelated Features

The two features I was working on were closely linked, and this can be problematic:

Since the two features would likely influence each other once added to the UI, testing them separately didn’t make sense. They would yield different test results than what would actually happen the day they’re launched together.

So here’s what I did instead:

Test 1: To measure the impact of Feature A in isolation.
Test 2: To measure both Feature A and Feature B combined to see how their interaction influenced metrics.

By splitting the test this way, we could accurately assess the individual performance of Feature A, while also understanding how Feature B might enhance or diminish its effect.

🚨 But remember: just because you’re testing the same feature in different setups doesn’t mean you’re testing the same hypothesis or using the same metrics.

We had to think carefully about how launching both features at once might affect the user experience. The key here was setting up hypotheses for both:

Will Feature A increase engagement on its own?
Will Feature B amplify or reduce the impact of Feature A?

And when it comes to metrics, the timeline matters.

If we’re expecting a feature to impact retention, we need to track how active users behave over a few weeks.
If we think it’ll increase consumption, we’d look at minutes played.

👉 Key takeaway: Depending on your assumptions, the hypothesis and the metrics will vary.

I’ll dig deeper into hypothesis formulation and metric setting in future posts—stay tuned for that!

Challenge #2: A/B/C Testing

Here’s where things got even more interesting: we had to decide where to place Feature A in the UI to maximize engagement.

Too high, and it could hurt the user experience.
Too low, and users might not even notice it’s there because they’d have to scroll down too much.

I first checked in the data how users were interacting with the UI where Feature A would be placed. But data alone wasn’t enough to figure out the best spot.

That’s when we decided to run an A/B/C test:

Control Group: Users see the standard UI without the new feature.
Variant A: The feature is placed in position x.
Variant B: The feature is placed in position y.

The goal was to figure out not just if the feature itself drives engagement, but where it should be placed to be most effective.

Why this matters

Running an A/B or A/B/C test can get complex, especially when you’re dealing with multiple variables like feature positioning or interactions between different elements.

So here are a few key takeaways that might help you in similar situations:

Split tests strategically: If features are closely linked, test them both individually and together to understand their interaction.
Tailor your hypotheses: Testing the same feature twice doesn’t mean you’re testing the same thing. Adjust your hypothesis and metrics based on what you want to learn.
Test placement, not just the feature: Sometimes where a feature lives on the screen is just as important as the feature itself. Use ABC tests to evaluate different positions.
Rely on more than just data: Data is critical, but it won’t always give you all the answers. Sometimes you need to combine data with design intuition to make decisions.

You might not be working with A/B tests right now, but trust me—if you end up in a tech company, it’ll become part of your reality. And when that day comes, you’ll be glad you already know a thing or two about them.

That’s it for today! I hope you found this helpful. If you did, please leave a ❤️ or drop a comment—I’d love to know I’m not just talking to a wall 🤓