K#17. 5 data science learnings from working on a project from A to Z
+ updates on the future of this newsletter
Hey all, I hope you’re all doing well.
I haven’t been around much, but welcome back to K’s DataLadder ⚡️. We’re now 3142 in this newsletter, so thanks a lot for being part of this adventure!
Agenda
Quick update
A year of building music videos at Spotify
5 learnings from leading data science on TV
Quick Update
I've decided to turn this newsletter into a public diary where I track my data science learnings at Spotify.
Writing data science content has been amazing, but it also stressed me out a lot. I’ve been doing all this (YouTube, Medium, LinkedIn, Substack) for a year now as a side project of my full-time job.
I love it but it has taken a toll on my personal life. I’ve had to sacrifice a lot to fulfill the vision of the type of “content creator” I wanted to be, but that vision involves doing all this full-time, which is impossible for me now.
So I’ve decided to put less pressure on myself and become more like someone who shares stuff with the world and less of a content creator. At least for now.
Fun fact: I actually reference my own newsletters during work. Everything I share is useful and hands-on – no bullshit.
If you haven’t done it yet:
A Year of Building Music Videos at Spotify 🎥
I’ve been working for a year now on the launch of Music Videos (MVs) at Spotify and it has been the best project I’ve ever worked on. I’ve led the data science efforts for the experience on TV and helped launch MVs worldwide!
For many years, Spotify had been my dream company – I play the violin and I love music more than anything in the world. So having the chance to work on music-related projects is sort of a dream come true.
More importantly:
Building new products in tech is a rare experience. Most of us spend our time fine-tuning what already exists. So being part of a project where we build something entirely new is a big chance, and I’m grateful for this opportunity.
5 Learnings from Leading Data Science on TV 🚀
I've grown so much this year as a data scientist working on music videos and TV, and to celebrate the culmination of my biggest project, here are my reflections on the whole experience from a data science perspective:
#1. One Feature = Many Mini-Features
It takes months to build a new product from scratch:
Each "simple" feature has multiple layers
We break them down, build them, test each piece
We iterate to make sure we're not breaking the existing experience.
Big learning: One product = Many small features. It’s like Russian nesting dolls.
#2. A/B testing rules everything
Every feature we experience on tech platforms has been A/B tested, with data scientists driving the engine. Here's how it works:
Data scientists collaborate closely with PMs to make sure the tests are well-prepared before they’re good to launch
PMs prepare test specs with all the technical and business details
Data scientists define the experiment type:
A/B tests: Full analysis with success metrics
Monitored rollouts: Making sure we don't break things
PS: If you’re confused, make sure to check out my newsletter on the difference between A/B Tests and Monitored Rollouts:
Big learning: All new features need to be tested, and data scientists always need to be a part of the process.
#3. Troubleshooting is tough
Whenever we test a new feature, we need to closely monitor metrics on day 1, week 1, and week 2. We do that to make sure the experience is working well:
Day 1 metrics: Serve as a proxy to crash metrics when crash data is not available. If they go down, it means something's breaking somewhere.
Week 1 metrics: Needed for urgent decisions. They give enough info on the impact of the feature being tested but there’s a caveat: it also suffers from the novelty effect – users are still discovering the experience. We usually need to wait for the novelty effect to fade before drawing conclusions.
Week 2 metrics: The gold standard. They approximate the true effect of the experience after the novelty effect has faded.
If the day 1/week 1 metrics start to go down, welcome to the rabbit hole of investigating what the hell happened.
Even worse: when crash data lives in inaccessible partner clouds, or we lack the technical capacity to track it, troubleshooting becomes a nightmare. For instance, if crash data is not instrumented, good luck figuring out why/where things are breaking.
Big learning: Make sure proper instrumentation (data collection) exists before launching a test.
Collaborate with engineers to validate that the metrics needed to troubleshoot things if shit goes down exist.
#4. Communication can make or break everything
I’ve always been somewhat of a good communicator but I learned this year that it’s only half of the battle. It takes two people for communication to go well.
In big organizations, crucial info gets lost or misinterpreted all the time. Two real-life examples from the last months:
Someone forgot to share crucial info which dramatically impacted a test halfway. We ended up having to cancel it due to technical oversight after weeks of work.
Someone shared the wrong info and threatened to ruin weeks of work. Thankfully we didn’t cancel the test because we double-checked the info directly at the source, but bad intel almost cost us weeks of work.
Big learning: Double-check critical info at the source. Skip the intermediaries.
#5. Balancing ambition with reality
Tech moves fast and wants it all, but the reality is different because resources are limited. We only have so many users to test on, and each test needs its piece of the pie:
Feature A needs 30% of users
Feature B takes 15%
And so on until we hit 100%
Once you're out of users, that's it. No more tests (cue PM tears). And it's not just about users, we've got limited engineers, data scientists, and time. Things go wrong, delays happen, and work spills over between cycles.
Big learning: As data scientists, we need to learn to push back on projects and say no during the planning phase.
Being ambitious is great, but someone's gotta be the reality check. It's not always fun (cue PM tears again), but hey, that's the job 🤷♀️.
That’s it for today. If you enjoyed this edition, please leave a ❤️ or drop a comment—I’d love to know I’m not just talking to a wall. Feel free to ask questions if something is not clear.
My socials: YouTube, Instagram, LinkedIn & Medium
Take care & see you soon 👋🏼