Unwrapping Wrapped 2019: Spotify VP of Engineering Tyson Singer Explains
As December 2019 drew near, people around the world were feeling excited—sure, for the holidays and the New Year—but also for Wrapped. For the past couple of years, Spotify has launched the elaborate end-of-year campaign showcasing the top songs, artists, albums, and, more recently, podcasts that users streamed the most throughout the year.
Last year’s Wrapped was no different, and with the dawn of a new decade, we also took a look back at the last 10 years of streaming.
But how was it all done? How were we able to deliver a seamless personalized experience for more than 200 million users across the globe around a decade’s worth of streaming data?
Well, we’re glad you asked.
Here to give you some behind-the-scenes insight about the infrastructure that enabled the beloved Wrapped experience for 2019 is Tyson Singer, VP of Engineering at Spotify. He talked to us about some of the technical nitty-gritty that allowed us to create a decade of Wrapped for hundreds of millions of users without skipping a beat.
Aside from the challenges of creating a personalized Wrapped experience for millions of users, what are the biggest foundational challenges in putting together something like this at the scale in which Spotify operates?
First and foremost, Wrapped requires a lot of data. More specifically, it involves an ability to process and manage it. The most challenging aspect when it comes to that much data is to determine how to best collect, process, optimize, and iterate on the massive quantities of data in a compliant and efficient way.
To give you a sense of the complexity and scale of what we’re talking about, this year the power required to deliver Wrapped to all of our users clocked in at 844 years of processing time, which in itself is rather extraordinary! And in 2019, we processed five times the amount of data compared to 2018.
So, how did you scale and optimize the data pull this year while also increasing the amount of data?
Tons more testing and QA work was done this year. So, to prepare, the Wrapped team ran several iterations of smaller data sets using employee data. We also worked more closely with the GCP (Google Cloud Platform) team to leverage their expertise. This caught issues and bugs earlier, which helped us avoid rerunning the larger, more heavy jobs. But this also meant that there was more time needed to set up the infrastructure and iterate, making this one of the most sophisticated data processing pipelines we’ve ever written.
How did Wrapped 2019 use previous years’ learnings to improve upon the process?
2018’s Wrapped Campaign had our largest Dataflow job ever run on GCP to date, which presented us with tons of learnings and opportunities for optimization. So this year, with five times the data we were processing and serving, we knew to break the large Dataflow jobs into smaller pieces. This allowed us to complete them faster than last year even though we had more data—we had a better system design. With Wrapped 2019, we ran the largest set of Cloud Dataflow jobs for a single campaign.
Why does a project like Wrapped excite you and your team?
Wrapped is a great opportunity to utilize our incredible data and present it back to fans all around the world. Without our engineering teams and sophisticated infrastructure, we would not be able to understand our users as deeply or serve them the personally tailored content they love. Wrapped is a way for us to show off our work in a way that everyone can enjoy.
What are you looking forward to in 2020?
From an infrastructure perspective, I’m especially looking forward to seeing how our systems stand the test of time and scale. We are growing every single day from a listener, catalogue, and data perspective. This presents challenging yet exciting opportunities for our engineering teams to scale.
As a service, users take for granted the complexity it takes to deliver seamless playback and meaningful data-driven experiences when they listen to Spotify—as they should! Part of my job at Spotify involves ensuring that on the back end, we have the best systems and technology in place to deliver against this expectation.
Did you know you can access your Wrapped playlists year-round? Pro tip: Head to the “Made for you” section of your Spotify to relive the best of 2019. And want to learn even more? Check out our tech blog.
