Why is this Song Popular? (feat. P̶i̶t̶b̶u̶l̶l̶ Spotify)

Have you ever wondered what makes a song popular? Why Despacito topped the charts in 47 countries, and why that indie bluegrass/hip-hop band you like (see: Gangstagrass) didn’t? What if I told you there was a way to find out?

Enter Spotify. You know Spotify, right? It’s that app automatically starts up with your computer for some reason. It’s that app that Taylor Swift got mad at that one time. It’s that app that let’s you listen to any music you want, except for Beyoncé’s Lemonade (we’re still mad about this). Well, it’s more than just those things. If you look into it’s API, it’s actually pretty cool.

You can use Spotify’s API to access a whole number of songs and their characteristics. Some make sense, like loudness or bpm. Others are a bit stranger, like danceability or acousticness. Spotify uses algorithms to determine these factors on various scales. You can also do things like analyze popularity on a scale from 0–100.

So, we had an idea: what if we could use these traits and regression analysis to see how song traits affect a song’s global popularity? We wanted to go further. We wanted to understand how the most popular song in each country (of the 52 countries that have Spotify) matches the global taste.

[Admittedly, we initially wanted to find the most popular features per country, but we realized there was a major error in Spotify’s API. Apart from the United States and the global values, every song has the same popularity in every other country. For example, the song rockstar by Post Malone was rated as having a popularity score of 97 in every country outside the US. It even had a 97 in Uruguay, the fraternal twin of Paraguay, despite the fact that it didn’t appear anywhere in Uraguay’s top 50 playlist. Something doesn’t quite add up here…]

Let’s break this down:

Getting the Data

First, we needed to get our data. We couldn’t pull all of our data from one genre, and we couldn’t just pick the 10,000 most popular songs. This wouldn’t give us information that was representative of trends across many different types of tracks. We needed a way to get a comprehensive list of songs across all genres and levels of popularity.

Spotify’s API allows you to pull songs based on a search for genre, so we found a list online of every genre that Spotify might have. There are 1522 genres. Highlights include: “deep deep tech house,” “cinematic dubstep,” “soda pop,” “fallen angel,” and something called “skiffle.” You can look at all of the genres here. After scraping this data, we checked out the Spotify search API request. By specifying the year parameter as 2017 and varying the genre parameter across the list, we could gather a wide range of tracks from the past year (we decided to limit it to 2017 so the popularity score is relevant).

We thought it would make the most sense to pull 40 tracks from each genre. We would pick the top 20 most popular songs in a genre, skip past 1000 songs using and offset, then pick the next 20 songs. This gives us a range of popularity both across genres and within genres. Theoretically, this strategy would give us 60,880 songs (1522*40 = 60,880); however, some genres didn’t have enough songs to offset by 1000, and some songs didn’t even have 20 songs. So we applied this method, collected what songs we could, and removed duplicates. This turned out to be 30,527 songs.

Creating a Dataframe

We corralled all these wonderful songs into a pandas dataframe, which is basically a matrix with labels. Using the track IDs, we gathered each song’s features with the “get audio features” API request. Each row contains a track, while each column contains the values for “name”, “id”, “popularity”, “acousticness”, “danceability”, “duration_ms”, “energy”, “instrumentalness”, “liveness”, “loudness”, “speechiness”, “tempo”, and “valence.” These are the features we’re looking at.

Cleaning the Data

Not all features are created equal, and not all song features are measured equally. Accousticness, for example, is measured from 1–10, while loudness is measured from -60–0. It’s hard to compare these traits when they’re on such different scales. So, we needed to normalize the data. We normalized all of the data on a scale from 0–1. For the sake of this post, we will present the non-normalized data on scatterplots.

Analyzing the Data

Plotting

Our goal was to determine which features are most significant in determining popularity. To get a general sense for this, we individually plotted each feature against popularity. Check out these lovely scatter plots. Which is your favorite? Tag your friends with the answer!

screenshot-medium.com-2018-05-23-17-36-42.png

Quadratic Regression

Next, we used sklearn’s linear regression model to determine how strongly a feature is correlated with popularity. We decided to make the regression quadratic using sklearn’s polynomial features, since it seems like most features peak somewhere in between its extreme values. To test how strong a linear regression fits the data, the R^2 value is a common metric. We extracted this value for each regression, giving a sense of how significantly tied the feature is to popularity. Here are the resulting R^2 values, converted to percentage and sorted for your convenience:

Loudness: 22.3% …. Instrumentalness: 12.2% …. Acousticness: 10.8%

Energy: 9.7% …. Danceability: 8.8% …. Duration: 6.0%

Speechiness: 4.7% …. Valence: 3.7% …. Liveness: 2.7% …. Tempo: 1.0%

Based on these values, LOUDNESS seems to be the most significant feature in determining popularity. It seems we should all turn it up to 11.

Runner up is instrumentalness, which is Spotify’s prediction for how likely it is for the song to be instrumental. Looking at the graph, it makes sense; the vast majority of songs are full of lyrics and score a whopping 0.0 on the instrumentalness scale, so the feature is heavily negatively correlated.

Some say that bronze medals are the best, since you don’t feel like you just missed gold and you’re just happy to get a medal. In any case, the bronze goes to acousticness. It also gets a gold for least interesting plot, so we’ll move on.

Wait a minute…

Although giving out medals is fun, we can’t get carried away. These R^2 values only indicate trends in specific features, and they don’t give a picture of how the features interrelate. It could be the case that instrumentalness is heavily negatively correlated, but changing its value has little effect on a song’s popularity with other popular features. Below are two graphs of our regressions:

There are several methods to determine relative feature importance, including lasso regression and neural networks. Both of these, however, require time that we didn’t have. To be continued…?

screenshot-medium.com-2018-05-24-10-34-36.png

Analyzing Global Taste

With all this data lying around, we thought it would be cool to create a model that predicts a song’s popularity based on its features. We could plug in each country’s top song, then compare the predicted value to the song’s actual popularity. It seems that countries with a big difference between these values are either hipsters or have bad taste… we’ll leave that up to you.

Creating the Model

We used the sklearn linear regression model as our predictor. This time, we fed in the entire dataframe of 30,000 songs and 10 features, so each feature gets a weight within the regression. Now, using regression.predict(), we can get the expected popularity values of a given set of song features.

Mista Worldwide

Next, we gathered the top track from each country that Spotify is available in (only 52). We plugged these feature values into the model, and got the model’s predicted popularities. The first striking thing about these predictions is that the model is a harsh critic. Rockstar by Post Malone got the highest predicted popularity, at a depressing 33/100. The average across all predicted scores was 29.2, while the average for the actual top songs was 89.6. This makes sense, though, in the context of the data. Check out this histogram of all the songs’ popularities:

screenshot-medium.com-2018-05-24-10-35-56.png

By far the majority of the songs we trained the model with are incredibly unpopular, so it makes sense that the predictions would follow suit. To make these vastly lower numbers for comparable to a top track’s popularity, we adjusted the values to have the same mean (89.6) by adding 60. Finally, we could take the difference between the actual popularity and the predicted popularity, getting a sense of which country is up to no good.

Results

Well, it took awhile, but we got results!

Puttin’ it in a Map

Next, we had to venture into the world of JavaScript. Once we had all the info, we created an interactive map that would allow users to select a country each country and see how the most popular song in each country relates to the overall trend.

Our notebook can be downloaded here.

In Conclusion

Can you predict the popularity of a song? Ahhhhh yes(ish). Our analysis of spotify features and prediction of global taste proves that there is general correlation between a song’s features and its general worldwide popularity. Why is this important? The music industry is a billion dollar market space and the ability to accurately predict the popularity of a certain song would be incredibly helpful in determining profitability before release. When we did this analysis Rockstar by Post Malone was the most popular song worldwide. Our model predicted that this song would generally be popular, but really is this song actually thaaaaaaat good and does it really deserve to be so popular? There’s a high chance that we all just have poor taste in music.

Signing off in the words of Post Malone: “Ayy, shit was legendary”