A Data Scientist Breaks Down All 10 Taylor Swift Albums

Taylor Swift’s music has pretty much been the soundtrack of my life. When she was waiting for her Romeo in Love Story, so was I. When she was ignoring the haters in Shake It Off, so was I. And when she was lying awake in the middle of the night in Anti-Hero, so was I.

As a data scientist, I’ve been wanting to analyze Taylor Swift’s songs for years. She’s known for her complex lyrics and Easter eggs sprinkled throughout her music and messages. Would I be able to uncover something new using data analysis? (Spoiler alert: yes!)

With her Eras Tour kicking off this month, I thought it’d be the perfect time to dig into her 10 studio albums in the medium that I know best — data visualizations.

1. Why The Eras Tour is in Such High Demand

Taylor Swift made national news when the demand for her upcoming tour overwhelmed Ticketmaster. We all know she’s a famous musician, but what made The Eras Tour especially popular? (Swipe through images)

Due to the pandemic, it’s been 5 years since Taylor last toured and she’s released 4 studio albums in the meantime. Plus, The Eras Tour covers all 10 studio albums – it’s basically a best-of tour!

2. Taylor Swift’s Sound Evolution Over Time

Taylor started out as a country singer, she transitioned to pop, and her sound keeps on changing with each new album. I wanted to see if I could quantify the evolution, so I used the Spotify API look at the sound characteristics of her music. (Swipe through images)

You can see that her earlier albums all cluster together, meaning that her sound was very similar on those albums. As she grows up, she’s become more experimental, with each album introducing new sounds and moods. I can’t wait to see where she goes next!

The technical details for those of you who are interested – I looked at the acousticness (instrumental vs electronic) and valence (positive sound vs negative sound) of all of her songs and averaged them for each album.

3. All the Colors in Taylor Swift’s Songs

Taylor is known for having extremely descriptive lyrics where she references specific people and dates, what she was wearing, how she was feeling and also lots of colors.

I loved this visualization capturing the timeline of wardrobe colors in Breaking Bad and wanted to create my own for TSwift. (Swipe through images)

The most interesting pattern that I found was that her earlier six albums mostly stuck to basic colors like red, gold, blue and black and her later albums, starting with Lover, started to explore more complex colors.

She goes from the song Red on her 4th album to the song Maroon on her 10th album, where her perception of love changes from “burning red” to a more mature “maroon”.

4. All the Swear Words in Taylor Swift’s Songs

When I was listening to Taylor’s latest album Midnights, one thing that stood out to me was the amount of swearing. That’s when I realized that she didn’t swear that much in her earlier albums.

The word damn was used in just one song in her debut album, Taylor Swift. No swear words appeared in her next two albums, Fearless and Speak Now. Compare that to Midnights, where half the songs are explicit.

I also wanted to call out the word sexy. While not a swear word, the only time she’s ever used the term sexy in all 10 albums is in the song Anti-Hero — “Sometimes, I feel like everybody is a sexy baby and I’m a monster on a hill”. I find it pretty impressive that she didn’t use the term at all during her first two decades as a songwriter!

5. The Last Line of Every Taylor Swift Album

I love the last song on every Taylor Swift album. The final song seems to capture her mental state upon completing the album and I feel resolved and content after listening to it.

When I looked at the last line of each of her albums, I was able to see her thought progression over time, from her teen years of hope, to her twenties learning to overcome hard times, to her thirties becoming more confident in herself.

The way the final lines capture her mindset over time is so fascinating and it’s one of those things where I can’t tell if it was done on purpose, or if it just sort of happened.

But I should know by now that with Taylor, everything is intentional. Like she said so herself, she is a mastermind.

More Taylor Swift Data Visualizations

These are just 5 of the 22 Taylor Swift data visualizations I created for this project.

For the Swifties:

  • To see all 22 data visualizations about her lyrics, collaborators and more, check out my Extended Version blog post.

Want to watch a video?

  • I explain all 22 Taylor Swift data visualizations in my YouTube video.

A Data Scientist Breaks Down All 10 Taylor Swift Albums (The Extended Version)

I don’t know about you, but I’m feeling 22 Taylor Swift data visualizations.

This is the Extended Version post. If you want to read just the highlights:

If you want to watch me talk about this post:


Table of Contents

Popularity

1. Billboard Hits
2. Demand for The Eras Tour
3. Audience Growth

Sound

4. Sound Evolution
5. Keys of Songs

Lyrics

6. Seasons
7. Days of the Week
8. Complex Words

Lyrics Over Time

9. Swear Words
10. Never vs Always
11. Girl vs Woman
12. Colors

Collaborators

13. Tour Openers
14. Co-Writers
15. Top Hits with Main Co-Writers

Midnights

16. Up and Down Theme
17. Song Connections

Mastermind

18. Album Titles From Song Lyrics
19. Song Lyrics From Album Titles
20. 3 Pen Types
21. Revenge Songs
22. Last Line of Every Album

How Did I Do This Analysis?


1. Billboard Hits

The majority of the songs on her albums make it to the Billboard Top 100.

2. Demand for The Eras Tour

Fans have been waiting years for The Eras Tour and can’t wait to hear all her greatest hits.

3. Audience Growth

She keeps expanding her fan base. Somehow both me and my niece who’s 20 years younger than me are both Swifties!

4. Sound Evolution

Her sound keeps evolving and she’s getting more experimental over time.

5. Keys of Songs

She writes a song in almost every key on folklore!

My guess is she’s transitioned from writing songs mostly on the guitar to the piano.

Even though she writes a lot of sad songs, most of her songs are in major keys.

6. Seasons

Not a lot is going on for Taylor in the spring.

7. Days of the Week

She loves singing about Tuesday nights.

8. Complex Words

She hides complex words inside pop songs.

9. Swear Words

She’s been slowly adding swear words to her vocabulary over the years.

10. Never vs Always

As she gets older, she uses the term never less and always more.

11. Girl vs Woman

Her lyrics have transitioned from talking about girls to women, and boys to men.

12. Colors

The colors in her songs have gotten more complex over time.

13. Tour Openers

Ed Sheeran and Shawn Mendes both made it big after touring with Taylor.

14. Co-Writers

Taylor wrote all of Speak Now on her own, and had the most number of co-writers on reputation.

15. Top Hits with Main Co-Writers

She writes her best sad songs with Liz Rose, her angsty songs with Jack Antonoff and love songs on her own.

16. Up and Down Theme

Every song on Midnights references something up or down, just like the time midnight, where both hands of the clock are pointing up.

17. Song Connections

The words in one song lead to the next song in Midnights, like “haze” in the first song of the album to “hazy” in the second song of the album.

18. Album Titles From Song Lyrics

Starting with 1989, her album titles have come from prior song lyrics (except for folklore, which was a surprise album).

19. Song Lyrics From Albums Titles

She references her album titles in later songs, like when she talks about being a “fearless leader” in the song, The Man (Lover).

20. 3 Pen Types

She imagines what type of pen she is holding (fountain, quill or glitter gel) as she writes her songs.

21. Revenge Songs

She says she writes in the style of 3 pen types (see previous visualization), but I think there may be a 4th pen type for her revenge songs.

22. Last Line of Every Album

The last line of every album shows how she’s grown over time to becoming more confident in herself.

She can finally admit that she is a mastermind. I hope you enjoyed these 22 Taylor Swift data visualizations!


How Did I Do This Analysis?

These data visualizations may look simple, but they were the product of a lot of technical work behind the scenes!

I want to thank the following people for their technical resources:

I did most of my data analysis in Excel, my data extraction and text analysis in Python and my data visualizations in Keynote.

You can find spreadsheets containing Taylor Swift’s song lyrics and metadata, along with the Python code I used to get the data from the Genius API and Spotify API on Github.

How Text Messages Change After Having A Baby

A few years ago, I wrote a blog post called “How Text Messages Change From Dating to Marriage”, where I compared my text messages with my husband before and after we got married. Several people suggested that I write a follow up post after we had kids. Well, we had our first baby.

Kerri Cohen Photography

We can’t believe it, but Henry turns one this weekend. This past year has been an eventful one, filled with many ups and downs, moments of overwhelming joy and sleepless nights that never end. But two things are certain: (1) our lives have been forever changed and (2) we’ve generated a ton of data to analyze, including our text messages. Here’s how our texts have changed from dating to marriage to parenthood.

My husband won the award for consistency, with his messages evolving from “home soon” to “home now” to “coming home”. For me though, my most frequent word became “just”. At first, I was surprised to see this, but then it all made sense when I looked at a few of my text messages.

Many people predicted that I would see words like “poop” or “milk” pop into my vocabulary, but those actually only appeared in about a dozen text messages. I realized that the biggest change for me and my husband after we became parents was that we started communicating more.

While I was on maternity leave and my husband had returned to work, the thing that we missed most was spending time together as a family. We used text messages to keep everyone connected and on the same page. I constantly gave my husband updates during the day about cute things that our son was doing for the first time. My husband would text me to let me know that he was leaving the office and excited to be coming home. Both of us now text each other 4 times as much as we did when we first started dating.

Not only are we communicating more over text, we’ve also been collecting and sharing a ton of data about the baby. Those of you with kids know that it’s often helpful to note the time of a baby’s last feeding / nap / diaper change to get an idea of why they’re uncomfortable and what they need next. We tracked that data for our son’s first six months.

The visualization below shows our son’s feeding times during my last month on maternity leave, when his schedule started getting a bit more consistent. During that time, he was eating 6-9 times per day, totaling 1.5 hours each day. I was never away from him for more than a few hours. On weekends, my husband would take the night shift, and then jump back into the work week on Mondays.

Parenthood is hard work. The hours are grueling, and I’m constantly trying to figure out how to best split my waking hours caring for my family, doing a good job at work, and squeezing in some time for friends, my husband and myself.

Now that I’m a parent, I have an incredible appreciation for all the parents before me. I also have a renewed appreciation for my husband, who’s going through the exact same struggles, while being so supportive through it all. So why doesn’t that show in our text messages? Taking a second look, I found that 1 in 12 of our texts is actually an image. Here are some examples:

It turns out a lot of the more interesting exchanges were done through photos, screenshots, bitmojis and gifs. When you’re short on time, why not send a thousand words with just a few clicks? After texting “coming home”, with one additional picture, you can say that you’re excited to do so, and let’s also have a dance party after having breakfast for dinner.

While my husband and I were dating, we used text messages to flirt. Once we got married, we spent a lot of time together and text messages were mainly used to coordinate logistics. As parents, we now use text messages to keep each other up to date, but also to brighten each others’ days and support each other in this phase of life where we’re limited on time, but have more love than ever before.

To my husband: Happy nine years since our first text message. Hope you enjoyed your second installment of #thegiftofdata.

To my little one: Happy first birthday. Your dad and I can’t wait to continue celebrating all of your firsts and showering you with love.

The Data Behind ‘The Bachelor’ & ‘The Bachelorette’

The year is 2002. I’m sitting in front of a TV along with a group of girls excited to watch a new reality TV show that’s come on air. There are twenty-some beautiful women all trying to win the love of a successful Harvard grad (who happened to be both homecoming king and valedictorian of his high school as well). Over the course of 7 weeks, he dates the women, gives roses to the ones he’d like to keep and in the end, chooses the woman he ultimately wants to marry — aka the winner of The Bachelor.

Thirteen years later, the show is still steadily bringing in an average of 8 million viewers per episode (myself being one of them!). And so, I dedicate this post to my favorite Monday night guilty pleasure, The Bachelor.

One of the reasons I keep coming back to the show every season is for the hope of a happy ending. The most famous success story is of Trista and Ryan who have been together for over ten years and have two kids together. I also love Jason and Molly who didn’t end up together at the end of their season, but got married later on. So just how many of the 28 seasons have ended in success?

success_funnel

It turns out not that many do. In fact, only two-thirds of the seasons end with proposals. Then out of those proposals, only 5 have led to marriage, with the Bachelorette having a better success rate at 30% versus the Bachelor at 11%. The first successful season occurred in 2003 (Trista & Ryan), the second one happened six years later in 2009 (Jason & Molly) and there have been quite a few more happy endings in recent years as well.

While I watch the show in part for the love stories, the other part is for the contestants (and all of their drama), of course! I decided to take a closer look at all of the contestants who go on the show, starting with what they do for a living. I looked at the professions of the contestants in the final four, since I figured these were the more desirable occupations as they had made it so far into the show.

occupations

I saw that the most desirable females tend to be models, assistants, students and teachers, while the top men are professional athletes or in finance. There were a few jobs that were shared by both men and women such as realtors and sales reps. Out of the 200+ male contestants I looked at, I found that none of them had “assistant” in their title and only 2 were students (both eliminated in episode 1). Out of the 300+ female contestants, there were no engineers and they often had job titles like “dog lover” or “free spirit”. [Note to self: Change job title from “data scientist” to “data diva”.]

It’s unclear the exact reason for this disparity. Perhaps there are few male assistants and female engineers that actually want to be on the show. Perhaps people in these professions aren’t dramatic enough to be on the show. Or perhaps they’ve all found the love of their life already. 😉 Whatever the case, please consider them for future seasons, ABC!

So my next question was, how old are these contestants, and which ones get picked?

age_females

I found that the average age of the bachelor is 31. What was surprising was that the average age of all of the female contestants that he gets to choose from is actually 5 years younger at 26. Given that age gap, the bachelor still tends to choose a women who is a year younger than the average age of the group. Note that no woman over 32 has ever made the final four, and while a few 30+ women have made it to that stage, the oldest winner was 28 years old and the youngest was 22.

While the average bachelor chooses a winner who is 6 years younger than himself, the average bachelorette tends to choose someone who is 1 year older than herself.

age_males

The average age of the bachelorette is 27 and the male contestants that she gets to choose from are 2 years older than her on average at 29. It’s worth noting that the two oldest contestants who were 38 and 41 both got eliminated during the first rose ceremony. What’s interesting is that again, the winner is a year younger than the average age of the group. It seems that when given the choice, both the bachelor and bachelorette tend to choose someone younger than the average of the contestants presented to them. [Pro Tip: Try to stand next to people older than you.]

Finally, I looked into the physical attributes of the women and men on the show and the most interesting tidbit I discovered was regarding height. I found that the Bachelorette is 5’5″ on average, which is just above the national female average of 5’4″, while the Bachelor is 6’1″ on average, which is a whole 3 inches above the national male average of 5’10”. [Pro Tip: To be successful in life, be tall or wear heels.]

I will end this post by sharing with you some of the youngest (and cutest) Bachelor contestants on Jimmy Kimmel’s The Baby Bachelor. Hopefully it will keep you busy as we all wait for next week’s Bachelor episode!

The data for this analysis was taken from Wikipedia (which had information on two-thirds of the seasons), Hollywood Life and Spotted Ratings.

EDIT (8/31/22): I get multiple requests for this Bachelor data set each month, so I decided to post it online. You can find the spreadsheet and csv files that I used to do the analysis here: https://github.com/adashofdata/bachelor_data

How Christmas Songs Have Evolved Over Time

‘Tis the season of giving and eating and of course… Christmas songs! When I stumbled upon the ASCAP’s list of the Top 30 Holidays Songs of the Century, I knew I had found my data set for this month’s blog post. It’s pretty awesome when blasting Christmas music = doing research for a blog. At one point my husband and I started speaking to each other in song lyrics, so I had to tone it down a bit on the Christmas music even though it’s the most wonderful [thing to listen to] of the year.

I decided to start by looking at when the top Christmas songs were written. It turns out that nearly two-thirds of them first came out in the 1940’s and 1950’s. When artists today release Christmas albums, it’s pretty much expected that they’ll include covers of songs from this time period.

It made me wonder, what’s the reason that classic Christmas songs are so much more memorable than newer Christmas songs? I decided to group the songs by theme and see if any patterns emerged.

I found that certain themes such as describing “Christmas in the Air” and wishing people “Merry Christmas” via song are timeless, while others seemed to evolve. For example, songs about “Being Together in the Cold” and being “Home” for the holidays were written up until the mid-1950’s when they switched over to songs about having fun and “Partying” at Christmastime.

One of my favorite trends was the transition from songs about Santa (“Santa Claus is Coming to Town” & “Here Comes Santa Claus”) in the 1930’s and 1940’s to songs about Santa being in love (“I Saw Mommy Kissing Santa Claus” & “Santa Baby”) in the 1950’s to songs straight up about love in the 1970’s and beyond (“Last Christmas” & “All I Want For Christmas Is You”).

When you look at the use of the word “Christmas” in songs over the past few decades, you see that every one of the top songs since 1963 has the word “Christmas” in it. It’s almost as if the word “Christmas” is put in a song just for the sake of telling you that it’s a Christmas song.

Take “Last Christmas” for example:

Last Christmas, I gave you my heart
But the very next day, you gave it away
This year, to save me from tears
I’ll give it to someone special

If you remove the word “Christmas” from the lyrics, the song actually has nothing to do with Christmas at all. It is simply a love song.

This made me wonder if Christmas songs have had less substance over time as well. After doing some digging, I found that there is no significant correlation between the year a song is written and the complexity of the song. It’s still fun to look at the vocabulary size of songs though, regardless of year. Here I’ve listed the top and bottom 5 songs in terms of the number of unique words.

“All I Want For Christmas Is You” has the most words and uses the most Christmas references of them all, which I guess is why the song feels extra Christmas-y (along with the choir and bells in the background). I also found it pretty humorous that “Feliz Navidad” has the smallest vocabulary of any Christmas song. It uses only 24 words over the course of 3:02 minutes.

Overall, we see that Christmas songs have evolved over time, from songs about fictional characters and being home for the holidays to ones about celebrating with friends and hoping to get your love interest for Christmas. While some might be concerned with the changes, I actually think it’s pretty impressive that the classic songs have prevailed. It shows that they are beautifully written and capture the spirit of the season better than any songs can do today. When else can you get kids from 1 to 92 all singing along to the same tunes? That’s the magic of Christmas songs.

Happy holidays, everyone!

Today’s Top Female Pop Artists (In Charts)

A couple weeks ago, I read an article on Billboard about how female artists have been ruling the charts. I took a moment to think about this phenomenon (as I paused Meghan Trainor’s “All About That Bass” on my phone) and realized that I really had been listening to a lot of female artists lately. It seemed like ages ago when I was telling the DJ at my wedding that she absolutely HAD to play “All of Me” by John Legend and “Happy” by Pharrell.

So I decided if female artists are ruling the Billboard charts, I might as well create some of my own charts about them too. I used Billboard as my inspiration and looked at female artists with #1 singles on the U.S. Billboard Hot 100 list within the past six years.

The first thing I wanted to do was figure out a way to map these 15 women onto one chart. I decided to look at the number of Hot 100 songs and #1 singles that they’ve had during their careers.

B Female Artists

I found a couple clusters of artists on the chart. On the bottom left you have the ladies that are new to the scene, such as Iggy Azalea, Lorde, Carly Rae Jepsen and Meghan Trainor. In the middle you have the artists that are well-established, having three or four #1 singles and around twenty songs reach the Hot 100 list. These are women such as Kelly Clarkson, Pink, Ke$ha and Lady Gaga.

B Female Artists Labels

Out of all these female singers, there are two that really stand out from the crowd – Taylor Swift and Rihanna. Since Taylor Swift’s debut single “Tim McGraw” made it onto the Hot 100 list back in 2006, she has had a whopping 67 Hot 100 songs. During Rihanna’s nine year career, she has had 13 #1 singles, tying her with Michael Jackson.

Let’s take a closer look at Taylor Swift. How is it possible that she’s had so many Hot 100 songs?

B Taylor Swift

It turns out that two-thirds of the songs on her five albums have made the Hot 100 list, with 100% of the songs on her 2010 “Speak Now” album making it onto the charts.

How about Rihanna? Just how significant are her 13 #1 singles?

B Rihanna

While it doesn’t look like much spread over nine years, if you total up all of her weeks at number one, she has spent almost a full year as the top artist in the U.S.

After doing all of this digging, I thought to myself, I can’t believe Taylor and Rihanna are so successful – they seem so young! So, I took a look at the age of all 15 women when their first song reached #1 on the Billboard charts, and I confirmed my theory – they are so young.

B Age

To my surprise, Carly Rae Jepsen was the oldest at 26! Most of today’s female pop stars make it to the top of the charts around 20 – 23 years old. After this analysis, I’ve learned that my chances of making it to the top of the Billboard charts are pretty slim, given my age (…and okay, my singing abilities too). It’s okay though, I’m just gonna shake it off (I shake it off, I, I, I shake it off). And then make some charts of my own.


Photo Credits (images were cropped and used in graphics)
CC BY 2.0: Taylor Swift (photo by Jana Zills on Flickr), Rihanna (photo by liammendez on Flickr), Britney Spears (photo by rhysadams on Flickr), Miley Cyrus (photo by MelissaRose14 on Flickr), Pink (photo by blumonkey14 on Flickr), Katy Perry (photo by ellasportfolio on Flickr), Lady Gaga (photo by aphrodite-in-nyc on Flickr), Ke$ha (photo by minglemediatv on Flickr), Adele (photo by Christopher Macsurak on Flickr), Iggy Azalea (photo by rarvesen on Flickr), Lorde (photo by Annette Geneva on Flickr)
Public Domain: Beyoncé, Kelly Clarkson
Fair Use: Carly Rae Jepsen (“Kiss” album cover), Meghan Trainor (“Title” album cover), Taylor Swift (“Taylor Swift” album cover, “Fearless” album cover, “Speak Now” album cover, “Red” album cover, “1989” album cover)

How Text Messages Change from Dating to Marriage

Way back in October 2008, my now husband and I went on our first date. On our one year anniversary, his gift to me was a Word doc of all of our text messages since our first date (what he likes to refer to as #thegiftofdata). This was especially high tech back in the day (given that we both had feature phones) and what I considered to be the most thoughtful gift ever (given that we are both nerds).

To celebrate our six year anniversary, I decided to take his present to the next level. I took a look at all of our text messages from our first year of dating and compared them with our text messages from the past year as an engaged couple and then newlyweds. I started by looking at the words we used in our text messages six years ago versus present day.

Text_Words2

First of all, we can clearly see that my husband has an obsession with the word “home”. As for me, my early twenties self frequently started conversations with the term “hey”, and more recently I seem to have decided to no longer greet my husband, but instead agree with most of the things he texts me.

I then looked at the frequency of specific terms we used in our text messages when we started dating compared to the past year.

content_final2

Our conversations changed from “hey, what’s up?” to “ok, sounds good”. We stopped saying each other’s names in our text messages. We don’t say in “love” as much anymore. Several words stayed relatively consistent over the years though, such as “home” and “dinner”. I took a look at the actual text messages that contained these words, and found that although the terms stayed the same, the context they were used in actually changed over time.

actualtexts_final

The main difference is that while we were dating, we didn’t see each other every day, so a lot of our communication had to happen via text. We’d often message the other person to see what they were doing or tell them that we were thinking of them. As a married couple, since we’re together all the time, we set up date nights and say sweet things to each other in person, so texting is mostly used to confirm logistics or share random thoughts.

Since the content of our text messages changed so much, I decided to look and see if the time of day we’d send text messages to each other changed as well. I focused on the messages we sent each other during the month after our first date, our engagement and our wedding.

timetext1

While we were dating, we started thinking about each other around 3pm in the afternoon and we’d send each other text messages until 3am in the morning. As a married couple, our texting schedule has pretty much flipped. We text all through the workday and never at night.

timetext2

We see the same story here. As a new couple, since we were apart the majority of the time, we had to check in with the other person every now and then, especially during the evening and late at night when we had no idea who they were with! It was also to tell the other person that we were out late doing something cool without them… and wishing they were there, of course.

As a committed couple, the only time of the day that we aren’t together is during the workday, so that’s when we text. We know exactly where the other person is each evening and if we’re doing something cool, it’s likely that we’re in it together and telling each other about it face to face.

Overall, our text messages started out very flirty and personal. Since we were new in our relationship, we made sure the things we said were interesting and thoughtful. As our relationship progressed, we spent more time together and got more comfortable with each other. Our text messages became more predictable, but only because all of the unpredictable things were said in person. We no longer have to text “I love you” from a distance in the middle of the night. I can now roll over, snuggle with my husband and whisper it into his ear.

To my husband: happy anniversary and I hope you enjoy your 2014 version of #thegiftofdata.