Text Mining to Analyze the Evolution of the most popular R&B and Hip-Hop Songs

In this project, Emre Devrim and William Lopez-Jaramillo extracted textual information from a collection of the most popular R&B and Hip-Hop songs in order to discover significant changes in the genre through the years. Such changes focused on the use of vocabulary, mention of places, persons, organizations, topics, sentiments and emotions.

In this project various text-mining techniques we used, such as but not limited to: Part-of-Speech (POS) Analysis, Named-entity Recognition, Topic Modeling, Sentiment Analysis, and Emotion Mining. Finally using this information we attempt to classify Hip-Hop songs from different time periods using various machine learning models.

A summary of this project and this blog can be found here in a short video: Text Mining to Analyze the Evolution of the most popular R&B and Hip-Hop Songs.

The data set consisted of 1515 instances representing songs with the Artist, Song Title, Lyrics, and Year information for each of them. The songs are from 2002 to 2018 since those are the years with available information from the Billboard website. The data set was collected using the top-100 songs list from Billboard in the Hip-Hop genre.

Hip-Hop Word Distribution

After some initial pre-processing (removing non-verbal textual markers such as [Intro] or lalalalalala) and text normalization (both lexical, syntactical and semantical), slang replacement (replace luv with love) and special character removal ($’s and double spaces)  various text-mining techniques were used to obtain new insights into the song lyrics.

Last year, in a related project, we already found that rappers have a much richer vocabulary than pop artists. The analysis of the number of unique words in the vocabulary in R&B and Hip-Hop can be found here under.

Hip-Hop Average Word

But more interesting insight can be obtained from the analysis of the most frequent words over time, as is shown next:

Hip-Hop Most Frequent Nouns

So which cities and states were mentioned in the songs:

Hip-Hop City Map

But more importantly, what do they sing about; well mostly about relationships, gangster life and, believe it or not, romantic love!

Hip-Hop Topics

A great project, well done!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s