In this project, Emre Devrim and William Lopez-Jaramillo extracted textual information from a collection of the most popular R&B and Hip-Hop songs in order to discover significant changes in the genre through the years. Such changes focused on the use of vocabulary, mention of places, persons, organizations, topics, sentiments and emotions.

In this project various text-mining techniques we used, such as but not limited to: Part-of-Speech (POS) Analysis, Named-entity Recognition, Topic Modeling, Sentiment Analysis, and Emotion Mining. Finally using this information we attempt to classify Hip-Hop songs from different time periods using various machine learning models.

A summary of this project and this blog can be found here in a short video: Text Mining to Analyze the Evolution of the most popular R&B and Hip-Hop Songs.

The data set consisted of 1515 instances representing songs with the Artist, Song Title, Lyrics, and Year information for each of them. The songs are from 2002 to 2018 since those are the years with available information from the Billboard website. The data set was collected using the top-100 songs list from Billboard in the Hip-Hop genre.

After some initial pre-processing (removing non-verbal textual markers such as [Intro] or lalalalalala) and text normalization (both lexical, syntactical and semantical), slang replacement (replace luv with love) and special character removal ($’s and double spaces)  various text-mining techniques were used to obtain new insights into the song lyrics.

Last year, in a related project, we already found that rappers have a much richer vocabulary than pop artists. The analysis of the number of unique words in the vocabulary in R&B and Hip-Hop can be found here under.

But more interesting insight can be obtained from the analysis of the most frequent words over time, as is shown next:

So which cities and states were mentioned in the songs:

But more importantly, what do they sing about; well mostly about relationships, gangster life and, believe it or not, romantic love!

A great project, well done!