1.2-Machine_Learning-FeaturesAndVisualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Humans are really good a finding patterns. Even from a young age, they can group together items that are similar. Grouping similar things is an essential step in learning concepts, even before language comes into play. \n",
    "\n",
    "In machine learning, we do things like find *similar things*, *classify*, *predict*, *generate*, and more. \n",
    "\n",
    "Today we'll learn about things called \"features\", measuring the difference between things, and finding similarities!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In Daily Life..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine learning today has taken off thanks to the large amounts of data available, thanks to the internet. One example includes Netflix being able to suggest new shows that you might like. \n",
    "\n",
    "Let's explore this idea more. The main concept behind it is that you might like things that are similar to what you already like. This has some caveats (we'll talk about it in the Ethics for AI section), but it is behind much of the technology we have around us today."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's talk about music."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine that we want to make an AI program that suggests new song that you might like. Let's say you are interested in discovering songs that you like from other countries, or in other languages.\n",
    "\n",
    "But how do you quantify or measure what you like? Could you even describe it?\n",
    "\n",
    "Let's explore the Spotify Dataset 2017 songs of 2017 to learn how an algorithm to suggest new songs might work.\n",
    "\n",
    "Consider this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADD CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset of 2017 songs, you will see that, alongside every song title, there are many characteristics or **features** extracted for each song. These features include numerical information like tempo, acousticness, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print data[\"tempo\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine that you were to search this entire database for a song that you might like. Some features will be useful for finding similar songs. Which features do you think would be most useful? Least useful?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What can we do with Features?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using features, we can already:\n",
    "\n",
    "- visualize our data, and\n",
    "- find similar items.\n",
    "\n",
    "We'll use a dataset to suggest songs that are similar to our favourite song. For the sake of this example, let's say that we like the song \"Mask Off\" by \"Future\" and would like to find songs that are similar. Let's just look at 2 features for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "raw_data = pandas.read_csv('data/spotify.csv')  # Top 2017 songs of 2017 from kaggle.com\n",
    "\n",
    "# Just get the columns we want\n",
    "dataset = pandas.DataFrame(raw_data, columns = ['tempo', 'acousticness', 'song_title', 'artist'])\n",
    "\n",
    "# Let's just get the top 20 songs\n",
    "# ADD CODE HERE\n",
    "\n",
    "# Display the dataset\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I want to find songs with a similar tempo!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by thinking about how to find similar songs if you only know one thing about each song, e.g. tempo. \n",
    "We'll compare each song to Mask Off.\n",
    "\n",
    "Let's add a new column that finds the differences in tempo from the top song, Mask Off (tempo=150.062)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Add a column tempo_difference. 150.062 refers to Mask Off's tempo\n",
    "# ADD CODE HERE\n",
    "\n",
    "# Display the dataset\n",
    "# ADD CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's visualize that"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Now let's visualize that \n",
    "import matplotlib.pyplot as plot\n",
    "plot.rcParams[\"figure.figsize\"] = 4,4\n",
    "\n",
    "plot.scatter(dataset['tempo_difference'], dataset['song_title'])\n",
    "plot.xlabel('Difference in Tempo')\n",
    "\n",
    "# Display the plot\n",
    "# ADD CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now let's find how different they are in terms of acousticness!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Add a new column that finds the differences in acousticness from top song, Mask Off (acousticness=0.01020)\n",
    "\n",
    "dataset['acoustic_difference'] = abs(0.01020 - dataset.acousticness)\n",
    "\n",
    "# Display the dataset\n",
    "# ADD CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now let's visualize that with a histogram\n",
    "import matplotlib.pyplot as plot\n",
    "\n",
    "plot.scatter(dataset['acoustic_difference'], dataset['song_title'])\n",
    "plot.xlabel('Difference in Acousticness')\n",
    "\n",
    "# Display the plot\n",
    "# ADD CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perceptually Similar Songs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we manually look at which song looked the closest over both **tempo** and **acousticness** to Mask Off, we get ________________. \n",
    "\n",
    "\n",
    "Did you make your guess? Let's see if that sounds right! \n",
    "    \n",
    "    https://youtu.be/CGt-rTDkMcM\n",
    "    \n",
    "    https://youtu.be/xvZqHgFz51I\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing The Songs in 2D Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also visualize the two features simultaneously in a 2D plot, like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Now let's visualize that by plotting all the original data points\n",
    "import matplotlib.pyplot as plot\n",
    "plot.rcParams[\"figure.figsize\"] = 10,10\n",
    "\n",
    "plot.scatter(dataset['tempo'], dataset['acousticness'])\n",
    "plot.xlabel('Tempo')\n",
    "plot.ylabel('Acousticness')\n",
    "\n",
    "for index, row in dataset.iterrows():\n",
    "    plot.annotate(unicode(row['song_title'], 'utf-8'), xy=(row['tempo'], row['acousticness']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculating Distance in 2D Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now that we plot the songs in 2D space, we can properly calculate the distances in 2D mathematically, using the **Euclidean distance** function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "dataset['euclidean_distance'] = numpy.sqrt( (0.01020 - dataset.acousticness)**2 + (150.062 - dataset.tempo)**2)\n",
    "\n",
    "# Sort the dataset\n",
    "sorted_dataset_distance = dataset.sort_values('euclidean_distance')\n",
    "\n",
    "# Display sorted dataset\n",
    "# ADD CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interesting..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the last plot above, it suddenly looks like **I've Seen Footage** is really *close* to Mask Off, and **One Night** is really *far* from Mask Off. But in our calculations of Euclidean distance above, both are ~20 units away from **Mask Off**. Can you guess what's wrong?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we calculated the distance, the tempo was on a scale of 100's, while the acousticness was on a scale from 0 to 1. This means that the distance in tempo completely overshadowed the minute differences in acousticness.\n",
    "\n",
    "So, we need to make them both on the same ___. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also known as *normalizing* or standardizing data, we should scale our data to make our comparisons fair. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Scale data to [0,1]\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Scale tempo to [0,1]\n",
    "tempo_column = dataset[['tempo']].values.astype(float)\n",
    "tempo_scaler = MinMaxScaler().fit(tempo_column)\n",
    "tempo_scaled = tempo_scaler.transform(tempo_column)\n",
    "\n",
    "# Add a column with normalized tempo data\n",
    "# ADD CODE HERE\n",
    "\n",
    "# Scale acousticness to [0,1] (just in case)\n",
    "acoustic_column = dataset[['acousticness']].values.astype(float)\n",
    "acoustic_scaler = MinMaxScaler().fit(acoustic_column)\n",
    "acousticness_scaled = acoustic_scaler.transform(acoustic_column)\n",
    "\n",
    "# Add a column with normalized tempo data\n",
    "# ADD CODE HERE\n",
    "\n",
    "# Display sorted dataset\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Now calculate the distances again using these normalized values\n",
    "dataset['norm_euclidean_distance'] = # ADD CODE HERE\n",
    "\n",
    "# Sort the dataset from closest to farthest\n",
    "sorted_dataset_distance = dataset.sort_values('norm_euclidean_distance')\n",
    "\n",
    "# Display sorted dataset\n",
    "# ADD CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see the closest songs are different! Which are the most similar songs to Mask Off?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing Our Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can start to see our data and notice that there are songs that are similar to Mask Off, but also very different! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Now let's visualize that by plotting all the original data points\n",
    "import matplotlib.pyplot as plot\n",
    "plot.rcParams[\"figure.figsize\"] = 10,10\n",
    "\n",
    "plot.scatter(dataset['tempo_normalized'], dataset['acoustic_normalized'])\n",
    "plot.xlabel('Tempo')\n",
    "plot.ylabel('Acousticness')\n",
    "\n",
    "plot.axis('equal')\n",
    "\n",
    "for index, row in dataset.iterrows():\n",
    "    plot.annotate(unicode(row['song_title'], 'utf-8'), xy=(row['tempo_normalized'], row['acoustic_normalized']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's finish by visualizing even more songs from our database."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's put everything together! And this time, let's use lots more data. 100 songs!\n",
    "\n",
    "**1. Pre-processing**\n",
    "- Reading the file\n",
    "- Getting the columns (features) we want\n",
    "- Choosing the rows (songs) we want\n",
    "- Scaling the features to [0,1]\n",
    "\n",
    "**2. Visualizing Data in Feature Space**\n",
    "- Plotting the songs in 2D space\n",
    "\n",
    "**3. Calculating Differences and Similarities**\n",
    "- Calculating distances and sorting them to find the most similar songs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import matplotlib.pyplot as plot\n",
    "\n",
    "# Read the data from the spreadsheet\n",
    "raw_data = pandas.read_csv('data/spotify.csv')  # Top 2017 songs of 2017 from kaggle.com\n",
    "\n",
    "# Just get the columns we want\n",
    "dataset = pandas.DataFrame(raw_data, columns = ['tempo', 'acousticness', 'song_title', 'artist'])\n",
    "\n",
    "# Let's get the top 100 songs\n",
    "dataset = dataset[:100]\n",
    "\n",
    "# Scale tempo to [0,1]\n",
    "tempo_column = dataset[['tempo']].values.astype(float)\n",
    "tempo_scaler = MinMaxScaler().fit(tempo_column)\n",
    "tempo_scaled = tempo_scaler.transform(tempo_column)\n",
    "dataset['tempo_normalized'] = pandas.DataFrame(tempo_scaled)\n",
    "\n",
    "# Scale acousticness to [0,1] (just in case)\n",
    "acoustic_column = dataset[['acousticness']].values.astype(float)\n",
    "acoustic_scaler = MinMaxScaler().fit(acoustic_column)\n",
    "acousticness_scaled = acoustic_scaler.transform(acoustic_column)\n",
    "dataset['acoustic_normalized'] = pandas.DataFrame(acousticness_scaled)\\"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now let's visualize that by plotting all the original data points\n",
    "plot.rcParams[\"figure.figsize\"] = 20,20\n",
    "\n",
    "plot.scatter(dataset['tempo_normalized'], dataset['acoustic_normalized'])\n",
    "plot.xlabel('Tempo')\n",
    "plot.ylabel('Acousticness')\n",
    "\n",
    "plot.axis('equal')\n",
    "\n",
    "for index, row in dataset.iterrows():\n",
    "    plot.annotate(unicode(row['song_title'], 'utf-8'), xy=(row['tempo_normalized'], row['acoustic_normalized']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Now calculate the distances again using these normalized values\n",
    "import numpy\n",
    "dataset['norm_euclidean_distance'] = numpy.sqrt( (0.011762 - dataset.acoustic_normalized)**2 \n",
    "                                                + (0.748899 - dataset.tempo_normalized)**2)\n",
    "# Sort the dataset from closest to farthest\n",
    "sorted_dataset_distance = dataset.sort_values('norm_euclidean_distance')\n",
    "\n",
    "# Display sorted dataset\n",
    "sorted_dataset_distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Your Turn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Play with this a little bit -- find a song you like, \n",
    "and change the code above to find other songs in the top 100 songs that you might also like!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can you say about the songs now that you've visualized it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you try to add another feature? How would you modify the Euclidean Distance to accomodate another feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discussion - Ethics and Bias in AI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you think about how features are chosen? Are all features equally important? Who chooses the features to use?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}