2021-03-29_unsupervised_learning_in_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Course\n",
    "\n",
    "- Notebook Author: [Trenton McKinney](https://trenton3983.github.io/)\n",
    "- Course: **[DataCamp: Unsupervised Learning in Python](https://learn.datacamp.com/courses/unsupervised-learning-in-python)**\n",
    "- This [notebook](https://github.com/trenton3983/DataCamp/blob/487ac2f732f4a4dfe7d1b6f9d3418cf4529486e3/2021-03-29_unsupervised_learning_in_python.ipynb) was created as a reproducible reference.\n",
    "- The material is from the course\n",
    "- The course website uses `scikit-learn v0.19.2`, `pandas v0.19.2`, and `numpy v1.17.4`\n",
    "- If you find the content beneficial, consider a [DataCamp Subscription](https://www.datacamp.com/pricing?period=yearly).\n",
    "- I added a function (**`create_dir_save_file`**) to automatically download and save the required data (`data/course_name`) and image (`Images/course_name`) files.\n",
    "- Package Versions:\n",
    "  - Pandas version: 2.2.1\n",
    "  - Matplotlib version: 3.8.1\n",
    "  - Seaborn version: 0.13.2\n",
    "  - SciPy version: 1.12.0\n",
    "  - Scikit-Learn version: 1.3.2\n",
    "  - NumPy version: 1.26.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description\n",
    "\n",
    "Say you have a collection of customers with a variety of characteristics such as age, location, and financial history, and you wish to discover patterns and sort them into clusters. Or perhaps you have a set of texts, such as Wikipedia pages, and you wish to segment them into categories based on their content. This is the world of unsupervised learning, called as such because you are not guiding, or supervising, the pattern discovery by some prediction task, but instead uncovering hidden structure from unlabeled data. Unsupervised learning encompasses a variety of techniques in machine learning, from clustering to dimension reduction to matrix factorization. In this course, you'll learn the fundamentals of unsupervised learning and implement the essential algorithms using scikit-learn and scipy. You will learn how to cluster, transform, visualize, and extract insights from unlabeled datasets, and end the course by building a recommender system to recommend popular musical artists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pprint import pprint as pp\n",
    "from itertools import combinations\n",
    "from zipfile import ZipFile\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "import requests\n",
    "import sys\n",
    "\n",
    "from scipy.sparse import csr_matrix\n",
    "from scipy.cluster.hierarchy import linkage, dendrogram, fcluster\n",
    "from scipy.stats import pearsonr\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.preprocessing import StandardScaler, Normalizer, normalize, MaxAbsScaler\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.decomposition import PCA, TruncatedSVD, NMF\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action=\"ignore\", category=UserWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration Options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_columns', 200)\n",
    "pd.set_option('display.max_rows', 300)\n",
    "pd.set_option('display.expand_frame_repr', True)\n",
    "plt.rcParams[\"patch.force_edgecolor\"] = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dir_save_file(dir_path: Path, url: str):\n",
    "    \"\"\"\n",
    "    Check if the path exists and create it if it does not.\n",
    "    Check if the file exists and download it if it does not.\n",
    "    \"\"\"\n",
    "    if not dir_path.parents[0].exists():\n",
    "        dir_path.parents[0].mkdir(parents=True)\n",
    "        print(f'Directory Created: {dir_path.parents[0]}')\n",
    "    else:\n",
    "        print('Directory Exists')\n",
    "        \n",
    "    if not dir_path.exists():\n",
    "        r = requests.get(url, allow_redirects=True)\n",
    "        open(dir_path, 'wb').write(r.content)\n",
    "        print(f'File Created: {dir_path.name}')\n",
    "    else:\n",
    "        print('File Exists')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = Path('data/2021-03-29_unsupervised_learning_python')\n",
    "images_dir = Path('Images/2021-03-29_unsupervised_learning_python')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# csv files\n",
    "base = 'https://assets.datacamp.com/production/repositories/655/datasets'\n",
    "file_spm = base + '/1304e66b1f9799e1a5eac046ef75cf57bb1dd630/company-stock-movements-2010-2015-incl.csv'\n",
    "file_ev = base + '/2a1f3ab7bcc76eef1b8e1eb29afbd54c4ebf86f2/eurovision-2016.csv'\n",
    "file_fish = base + '/fee715f8cf2e7aad9308462fea5a26b791eb96c4/fish.csv'\n",
    "file_lcd = base + '/effd1557b8146ab6e620a18d50c9ed82df990dce/lcd-digits.csv'\n",
    "file_wine = base + '/2b27d4c4bdd65801a3b5c09442be3cb0beb9eae0/wine.csv'\n",
    "file_artists_sparse = 'https://raw.githubusercontent.com/trenton3983/DataCamp/master/data/2021-03-29_unsupervised_learning_python/artists_sparse.csv'\n",
    "\n",
    "# zip files\n",
    "file_grain = base + '/bb87f0bee2ac131042a01307f7d7e3d4a38d21ec/Grains.zip'\n",
    "file_musicians = base + '/c974f2f2c4834958cbe5d239557fbaf4547dc8a3/Musical%20artists.zip'\n",
    "file_wiki = base + '/8e2fbb5b8240c06602336f2148f3c42e317d1fdb/Wikipedia%20articles.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_links = [file_spm, file_ev, file_fish, file_lcd, file_wine, file_grain, file_musicians, file_wiki, file_artists_sparse]\n",
    "file_paths = list()\n",
    "\n",
    "for file in file_links:\n",
    "    file_name = file.split('/')[-1].replace('?raw=true', '').replace('%20', '_')\n",
    "    data_path = data_dir / file_name\n",
    "    create_dir_save_file(data_path, file)\n",
    "    file_paths.append(data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unzip the zipped files\n",
    "zip_files = [v for v in file_paths if v.suffix == '.zip']\n",
    "for file in zip_files:\n",
    "    with ZipFile(file, 'r') as zip_:\n",
    "        zip_.extractall(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp = [v for v in data_dir.rglob('*') if v.suffix in ['.csv', '.txt']]\n",
    "dp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `stk`: Company Stock Movements 2010 - 2015"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stk = pd.read_csv(dp[1], index_col=[0])\n",
    "stk.iloc[:2, :5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `euv`: Eurovision 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "euv = pd.read_csv(dp[2])\n",
    "euv.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `fsh`: Fish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fsh = pd.read_csv(dp[3], header=None)\n",
    "fsh.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `lcd`: LCD Digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lcd = pd.read_csv(dp[4], header=None)\n",
    "lcd.iloc[:2, :5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `win`: Wine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "win = pd.read_csv(dp[5])\n",
    "win.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `swl`: Seeds Width vs. Length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "swl = pd.read_csv(dp[6], header=None)\n",
    "swl.columns = ['width', 'length']\n",
    "swl.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `sed`: Seeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sed = pd.read_csv(dp[7], header=None)\n",
    "\n",
    "sed['varieties'] = sed[7].map({1: 'Kama wheat', 2: 'Rosa wheat', 3: 'Canadian wheat'})\n",
    "\n",
    "sed.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `mus1`: Musical Artists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mus1 = pd.read_csv(dp[8])\n",
    "mus1.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `mus2`: Musical Artists - Scrobbler Small Sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mus2 = pd.read_csv(dp[9])\n",
    "mus2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `artists_sparse`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "artist_df = pd.read_csv(dp[0], header=None, index_col=[0])\n",
    "artist_names = artist_df.index.tolist()\n",
    "artists_sparse = csr_matrix(artist_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `wik1`: Wikipedia Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wik1 = pd.read_csv(dp[10], index_col=0).T\n",
    "wik1.iloc[:4, :10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wik1_sparse = csr_matrix(wik1)\n",
    "wik1_sparse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `wik2`: Wikipedia Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wik2 = pd.read_csv(dp[11], header=None)\n",
    "wik2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Memory Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# These are the usual ipython objects, including this one you are creating\n",
    "ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']  # list a variables\n",
    "\n",
    "# Get a sorted list of the objects and their sizes\n",
    "sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)[:11]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering for dataset exploration\n",
    "\n",
    "Learn how to discover the underlying groups (or \"clusters\") in a dataset. By the end of this chapter, you'll be clustering companies using their stock market prices, and distinguishing different species by clustering their measurements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised Learning\n",
    "\n",
    "- We're here to learn about unsupervised learning in Python.\n",
    "- Unsupervised learning is a class of machine learning techniques for discovering patterns in data. For instance, finding the natural \"clusters\" of customers based on their purchase histories, or searching for patterns and correlations among these purchases, and using these patterns to express the data in a compressed form. These are examples of unsupervised learning techniques called \"clustering\" and \"dimension reduction\".\n",
    "\n",
    "1. Supervised vs unsupervised learning\n",
    "   - Unsupervised learning is defined in opposition to supervised learning.\n",
    "   - An example of supervised learning is using the measurements of tumors to classify them as benign or cancerous.\n",
    "     - In this case, the pattern discovery is guided, or \"supervised\", so that the patterns are as useful as possible for predicting the label: benign or cancerous.\n",
    "   - Unsupervised learning, in contrast, is learning without labels.\n",
    "     - It is pure pattern discovery, unguided by a prediction task. You'll start by learning about clustering.\n",
    "\n",
    "1. Iris dataset\n",
    "   - The [iris dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html) consists of the measurements of many iris plants of three different species. \n",
    "     - _setosa_\n",
    "     - _versicolor_\n",
    "     - _virginica_\n",
    "   - There are four measurements: _petal length_, _petal width_, _sepal length_ and _sepal width_. These are the features of the dataset.\n",
    "\n",
    "1. Arrays, features & samples\n",
    "   - Throughout this course, datasets like this will be written as two-dimensional numpy arrays.\n",
    "   - The columns of the array will correspond to the features.\n",
    "   - The measurements for individual plants are the samples of the dataset. These correspond to rows of the array.\n",
    "\n",
    "1. Iris data is 4-dimensional\n",
    "   - The samples of the iris dataset have four measurements, and so correspond to points in a four-dimensional space.\n",
    "   - This is the dimension of the dataset.\n",
    "   - We can't visualize four dimensions directly, but using unsupervised learning techniques we can still gain insight.\n",
    "\n",
    "1. k-means clustering\n",
    "   - In this chapter, we'll cluster these samples using k-means clustering.\n",
    "   - k-means finds a specified number of clusters in the samples.\n",
    "   - It's implemented in the scikit-learn or \"sklearn\" library. Let's see `kmeans` in action on some samples from the iris dataset.\n",
    "     - [`sklearn.cluster.kmeans`](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html)\n",
    "     - [sklearn K-means clustering guide](https://scikit-learn.org/stable/modules/clustering.html#k-means)\n",
    "\n",
    "1. k-means clustering with scikit-learn\n",
    "   - The iris samples are represented as an array. To start, import kmeans from scikit-learn.\n",
    "   - Then create a kmeans model, specifying the number of clusters you want to find.\n",
    "   - Let's specify 3 clusters, since there are three species of iris.\n",
    "   - Now call the fit method of the model, passing the array of samples.\n",
    "   - This fits the model to the data, by locating and remembering the regions where the different clusters occur.\n",
    "   - Then we can use the predict method of the model on these same samples.\n",
    "   - This returns a cluster label for each sample, indicating to which cluster a sample belongs.\n",
    "   - Let's assign the result to labels, and print it out.\n",
    "\n",
    "1. Cluster labels for new samples\n",
    "   - If someone comes along with some new iris samples, k-means can determine to which clusters they belong without starting over.\n",
    "   - k-means does this by remembering the mean of the samples in each cluster.\n",
    "   - These are called the \"centroids\".\n",
    "   - New samples are assigned to the cluster whose centroid is closest.\n",
    "   - Suppose you've got an array of new samples.\n",
    "   - To assign the new samples to the existing clusters, pass the array of new samples to the predict method of the kmeans model.\n",
    "   - This returns the cluster labels of the new samples.\n",
    "\n",
    "1. Scatter plots\n",
    "   - In the next video, you'll learn how to evaluate the quality of your clustering.\n",
    "   - Let's visualize our clustering of the iris samples using scatter plots.\n",
    "   - Here is a scatter plot of the sepal length vs petal length of the iris samples. Each point represents an iris sample, and is colored according to the cluster of the sample.\n",
    "   - To create a scatter plot like this, use PyPlot.\n",
    "   - Firstly, import PyPlot. It is conventionally imported as plt.\n",
    "   - Now get the x- and y- co-ordinates of each sample.\n",
    "   - Sepal length is in the 0th column of the array, while petal length is in the 2nd column.\n",
    "   - Now call the plt.scatter function, passing the x- and y- co-ordinates and specifying c=labels to color by cluster label.\n",
    "   - When you are ready to show your plot, call plt.show()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = sns.load_dataset('iris')\n",
    "iris_samples = iris.sample(n=75, replace=False, random_state=3)\n",
    "X_iris = iris_samples.iloc[:, :4]\n",
    "y_iris = iris_samples.species"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_samples.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_model = KMeans(n_clusters=3, n_init=10)\n",
    "iris_model.fit(X_iris)\n",
    "\n",
    "iris_labels = iris_model.predict(X_iris)\n",
    "iris_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_new_samples = iris[~iris.index.isin(iris_samples.index)].copy()\n",
    "\n",
    "X_iris_new = iris_new_samples.iloc[:, :4]\n",
    "y_iris_new = iris_new_samples.species\n",
    "iris_new_labels = iris_model.predict(X_iris_new)\n",
    "iris_new_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_new_samples['pred_labels'] = iris_new_labels\n",
    "iris_samples['pred_labels'] = iris_labels\n",
    "\n",
    "pred_labels = pd.concat([iris_new_samples[['species', 'pred_labels']], iris_samples[['species', 'pred_labels']]]).sort_index()\n",
    "pred_labels.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))\n",
    "\n",
    "xs = X_iris.sepal_length\n",
    "ys = X_iris.petal_length\n",
    "xs_new = X_iris_new.sepal_length\n",
    "ys_new = X_iris_new.petal_length\n",
    "\n",
    "ax1.scatter(xs, ys, c=iris_labels)\n",
    "ax1.set_ylabel('Petal Length')\n",
    "ax1.set_xlabel('Sepal Length')\n",
    "ax1.set_title('Sample')\n",
    "\n",
    "ax2.scatter(xs_new, ys_new, c=iris_new_labels)\n",
    "ax2.set_ylabel('Petal Length')\n",
    "ax2.set_xlabel('Sepal Length')\n",
    "ax2.set_title('New Sample')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many clusters?\n",
    "\n",
    "You are given an array `points` of size 300x2, where each row gives the (x, y) co-ordinates of a point on a map. Make a scatter plot of these points, and use the scatter plot to guess how many clusters there are.\n",
    "\n",
    "`matplotlib.pyplot` has already been imported as `plt`. In the IPython Shell:\n",
    "\n",
    "- Create an array called `xs` that contains the values of `points[:,0]` - that is, column `0` of `points`.\n",
    "- Create an array called `ys` that contains the values of `points[:,1]` - that is, column `1` of `points`.\n",
    "- Make a scatter plot by passing `xs` and `ys` to the `plt.scatter()` function.\n",
    "- Call the `plt.show()` function to show your plot.\n",
    "\n",
    "![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/how_many_clusters.JPG)\n",
    "\n",
    "How many clusters do you see?\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~2~~\n",
    "- **3**\n",
    "- ~~300~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pen = sns.load_dataset('penguins').dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "points = pen.iloc[:, 2:4]\n",
    "points.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = points.bill_length_mm\n",
    "ys = points.bill_depth_mm\n",
    "\n",
    "sns.scatterplot(x=xs, y=ys, hue=pen.species)\n",
    "plt.legend(title='Species', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
    "plt.title('Species With Real Labels')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering 2D points\n",
    "\n",
    "From the scatter plot of the previous exercise, you saw that the points seem to separate into 3 clusters. You'll now create a KMeans model to find 3 clusters, and fit it to the data points from the previous exercise. After the model has been fit, you'll obtain the cluster labels for some new points using the `.predict()` method.\n",
    "\n",
    "You are given the array `points` from the previous exercise, and also an array `new_points`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `KMeans` from `sklearn.cluster`.\n",
    "- Using `KMeans()`, create a `KMeans` instance called `model` to find `3` clusters. To specify the number of clusters, use the `n_clusters` keyword argument.\n",
    "- Use the `.fit()` method of `model` to fit the model to the array of points `points`.\n",
    "- Use the `.predict()` method of `model` to predict the cluster labels of `new_points`, assigning the result to `labels`.\n",
    "- Hit 'Submit Answer' to see the cluster labels of `new_points`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create points \n",
    "points = pen.iloc[:, 2:4].sample(n=177, random_state=3)\n",
    "new_points = pen[~pen.index.isin(points.index)].iloc[:, 2:4]\n",
    "\n",
    "# Import KMeans\n",
    "# from sklearn.cluster import KMeans\n",
    "\n",
    "# Create a KMeans instance with 3 clusters: model\n",
    "model = KMeans(n_clusters=3, n_init=10)\n",
    "\n",
    "# Fit model to points\n",
    "model.fit(points)\n",
    "\n",
    "labels = model.predict(points)\n",
    "\n",
    "# Determine the cluster labels of new_points: labels\n",
    "new_labels = model.predict(new_points)\n",
    "\n",
    "# Print cluster labels of new_points\n",
    "print(new_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "points.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))\n",
    "\n",
    "xs = points.bill_length_mm\n",
    "ys = points.bill_depth_mm\n",
    "xs_new = new_points.bill_length_mm\n",
    "ys_new = new_points.bill_depth_mm\n",
    "\n",
    "ax1.scatter(xs, ys, c=labels)\n",
    "ax1.set_ylabel('bill Depth (mm)')\n",
    "ax1.set_xlabel('bill Length (mm)')\n",
    "ax1.set_title('Points: Predicted Labels')\n",
    "\n",
    "ax2.scatter(xs_new, ys_new, c=new_labels)\n",
    "ax2.set_ylabel('bill Depth (mm)')\n",
    "ax2.set_xlabel('bill Length (mm)')\n",
    "ax2.set_title('New Points: Predicted Labels')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You've successfully performed k-Means clustering and predicted the labels of new points. But it is not easy to inspect the clustering by just looking at the printed labels. A visualization would be far more useful. In the next exercise, you'll inspect your clustering with a scatter plot!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect your clustering\n",
    "\n",
    "Let's now inspect the clustering you performed in the previous exercise!\n",
    "\n",
    "A solution to the previous exercise has already run, so `new_points` is an array of points and `labels` is the array of their cluster labels.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `matplotlib.pyplot` as `plt`.\n",
    "- Assign column `0` of `new_points` to `xs`, and column `1` of `new_point`s to `ys`.\n",
    "- Make a scatter plot of `xs` and `ys`, specifying the `c=labels` keyword arguments to color the points by their cluster label. Also specify `alpha=0.5`.\n",
    "- Compute the coordinates of the centroids using the `.cluster_centers_` attribute of `model`.\n",
    "- Assign column `0` of `centroids` to `centroids_x`, and column `1` of `centroids` to `centroids_y`.\n",
    "- Make a scatter plot of `centroids_x` and `centroids_y`, using `'D'` (a diamond) as a `marker` by specifying the marker parameter. Set the size of the markers to be 50 using `s=50`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pyplot\n",
    "# import matplotlib.pyplot as plt\n",
    "\n",
    "new_points = new_points.to_numpy()\n",
    "\n",
    "# Assign the columns of new_points: xs and ys\n",
    "xs = new_points[:, 0]\n",
    "ys = new_points[:, 1]\n",
    "\n",
    "# Make a scatter plot of xs and ys, using labels to define the colors\n",
    "plt.scatter(xs, ys, c=new_labels, alpha=0.5)\n",
    "\n",
    "# Assign the cluster centers: centroids\n",
    "centroids = model.cluster_centers_\n",
    "\n",
    "# Assign the columns of centroids: centroids_x, centroids_y\n",
    "centroids_x = centroids[:,0]\n",
    "centroids_y = centroids[:,1]\n",
    "\n",
    "# Make a scatter plot of centroids_x and centroids_y\n",
    "plt.scatter(centroids_x, centroids_y, marker='D', s=50)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The clustering looks great! But how can you be sure that 3 clusters is the correct choice? In other words, how can you evaluate the quality of a clustering? Tune into the next video in which Ben will explain how to evaluate a clustering!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating a clustering\n",
    "\n",
    "- In the previous video, we used k-means to cluster the iris samples into three clusters.\n",
    "- But how can we evaluate the quality of this clustering?\n",
    "\n",
    "1. Evaluating a clustering\n",
    "   - A direct approach is to compare the clusters with the iris species.\n",
    "   - You'll learn about this first, before considering the problem of how to measure the quality of a clustering in a way that doesn't require our samples to come pre-grouped into species.\n",
    "   - This measure of quality can then be used to make an informed choice about the number of clusters to look for.\n",
    "\n",
    "1. Iris: clusters vs species\n",
    "   - Firstly, let's check whether the 3 clusters of iris samples have any correspondence to the iris species.\n",
    "   - The correspondence is described by this table.\n",
    "   - There is one column for each of the three species of iris: setosa, versicolor and virginica, and one row for each of the three cluster labels: 0, 1 and 2.\n",
    "     - The table shows the number of samples that have each possible cluster label/species combination.\n",
    "     - For example, we see that cluster 1 corresponds perfectly with the species setosa.\n",
    "     - On the other hand, while cluster 0 contains mainly virginica samples, there are also some virginica samples in cluster 2.\n",
    "\n",
    "1. Cross tabulation with pandas\n",
    "   - Tables like these are called \"cross-tabulations\".\n",
    "   - To construct one, we are going to use the pandas library.\n",
    "   - Let's assume the species of each sample is given as a list of strings.\n",
    "\n",
    "1. Aligning labels and species\n",
    "   - Import pandas, and then create a two-column DataFrame, where the first column is cluster labels and the second column is the iris species, so that each row gives the cluster label and species of a single sample.\n",
    "\n",
    "1. Crosstab of labels and species\n",
    "   - Now use the pandas crosstab function to build the cross tabulation, passing the two columns of the DataFrame.\n",
    "   - Cross tabulations like these provide great insights into which sort of samples are in which cluster.\n",
    "   - But in most datasets, the samples are not labeled by species.\n",
    "   - How can the quality of a clustering be evaluated in these cases?\n",
    "\n",
    "1. Measuring clustering quality\n",
    "   - We need a way to measure the quality of a clustering that uses only the clusters and the samples themselves.\n",
    "   - A good clustering has tight clusters, meaning that the samples in each cluster are bunched together, not spread out.\n",
    "\n",
    "1. Inertia measures clustering quality\n",
    "   - How spread out the samples within each cluster are can be measured by the \"inertia\".\n",
    "   - Intuitively, inertia measures how far samples are from their centroids.\n",
    "   - You can find the precise definition in the scikit-learn documentation.\n",
    "     - [SKLearn Clustering Guide: KMeans](https://scikit-learn.org/stable/modules/clustering.html#k-means)\n",
    "   - We want clusters that are not spread out, so _lower values of the inertia are better_.\n",
    "   - The inertia of a kmeans model is measured automatically when any of the `.fit()` methods are called, and is available afterwards as the `.inertia_` attribute.\n",
    "   - In fact, kmeans aims to place the clusters in a way that minimizes the inertia.\n",
    "\n",
    "1. The number of clusters\n",
    "   - Here is a plot of the inertia values of clusterings of the iris dataset with different numbers of clusters.\n",
    "   - Our kmeans model with 3 clusters has relatively low inertia, which is great.\n",
    "   - But notice that the inertia continues to decrease slowly.\n",
    "   - So what's the best number of clusters to choose?\n",
    "\n",
    "1. How many clusters to choose?\n",
    "   - Ultimately, this is a trade-off.\n",
    "   - A good clustering has tight clusters (meaning low inertia).\n",
    "   - But it also doesn't have too many clusters.\n",
    "   - A good rule of thumb is to choose an elbow in the inertia plot, that is, a point where the inertia begins to decrease more slowly.\n",
    "   - For example, by this criterion, 3 is a good number of clusters for the iris dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct = pd.crosstab(pred_labels.pred_labels, pred_labels.species)\n",
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_model.inertia_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Sum_of_squared_distances = list()\n",
    "K = range(1, 10)\n",
    "for k in K:\n",
    "    km = KMeans(n_clusters=k, n_init=10)\n",
    "    km = km.fit(X_iris)\n",
    "    Sum_of_squared_distances.append(km.inertia_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 5))\n",
    "plt.plot(K, Sum_of_squared_distances, 'bx-')\n",
    "plt.xlabel('k')\n",
    "plt.ylabel('Sum_of_squared_distances')\n",
    "plt.title('Elbow Method For Optimal k')\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many clusters of grain?\n",
    "\n",
    "In the video, you learned how to choose a good number of clusters for a dataset using the k-means inertia graph. You are given an `array` samples containing the measurements (such as area, perimeter, length, and several others) of samples of grain. What's a good number of clusters in this case?\n",
    "\n",
    "`KMeans` and PyPlot (`plt`) have already been imported for you.\n",
    "\n",
    "This dataset was sourced from the [UCI Machine Learning Repository.](https://archive.ics.uci.edu/ml/datasets/seeds)\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- For each of the given values of `k`, perform the following steps:\n",
    "- Create a `KMeans` instance called `model` with `k` clusters.\n",
    "- Fit the model to the grain data `samples`.\n",
    "- Append the value of the `inertia_` attribute of `model` to the list `inertias`.\n",
    "- The code to plot `ks` vs `inertias` has been written for you, so hit 'Submit Answer' to see the plot!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = sed.iloc[:, :-2]\n",
    "\n",
    "ks = range(1, 6)\n",
    "inertias = list()\n",
    "\n",
    "for k in ks:\n",
    "    # Create a KMeans instance with k clusters: model\n",
    "    model = KMeans(n_clusters=k, n_init=10)\n",
    "    \n",
    "    # Fit model to samples\n",
    "    model.fit(samples)\n",
    "    \n",
    "    # Append the inertia to the list of inertias\n",
    "    inertias.append(model.inertia_)\n",
    "    \n",
    "# Plot ks vs inertias\n",
    "plt.plot(ks, inertias, '-o')\n",
    "plt.xlabel('number of clusters, k')\n",
    "plt.ylabel('inertia')\n",
    "plt.xticks(ks)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The inertia decreases very slowly from 3 clusters to 4, so it looks like 3 clusters would be a good choice for this data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the grain clustering\n",
    "\n",
    "In the previous exercise, you observed from the inertia plot that 3 is a good number of clusters for the grain data. In fact, the grain samples come from a mix of 3 different grain varieties: \"Kama\", \"Rosa\" and \"Canadian\". In this exercise, cluster the grain samples into three clusters, and compare the clusters to the grain varieties using a cross-tabulation.\n",
    "\n",
    "You have the array samples of grain `samples`, and a list `varieties` giving the grain variety for each sample. Pandas (`pd`) and `KMeans` have already been imported for you.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Create a `KMeans` model called `model` with `3` clusters.\n",
    "- Use the `.fit_predict()` method of `model` to fit it to `samples` and derive the cluster labels. Using `.fit_predict()` is the same as using `.fit()` followed by `.predict()`.\n",
    "- Create a DataFrame `df` with two columns named `'labels'` and `'varieties'`, using `labels` and `varieties`, respectively, for the column values. This has been done for you.\n",
    "- Use the `pd.crosstab()` function on `df['labels']` and `df['varieties']` to count the number of times each grain variety coincides with each cluster label. Assign the result to `ct`.\n",
    "- Hit 'Submit Answer' to see the cross-tabulation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "varieties = sed.varieties\n",
    "\n",
    "# Create a KMeans model with 3 clusters: model\n",
    "model = KMeans(n_clusters=3, n_init=10)\n",
    "\n",
    "# Use fit_predict to fit model and obtain cluster labels: labels\n",
    "labels = model.fit_predict(samples)\n",
    "\n",
    "# Create a DataFrame with labels and varieties as columns: df\n",
    "df = pd.DataFrame({'labels': labels, 'varieties': varieties})\n",
    "\n",
    "# Create crosstab: ct\n",
    "ct = pd.crosstab(df.labels, df.varieties)\n",
    "\n",
    "# Display ct\n",
    "ct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The cross-tabulation shows that the 3 varieties of grain separate really well into 3 clusters. But depending on the type of data you are working with, the clustering may not always be this good. Is there anything you can do in such situations to improve your clustering?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transforming features for better clusterings\n",
    "\n",
    "1. Piedmont wines dataset\n",
    "   - The Piedmont [wines dataset][wine].\n",
    "   - We have 178 samples of red wine from the Piedmont region of Italy.\n",
    "   - The features measure chemical composition (like alcohol content) and visual properties like color intensity.\n",
    "   - The samples come from 3 distinct varieties of wine.\n",
    "\n",
    "1. Clustering the wines\n",
    "   - Let's take the array of samples and use KMeans to find 3 clusters.\n",
    "\n",
    "1. Clusters vs. varieties\n",
    "   - There are three varieties of wine, so let's use pandas crosstab to check the cluster label - wine variety correspondence.\n",
    "   - As you can see, this time things haven't worked out so well.\n",
    "   - The KMeans clusters don't correspond well with the wine varieties.\n",
    "\n",
    "1. Feature variances\n",
    "   - The problem is that the features of the wine dataset have very different variances.\n",
    "   - The variance of a feature measures the spread of its values.\n",
    "   - For example, the malic acid feature has a higher variance than the od280 feature, and this can also be seen in their scatter plot.\n",
    "   - The differences in some of the feature variances is enormous, as seen here, for example, in the scatter plot of the od280 and proline features.\n",
    "\n",
    "1. StandardScaler\n",
    "   - In KMeans clustering, the variance of a feature corresponds to its influence on the clustering algorithm.\n",
    "   - To give every feature a chance, the data needs to be transformed so that features have equal variance.\n",
    "   - This can be achieved with the StandardScaler from scikit-learn.\n",
    "   - It transforms every feature to have mean 0 and variance 1.\n",
    "   - The resulting \"standardized\" features can be very informative.\n",
    "   - Using standardized od280 and proline, for example, the three wine varieties are much more distinct.\n",
    "\n",
    "1. sklearn StandardScaler\n",
    "   - Let's see the StandardScaler in action.\n",
    "   - First, import StandardScaler from sklearn.preprocessing.\n",
    "   - Then create a StandardScaler object, and fit it to the samples.\n",
    "   - The transform method can now be used to standardize any samples, either the same ones, or completely new ones.\n",
    "\n",
    "1. Similar methods\n",
    "   - The APIs of StandardScaler and KMeans are similar, but there is an important difference.\n",
    "   - StandardScaler transforms data, and so has a transform method.\n",
    "   - KMeans, in contrast, assigns cluster labels to samples, and this done using the predict method.\n",
    "\n",
    "1. StandardScaler, then KMeans\n",
    "   - Let's return to the problem of clustering the wines.\n",
    "   - We need to perform two steps.\n",
    "   - Firstly, to standardize the data using StandardScaler, and secondly to take the standardized data and cluster it using KMeans.\n",
    "   - This can be conveniently achieved by combining the two steps using a scikit-learn pipeline.\n",
    "   - Data then flows from one step into the next, automatically.\n",
    "\n",
    "1. Pipelines combine multiple steps\n",
    "   - The first steps are the same: creating a StandardScaler and a KMeans object.\n",
    "   - After that, import the make_pipeline function from sklearn.pipeline.\n",
    "   - Apply the make_pipeline function to the steps that you want to compose in this case, the scaler and the kmeans objects.\n",
    "   - Now use the fit method of the pipeline to fit both the scaler and kmeans, and use its predict method to obtain the cluster labels.\n",
    "\n",
    "1. Feature standardization improves clustering\n",
    "   - Checking the correspondence between the cluster labels and the wine varieties reveals that this new clustering, incorporating standardization, is fantastic.\n",
    "   - Its three clusters correspond almost exactly to the three wine varieties.\n",
    "   - This is a huge improvement on the clustering without standardization.\n",
    "\n",
    "1. sklearn preprocessing steps\n",
    "   - StandardScaler is an example of a \"preprocessing\" step.\n",
    "   - There are several of these available in scikit-learn, for example MaxAbsScaler and Normalizer.\n",
    "\n",
    "\n",
    "  [wine]: https://archive.ics.uci.edu/ml/datasets/Wine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "win.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_samples = win.iloc[:, 2:]\n",
    "wine_model = KMeans(n_clusters=3, n_init=10)\n",
    "wine_labels = wine_model.fit_predict(wine_samples)\n",
    "\n",
    "wine_pred = pd.DataFrame({'labels': wine_labels, 'varieties': win.class_name})\n",
    "wine_ct = pd.crosstab(wine_pred.labels, wine_pred.varieties)\n",
    "wine_ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_samples.var().round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 4))\n",
    "sns.scatterplot(data=win, x='od280', y='malic_acid', hue='class_name', ax=ax1)\n",
    "ax1.legend(title='Variety', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
    "ax1.set_xlim(0, 8)\n",
    "ax1.set_title('Real Labels')\n",
    "\n",
    "sns.scatterplot(data=win, x='od280', y='malic_acid', hue=wine_pred.labels, palette=\"tab10\", ax=ax2)\n",
    "ax2.set_xlim(0, 8)\n",
    "ax2.set_title('Predicted Labels')\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p1 = sns.scatterplot(data=win, x='od280', y='proline', hue='class_name')\n",
    "p1.set_xlim(-7.5, 7.5)\n",
    "p1.set_title('Unscaled Values');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_scaler = StandardScaler()\n",
    "wine_scaler.fit(wine_samples)\n",
    "StandardScaler(copy=True, with_mean=True, with_std=True)\n",
    "wine_samples_scaled = wine_scaler.transform(wine_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_samples_scaled = pd.DataFrame(wine_samples_scaled, columns=win.columns[2:])\n",
    "wine_samples_scaled.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p2 = sns.scatterplot(data=wine_samples_scaled, x='od280', y='proline', hue=win.class_name)\n",
    "p2.set_xlim(-7.5, 7.5)\n",
    "p2.set_title('Scaled Values');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "kmeans = KMeans(n_clusters=3, n_init=10)\n",
    "pipeline = make_pipeline(scaler, kmeans)\n",
    "pipeline.fit(wine_samples_scaled)\n",
    "wine_scaled_labels = pipeline.predict(wine_samples_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_pred_scaled = pd.DataFrame({'labels': wine_scaled_labels, 'varieties': win.class_name})\n",
    "wine_scaled_ct = pd.crosstab(wine_pred_scaled.labels, wine_pred_scaled.varieties)\n",
    "wine_scaled_ct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scaling fish data for clustering\n",
    "\n",
    "You are given an array `samples` giving measurements of fish. Each row represents an individual fish. The measurements, such as weight in grams, length in centimeters, and the percentage ratio of height to length, have very different scales. In order to cluster this data effectively, you'll need to standardize these features first. In this exercise, you'll build a pipeline to standardize and cluster the data.\n",
    "\n",
    "These fish measurement data were sourced from the [Journal of Statistics Education](https://jse.amstat.org/jse_data_archive.htm).\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import:\n",
    "  - `make_pipeline` from `sklearn.pipeline`.\n",
    "  - `StandardScaler` from `sklearn.preprocessing`.\n",
    "  - `KMeans` from `sklearn.cluster`.\n",
    "- Create an instance of `StandardScaler` called `scaler`.\n",
    "- Create an instance of `KMeans` with `4` clusters called `kmeans`.\n",
    "- Create a pipeline called `pipeline` that chains `scaler` and `kmeans`. To do this, you just need to pass them in as arguments to `make_pipeline()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# from sklearn.pipeline import make_pipeline\n",
    "# from sklearn.preprocessing import StandardScaler\n",
    "# from sklearn.cluster import KMeans\n",
    "\n",
    "# Create scaler: scaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Create KMeans instance: kmeans\n",
    "kmeans = KMeans(n_clusters=4, n_init=10)\n",
    "\n",
    "# Create pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, kmeans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now that you've built the pipeline, you'll use it in the next exercise to cluster the fish by their measurements.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering the fish data\n",
    "\n",
    "You'll now use your standardization and clustering pipeline from the previous exercise to cluster the fish by their measurements, and then create a cross-tabulation to compare the cluster labels with the fish species.\n",
    "\n",
    "As before, `samples` is the 2D array of fish measurements. Your pipeline is available as `pipeline`, and the species of every fish sample is given by the list `species`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `pandas` as `pd`.\n",
    "- Fit the pipeline to the fish measurements `samples`.\n",
    "- Obtain the cluster labels for `samples` by using the `.predict()` method of `pipeline`.\n",
    "- Using `pd.DataFrame()`, create a DataFrame `df` with two columns named `'labels'` and `'species'`, using `labels` and `species`, respectively, for the column values.\n",
    "- Using `pd.crosstab()`, create a cross-tabulation `ct` of `df['labels']` and `df['species']`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = fsh.iloc[:, 1:]\n",
    "species = fsh[0]\n",
    "\n",
    "# Fit the pipeline to samples\n",
    "pipeline.fit(samples)\n",
    "\n",
    "# Calculate the cluster labels: labels\n",
    "labels = pipeline.predict(samples)\n",
    "\n",
    "# Create a DataFrame with labels and species as columns: df\n",
    "df = pd.DataFrame({'labels': labels, 'species': species})\n",
    "\n",
    "# Create crosstab: ct\n",
    "ct = pd.crosstab(df.labels, df.species)\n",
    "\n",
    "# Display ct\n",
    "ct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering stocks using KMeans\n",
    "\n",
    "In this exercise, you'll cluster companies using their daily stock price movements (i.e. the dollar difference between the closing and opening prices for each trading day). You are given a NumPy array `movements` of daily price movements from 2010 to 2015 (obtained from Yahoo! Finance), where each row corresponds to a company, and each column corresponds to a trading day.\n",
    "\n",
    "Some stocks are more expensive than others. To account for this, include a `Normalizer` at the beginning of your pipeline. The Normalizer will separately transform each company's stock price to a relative scale before the clustering begins.\n",
    "\n",
    "Note that [`Normalizer()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Normalizer.html) is different to [`StandardScaler()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html), which you used in the previous exercise. While `StandardScaler()` standardizes **features** (such as the features of the fish data from the previous exercise) by removing the mean and scaling to unit variance, `Normalizer()` rescales `each sample` - here, each company's stock price - independently of the other.\n",
    "\n",
    "`KMeans` and `make_pipeline` have already been imported for you.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `Normalizer` from `sklearn.preprocessing`.\n",
    "- Create an instance of `Normalizer` called `normalizer`.\n",
    "- Create an instance of `KMeans` called `kmeans` with `10` clusters.\n",
    "- Using `make_pipeline()`, create a pipeline called `pipeline` that chains `normalizer` and `kmeans`.\n",
    "- Fit the pipeline to the `movements` array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movements = stk.to_numpy()\n",
    "companies = stk.index.to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Normalizer\n",
    "# from sklearn.preprocessing import Normalizer\n",
    "\n",
    "# Create a normalizer: normalizer\n",
    "normalizer = Normalizer()\n",
    "\n",
    "# Create a KMeans model with 10 clusters: kmeans\n",
    "kmeans = KMeans(n_clusters=10, random_state=12, n_init=10)\n",
    "\n",
    "# Make a pipeline chaining normalizer and kmeans: pipeline\n",
    "pipeline = make_pipeline(normalizer, kmeans)\n",
    "\n",
    "# Fit pipeline to the daily price movements\n",
    "pipeline.fit(movements)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now that your pipeline has been set up, you can find out which stocks move together in the next exercise!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which stocks move together?\n",
    "\n",
    "In the previous exercise, you clustered companies by their daily stock price movements. So which company have stock prices that tend to change in the same way? You'll now inspect the cluster labels from your clustering to find out.\n",
    "\n",
    "Your solution to the previous exercise has already been run. Recall that you constructed a Pipeline `pipeline` containing a `KMeans` model and fit it to the NumPy array `movements` of daily stock movements. In addition, a list `companies` of the company names is available.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `pandas` as `pd`.\n",
    "- Use the `.predict()` method of the pipeline to predict the labels for `movements`.\n",
    "- Align the cluster labels with the list of company names `companies` by creating a DataFrame `df` with `labels` and `companies` as columns. This has been done for you.\n",
    "- Use the `.sort_values()` method of `df` to sort the DataFrame by the `'labels'` column, and print the result.\n",
    "- Hit 'Submit Answer' and take a moment to see which companies are together in each cluster!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict the cluster labels: labels\n",
    "labels = pipeline.predict(movements)\n",
    "\n",
    "# Create a DataFrame aligning labels and companies: df\n",
    "df = pd.DataFrame({'labels': labels, 'companies': companies})\n",
    "\n",
    "# Display df sorted by cluster label\n",
    "df = df.sort_values('labels')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Take a look at the clusters. Are you surprised by any of the results? In the next chapter, you'll learn about how to communicate results such as this through visualizations.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stk_t = stk.T.copy()\n",
    "stk_t.index = pd.to_datetime(stk_t.index)\n",
    "stk_t = stk_t.rolling(30).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=5, ncols=2, figsize=(16, 16))\n",
    "axes = axes.ravel()\n",
    "for i, (g, d) in enumerate(df.groupby('labels')):\n",
    "    cols = d.companies.tolist()\n",
    "    sns.lineplot(data=stk_t[cols], ax=axes[i])\n",
    "    axes[i].legend(bbox_to_anchor=(1.05, 1), loc='upper left')\n",
    "    axes[i].set_title(f'30-Day Rolling Mean: Group {g}')\n",
    "    axes[i].set_ylim(-3, 3)\n",
    "\n",
    "fig.autofmt_xdate(rotation=90, ha='center')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization with hierarchical clustering and t-SNE\n",
    "\n",
    "In this chapter, you'll learn about two unsupervised learning techniques for data visualization, hierarchical clustering and t-SNE. Hierarchical clustering merges the data samples into ever-coarser clusters, yielding a tree visualization of the resulting cluster hierarchy. t-SNE maps the data samples into 2d space so that the proximity of the samples to one another can be visualized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing hierarchies\n",
    "\n",
    "- A huge part of your work as a data scientist will be the communication of your insights to other people.\n",
    "\n",
    "1. Visualizations communicate insight\n",
    "   - Visualizations are an excellent way to share your findings, particularly with a non-technical audience.\n",
    "   - In this chapter, you'll learn about two unsupervised learning techniques for visualization: t-SNE and hierarchical clustering.\n",
    "   - t-SNE, which we'll consider later, creates a 2d map of any dataset, and conveys useful information about the proximity of the samples to one another.\n",
    "   - First up, however, let's learn about hierarchical clustering.\n",
    "\n",
    "1. A hierarchy of groups\n",
    "   - You've already seen many hierarchical clusterings in the real world.\n",
    "   - For example, living things can be organized into small narrow groups, like humans, apes, snakes and lizards, or into larger, broader groups like mammals and reptiles, or even broader groups like animals and plants.\n",
    "   - These groups are contained in one another, and form a hierarchy.\n",
    "   - Analogously, hierarchical clustering arranges samples into a hierarchy of clusters.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/hierarchy_groups.JPG)\n",
    "\n",
    "1. Eurovision scoring dataset\n",
    "   - Hierarchical clustering can organize any sort of data into a hierarchy, not just samples of plants and animals.\n",
    "   - Let's consider a new type of dataset, describing how countries scored performances at the [Eurovision 2016 song contest](https://eurovision.tv/history/full-split-results).\n",
    "   - The data is arranged in a rectangular array, where the rows of the array show how many points a country gave to each song.\n",
    "   - The \"samples\" in this case are the countries.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/eurovision_scoring.JPG)\n",
    "\n",
    "1. Hierarchical clustering of voting countries\n",
    "   - The result of applying hierarchical clustering to the Eurovision scores can be visualized as a tree-like diagram called a \"dendrogram\".\n",
    "   - This single picture reveals a great deal of information about the voting behavior of countries at the Eurovision.\n",
    "   - The dendrogram groups the countries into larger and larger clusters, and many of these clusters are immediately recognizable as containing countries that are close to one another geographically, or that have close cultural or political ties, or that belong to single language group.\n",
    "   - So hierarchical clustering can produce great visualizations. But how does it work?\n",
    "\n",
    "1. Hierarchical clustering\n",
    "   - Hierarchical clustering proceeds in steps.\n",
    "   - In the beginning, every country is its own cluster - so there are as many clusters as there are countries!\n",
    "   - At each step, the two closest clusters are merged.\n",
    "   - This decreases the number of clusters, and eventually, there is only one cluster left, and it contains all the countries.\n",
    "   - This process is actually a particular type of hierarchical clustering called _\"agglomerative clustering\"_ - there is also _\"divisive clustering\"_, which works the other way around.\n",
    "   - We haven't defined yet what it means for two clusters to be close, but we'll revisit that later on.\n",
    "\n",
    "1. The dendrogram of a hierarchical clustering\n",
    "   - [`scipy.cluster.hierarchy.dendrogram`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.dendrogram.html)\n",
    "   - The entire process of the hierarchical clustering is encoded in the dendrogram.\n",
    "   - At the bottom, each country is in a cluster of its own.\n",
    "   - The clustering then proceeds from the bottom up.\n",
    "   - Clusters are represented as vertical lines, and a joining of vertical lines indicates a merging of clusters.\n",
    "   - To understand better, let's zoom in and look at just one part of this dendrogram.\n",
    "\n",
    "1. Dendrograms, step-by-step\n",
    "   - In the beginning, there are six clusters, each containing only one country.\n",
    "   - The first merging is here, where the clusters containing Cyprus and Greece are merged together in a single cluster.\n",
    "   - Later on, this new cluster is merged with the cluster containing Bulgaria.\n",
    "   - Shortly after that, the clusters containing Moldova and Russia are merged, which later is in turn merged with the cluster containing Armenia.\n",
    "   - Later still, the two big composite clusters are merged together. This process continues\n",
    "   - until there is only one cluster left, and it contains all the countries.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/dendrogram_step_by_step.jpg)\n",
    "\n",
    "1. Hierarchical clustering with SciPy\n",
    "   - We'll use functions from scipy to perform a hierarchical clustering on the array of scores.\n",
    "   - For the dendrogram, we'll also need a list of country names.\n",
    "   - Firstly, import the linkage and dendrogram functions.\n",
    "   - Then, apply the linkage function to the sample array.\n",
    "   - Its the linkage function that performs the hierarchical clustering.\n",
    "   - Notice there is an extra method parameter - we'll cover that in the next video.\n",
    "   - Now pass the output of linkage to the dendrogram function, specifying the list of country names as the labels parameter.\n",
    "   - In the next video, you'll learn how to extract information from a hierarchical clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A Note Regarding the Data**\n",
    "\n",
    "- The Eurovision data, `euv`, is used for the lecture and some of the following exercises.\n",
    "- The `.shape` of the Eurovision `samples` is `(42, 26)`\n",
    "- The Eurovision DataFrame must be pivoted to achieve the correct shape\n",
    "  - `'From country'` is `index`\n",
    "  - `'To country'` is `columns`\n",
    "  - `'Jury Points'` is `values`\n",
    "    - In `samples` produced by DataCamp, they have changed the order of the values for every row, so that the correct data point does not correctly correspond to `'To country'`\n",
    "    - Other than copying `samples` from the `iPython` shell, there isn't an automated way, that I can see, to sort the rows to match the DataCamp example, so the Dendrogram will not look the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "euvp = euv.pivot(index='From country', columns='To country', values='Jury Points').fillna(0)\n",
    "euv_samples = euvp.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "euvp.iloc[:5, :5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16, 6))\n",
    "euv_mergings = linkage(euv_samples, method='complete')\n",
    "dendrogram(euv_mergings, labels=euvp.index, leaf_rotation=90, leaf_font_size=12)\n",
    "plt.title('Countries Hierarchically Clustered by Eurovision 2016 Voting')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many merges?\n",
    "\n",
    "If there are 5 data samples, how many merge operations will occur in a hierarchical clustering?\n",
    "\n",
    "(To help answer this question, think back to the video, in which Ben walked through an example of hierarchical clustering using 6 countries.)\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- **4 merges.**\n",
    "  - _With 5 data samples, there would be 4 merge operations, and with 6 data samples, there would be 5 merges, and so on._\n",
    "- ~~3 merges.~~\n",
    "- ~~This can't be known in advance.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hierarchical clustering of the grain data\n",
    "\n",
    "In the video, you learned that the SciPy `linkage()` function performs hierarchical clustering on an array of samples. Use the `linkage()` function to obtain a hierarchical clustering of the grain samples, and use `dendrogram()` to visualize the result. A sample of the grain measurements is provided in the array `samples`, while the variety of each grain sample is given by the list `varieties`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import:\n",
    "  - `linkage` and `dendrogram` from `scipy.cluster.hierarchy`.\n",
    "  - `matplotlib.pyplot` as `plt`.\n",
    "- Perform hierarchical clustering on `samples` using the `linkage()` function with the `method='complete'` keyword argument. Assign the result to `mergings`.\n",
    "- Plot a dendrogram using the `dendrogram()` function on `mergings`. Specify the keyword arguments `labels=varieties`, `leaf_rotation=90`, and `leaf_font_size=6`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the DataCamp sample uses a subset of the seed data; the linkage result is very dependant upon the random_state\n",
    "seed_sample = sed.groupby('varieties').sample(n=14, random_state=250)\n",
    "samples = seed_sample.iloc[:, :7]\n",
    "varieties = seed_sample.varieties.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# from scipy.cluster.hierarchy import linkage, dendrogram\n",
    "# import matplotlib.pyplot as plt\n",
    "\n",
    "# Calculate the linkage: mergings\n",
    "mergings = linkage(samples, method='complete')\n",
    "\n",
    "# Plot the dendrogram, using varieties as labels\n",
    "plt.figure(figsize=(15, 6))\n",
    "dendrogram(mergings, labels=varieties, leaf_rotation=90, leaf_font_size=10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dendrograms are a great way to illustrate the arrangement of the clusters produced by hierarchical clustering.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hierarchies of stocks\n",
    "\n",
    "In chapter 1, you used k-means clustering to cluster companies according to their stock price movements. Now, you'll perform hierarchical clustering of the companies. You are given a NumPy array of price movements `movements`, where the rows correspond to companies, and a list of the company names `companies`. SciPy hierarchical clustering doesn't fit into a sklearn pipeline, so you'll need to use the `normalize()` function from `sklearn.preprocessing` instead of `Normalizer`.\n",
    "\n",
    "`linkage` and `dendrogram` have already been imported from `scipy.cluster.hierarchy`, and PyPlot has been imported as `plt`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `normalize` from `sklearn.preprocessing`.\n",
    "- Rescale the price movements for each stock by using the `normalize()` function on `movements`.\n",
    "- Apply the `linkage()` function to `normalized_movements`, using `'complete'` linkage, to calculate the hierarchical clustering. Assign the result to `mergings`.\n",
    "- Plot a dendrogram of the hierarchical clustering, using the list `companies` of company names as the `labels`. In addition, specify the `leaf_rotation=90`, and `leaf_font_size=6` keyword arguments as you did in the previous exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import normalize\n",
    "# from sklearn.preprocessing import normalize\n",
    "\n",
    "# Normalize the movements: normalized_movements\n",
    "normalized_movements = normalize(stk)\n",
    "\n",
    "# Calculate the linkage: mergings\n",
    "mergings = linkage(normalized_movements, method='complete')\n",
    "\n",
    "# Plot the dendrogram\n",
    "plt.figure(figsize=(15, 6))\n",
    "dendrogram(mergings, labels=stk.index, leaf_rotation=90, leaf_font_size=10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cluster labels in hierarchical clustering\n",
    "\n",
    "1. Cluster labels in hierarchical clustering\n",
    "   - To create a great visualization of the voting behavior at the Eurovision.\n",
    "   - But hierarchical clustering is not only a visualization tool.\n",
    "   - In this video, you'll learn how to extract the clusters from intermediate stages of a hierarchical clustering.\n",
    "   - The cluster labels for these intermediate clusterings can then be used in further computations, such as cross tabulations, just like the cluster labels from k-means.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/eurovision_2016.jpg)\n",
    "\n",
    "1. Intermediate clusterings & height on dendrogram\n",
    "   - An intermediate stage in the hierarchical clustering is specified by choosing a height on the dendrogram.\n",
    "   - For example, choosing a height of 15 defines a clustering in which Bulgaria, Cyprus and Greece are in one cluster, Russia and Moldova are in another, and Armenia is in a cluster on its own.\n",
    "   - But what is the meaning of the height?\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/dendrogram_height.JPG)\n",
    "\n",
    "1. Dendrograms show cluster distances\n",
    "   - The y-axis of the dendrogram encodes the distance between merging clusters.\n",
    "   - For example, the distance between the cluster containing Cyprus and the cluster containing Greece was approximately 6 when they were merged into a single cluster.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/dendrogram_distance1.JPG)\n",
    "   - When this new cluster was merged with the cluster containing Bulgaria, the distance between them was 12.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/dendrogram_distance2.JPG)\n",
    "\n",
    "1. Intermediate clusterings & height on dendrogram\n",
    "   - So the height that specifies an intermediate clustering corresponds to a distance.\n",
    "   - This specifies that the hierarchical clustering should stop merging clusters when all clusters are at least this far apart.\n",
    "\n",
    "1. Distance between clusters\n",
    "   - The distance between two clusters is measured using a \"linkage method\".\n",
    "   - In our example, we used \"complete\" linkage, where the distance between two clusters is the maximum of the distances between their samples.\n",
    "   - This was specified via the \"method\" parameter.\n",
    "   - There are many other linkage methods, and you'll see in the exercises that different linkage methods give different hierarchical clusterings!\n",
    "\n",
    "1. Extracting cluster labels\n",
    "   - The cluster labels for any intermediate stage of the hierarchical clustering can be extracted using the fcluster function.\n",
    "   - Let's try it out, specifying the height of 15.\n",
    "\n",
    "1. Extracting cluster labels using fcluster\n",
    "   - After performing the hierarchical clustering of the Eurovision data, import the fcluster function.\n",
    "   - Then pass the result of the linkage function to the fcluster function, specifying the height as the second argument.\n",
    "   - This returns a numpy array containing the cluster labels for all the countries.\n",
    "\n",
    "1. Aligning cluster labels with country names\n",
    "   - To inspect cluster labels, let's use a DataFrame to align the labels with the country names.\n",
    "   - Firstly, import pandas, then create the data frame, and then sort by cluster label, printing the result.\n",
    "   - As expected, the cluster labels group Bulgaria, Greece and Cyprus in the same cluster.\n",
    "   - But do note that the scipy cluster labels start at 1, not at 0 like they do in scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mergings = linkage(euv_samples, method='complete')\n",
    "labels = fcluster(mergings, 15, criterion='distance')\n",
    "print(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pairs = pd.DataFrame({'labels': labels, 'countries': euvp.index}).sort_values('labels')\n",
    "pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which clusters are closest?\n",
    "\n",
    "In the video, you learned that the linkage method defines how the distance between clusters is measured. In complete linkage, the distance between clusters is the distance between the furthest points of the clusters. In single linkage, the distance between clusters is the distance between the closest points of the clusters.\n",
    "\n",
    "Consider the three clusters in the diagram. Which of the following statements are true?\n",
    "\n",
    "![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/cluster_linkage_riddle.png)\n",
    "\n",
    "A. In single linkage, Cluster 3 is the closest to Cluster 2.\n",
    "\n",
    "B. In complete linkage, Cluster 1 is the closest to Cluster 2.\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~Neither A nor B~~.\n",
    "- ~~A only~~.\n",
    "- **Both A and B**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Different linkage, different hierarchical clustering\n",
    "\n",
    "In the video, you saw a hierarchical clustering of the voting countries at the Eurovision song contest using `'complete'` linkage. Now, perform a hierarchical clustering of the voting countries with `'single'` linkage, and compare the resulting dendrogram with the one in the video. Different linkage, different hierarchical clustering!\n",
    "\n",
    "You are given an array `samples`. Each row corresponds to a voting country, and each column corresponds to a performance that was voted for. The list `country_names` gives the name of each voting country. This dataset was obtained from [Eurovision](https://eurovision.tv/history/full-split-results).\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `linkage` and `dendrogram` from `scipy.cluster.hierarchy`.\n",
    "- Perform hierarchical clustering on `samples` using the `linkage()` function with the `method='single'` keyword argument. Assign the result to `mergings`.\n",
    "- Plot a dendrogram of the hierarchical clustering, using the list `country_names` as the `labels`. In addition, specify the `leaf_rotation=90`, and `leaf_font_size=6` keyword arguments as you have done earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "country_names = euv['From country'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# import matplotlib.pyplot as plt\n",
    "# from scipy.cluster.hierarchy import linkage, dendrogram\n",
    "\n",
    "# Calculate the linkage: mergings\n",
    "mergings = linkage(euv_samples, method='single')\n",
    "\n",
    "# Plot the dendrogram\n",
    "plt.figure(figsize=(16, 6))\n",
    "dendrogram(mergings, labels=country_names, leaf_rotation=90, leaf_font_size=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**As you can see, performing single linkage hierarchical clustering produces a different dendrogram!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intermediate clusterings\n",
    "\n",
    "Displayed on the right is the dendrogram for the hierarchical clustering of the grain samples that you computed earlier. If the hierarchical clustering were stopped at height 6 on the dendrogram, how many clusters would there be?\n",
    "\n",
    "![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/intermediate_clusterings.JPG)\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~1~~\n",
    "- **3**\n",
    "- ~~As many as there were at the beginning.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting the cluster labels\n",
    "\n",
    "In the previous exercise, you saw that the intermediate clustering of the grain samples at height 6 has 3 clusters. Now, use the `fcluster()` function to extract the cluster labels for this intermediate clustering, and compare the labels with the grain varieties using a cross-tabulation.\n",
    "\n",
    "The hierarchical clustering has already been performed and `mergings` is the result of the `linkage()` function. The list `varieties` gives the variety of each grain sample.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import:\n",
    "  - `pandas` as `pd`.\n",
    "  - `fcluster` from `scipy.cluster.hierarchy`.\n",
    "- Perform a flat hierarchical clustering by using the `fcluster()` function on `mergings`. Specify a maximum height of `6` and the keyword argument `criterion='distance'`.\n",
    "- Create a DataFrame `df` with two columns named `'labels'` and `'varieties'`, using `labels` and `varieties`, respectively, for the column values. This has been done for you.\n",
    "- Create a cross-tabulation `ct` between `df['labels']` and `df['varieties']` to count the number of times each grain variety coincides with each cluster label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the DataCamp sample uses a subset of the seed data; the linkage result is very dependant upon the random_state\n",
    "seed_sample = sed.groupby('varieties').sample(n=14, random_state=250)\n",
    "samples = seed_sample.iloc[:, :7]\n",
    "varieties = seed_sample.varieties.tolist()\n",
    "\n",
    "# Calculate the linkage: mergings\n",
    "mergings = linkage(samples, method='complete')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# import pandas as pd\n",
    "# from scipy.cluster.hierarchy import fcluster\n",
    "\n",
    "# Use fcluster to extract labels: labels\n",
    "labels = fcluster(mergings, 6, criterion='distance')\n",
    "\n",
    "# Create a DataFrame with labels and varieties as columns: df\n",
    "df = pd.DataFrame({'labels': labels, 'varieties': varieties})\n",
    "\n",
    "# Create crosstab: ct\n",
    "ct = pd.crosstab(df.labels, df.varieties)\n",
    "\n",
    "# Display ct\n",
    "ct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## t-SNE for 2-dimensional maps\n",
    "\n",
    "- In this video, you'll learn about an unsupervised learning method for visualization called \"t-SNE\".\n",
    "\n",
    "1. t-SNE for 2-dimensional maps\n",
    "   - t-SNE stands for \"t-distributed stochastic neighbor embedding\".\n",
    "   - It has a complicated name, but it serves a very simple purpose.\n",
    "   - It maps samples from their high-dimensional space into a 2- or 3-dimensional space so they can visualized.\n",
    "   - While some distortion is inevitable, t-SNE does a great job of approximately representing the distances between the samples.\n",
    "   - For this reason, t-SNE is an invaluable visual aid for understanding a dataset.\n",
    "\n",
    "1. t-SNE on the iris dataset\n",
    "   - To see what sorts of insights are possible with t-SNE, let's look at how it performs on the iris dataset.\n",
    "   - The iris samples are in a four dimensional space, where each dimension corresponds to one of the four iris measurements, such as petal length and petal width.\n",
    "   - Now t-SNE was given only the measurements of the iris samples.\n",
    "   - In particular it wasn't given any information about the three species of iris.\n",
    "   - But if we color the species differently on the scatter plot, we see that t-SNE has kept the species separate.\n",
    "\n",
    "1. Interpreting t-SNE scatter plots\n",
    "   - This scatter plot gives us a new insight.\n",
    "   - We learn that there are two iris species, versicolor and virginica, whose samples are close together in space.\n",
    "   - So it could happen that the iris dataset appears to have two clusters, instead of three.\n",
    "   - This is compatible with our previous examples using k-means, where we saw that a clustering with 2 clusters also had relatively low inertia, meaning tight clusters.\n",
    "\n",
    "1. t-SNE in sklearn\n",
    "   - t-SNE is available in scikit-learn, but it works a little differently to the fit/transform components you've already met.\n",
    "   - Let's see it in action on the iris dataset.\n",
    "   - The samples are in a 2-dimensional numpy array, and there is a list giving the species of each sample.\n",
    "   - To start with, import TSNE and create a TSNE object.\n",
    "   - Apply the fit_transform method to the samples, and then make a scatter plot of the result, coloring the points using the species.\n",
    "   - There are two aspects that deserve special attention: the fit_transform method, and the learning rate.\n",
    "\n",
    "1. t-SNE has only fit_transform()\n",
    "   - t-SNE only has a fit_transform method.\n",
    "   - As you might expect, the fit_transform method simultaneously fits the model and transforms the data.\n",
    "   - However, t-SNE does not have separate fit and transform methods.\n",
    "   - This means that you can't extend a t-SNE map to include new samples.\n",
    "   - Instead, you have to start over each time.\n",
    "\n",
    "1. t-SNE learning rate\n",
    "   - The second thing to notice is the learning rate.\n",
    "   - The learning rate makes the use of t-SNE more complicated than some other techniques.\n",
    "   - You may need to try different learning rates for different datasets.\n",
    "   - It is clear, however, when you've made a bad choice, because all the samples appear bunched together in the scatter plot.\n",
    "   - Normally it's enough to try a few values between 50 and 200.\n",
    "\n",
    "1. Different every time\n",
    "   - A final thing to be aware of is that the axes of a t-SNE plot do not have any interpretable meaning.\n",
    "   - In fact, they are different every time t-SNE is applied, even on the same data.\n",
    "   - For example, here are three t-SNE plots of the scaled Piedmont wine samples, generated using the same code.\n",
    "   - Note that while the orientation of the plot is different each time, the three wine varieties, represented here using colors, have the same position relative to one another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rs = [100, 200, 300]\n",
    "fig, axes = plt.subplots(ncols=3, figsize=(15, 3))\n",
    "axes = axes.ravel()\n",
    "\n",
    "for i, state in enumerate(rs):\n",
    "    ax = axes[i]\n",
    "    \n",
    "    model = TSNE(learning_rate=100, random_state=state)\n",
    "    transformed = model.fit_transform(iris.iloc[:, :4])\n",
    "\n",
    "    xs = transformed[:, 0]\n",
    "    ys = transformed[:, 1]\n",
    "\n",
    "    sns.scatterplot(x=xs, y=ys, hue=iris.species, ax=ax)\n",
    "    ax.set_title(f't-SNE applied to Iris with random_state={state}')\n",
    "    \n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### t-SNE visualization of grain dataset\n",
    "\n",
    "In the video, you saw t-SNE applied to the iris dataset. In this exercise, you'll apply t-SNE to the grain samples data and inspect the resulting t-SNE features using a scatter plot. You are given an array `samples` of grain samples and a list `variety_numbers` giving the variety number of each grain sample.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `TSNE` from `sklearn.manifold`.\n",
    "- Create a TSNE instance called `model` with `learning_rate=200`.\n",
    "- Apply the `.fit_transform()` method of `model` to `samples`. Assign the result to `tsne_features`.\n",
    "- Select the column `0` of `tsne_features`. Assign the result to `xs`.\n",
    "- Select the column `1` of `tsne_features`. Assign the result to `ys`.\n",
    "- Make a scatter plot of the t-SNE features `xs` and `ys`. To color the points by the grain variety, specify the additional keyword argument `c=variety_numbers`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = sed.iloc[:, :7]\n",
    "variety_numbers = sed[7]\n",
    "variety_names = sed.varieties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import TSNE\n",
    "# from sklearn.manifold import TSNE\n",
    "\n",
    "# Create a TSNE instance: model\n",
    "model = TSNE(learning_rate=200, random_state=300)\n",
    "\n",
    "# Apply fit_transform to samples: tsne_features\n",
    "tsne_features = model.fit_transform(samples)\n",
    "\n",
    "# Select the 0th feature: xs\n",
    "xs = tsne_features[:,0]\n",
    "\n",
    "# Select the 1st feature: ys\n",
    "ys = tsne_features[:,1]\n",
    "\n",
    "# Scatter plot, coloring by variety_numbers\n",
    "# plt.scatter(xs, ys, c=variety_numbers)\n",
    "sns.scatterplot(x=xs, y=ys, hue=variety_names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A t-SNE map of the stock market\n",
    "\n",
    "t-SNE provides great visualizations when the individual samples can be labeled. In this exercise, you'll apply t-SNE to the company stock price data. A scatter plot of the resulting t-SNE features, labeled by the company names, gives you a map of the stock market! The stock price movements for each company are available as the array `normalized_movements` (these have already been normalized for you). The list `companies` gives the name of each company. PyPlot (`plt`) has been imported for you.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `TSNE` from `sklearn.manifold`.\n",
    "- Create a TSNE instance called `model` with `learning_rate=50`.\n",
    "- Apply the `.fit_transform()` method of `model` to `normalized_movements`. Assign the result to `tsne_features`.\n",
    "- Select column `0` and column `1` of `tsne_features`.\n",
    "- Make a scatter plot of the t-SNE features `xs` and `ys`. Specify the additional keyword argument `alpha=0.5`.\n",
    "- Code to label each point with its company name has been written for you using `plt.annotate()`, so just hit 'Submit Answer' to see the visualization!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import TSNE\n",
    "# from sklearn.manifold import TSNE\n",
    "\n",
    "# Create a TSNE instance: model\n",
    "model = TSNE(learning_rate=50, random_state=300)\n",
    "\n",
    "# Apply fit_transform to normalized_movements: tsne_features\n",
    "tsne_features = model.fit_transform(normalized_movements)\n",
    "\n",
    "# Select the 0th feature: xs\n",
    "xs = tsne_features[:, 0]\n",
    "\n",
    "# Select the 1th feature: ys\n",
    "ys = tsne_features[:, 1]\n",
    "\n",
    "# Scatter plot\n",
    "plt.figure(figsize=(16, 10))\n",
    "plt.scatter(xs, ys, alpha=0.5)\n",
    "\n",
    "# Annotate the points\n",
    "for x, y, company in zip(xs, ys, companies):\n",
    "    plt.annotate(company, (x, y), fontsize=10, alpha=0.75)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**It's visualizations such as this that make t-SNE such a powerful tool for extracting quick insights from high dimensional data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decorrelating your data and dimension reduction\n",
    "\n",
    "Dimension reduction summarizes a dataset using its common occuring patterns. In this chapter, you'll learn about the most fundamental of dimension reduction techniques, \"Principal Component Analysis\" (\"PCA\"). PCA is often used before supervised learning to improve model performance and generalization. It can also be useful for unsupervised learning. For example, you'll employ a variant of PCA will allow you to cluster Wikipedia articles by their content!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the PCA transformation\n",
    "\n",
    "- In the next two chapters you'll learn techniques for dimension reduction.\n",
    "\n",
    "1. Dimension reduction\n",
    "   - Dimension reduction finds patterns in data, and uses these patterns to re-express it in a compressed form.\n",
    "   - This makes subsequent computation with the data much more efficient, and this can be a big deal in a world of big datasets.\n",
    "   - However, the most important function of dimension reduction is to reduce a dataset to its \"bare bones\", discarding noisy features that cause big problems for supervised learning tasks like regression and classification.\n",
    "   - In many real-world applications, it's dimension reduction that makes prediction possible.\n",
    "\n",
    "1. Principal Component Analysis\n",
    "   - In this chapter, you'll learn about the most fundamental of dimension reduction techniques.\n",
    "   - It's called \"Principal Component Analysis\", or \"PCA\" for short.\n",
    "   - PCA performs dimension reduction in two steps, and the first one, called \"de-correlation\", doesn't change the dimension of the data at all.\n",
    "   - It's this first step that we'll focus on in this video.\n",
    "\n",
    "1. PCA aligns data with axes\n",
    "   - In this first step, PCA rotates the samples so that they are aligned with the coordinate axes.\n",
    "   - In fact, it does more than this: PCA also shifts the samples so that they have mean zero.\n",
    "   - These scatter plots show the effect of PCA applied to two features of the wine dataset.\n",
    "   - Notice that no information is lost - this is true no matter how many features your dataset has.\n",
    "   - You'll practice visualizing this transformation in the exercises.\n",
    "\n",
    "1. PCA follows the fit/transform pattern\n",
    "   - scikit-learn has an implementation of `PCA`, and it has `fit()` and `transform()` methods just like `StandardScaler`.\n",
    "   - The fit method learns how to shift and how to rotate the samples, but doesn't actually change them.\n",
    "   - The transform method, on the other hand, applies the transformation that fit learned.\n",
    "   - In particular, the transform method can be applied to new, unseen samples.\n",
    "\n",
    "1. Using scikit-learn PCA\n",
    "   - `from sklearn.decomposition import PCA`\n",
    "   - Let's see PCA in action on the some features of the wine dataset.\n",
    "   - Firstly, import PCA.\n",
    "   - Now create a PCA object, and fit it to the samples.\n",
    "   - Then use the fit PCA object to transform the samples.\n",
    "   - This returns a new array of transformed samples.\n",
    "\n",
    "1. PCA features\n",
    "   - This new array has the same number of rows and columns as the original sample array.\n",
    "   - In particular, there is one row for each transformed sample.\n",
    "   - The columns of the new array correspond to \"PCA features\", just as the original features corresponded to columns of the original array.\n",
    "\n",
    "1. PCA features are not correlated\n",
    "   - It is often the case that the features of a dataset are correlated.\n",
    "   - This is the case with many of the features of the wine dataset, for instance.\n",
    "   - However, PCA, due to the rotation it performs, \"de-correlates\" the data, in the sense that the columns of the transformed array are not linearly correlated.\n",
    "\n",
    "1. Pearson correlation\n",
    "   - Linear correlation can be measured with the Pearson correlation.\n",
    "   - It takes values between -1 and 1, where larger values indicate a stronger correlation, and 0 indicates no linear correlation.\n",
    "   - Here are some examples of features with varying degrees of correlation.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/pearson_corr_ex.jpg)\n",
    "\n",
    "1. Principal components\n",
    "   - Finally, PCA is called \"principal component analysis\" because it learns the \"principal components\" of the data.\n",
    "   - These are the directions in which the samples vary the most, depicted here in red.\n",
    "     - \"Principal components\" = directions of variance\n",
    "   - It is the principal components that PCA aligns with the coordinate axes.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/principal_components_2.JPG)\n",
    "   - After a PCA model has been fit, the principal components are available as the components attribute.\n",
    "   - This is numpy array with one row for each principal component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_samples = win[['total_phenols', 'od280']]\n",
    "wine_samples.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_samples.corr().round(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_model = PCA()\n",
    "wine_model.fit(wine_samples)\n",
    "wine_transformed = wine_model.transform(wine_samples)\n",
    "\n",
    "wine_transformed_df = pd.DataFrame(wine_transformed, columns=['total_phenols', 'od280'])\n",
    "wine_transformed_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_transformed_df.corr().round(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_model.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 4))\n",
    "sns.scatterplot(data=wine_samples, x='total_phenols', y='od280', hue=win.class_name, ax=ax1)\n",
    "ax1.set_ylim(-4, 6)\n",
    "ax1.set_xlim(-4, 6)\n",
    "ax1.set_title('Not Scaled')\n",
    "\n",
    "sns.scatterplot(data=wine_transformed_df, x='total_phenols', y='od280', hue=win.class_name, ax=ax2)\n",
    "ax2.set_ylim(-4, 6)\n",
    "ax2.set_xlim(-4, 6)\n",
    "ax2.set_title('PCA Scaled')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlated data in nature\n",
    "\n",
    "You are given an array `grains` giving the width and length of samples of grain. You suspect that width and length will be correlated. To confirm this, make a scatter plot of width vs length and measure their Pearson correlation.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import:\n",
    "  - `matplotlib.pyplot` as `plt`.\n",
    "  - `pearsonr` from `scipy.stats`.\n",
    "- Assign column `0` of `grains` to `width` and column `1` of `grains` to `length`.\n",
    "- Make a scatter plot with `width` on the x-axis and `length` on the y-axis.\n",
    "- Use the `pearsonr()` function to calculate the Pearson correlation of `width` and `length`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grains = sed[[4, 3]].to_numpy()\n",
    "varieties = sed[7]\n",
    "grains[:2, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# import matplotlib.pyplot as plt\n",
    "# from scipy.stats import pearsonr\n",
    "\n",
    "# Assign the 0th column of grains: width\n",
    "width = grains[:, 0]\n",
    "\n",
    "# Assign the 1st column of grains: length\n",
    "length = grains[:, 1]\n",
    "\n",
    "# Scatter plot width vs length\n",
    "plt.scatter(width, length, c=varieties)\n",
    "plt.axis('equal')\n",
    "plt.show()\n",
    "\n",
    "# Calculate the Pearson correlation\n",
    "correlation, pvalue = pearsonr(width, length)\n",
    "\n",
    "# Display the correlation\n",
    "print(correlation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = sns.scatterplot(data=sed, x=4, y=3, hue='varieties')\n",
    "p.set_xlabel('width')\n",
    "p.set_ylabel('length')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sed[[4, 3]].corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decorrelating the grain measurements with PCA\n",
    "\n",
    "You observed in the previous exercise that the width and length measurements of the grain are correlated. Now, you'll use PCA to decorrelate these measurements, then plot the decorrelated points and measure their Pearson correlation.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `PCA` from `sklearn.decomposition`.\n",
    "- Create an instance of `PCA` called `model`.\n",
    "- Use the `.fit_transform()` method of `model` to apply the PCA transformation to `grains`. Assign the result to `pca_features`.\n",
    "- The subsequent code to extract, plot, and compute the Pearson correlation of the first two columns `pca_features` has been written for you, so hit 'Submit Answer' to see the result!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PCA\n",
    "# from sklearn.decomposition import PCA\n",
    "\n",
    "# Create PCA instance: model\n",
    "model = PCA()\n",
    "\n",
    "# Apply the fit_transform method of model to grains: pca_features\n",
    "pca_features = model.fit_transform(grains)\n",
    "\n",
    "# Assign 0th column of pca_features: xs\n",
    "xs = pca_features[:,0]\n",
    "\n",
    "# Assign 1st column of pca_features: ys\n",
    "ys = pca_features[:,1]\n",
    "\n",
    "# Scatter plot xs vs ys\n",
    "plt.scatter(xs, ys, c=varieties)\n",
    "plt.axis('equal')\n",
    "plt.show()\n",
    "\n",
    "# Calculate the Pearson correlation of xs and ys\n",
    "correlation, pvalue = pearsonr(xs, ys)\n",
    "\n",
    "# Display the correlation\n",
    "print(f'Correlation: {round(correlation)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Principal components\n",
    "\n",
    "There are three scatter plots of the same point cloud. Each scatter plot shows a different set of axes (in red). In which of the plots could the axes represent the principal components of the point cloud?\n",
    "\n",
    "Recall that the principal components are the directions along which the the data varies.\n",
    "\n",
    "![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/principal_components.JPG)\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~None of them.~~\n",
    "- **Both plot 1 and plot 3.**\n",
    "  - **You've correctly inferred that the principal components have to align with the axes of the point cloud. This happens in both plot 1 and plot 3.**\n",
    "- ~~Plot 2.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intrinsic dimension\n",
    "\n",
    "1. Intrinsic dimension of a flight path\n",
    "   - Consider this dataset with 2 features: latitude and longitude.\n",
    "   - These two features might track the flight of an airplane, for example.\n",
    "   - This dataset is 2-dimensional, yet it turns out that it can be closely approximated using only one feature: the displacement along the flight path.\n",
    "   - This dataset is intrinsically one-dimensional.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/flight_path_dim.jpg)\n",
    "\n",
    "1. Intrinsic dimension\n",
    "   - The intrinsic dimension of a dataset is the number of features required to approximate it.\n",
    "   - The intrinsic dimension informs dimension reduction, because it tells us how much a dataset can be compressed.\n",
    "   - In this video, you'll gain a solid understanding of the intrinsic dimension, and be able to use PCA to identify it in real-world datasets that have thousands of features.\n",
    "\n",
    "1. Versicolor dataset\n",
    "   - To better illustrate the intrinsic dimension, let's consider an example dataset containing only some of the samples from the iris dataset.\n",
    "   - Specifically, let's take three measurements from the iris versicolor samples: sepal length, sepal width, and petal width.\n",
    "   - So each sample is represented as a point in 3-dimensional space.\n",
    "\n",
    "1. Versicolor dataset has intrinsic dimension 2\n",
    "   - However, if we make a 3d scatter plot of the samples, we see that they all lie very close to a flat, 2-dimensional sheet.\n",
    "   - This means that the data can be approximated by using only two coordinates, without losing much information.\n",
    "   - So this dataset has intrinsic dimension 2.\n",
    "\n",
    "1. PCA identifies intrinsic dimension\n",
    "   - But scatter plots are only possible if there are 3 features or less.\n",
    "   - So how can the intrinsic dimension be identified, even if there are many features?\n",
    "   - This is where PCA is really helpful.\n",
    "   - The intrinsic dimension can be identified by counting the PCA features that have high variance.\n",
    "   - To see how, let's see what happens when PCA is applied to the dataset of versicolor samples.\n",
    "\n",
    "1. PCA of the versicolor samples\n",
    "   - PCA rotates and shifts the samples to align them with the coordinate axes.\n",
    "   - This expresses the samples using three PCA features.\n",
    "\n",
    "1. PCA features are ordered by variance descending\n",
    "   - The PCA features are in a special order.\n",
    "   - Here is a bar graph showing the variance of each of the PCA features.\n",
    "   - As you can see, each PCA feature has less variance than the last, and in this case the last PCA feature has very low variance.\n",
    "   - This agrees with the scatter plot of the PCA features, where the samples don't vary much in the vertical direction.\n",
    "   - In the other two directions, however, the variance is apparent.\n",
    "\n",
    "1. Variance and intrinsic dimension\n",
    "   - The intrinsic dimension is the number of PCA features that have significant variance.\n",
    "   - In our example, only the first two PCA features have significant variance.\n",
    "   - So this dataset has intrinsic dimension 2, which agrees with what we observed when inspecting the scatter plot.\n",
    "\n",
    "1. Plotting the variances of PCA features\n",
    "   - Let's see how to plot the variances of the PCA features in practice.\n",
    "   - Firstly, make the necessary imports.\n",
    "   - Then create a PCA model, and fit it to the samples.\n",
    "   - Now create a range enumerating the PCA features, and make a bar plot of the variances; the variances are available as the explained_variance attribute of the PCA model.\n",
    "\n",
    "1. Intrinsic dimension can be ambiguous\n",
    "   - The intrinsic dimension is a useful idea that helps to guide dimension reduction.\n",
    "   - However, it is not always unambiguous.\n",
    "   - Here is a graph of the variances of the PCA features for the wine dataset.\n",
    "   - We could argue for an intrinsic dimension of 2, of 3, or even more, depending upon the threshold you chose.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/piedmont_var.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = sns.load_dataset('iris')\n",
    "iris.head()\n",
    "y = iris.species.astype('category').cat.codes\n",
    "\n",
    "vers = iris[iris.species.eq('versicolor')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(1, figsize=(8, 6))\n",
    "ax = Axes3D(fig, elev=15, azim=40)\n",
    "ax.scatter(iris.sepal_length, iris.sepal_width, iris.petal_width, c=y, edgecolor='k', s=40)\n",
    "\n",
    "ax.set_title(\"Iris\")\n",
    "\n",
    "ax.set_xlabel(\"Sepal Length\")\n",
    "ax.set_ylabel(\"Sepal Width\")\n",
    "ax.set_zlabel(\"Petal Width\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA()\n",
    "iris_reduced = pca.fit_transform(iris[['sepal_length', 'sepal_width', 'petal_width']])\n",
    "\n",
    "fig = plt.figure(1, figsize=(8, 6))\n",
    "ax = Axes3D(fig, elev=25, azim=55)\n",
    "ax.scatter(iris_reduced[:, 0], iris_reduced[:, 1], iris_reduced[:, 2], c=y, edgecolor='k', s=40)\n",
    "\n",
    "ax.set_title(\"Iris Reduced\")\n",
    "\n",
    "ax.set_xlabel(\"Sepal Length\")\n",
    "ax.set_ylabel(\"Sepal Width\")\n",
    "ax.set_zlabel(\"Petal Width\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(1, figsize=(8, 6))\n",
    "ax = Axes3D(fig, elev=25, azim=-235)\n",
    "ax.scatter(vers.sepal_length, vers.sepal_width, vers.petal_width, edgecolor='k', s=40)\n",
    "\n",
    "ax.set_title(\"Versicolor\")\n",
    "\n",
    "ax.set_xlabel(\"Sepal Length\")\n",
    "ax.set_ylabel(\"Sepal Width\")\n",
    "ax.set_zlabel(\"Petal Width\")\n",
    "\n",
    "ax.set_xlim(4.5, 7.5)\n",
    "ax.set_ylim(1.5, 4.0)\n",
    "ax.set_zlim(0, 2.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA()\n",
    "pca.fit(vers[['sepal_length', 'sepal_width', 'petal_width']])\n",
    "\n",
    "vers_reduced = pca.transform(vers[['sepal_length', 'sepal_width', 'petal_width']])\n",
    "\n",
    "fig = plt.figure(1, figsize=(8, 6))\n",
    "ax = Axes3D(fig, elev=15, azim=-245)\n",
    "ax.scatter(vers_reduced[:, 0], vers_reduced[:, 1], vers_reduced[:, 2], edgecolor='k', s=40)\n",
    "\n",
    "ax.set_title(\"Versicolor Reduced\")\n",
    "\n",
    "ax.set_xlabel(\"Sepal Length\")\n",
    "ax.set_ylabel(\"Sepal Width\")\n",
    "ax.set_zlabel(\"Petal Width\")\n",
    "\n",
    "ax.set_xlim(-1.5, 1.5)\n",
    "ax.set_ylim(-1.5, 1.5)\n",
    "ax.set_zlim(-1.5, 1.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = range(pca.n_components_)\n",
    "features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca.explained_variance_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "versi_df = pd.DataFrame(vers_reduced, columns=['sepal_length', 'sepal_width', 'petal_width'])\n",
    "versi_df.var().plot(kind='bar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "versi_df.var()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The first principal component\n",
    "\n",
    "The first principal component of the data is the direction in which the data varies the most. In this exercise, your job is to use PCA to find the first principal component of the length and width measurements of the grain samples, and represent it as an arrow on the scatter plot.\n",
    "\n",
    "The array `grains` gives the length and width of the grain samples. PyPlot (`plt`) and `PCA` have already been imported for you.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Make a scatter plot of the grain measurements. This has been done for you.\n",
    "- Create a `PCA` instance called `model`.\n",
    "- Fit the model to the `grains` data.\n",
    "- Extract the coordinates of the mean of the data using the `.mean_` attribute of `model`.\n",
    "- Get the first principal component of `model` using the `.components_[0,:]` attribute.\n",
    "- Plot the first principal component as an arrow on the scatter plot, using the `plt.arrow()` function. You have to specify the first two arguments - `mean[0]` and `mean[1]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a scatter plot of the untransformed points\n",
    "plt.scatter(grains[:,0], grains[:,1])\n",
    "\n",
    "# Create a PCA instance: model\n",
    "model = PCA()\n",
    "\n",
    "# Fit model to points\n",
    "model.fit(grains)\n",
    "\n",
    "# Get the mean of the grain samples: mean\n",
    "mean = model.mean_\n",
    "\n",
    "# Get the first principal component: first_pc\n",
    "first_pc = model.components_[0, :]\n",
    "\n",
    "# Plot first_pc as an arrow, starting at mean\n",
    "plt.arrow(mean[0], mean[1], first_pc[0], first_pc[1], color='red', width=0.01)\n",
    "\n",
    "# Keep axes on same scale\n",
    "plt.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This is the direction in which the grain data varies the most.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variance of the PCA features\n",
    "\n",
    "The fish dataset is 6-dimensional. But what is its _intrinsic_ dimension? Make a plot of the variances of the PCA features to find out. As before, `samples` is a 2D array, where each row represents a fish. You'll need to standardize the features first.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Create an instance of `StandardScaler` called `scaler`.\n",
    "- Create a `PCA` instance called `pca`.\n",
    "- Use the `make_pipeline()` function to create a pipeline chaining `scaler` and `pca`.\n",
    "- Use the `.fit()` method of `pipeline` to fit it to the fish samples `samples`.\n",
    "- Extract the number of components used using the `.n_components_` attribute of `pca`. Place this inside a `range()` function and store the result as `features`.\n",
    "- Use the `plt.bar()` function to plot the explained variances, with `features` on the x-axis and `pca.explained_variance_` on the y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = fsh.iloc[:, 1:].to_numpy()\n",
    "samples[:3, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# from sklearn.decomposition import PCA\n",
    "# from sklearn.preprocessing import StandardScaler\n",
    "# from sklearn.pipeline import make_pipeline\n",
    "# import matplotlib.pyplot as plt\n",
    "\n",
    "# Create scaler: scaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Create a PCA instance: pca\n",
    "pca = PCA()\n",
    "\n",
    "# Create pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, pca)\n",
    "\n",
    "# Fit the pipeline to 'samples'\n",
    "pipeline.fit(samples)\n",
    "\n",
    "# Plot the explained variances\n",
    "features = range(pca.n_components_)\n",
    "plt.bar(features, pca.explained_variance_)\n",
    "plt.xlabel('PCA feature')\n",
    "plt.ylabel('variance')\n",
    "plt.xticks(features)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**It looks like PCA features 0 and 1 have significant variance.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intrinsic dimension of the fish data\n",
    "\n",
    "In the previous exercise, you plotted the variance of the PCA features of the fish measurements. Looking again at your plot, what do you think would be a reasonable choice for the \"intrinsic dimension\" of the fish measurements? Recall that the intrinsic dimension is the number of PCA features with significant variance.\n",
    "\n",
    "![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/intrinsic_dim_fish.JPG)\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~1~~\n",
    "- **2**\n",
    "  - **Since PCA features 0 and 1 have significant variance, the intrinsic dimension of this dataset appears to be 2.**\n",
    "- ~~5~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimension reduction with PCA\n",
    "\n",
    "1. Dimension reduction\n",
    "   - Dimension reduction represents the same data using less features and is vital for building machine learning pipelines using real-world data.\n",
    "   - Finally, in this video, you'll learn how to perform dimension reduction using PCA.\n",
    "\n",
    "1. Dimension reduction with PCA\n",
    "   - We've already seen that the PCA features are in decreasing order of variance.\n",
    "   - PCA performs dimension reduction by discarding the PCA features with lower variance, which it assumes to be noise, and retaining the higher variance PCA features, which it assumes to be informative.\n",
    "   - To use PCA for dimension reduction, you need to specify how many PCA features to keep.\n",
    "   - For example, specifying n_components=2 when creating a PCA model tells it to keep only the first two PCA features.\n",
    "   - A good choice is the intrinsic dimension of the dataset, if you know it.\n",
    "   - Let's consider an example right away.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/dimension_reduction_pca.jpg)\n",
    "\n",
    "1. Dimension reduction of iris dataset\n",
    "   - The iris dataset has 4 features representing the 4 measurements.\n",
    "   - Here, the measurements are in a numpy array called samples.\n",
    "   - Let's use PCA to reduce the dimension of the iris dataset to only 2.\n",
    "   - Begin by importing PCA as usual.\n",
    "   - Create a PCA model specifying n_components=2, and then fit the model and transform the samples as usual.\n",
    "   - Printing the shape of the transformed samples, we see that there are only two features, as expected.\n",
    "\n",
    "1. Iris dataset in 2 dimensions\n",
    "   - Here is a scatterplot of the two PCA features, where the colors represent the three species of iris.\n",
    "   - Remarkably, despite having reduced the dimension from 4 to 2, the species can still be distinguished.\n",
    "   - Remember that PCA didn't even know that there were distinct species.\n",
    "   - PCA simply took the 2 PCA features with highest variance.\n",
    "   - As we can see, these two features are very informative.\n",
    "\n",
    "1. Dimension reduction with PCA\n",
    "   - PCA discards the low variance features, and assumes that the higher variance features are informative.\n",
    "   - Like all assumptions, there are cases where this doesn't hold.\n",
    "   - As we saw with the iris dataset, however, it often does in practice.\n",
    "\n",
    "1. Word frequency arrays\n",
    "   - In some cases, an alternative implementation of PCA needs to be used.\n",
    "   - Word frequency arrays are a great example.\n",
    "   - In a word-frequency array, each row corresponds to a document, and each column corresponds to a word from a fixed vocabulary.\n",
    "   - The entries of the word-frequency array measure how often each word appears in each document.\n",
    "   - **Only some of the words from the vocabulary appear in any one document, so most entries of the word frequency array are zero.**\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/word_freq_array.jpg)\n",
    "\n",
    "1. Sparse arrays and `csr_matrix`\n",
    "   - Arrays like this are said to be **sparse**, and are often represented using a special type of array called a **csr_matrix**.\n",
    "   - **Sparse**: most entries are zero\n",
    "   - CSR: compressed sparse row\n",
    "   - Can use [`scipy.sparse.csr_matrix`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) instead of NumPy array\n",
    "   - `csr_matrices` save space by remembering only the non-zero entries of the array.\n",
    "\n",
    "1. TruncatedSVD and csr_matrix\n",
    "   - **Scikit-learn's `PCA` doesn't support`csr_matrices`, and you'll need to use `TruncatedSVD` instead.**\n",
    "   - `TruncatedSVD` performs the same transformation as PCA, but accepts csr matrices as input.\n",
    "     - [`sklearn.decomposition.TruncatedSVD`](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html)\n",
    "   - Other than that, you interact with TruncatedSVD and PCA in exactly the same way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dimension Reduction of the Iris Dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.iloc[:, :4].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=2)\n",
    "pca.fit(iris.iloc[:, :4])\n",
    "transformed = pca.transform(iris.iloc[:, :4])\n",
    "transformed.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = transformed[:,0]\n",
    "ys = transformed[:,1]\n",
    "sns.scatterplot(x=xs, y=ys, hue=iris.species)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TruncatedSVD and csr_matrix**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wik1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wik1.iloc[:3, :6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = TruncatedSVD(n_components=3)\n",
    "model.fit(wik1) # documents is csr_matrix\n",
    "TruncatedSVD(algorithm='randomized')\n",
    "transformed = model.transform(wik1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed[:3, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dimension reduction of the fish measurements\n",
    "\n",
    "In a previous exercise, you saw that `2` was a reasonable choice for the \"intrinsic dimension\" of the fish measurements. Now use PCA for dimensionality reduction of the fish measurements, retaining only the 2 most important components.\n",
    "\n",
    "The fish measurements have already been scaled for you, and are available as `scaled_samples`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `PCA` from `sklearn.decomposition`.\n",
    "- Create a PCA instance called `pca` with `n_components=2`.\n",
    "- Use the `.fit()` method of `pca` to fit it to the scaled fish measurements `scaled_samples`.\n",
    "- Use the `.transform()` method of `pca` to transform the `scaled_samples`. Assign the result to `pca_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fsh.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "scaler.fit(fsh.iloc[:, 1:])\n",
    "scaled_samples = scaler.transform(fsh.iloc[:, 1:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaled_samples.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaled_samples[:3, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PCA\n",
    "# from sklearn.decomposition import PCA\n",
    "\n",
    "# Create a PCA model with 2 components: pca\n",
    "pca = PCA(n_components=2)\n",
    "\n",
    "# Fit the PCA instance to the scaled samples\n",
    "pca.fit(scaled_samples)\n",
    "\n",
    "# Transform the scaled samples: pca_features\n",
    "pca_features = pca.transform(scaled_samples)\n",
    "\n",
    "# Print the shape of pca_features\n",
    "pca_features.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A tf-idf word-frequency array\n",
    "\n",
    "In this exercise, you'll create a tf-idf word frequency array for a toy collection of documents. For this, use the `TfidfVectorizer` from sklearn. It transforms a list of documents into a word frequency array, which it outputs as a csr_matrix. It has `fit()` and `transform()` methods like other sklearn objects.\n",
    "\n",
    "You are given a list documents of toy documents about pets. Its contents have been printed in the IPython Shell.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `TfidfVectorizer` from `sklearn.feature_extraction.text`.\n",
    "- Create a `TfidfVectorizer` instance called `tfidf`.\n",
    "- Apply `.fit_transform()` method of `tfidf` to `documents` and assign the result to `csr_mat`. This is a word-frequency array in csr_matrix format.\n",
    "- Inspect `csr_mat` by calling its `.toarray()` method and printing the result. This has been done for you.\n",
    "- The columns of the array correspond to words. Get the list of words by calling the `.get_feature_names_out()` method of `tfidf`, and assign the result to `words`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = ['cats say meow', 'dogs say woof', 'dogs chase cats']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import TfidfVectorizer\n",
    "# from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# Create a TfidfVectorizer: tfidf\n",
    "tfidf = TfidfVectorizer() \n",
    "\n",
    "# Apply fit_transform to document: csr_mat\n",
    "csr_mat = tfidf.fit_transform(documents)\n",
    "\n",
    "# Print result of toarray() method\n",
    "print(csr_mat.toarray())\n",
    "\n",
    "# Get the words: words\n",
    "words = tfidf.get_feature_names_out()\n",
    "\n",
    "# Print words\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering Wikipedia part I\n",
    "\n",
    "You saw in the video that `TruncatedSVD` is able to perform PCA on sparse arrays in csr_matrix format, such as word-frequency arrays. Combine your knowledge of TruncatedSVD and k-means to cluster some popular pages from Wikipedia. In this exercise, build the pipeline. In the next exercise, you'll apply it to the word-frequency array of some Wikipedia articles.\n",
    "\n",
    "Create a Pipeline object consisting of a TruncatedSVD followed by KMeans. (This time, we've precomputed the word-frequency matrix for you, so there's no need for a TfidfVectorizer).\n",
    "\n",
    "The Wikipedia dataset you will be working with was obtained from [here](https://blog.lateral.io/2015/06/the-unknown-perils-of-mining-wikipedia/).\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import:\n",
    "  - `TruncatedSVD` from `sklearn.decomposition`.\n",
    "  - `KMeans` from `sklearn.cluster`.\n",
    "  - `make_pipeline` from `sklearn.pipeline`.\n",
    "- Create a `TruncatedSVD` instance called `svd` with `n_components=50`.\n",
    "- Create a `KMeans` instance called `kmeans` with `n_clusters=6`.\n",
    "- Create a pipeline called `pipeline` consisting of `svd` and `kmeans`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# from sklearn.decomposition import TruncatedSVD\n",
    "# from sklearn.cluster import KMeans\n",
    "# from sklearn.pipeline import make_pipeline\n",
    "\n",
    "# Create a TruncatedSVD instance: svd\n",
    "svd = TruncatedSVD(n_components=50)\n",
    "\n",
    "# Create a KMeans instance: kmeans\n",
    "kmeans = KMeans(n_clusters=6, n_init=10)\n",
    "\n",
    "# Create a pipeline: pipeline\n",
    "pipeline = make_pipeline(svd, kmeans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering Wikipedia part II\n",
    "\n",
    "It is now time to put your pipeline from the previous exercise to work! You are given an array `articles` of tf-idf word-frequencies of some popular Wikipedia articles, and a list `titles` of their titles. Use your pipeline to cluster the Wikipedia articles.\n",
    "\n",
    "A solution to the previous exercise has been pre-loaded for you, so a Pipeline `pipeline` chaining TruncatedSVD with KMeans is available.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `pandas` as `pd`.\n",
    "- Fit the pipeline to the word-frequency array `articles`.\n",
    "- Predict the cluster labels.\n",
    "- Align the cluster labels with the list `titles` of article titles by creating a DataFrame `df` with `labels` and `titles` as columns. This has been done for you.\n",
    "- Use the `.sort_values()` method of `df` to sort the DataFrame by the `'label'` column, and print the result.\n",
    "- Hit 'Submit Answer' and take a moment to investigate your amazing clustering of Wikipedia pages!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wik1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wik1.iloc[:5, :5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = csr_matrix(wik1)\n",
    "articles.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titles = wik1.index\n",
    "print(titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas\n",
    "# import pandas as pd\n",
    "\n",
    "# Fit the pipeline to articles\n",
    "pipeline.fit(wik1)\n",
    "\n",
    "# Calculate the cluster labels: labels\n",
    "labels = pipeline.predict(wik1)\n",
    "\n",
    "# Create a DataFrame aligning labels and titles: df\n",
    "df = pd.DataFrame({'label': labels, 'article': titles})\n",
    "\n",
    "# Display df sorted by cluster label\n",
    "df.sort_values(['label', 'article'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discovering interpretable features\n",
    "\n",
    "In this chapter, you'll learn about a dimension reduction technique called \"Non-negative matrix factorization\" (\"NMF\") that expresses samples as combinations of interpretable parts. For example, it expresses documents as combinations of topics, and images in terms of commonly occurring visual patterns. You'll also learn to use NMF to build recommender systems that can find you similar articles to read, or musical artists that match your listening history!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Non-negative matrix factorization (NMF)\n",
    "\n",
    "- NMF stands for \"non-negative matrix factorization\".\n",
    "- NMF, like PCA, is a dimension reduction technique.\n",
    "- In constract to PCA, however, NMF models are interpretable.\n",
    "- This means an NMF models are easier to understand yourself, and much easier for you to explain to others.\n",
    "- **NMF can not be applied to every dataset, however.**\n",
    "- **It is required that the sample features be \"non-negative\", so greater than or equal to 0.**\n",
    "\n",
    "1. Interpretable parts\n",
    "   - NMF achieves its interpretability by decomposing samples as sums of their parts.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_1.jpg)\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_2.jpg)\n",
    "   - For example, NMF decomposes documents as combinations of common themes, and images as combinations of common patterns.\n",
    "   - You'll learn about both these examples in detail later.\n",
    "   - For now, let's focus on getting started.\n",
    "\n",
    "1. Using scikit-learn NMF\n",
    "   - NMF is available in scikit learn, and follows the same `fit`/`transform` pattern as PCA.\n",
    "   - However, unlike PCA, the desired number of components must always be specified.\n",
    "   - NMF works both with numpy arrays and sparse arrays in the csr_matrix format.\n",
    "\n",
    "1. Example word-frequency array\n",
    "   - Let's see an application of NMF to a toy example of a word-frequency array.\n",
    "   - In this toy dataset, there are only 4 words in the vocabulary, and these correspond to the four columns of the word-frequency array.\n",
    "   - Each row represents a document, and the entries of the array measure the frequency of each word in the document using what's known as \"tf-idf\".\n",
    "   - \"tf\" is the frequency of the word in the document.\n",
    "   - So if 10% of the words in the document are \"datacamp\", then the tf of \"datacamp\" for that document is point-1.\n",
    "   - \"idf\" is a weighting scheme that reduces the influence of frequent words like \"the\".\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_3.jpg)\n",
    "\n",
    "1. Example usage of NMF\n",
    "   - Let's now see how to use NMF in Python.\n",
    "   - Firstly, import NMF. Create a model, specifying the desired number of components.\n",
    "   - Let's specify 2. Fit the model to the samples, then use the fit model to perform the transformation.\n",
    "\n",
    "1. NMF components\n",
    "   - Just as PCA has principal components, NMF has components which it learns from the samples, and as with PCA, the dimension of the components is the same as the dimension of the samples.\n",
    "   - In our example, for instance, there are 2 components, and they live in 4 dimensional space, corresponding to the 4 words in the vocabulary.\n",
    "   - The entries of the NMF components are always non-negative.\n",
    "\n",
    "1. NMF features\n",
    "   - The NMF feature values are non-negative, as well.\n",
    "   - As we saw with PCA, our transformed data in this example will have two columns, corresponding to our two new features.\n",
    "   - The features and the components of an NMF model can be combined to approximately reconstruct the original data samples.\n",
    "\n",
    "1. Reconstruction of a sample\n",
    "   - Let's see how this works with a single data sample.\n",
    "   - Here is a sample representing a document from our toy dataset, and here are its NMF feature values.\n",
    "   - Now if we multiply each NMF components by the corresponding NMF feature value, and add up each column, we get something very close to the original sample.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_4.jpg)\n",
    "\n",
    "1. Sample reconstruction\n",
    "   - So a sample can be reconstructed by multiplying the NMF components by the NMF feature values of the sample, and adding up.\n",
    "   - This calculation also can be expressed as what is known as a product of matrices.\n",
    "   - We won't be using that point of view, but that's where the \"matrix factorization\", or \"MF\", in NMF comes from.\n",
    "\n",
    "1. NMF fits to non-negative data only\n",
    "   - Finally, remember that NMF can only be applied to arrays of non-negative data, such as word-frequency arrays.\n",
    "   - In the next video, you'll construct another example by encoding collections of images as non-negative arrays.\n",
    "   - There are many other great examples as well, such as arrays encoding audio spectrograms, and arrays representing the purchase histories on e-Commerce sites."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The data associated to the example from the slides/lecture is not provided so the `wik1` dataset is used.\n",
    "- `wik1` has 13125 columns, while the toy example had 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = NMF(n_components=6, init=None)\n",
    "model.fit(wik1_sparse)\n",
    "nmf_features = model.transform(wik1_sparse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# just the first 6 features\n",
    "nmf_features[:6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_row = wik1.loc['Climate change', :].to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf_features[14, :].reshape((6, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reconstruction = np.sum(nmf_features[14, :].reshape((6, 1)) * model.components_, axis=0)\n",
    "reconstruction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The reconstructed data isn't nearly as close is the original example with 4 features transformed into 2 principal components\n",
    "- In this case, 13125 features were transformed into 6 principal components, which doesn't reconstruct the original values that well\n",
    "- Increasing the number of principal components increases the accuracy of the reconstructed value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_exp = pd.DataFrame({'original value': sample_row, 'reconstructed value': reconstruction})\n",
    "df_exp[df_exp['original value'].gt(0.15)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `wik2` contains the columns names of `wik1` (i.e. the feature terms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wik2.iloc[[1865, 2078, 5216, 5818, 11866], :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Non-negative data\n",
    "\n",
    "Which of the following 2-dimensional arrays are examples of non-negative data?\n",
    "\n",
    "A tf-idf word-frequency array.\n",
    "An array daily stock market price movements (up and down), where each row represents a company.\n",
    "An array where rows are customers, columns are products and entries are 0 or 1, indicating whether a customer has purchased a product.\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- ~~1 only~~\n",
    "- ~~2 and 3~~\n",
    "- **1 and 3**\n",
    "  - **Stock prices can go down as well as up, so an array of daily stock market price movements is not an example of non-negative data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NMF applied to Wikipedia articles\n",
    "\n",
    "In the video, you saw NMF applied to transform a toy word-frequency array. Now it's your turn to apply NMF, this time using the tf-idf word-frequency array of Wikipedia articles, given as a csr matrix `articles`. Here, fit the model and transform the articles. In the next exercise, you'll explore the result.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `NMF` from `sklearn.decomposition`.\n",
    "- Create an `NMF` instance called `model` with `6` components.\n",
    "- Fit the model to the word count data `articles`.\n",
    "- Use the `.transform()` method of `model` to transform `articles`, and assign the result to `nmf_features`.\n",
    "- Print `nmf_features` to get a first idea what it looks like (`.round(2)` rounds the entries to 2 decimal places.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = wik1_sparse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import NMF\n",
    "# from sklearn.decomposition import NMF\n",
    "\n",
    "# Create an NMF instance: model\n",
    "model = NMF(n_components=6, init=None)\n",
    "\n",
    "# Fit the model to articles\n",
    "model.fit(articles)\n",
    "\n",
    "# Transform the articles: nmf_features\n",
    "nmf_features = model.transform(articles)\n",
    "\n",
    "# Print the NMF features\n",
    "print(nmf_features.round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NMF features of the Wikipedia articles\n",
    "\n",
    "Now you will explore the NMF features you created in the previous exercise. A solution to the previous exercise has been pre-loaded, so the array `nmf_features` is available. Also available is a list `titles` giving the title of each Wikipedia article.\n",
    "\n",
    "When investigating the features, notice that for both actors, the NMF feature 3 has by far the highest value. This means that both articles are reconstructed using mainly the 3rd NMF component. In the next video, you'll see why: NMF components represent topics (for instance, acting!).\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `pandas` as `pd`.\n",
    "- Create a DataFrame `df` from `nmf_features` using `pd.DataFrame()`. Set the index to `titles` using `index=titles`.\n",
    "- Use the `.loc[]` accessor of `df` to select the row with title `'Anne Hathaway'`, and print the result. These are the NMF features for the article about the actress Anne Hathaway.\n",
    "- Repeat the last step for `'Denzel Washington'` (another actor)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pandas DataFrame: df\n",
    "df = pd.DataFrame(nmf_features, index=wik1.index)\n",
    "\n",
    "display(df.head())\n",
    "\n",
    "# Print the row for 'Anne Hathaway'\n",
    "display(df.loc['Anne Hathaway'].to_frame())\n",
    "\n",
    "# Print the row for 'Denzel Washington'\n",
    "display(df.loc['Denzel Washington'].to_frame())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice that for both actors, the NMF feature 3 has by far the highest value. This means that both articles are reconstructed using mainly the 3rd NMF component. In the next video, you'll see why: NMF components represent topics (for instance, acting!).**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NMF reconstructs samples\n",
    "\n",
    "In this exercise, you'll check your understanding of how NMF reconstructs samples from its components using the NMF feature values. On the right are the components of an NMF model. If the NMF feature values of a sample are `[2, 1]`, then which of the following is _most likely_ to represent the original sample? A pen and paper will help here! You have to apply the same technique Ben used in the video to reconstruct the sample `[0.1203 0.1764 0.3195 0.141]`.\n",
    "\n",
    "**Possible Answers**\n",
    "\n",
    "- **`[2.2, 1.1, 2.1]`**\n",
    "- ~~`[0.5, 1.6, 3.1]`~~\n",
    "- ~~`[-4.0, 1.0, -2.0]`~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc = np.array([[1., 0.5, 0. ], [0.2, 0.1, 2.1]])\n",
    "f = np.array([[2], [1]])\n",
    "\n",
    "np.sum(f * mc, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NMF learns interpretable parts\n",
    "\n",
    "- In this video, you'll learn that the components of NMF represent patterns that frequently occur in the samples.\n",
    "\n",
    "1. Example: NMF learns interpretable parts\n",
    "   - Let's consider a concrete example, where scientific articles are represented by their word frequencies.\n",
    "   - There are 20000 articles, and 800 words.\n",
    "   - So the array has 800 columns.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_5.jpg)\n",
    "\n",
    "1. Applying NMF to the articles\n",
    "   - Let's fit an NMF model with 10 components to the articles.\n",
    "   - The 10 components are stored as the 10 rows of a 2-dimensional numpy array.\n",
    "\n",
    "1. NMF components are topics\n",
    "   - The rows, or components, live in an 800-dimensional space - there is one dimension for each of the words.\n",
    "   - Aligning the words of our vocabulary with the columns of the NMF components allows them to be interpreted.\n",
    "   - Choosing a component, such as this one, and looking at which words have the highest values, we see that they fit a theme: the words are 'species', 'plant', 'plants', 'genetic', 'evolution' and 'life'.\n",
    "   - The same happens if any other component is considered.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_6.jpg)\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_7.jpg)\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_8.jpg)\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_9.jpg)\n",
    "\n",
    "1. NMF components\n",
    "   - So if NMF is applied to documents, then the components correspond to topics, and the NMF features reconstruct the documents from the topics.\n",
    "   - If NMF is applied to a collection of images, on the other hand, then the NMF components represent patterns that frequently occur in the images.\n",
    "   - In this example, for instance, NMF decomposes images from an LCD display into the individual cells of the display.\n",
    "   - This example you'll investigate for yourself in the exercises.\n",
    "   - To do this, you'll need to know how to represent a collection of images as a non-negative array.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_2.jpg)\n",
    "\n",
    "1. Grayscale images\n",
    "   - An image in which all the pixels are shades of gray ranging from black to white is called a \"grayscale image\".\n",
    "   - Since there are only shades of grey, a grayscale image can be encoded by the brightness of every pixel.\n",
    "   - Representing the brightness as a number between 0 and 1, where 0 is totally black and 1 is totally white, the image can be represented as 2-dimensional array of numbers.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_10.jpg)\n",
    "\n",
    "1. Grayscale image example\n",
    "   - Here, for example, is a grayscale photo of the moon!\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_11.jpg)\n",
    "\n",
    "1. Grayscale images as flat arrays\n",
    "   - These 2-dimensional arrays of numbers can then be flattened by enumerating the entries.\n",
    "   - For instance, we could read-off the values row-by-row, from left-to-right and top to bottom.\n",
    "   - The grayscale image is now represented by a flat array of non-negative numbers.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_12.jpg)\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_13.jpg)\n",
    "\n",
    "1. Encoding a collection of images\n",
    "   - A collection of grayscale images of the same size can thus be encoded as a 2-dimensional array, in which each row represents an image as a flattened array, and each column represents a pixel.\n",
    "   - Viewing the images as samples, and the pixels as features, we see that the data is arranged similarly to the word frequency array.\n",
    "   - Indeed, the entries of this array are non-negative, so NMF can be used to learn the parts of the images.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_14.jpg)\n",
    "\n",
    "1. Visualizing samples\n",
    "   - It's difficult to visualize an image by just looking at the flattened array.\n",
    "   - To recover the image, use the reshape method of the sample, specifying the dimensions of the original image as a tuple.\n",
    "   - This yields the 2-dimensional array of pixel brightnesses.\n",
    "   - To display the corresponding image, import pyplot, and pass the 2-dimensional array to the plt dot imshow function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = np.array([0, 1, 0.5, 1, 0, 1])\n",
    "bitmap = sample.reshape((2, 3))\n",
    "\n",
    "plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NMF learns topics of documents\n",
    "\n",
    "In the video, you learned when NMF is applied to documents, the components correspond to topics of documents, and the NMF features reconstruct the documents from the topics. Verify this for yourself for the NMF model that you built earlier using the Wikipedia articles. Previously, you saw that the 3rd NMF feature value was high for the articles about actors Anne Hathaway and Denzel Washington. In this exercise, identify the topic of the corresponding NMF component.\n",
    "\n",
    "The NMF model you built earlier is available as `model`, while `words` is a list of the words that label the columns of the word-frequency array.\n",
    "\n",
    "After you are done, take a moment to recognise the topic that the articles about Anne Hathaway and Denzel Washington have in common!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `pandas` as `pd`.\n",
    "- Create a DataFrame `components_df` from `model.components_`, setting `columns=words` so that columns are labeled by the words.\n",
    "- Print `components_df.shape` to check the dimensions of the DataFrame.\n",
    "- Use the `.iloc[]` accessor on the DataFrame `components_df` to select row `3`. Assign the result to `component`.\n",
    "- Call the `.nlargest()` method of `component`, and print the result. This gives the five words with the highest values for that component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = wik2[0].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a DataFrame: components_df\n",
    "components_df = pd.DataFrame(model.components_, columns=words)\n",
    "\n",
    "display(components_df.iloc[:5, :5])\n",
    "\n",
    "# Print the shape of the DataFrame\n",
    "print(components_df.shape)\n",
    "\n",
    "# Select row 3: component\n",
    "component = components_df.iloc[3, :]\n",
    "\n",
    "# Print result of nlargest\n",
    "component.nlargest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Take a moment to recognise the topics that the articles about Anne Hathaway and Denzel Washington have in common!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the LED digits dataset\n",
    "\n",
    "In the following exercises, you'll use NMF to decompose grayscale images into their commonly occurring patterns. Firstly, explore the image dataset and see how it is encoded as an array. You are given 100 images as a 2D array `samples`, where each row represents a single 13x8 image. The images in your dataset are pictures of a LED digital display.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `matplotlib.pyplot` as `plt`.\n",
    "- Select row `0` of `samples` and assign the result to `digit`. For example, to select column `2` of an array `a`, you could use `a[:,2]`. Remember that since `samples` is a NumPy array, you can't use the `.loc[]` or `iloc[]` accessors to select specific rows or columns.\n",
    "- Print `digit`. This has been done for you. Notice that it is a 1D array of 0s and 1s.\n",
    "- Use the `.reshape()` method of `digit` to get a 2D array with shape `(13, 8)`. Assign the result to `bitmap`.\n",
    "- Print `bitmap`, and notice that the 1s show the digit 7!\n",
    "- Use the `plt.imshow()` function to display `bitmap` as an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = lcd.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the 0th row: digit\n",
    "digit = samples[0]\n",
    "\n",
    "# Print digit\n",
    "print(digit)\n",
    "\n",
    "# Reshape digit to a 13x8 array: bitmap\n",
    "bitmap = digit.reshape((13, 8))\n",
    "\n",
    "# Print bitmap\n",
    "print(bitmap)\n",
    "\n",
    "# Use plt.imshow to display bitmap\n",
    "plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You'll explore this dataset further in the next exercise and see for yourself how NMF can learn the parts of images.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NMF learns the parts of images\n",
    "\n",
    "Now use what you've learned about NMF to decompose the digits dataset. You are again given the digit images as a 2D array `samples`. This time, you are also provided with a function `show_as_image()` that displays the image encoded by any 1D array:\n",
    "\n",
    "```python\n",
    "def show_as_image(sample):\n",
    "    bitmap = sample.reshape((13, 8))\n",
    "    plt.figure()\n",
    "    plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "    plt.colorbar()\n",
    "    plt.show()\n",
    "```\n",
    "\n",
    "After you are done, take a moment to look through the plots and notice how NMF has expressed the digit as a sum of the components!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `NMF` from `sklearn.decomposition`.\n",
    "- Create an `NMF` instance called `model` with `7` components. (7 is the number of cells in an LED display).\n",
    "- Apply the `.fit_transform()` method of `model` to `samples`. Assign the result to `features`.\n",
    "- To each component of the model (accessed via `model.components_`), apply the `show_as_image()` function to that component inside the loop.\n",
    "- Assign the row `0` of features to `digit_features`.\n",
    "- Print `digit_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_as_image(sample):\n",
    "    bitmap = sample.reshape((13, 8))\n",
    "    plt.figure()\n",
    "    plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "    plt.colorbar()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import NMF\n",
    "# from sklearn.decomposition import NMF\n",
    "\n",
    "# Create an NMF model: model\n",
    "model = NMF(n_components=7, init=None)\n",
    "\n",
    "# Apply fit_transform to samples: features\n",
    "features = model.fit_transform(samples)\n",
    "\n",
    "# Call show_as_image on each component\n",
    "for component in model.components_:\n",
    "    show_as_image(component)\n",
    "\n",
    "# Assign the 0th row of features: digit_features\n",
    "digit_features = features[0, :]\n",
    "\n",
    "# Print digit_features\n",
    "print(digit_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Take a moment to look through the plots and notice how NMF has expressed the digit as a sum of the components!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PCA doesn't learn parts\n",
    "\n",
    "Unlike NMF, PCA doesn't learn the parts of things. Its components do not correspond to topics (in the case of documents) or to parts of images, when trained on images. Verify this for yourself by inspecting the components of a PCA model fit to the dataset of LED digit images from the previous exercise. The images are available as a 2D array `samples`. Also available is a modified version of the `show_as_image()` function which colors a pixel red if the value is negative.\n",
    "\n",
    "After submitting the answer, notice that the components of PCA do not represent meaningful parts of images of LED digits!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `PCA` from `sklearn.decomposition`.\n",
    "- Create a `PCA` instance called `model` with `7` components.\n",
    "- Apply the `.fit_transform()` method of `model` to `samples`. Assign the result to `features`.\n",
    "- To each component of the model (accessed via `model.components_`), apply the `show_as_image()` function to that component inside the loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PCA\n",
    "# from sklearn.decomposition import PCA\n",
    "\n",
    "# Create a PCA instance: model\n",
    "model = PCA(n_components=7)\n",
    "\n",
    "# Apply fit_transform to samples: features\n",
    "features = model.fit_transform(samples)\n",
    "\n",
    "# Call show_as_image on each component\n",
    "for component in model.components_:\n",
    "    show_as_image(component)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice that the components of PCA do not represent meaningful parts of images of LED digits!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building recommender systems using NMF\n",
    "\n",
    "1. Finding similar articles\n",
    "   - Suppose that you are an engineer at a large online newspaper.\n",
    "   - You've been given the task of recommending articles that are similar to the article currently being read by a customer.\n",
    "   - Given an article, how can you find articles that have similar topics?\n",
    "   - In this video, you'll learn how to solve this problem, and others like it, by using NMF.\n",
    "\n",
    "1. Strategy\n",
    "   - Our strategy for solving this problem is to apply NMF to the word-frequency array of the articles, and to use the resulting NMF features.\n",
    "   - You learned in the previous videos these NMF features describe the topic mixture of an article.\n",
    "   - So similar articles will have similar NMF features.\n",
    "   - But how can two articles be compared using their NMF features?\n",
    "   - Before answering this question, let's set the scene by doing the first step.\n",
    "\n",
    "1. Apply NMF to the word-frequency array\n",
    "   - You are given a word frequency array articles corresponding to the collection of newspaper articles in question. Import NMF, create the model, and use the fit_transform method to obtain the transformed articles. Now we've got NMF features for every article, given by the columns of the new array.\n",
    "\n",
    "1. Strategy\n",
    "   - Now we need to define how to compare articles using their NMF features.\n",
    "\n",
    "1. Versions of articles\n",
    "   - Similar documents have similar topics, but it isn't always the case that the NMF feature values are exactly the same.\n",
    "   - For instance, one version of a document might use very direct language, whereas other versions might interleave the same content with meaningless chatter.\n",
    "   - Meaningless chatter reduces the frequency of the topic words overall, which reduces the values of the NMF features representing the topics.\n",
    "   - However, on a scatter plot of the NMF features, all these versions lie on a single line passing through the origin.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_15.jpg)\n",
    "\n",
    "1. Cosine similarity\n",
    "   - For this reason, when comparing two documents, it's a good idea to compare these lines.\n",
    "   - We'll compare them using what is known as the cosine similarity, which uses the angle between the two lines.\n",
    "   - Higher values indicate greater similarity.\n",
    "   - The technical definition of the cosine similarity is out the scope of this course, but we've already gained an intuition.\n",
    "   - ![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/nmf_16.jpg)\n",
    "\n",
    "1. Calculating the cosine similarities\n",
    "   - Let's see now how to compute the cosine similarity.\n",
    "   - Firstly, import the normalize function, and apply it to the array of all NMF features.\n",
    "   - Now select the row corresponding to the current article, and pass it to the dot method of the array of all normalized features.\n",
    "   - This results in the cosine similarities.\n",
    "\n",
    "1. DataFrames and labels\n",
    "   - With the help of a pandas DataFrame, we can label the similarities with the article titles.\n",
    "   - Start by importing pandas. After normalizing the NMF features, create a DataFrame whose rows are the normalized features, using the titles as an index.\n",
    "   - Now use the loc method of the DataFrame to select the normalized feature values for the current article, using its title 'Dog bites man'.\n",
    "   - Calculate the cosine similarities using the dot method of the DataFrame.\n",
    "\n",
    "1. DataFrames and labels\n",
    "   - Finally, use the nlargest method of the resulting pandas Series to find the articles with the highest cosine similarity.\n",
    "   - We see that all of them are concerned with 'domestic animals' and/or 'danger'!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The data associated to the example from the slides/lecture is not provided so the `wik1` dataset is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf = NMF(n_components=6, init=None)\n",
    "nmf_features = nmf.fit_transform(wik1_sparse)\n",
    "\n",
    "norm_features = normalize(nmf_features)\n",
    "current_article = norm_features[45, :]\n",
    "similarities = norm_features.dot(current_article)\n",
    "print(similarities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "norm_features = normalize(nmf_features)\n",
    "df = pd.DataFrame(norm_features, index=wik1.index)\n",
    "current_article = df.loc['Hepatitis C']\n",
    "similarities = df.dot(current_article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarities.nlargest(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which articles are similar to 'Cristiano Ronaldo'?\n",
    "\n",
    "In the video, you learned how to use NMF features and the cosine similarity to find similar articles. Apply this to your NMF model for popular Wikipedia articles, by finding the articles most similar to the article about the footballer Cristiano Ronaldo. The NMF features you obtained earlier are available as `nmf_features`, while `titles` is a list of the article titles.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `normalize` from `sklearn.preprocessing`.\n",
    "- Apply the `normalize()` function to `nmf_features`. Store the result as `norm_features`.\n",
    "- Create a DataFrame `df` from `norm_features`, using `titles` as an index.\n",
    "- Use the `.loc[]` accessor of `df` to select the row of `'Cristiano Ronaldo'`. Assign the result to `article`.\n",
    "- Apply the `.dot()` method of `df` to `article` to calculate the cosine similarity of every row with `article`.\n",
    "- Print the result of the `.nlargest()` method of `similarities` to display the most similiar articles. This has been done for you, so hit 'Submit Answer' to see the result!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# import pandas as pd\n",
    "# from sklearn.preprocessing import normalize\n",
    "\n",
    "# Normalize the NMF features: norm_features\n",
    "norm_features = normalize(nmf_features)\n",
    "\n",
    "# Create a DataFrame: df\n",
    "df = pd.DataFrame(norm_features, index=wik1.index)\n",
    "\n",
    "# Select the row corresponding to 'Cristiano Ronaldo': article\n",
    "article = df.loc['Cristiano Ronaldo']\n",
    "\n",
    "# Compute the dot products: similarities\n",
    "similarities = df.dot(article)\n",
    "\n",
    "# Display those with the largest cosine similarity\n",
    "print(similarities.nlargest())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You may need to know a little about football (or soccer, depending on where you're from!) to be able to evaluate for yourself the quality of the computed similarities!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommend musical artists part II\n",
    "\n",
    "In this exercise and the next, you'll use what you've learned about NMF to recommend popular music artists! You are given a sparse array `artists` whose rows correspond to artists and whose columns correspond to users. The entries give the number of times each artist was listened to by each user.\n",
    "\n",
    "In this exercise, build a pipeline and transform the array into normalized NMF features. The first step in the pipeline, `MaxAbsScaler`, transforms the data so that all users have the same influence on the model, regardless of how many different artists they've listened to. In the next exercise, you'll use the resulting normalized NMF features for recommendation!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import:\n",
    "  - `NMF` from `sklearn.decomposition`.\n",
    "  - `Normalizer` and `MaxAbsScaler` from `sklearn.preprocessing`.\n",
    "  - `make_pipeline` from `sklearn.pipeline`.\n",
    "- Create an instance of `MaxAbsScaler` called `scaler`.\n",
    "- Create an `NMF` instance with `20` components called `nmf`.\n",
    "- Create an instance of `Normalizer` called `normalizer`.\n",
    "- Create a pipeline called `pipeline` that chains together `scaler`, `nmf`, and `normalizer`.\n",
    "- Apply the `.fit_transform()` method of `pipeline` to `artists`. Assign the result to `norm_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "# from sklearn.decomposition import NMF\n",
    "# from sklearn.preprocessing import Normalizer, MaxAbsScaler\n",
    "# from sklearn.pipeline import make_pipeline\n",
    "\n",
    "# Create a MaxAbsScaler: scaler\n",
    "scaler = MaxAbsScaler()\n",
    "\n",
    "# Create an NMF model: nmf\n",
    "nmf = NMF(n_components=20, init=None)\n",
    "\n",
    "# Create a Normalizer: normalizer\n",
    "normalizer = Normalizer()\n",
    "\n",
    "# Create a pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, nmf, normalizer)\n",
    "\n",
    "# Apply fit_transform to artists: norm_features\n",
    "norm_features = pipeline.fit_transform(artists_sparse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommend musical artists part II\n",
    "\n",
    "Suppose you were a big fan of Bruce Springsteen - which other musicial artists might you like? Use your NMF features from the previous exercise and the cosine similarity to find similar musical artists. A solution to the previous exercise has been run, so `norm_features` is an array containing the normalized NMF features as rows. The names of the musical artists are available as the list `artist_names`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `pandas` as `pd`.\n",
    "- Create a DataFrame `df` from `norm_features`, using `artist_names` as an index.\n",
    "- Use the `.loc[]` accessor of `df` to select the row of `'Bruce Springsteen'`. Assign the result to `artist`.\n",
    "- Apply the `.dot()` method of `df` to `artist` to calculate the dot product of every row with `artist`. Save the result as `similarities`.\n",
    "- Print the result of the `.nlargest()` method of `similarities` to display the artists most similar to `'Bruce Springsteen'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas\n",
    "# import pandas as pd\n",
    "\n",
    "# Create a DataFrame: df\n",
    "df = pd.DataFrame(norm_features, index=artist_names)\n",
    "\n",
    "# Select row of 'Bruce Springsteen': artist\n",
    "artist = df.loc['Bruce Springsteen']\n",
    "\n",
    "# Compute cosine similarities: similarities\n",
    "similarities = df.dot(artist)\n",
    "\n",
    "# Display those with highest cosine similarity\n",
    "similarities.nlargest()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Thoughts\n",
    "\n",
    "You've learned all about Unsupervised Learning, and applied the techniques to real-world datasets, and built your knowledge of Python along the way. In particular, you've become a whiz at using scikit-learn and scipy for unsupervised learning challenges. You have harnessed both clustering and dimension reduction techniques to tackle serious problems with real-world datasets, such as clustering Wikipedia documents by the words they contain, and recommending musical artists to consumers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Certificate\n",
    "\n",
    "![](https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2021-03-29_unsupervised_learning_python/2021-04-19_DataCamp_unsupervised_learning_in_python_certificate.jpg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py312",
   "language": "python",
   "name": "py312"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  },
  "toc-autonumbering": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}