MyAnimeList Web Scraping/MAL Data Collection (Scraping).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3bc81b58",
   "metadata": {},
   "source": [
    "# MyAnimeList Web Scraper"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9abfb32",
   "metadata": {},
   "source": [
    "**Goal:** Given a valid MyAnimeList username, produce a list of that user's \"anime list\" and pertinent details about each listed show."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f76fe68e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "from requests_html import AsyncHTMLSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "706fb593",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate an Asynchronous Session\n",
    "session = AsyncHTMLSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "412cfe85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify a MAL user and the corresponding anime list URL\n",
    "username = \"ABPhanatic\"\n",
    "url = \"https://myanimelist.net/animelist/\" + username"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ec0c0051",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Connect to the webpage\n",
    "r = await session.get(url)\n",
    "r.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c5bb044",
   "metadata": {},
   "source": [
    "### Examining the page's HTML code before rendering the JavaScript"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "31d043f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "html = r.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "090dcafb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of div elements PRE-JS render\n",
    "len(html.find('div'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e824e2be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of anchor elements PRE-JS render\n",
    "len(html.find('a'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "69e2f875",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of span elements PRE-JS render\n",
    "len(html.find('span'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1f8d579c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving the pre-rendered HTML to a file so we can inspect it\n",
    "\n",
    "with open('MAL_PreRender_Scrape.html', 'wb') as file:\n",
    "    file.write(html.raw_html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e762b93",
   "metadata": {},
   "source": [
    "### Examining the page's HTML code after rendering the JavaScript"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6f129add",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rendering the JavaScript of the webpage\n",
    "await r.html.arender()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b8cf1892",
   "metadata": {},
   "outputs": [],
   "source": [
    "html = r.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "01904aa0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "423"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of div elements POST-JS render\n",
    "len(html.find('div'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "38784e3b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "573"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of anchor elements POST-JS render\n",
    "len(html.find('a'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "cd62f869",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "605"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of span elements POST-JS render\n",
    "len(html.find('span'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7bbec62b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving the post-rendered HTML to a file as well\n",
    "\n",
    "with open('MAL_PostRender_Scrape.html', 'wb') as file:\n",
    "    file.write(html.raw_html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6c50b09",
   "metadata": {},
   "source": [
    "Clearly, the webpage contains much more content once the JavaScript code is rendered. As seen above, the number of divs, anchors, and span tags increase more than ten-fold after rendering the JS. \n",
    "<br><br>\n",
    "**Note:** If we examine the the pre-render and post-render HTML code that we saved above, we see that the pre-rendered HTML does not provide any information about the shows in the user's anime list  (the exact information I was interested in). The post-rendered HTML however, contains the information we want so that is what we use for the rest of this project."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fcf2f08",
   "metadata": {},
   "source": [
    "# Using Pandas to Scrape HTML Table"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c017813e",
   "metadata": {},
   "source": [
    "It is important to acknowledge that MAL uses a regular HTML table to present the data of a user's anime list. Therefore, my first choice was to see what information I could collect using **Pandas** convenient _read_html()_ method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5e8dd793",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "cbda7403",
   "metadata": {},
   "outputs": [],
   "source": [
    "tables = pd.read_html(html.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "938dde84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>#</th>\n",
       "      <th>Image</th>\n",
       "      <th>Anime Title</th>\n",
       "      <th>Score</th>\n",
       "      <th>Type</th>\n",
       "      <th>Progress</th>\n",
       "      <th>Tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hajime no Ippo Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>30 /  75</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>3 /  -</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ousama Ranking Airing  Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>11 /  23</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>NaN</td>\n",
       "      <td>90.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Wu Shan Wu Xing Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>ONA</td>\n",
       "      <td>- /  3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>NaN</td>\n",
       "      <td>91.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yuu☆Yuu☆Hakusho Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  112</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>182 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Unnamed: 0     #  Image  \\\n",
       "0           NaN   1.0    NaN   \n",
       "1           NaN   NaN    NaN   \n",
       "2           NaN   2.0    NaN   \n",
       "3           NaN   NaN    NaN   \n",
       "4           NaN   3.0    NaN   \n",
       "..          ...   ...    ...   \n",
       "177         NaN   NaN    NaN   \n",
       "178         NaN  90.0    NaN   \n",
       "179         NaN   NaN    NaN   \n",
       "180         NaN  91.0    NaN   \n",
       "181         NaN   NaN    NaN   \n",
       "\n",
       "                                           Anime Title Score Type  Progress  \\\n",
       "0                          Hajime no Ippo Add  -  More     9   TV  30 /  75   \n",
       "1                                                  NaN   NaN  NaN       NaN   \n",
       "2    Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...     8   TV    3 /  -   \n",
       "3                                                  NaN   NaN  NaN       NaN   \n",
       "4                  Ousama Ranking Airing  Add  -  More     9   TV  11 /  23   \n",
       "..                                                 ...   ...  ...       ...   \n",
       "177                                                NaN   NaN  NaN       NaN   \n",
       "178                       Wu Shan Wu Xing Add  -  More     -  ONA    - /  3   \n",
       "179                                                NaN   NaN  NaN       NaN   \n",
       "180                       Yuu☆Yuu☆Hakusho Add  -  More     -   TV  - /  112   \n",
       "181                                                NaN   NaN  NaN       NaN   \n",
       "\n",
       "     Tags  \n",
       "0     NaN  \n",
       "1     NaN  \n",
       "2     NaN  \n",
       "3     NaN  \n",
       "4     NaN  \n",
       "..    ...  \n",
       "177   NaN  \n",
       "178   NaN  \n",
       "179   NaN  \n",
       "180   NaN  \n",
       "181   NaN  \n",
       "\n",
       "[182 rows x 8 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table = tables[0] \n",
    "table\n",
    "\n",
    "# For the purposes of this project, the columns \"Unnamed: 0\", \"#\", \"Image\", and \"Tags\" are not needed\n",
    "# Also, it appears that every other row in this table contains unnecessary missing information which we should get rid of"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "14e79b25",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>#</th>\n",
       "      <th>Image</th>\n",
       "      <th>Anime Title</th>\n",
       "      <th>Score</th>\n",
       "      <th>Type</th>\n",
       "      <th>Progress</th>\n",
       "      <th>Tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hajime no Ippo Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>30 /  75</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>3 /  -</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ousama Ranking Airing  Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>11 /  23</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tensei shitara Slime Datta Ken 2nd Season Add ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>2 /  12</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Akame ga Kill! Add  -  More</td>\n",
       "      <td>7</td>\n",
       "      <td>TV</td>\n",
       "      <td>24</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>NaN</td>\n",
       "      <td>87.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Stranger: Mukou Hadan Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>Movie</td>\n",
       "      <td>- /  1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>NaN</td>\n",
       "      <td>88.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tokyo Ghoul Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  12</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>NaN</td>\n",
       "      <td>89.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Violet Evergarden Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  13</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>NaN</td>\n",
       "      <td>90.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Wu Shan Wu Xing Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>ONA</td>\n",
       "      <td>- /  3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>NaN</td>\n",
       "      <td>91.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yuu☆Yuu☆Hakusho Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  112</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>91 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Unnamed: 0     #  Image  \\\n",
       "0           NaN   1.0    NaN   \n",
       "2           NaN   2.0    NaN   \n",
       "4           NaN   3.0    NaN   \n",
       "6           NaN   4.0    NaN   \n",
       "8           NaN   5.0    NaN   \n",
       "..          ...   ...    ...   \n",
       "172         NaN  87.0    NaN   \n",
       "174         NaN  88.0    NaN   \n",
       "176         NaN  89.0    NaN   \n",
       "178         NaN  90.0    NaN   \n",
       "180         NaN  91.0    NaN   \n",
       "\n",
       "                                           Anime Title Score   Type  Progress  \\\n",
       "0                          Hajime no Ippo Add  -  More     9     TV  30 /  75   \n",
       "2    Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...     8     TV    3 /  -   \n",
       "4                  Ousama Ranking Airing  Add  -  More     9     TV  11 /  23   \n",
       "6    Tensei shitara Slime Datta Ken 2nd Season Add ...     8     TV   2 /  12   \n",
       "8                          Akame ga Kill! Add  -  More     7     TV        24   \n",
       "..                                                 ...   ...    ...       ...   \n",
       "172                 Stranger: Mukou Hadan Add  -  More     -  Movie    - /  1   \n",
       "174                           Tokyo Ghoul Add  -  More     -     TV   - /  12   \n",
       "176                     Violet Evergarden Add  -  More     -     TV   - /  13   \n",
       "178                       Wu Shan Wu Xing Add  -  More     -    ONA    - /  3   \n",
       "180                       Yuu☆Yuu☆Hakusho Add  -  More     -     TV  - /  112   \n",
       "\n",
       "     Tags  \n",
       "0     NaN  \n",
       "2     NaN  \n",
       "4     NaN  \n",
       "6     NaN  \n",
       "8     NaN  \n",
       "..    ...  \n",
       "172   NaN  \n",
       "174   NaN  \n",
       "176   NaN  \n",
       "178   NaN  \n",
       "180   NaN  \n",
       "\n",
       "[91 rows x 8 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Removing all the unnecessary rows\n",
    "anime_list = table.iloc[0:-1:2].copy()\n",
    "anime_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ac46e3c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Unnamed: 0', '#', 'Image', 'Anime Title', 'Score', 'Type', 'Progress',\n",
       "       'Tags'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_list.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "dd3dcadd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Anime Title</th>\n",
       "      <th>Score</th>\n",
       "      <th>Type</th>\n",
       "      <th>Progress</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hajime no Ippo Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>30 /  75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>3 /  -</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ousama Ranking Airing  Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>11 /  23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Tensei shitara Slime Datta Ken 2nd Season Add ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>2 /  12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Akame ga Kill! Add  -  More</td>\n",
       "      <td>7</td>\n",
       "      <td>TV</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>Stranger: Mukou Hadan Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>Movie</td>\n",
       "      <td>- /  1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>Tokyo Ghoul Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>Violet Evergarden Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>Wu Shan Wu Xing Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>ONA</td>\n",
       "      <td>- /  3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>Yuu☆Yuu☆Hakusho Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  112</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>91 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           Anime Title Score   Type  Progress\n",
       "0                          Hajime no Ippo Add  -  More     9     TV  30 /  75\n",
       "2    Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...     8     TV    3 /  -\n",
       "4                  Ousama Ranking Airing  Add  -  More     9     TV  11 /  23\n",
       "6    Tensei shitara Slime Datta Ken 2nd Season Add ...     8     TV   2 /  12\n",
       "8                          Akame ga Kill! Add  -  More     7     TV        24\n",
       "..                                                 ...   ...    ...       ...\n",
       "172                 Stranger: Mukou Hadan Add  -  More     -  Movie    - /  1\n",
       "174                           Tokyo Ghoul Add  -  More     -     TV   - /  12\n",
       "176                     Violet Evergarden Add  -  More     -     TV   - /  13\n",
       "178                       Wu Shan Wu Xing Add  -  More     -    ONA    - /  3\n",
       "180                       Yuu☆Yuu☆Hakusho Add  -  More     -     TV  - /  112\n",
       "\n",
       "[91 rows x 4 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dropping the columns we don't need\n",
    "anime_list = anime_list.drop(labels=['Unnamed: 0','Tags','#','Image'], axis=1)\n",
    "anime_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a214202b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>Anime Title</th>\n",
       "      <th>Score</th>\n",
       "      <th>Type</th>\n",
       "      <th>Progress</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Hajime no Ippo Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>30 /  75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>3 /  -</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>Ousama Ranking Airing  Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>11 /  23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>Tensei shitara Slime Datta Ken 2nd Season Add ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>2 /  12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>Akame ga Kill! Add  -  More</td>\n",
       "      <td>7</td>\n",
       "      <td>TV</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>172</td>\n",
       "      <td>Stranger: Mukou Hadan Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>Movie</td>\n",
       "      <td>- /  1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>174</td>\n",
       "      <td>Tokyo Ghoul Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>176</td>\n",
       "      <td>Violet Evergarden Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>178</td>\n",
       "      <td>Wu Shan Wu Xing Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>ONA</td>\n",
       "      <td>- /  3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>180</td>\n",
       "      <td>Yuu☆Yuu☆Hakusho Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  112</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>91 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    index                                        Anime Title Score   Type  \\\n",
       "0       0                        Hajime no Ippo Add  -  More     9     TV   \n",
       "1       2  Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...     8     TV   \n",
       "2       4                Ousama Ranking Airing  Add  -  More     9     TV   \n",
       "3       6  Tensei shitara Slime Datta Ken 2nd Season Add ...     8     TV   \n",
       "4       8                        Akame ga Kill! Add  -  More     7     TV   \n",
       "..    ...                                                ...   ...    ...   \n",
       "86    172                 Stranger: Mukou Hadan Add  -  More     -  Movie   \n",
       "87    174                           Tokyo Ghoul Add  -  More     -     TV   \n",
       "88    176                     Violet Evergarden Add  -  More     -     TV   \n",
       "89    178                       Wu Shan Wu Xing Add  -  More     -    ONA   \n",
       "90    180                       Yuu☆Yuu☆Hakusho Add  -  More     -     TV   \n",
       "\n",
       "    Progress  \n",
       "0   30 /  75  \n",
       "1     3 /  -  \n",
       "2   11 /  23  \n",
       "3    2 /  12  \n",
       "4         24  \n",
       "..       ...  \n",
       "86    - /  1  \n",
       "87   - /  12  \n",
       "88   - /  13  \n",
       "89    - /  3  \n",
       "90  - /  112  \n",
       "\n",
       "[91 rows x 5 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Resetting the indexes of the final rows\n",
    "anime_list = anime_list.reset_index()\n",
    "anime_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "12f2e295",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Anime Title</th>\n",
       "      <th>Score</th>\n",
       "      <th>Type</th>\n",
       "      <th>Progress</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hajime no Ippo Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>30 /  75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>3 /  -</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ousama Ranking Airing  Add  -  More</td>\n",
       "      <td>9</td>\n",
       "      <td>TV</td>\n",
       "      <td>11 /  23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tensei shitara Slime Datta Ken 2nd Season Add ...</td>\n",
       "      <td>8</td>\n",
       "      <td>TV</td>\n",
       "      <td>2 /  12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Akame ga Kill! Add  -  More</td>\n",
       "      <td>7</td>\n",
       "      <td>TV</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>Stranger: Mukou Hadan Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>Movie</td>\n",
       "      <td>- /  1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>Tokyo Ghoul Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>Violet Evergarden Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>Wu Shan Wu Xing Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>ONA</td>\n",
       "      <td>- /  3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>Yuu☆Yuu☆Hakusho Add  -  More</td>\n",
       "      <td>-</td>\n",
       "      <td>TV</td>\n",
       "      <td>- /  112</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>91 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          Anime Title Score   Type  Progress\n",
       "0                         Hajime no Ippo Add  -  More     9     TV  30 /  75\n",
       "1   Kimetsu no Yaiba: Yuukaku-hen Airing  Add  -  ...     8     TV    3 /  -\n",
       "2                 Ousama Ranking Airing  Add  -  More     9     TV  11 /  23\n",
       "3   Tensei shitara Slime Datta Ken 2nd Season Add ...     8     TV   2 /  12\n",
       "4                         Akame ga Kill! Add  -  More     7     TV        24\n",
       "..                                                ...   ...    ...       ...\n",
       "86                 Stranger: Mukou Hadan Add  -  More     -  Movie    - /  1\n",
       "87                           Tokyo Ghoul Add  -  More     -     TV   - /  12\n",
       "88                     Violet Evergarden Add  -  More     -     TV   - /  13\n",
       "89                       Wu Shan Wu Xing Add  -  More     -    ONA    - /  3\n",
       "90                       Yuu☆Yuu☆Hakusho Add  -  More     -     TV  - /  112\n",
       "\n",
       "[91 rows x 4 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dropping the old \"index\" column\n",
    "anime_list = anime_list.drop(labels='index', axis=1)\n",
    "anime_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "023fb86b",
   "metadata": {},
   "source": [
    "Although we are able to collect some information with this method, there is more information about each show present in the HTML code of the web page that we need (such as the \"status\" of each show i.e. Completed, Watching, Plan To Watch, etc., as well as the URLs of each anime which we will use to gather more information in the future).  \n",
    "Therefore, we will collect our data through web-scraping instead."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b502f08",
   "metadata": {},
   "source": [
    "# Scraping the HTML code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b16696c",
   "metadata": {},
   "source": [
    "## Getting all anime names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f7b878a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_entries = html.find('tr.list-table-data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6c95f60d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list_entries) # After checking the actual webpage, we can confirm that this number seems correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "09ef0cec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<tr class=\"list-table-data\"><td class=\"data status watching\"></td> <td class=\"data number\">1</td> <td class=\"data image\"><a href=\"/anime/263/Hajime_no_Ippo\" class=\"link sort\"><img src=\"https://cdn.myanimelist.net/r/96x136/images/anime/4/86334.webp?s=ef3213de769a55ea677d4536af3dfb87\" class=\"hover-info image\" /></a></td> <td class=\"data title clearfix\"><a href=\"/anime/263/Hajime_no_Ippo\" class=\"link sort\">Hajime no Ippo</a> <div class=\"icon-watch2\"><a href=\"/anime/263/Hajime_no_Ippo/video\" title=\"Watch Episode Video\" class=\"mal-icon ml4\"><i class=\"malicon malicon-movie-episode\"></i></a></div> <span class=\"rewatching\" style=\"display: none;\">\\n</span> <span class=\"content-status\" style=\"display: none;\">\\n</span> <div class=\"add-edit-more\"><!-- --> <span class=\"add\"><a href=\"/ownlist/anime/add?selected_series_id=263&amp;hideLayout\" class=\"List_LightBox\">Add</a></span>\\n            -\\n            <span class=\"more\"><a href=\"#\">More</a></span></div></td> <td class=\"data score\"><a href=\"#\" class=\"link edit-disabled\"><span class=\"score-label score-9\">\\n              9\\n            </span></a> <!-- --></td> <td class=\"data type\">\\n          TV\\n        </td> <td class=\"data progress\"><div class=\"progress-263\"><span><a href=\"#\" class=\"link edit-disabled\">30</a> <!-- -->\\n              /\\n            </span> <span>75</span> <!-- --></div></td> <td id=\"tags-263\" class=\"data tags\"><div class=\"tags-263\"></div> <!-- --> <!-- --></td></tr>'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Examining the structure of one entry\n",
    "list_entries[0].html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e5d34921",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The element containing the anime title is located in this tag: \n",
    "# <a href=\"/anime/263/Hajime_no_Ippo\" class=\"link sort\">Hajime no Ippo</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1253c4c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Isolating only the anime names\n",
    "anime_names = [entry.find('td.data.title.clearfix')[0].find('a.link.sort')[0].text for entry in list_entries]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e516e3ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(anime_names) # Still the correct number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "20b07dcc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Hajime no Ippo',\n",
       " 'Kimetsu no Yaiba: Yuukaku-hen',\n",
       " 'Ousama Ranking',\n",
       " 'Tensei shitara Slime Datta Ken 2nd Season',\n",
       " 'Akame ga Kill!',\n",
       " 'Banana Fish',\n",
       " 'Black Clover',\n",
       " 'Boku no Hero Academia',\n",
       " 'Boku no Hero Academia 2nd Season',\n",
       " 'Boku no Hero Academia 3rd Season']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_names[:10] # Looks good"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "791e77dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#anime_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f923aeb8",
   "metadata": {},
   "source": [
    "## Getting all anime urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "121bac9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The relative URL for each anime can be found as the 'href' attribute of the anchor tags containing their titles\n",
    "\n",
    "anime_urls = [entry.find('td.data.title.clearfix')[0].find('a.link.sort')[0].attrs['href'] for entry in list_entries]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "da7521a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(anime_urls) # 90 titles --> 90 URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "9abac2b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/anime/263/Hajime_no_Ippo',\n",
       " '/anime/47778/Kimetsu_no_Yaiba__Yuukaku-hen',\n",
       " '/anime/40834/Ousama_Ranking',\n",
       " '/anime/39551/Tensei_shitara_Slime_Datta_Ken_2nd_Season',\n",
       " '/anime/22199/Akame_ga_Kill',\n",
       " '/anime/36649/Banana_Fish',\n",
       " '/anime/34572/Black_Clover',\n",
       " '/anime/31964/Boku_no_Hero_Academia',\n",
       " '/anime/33486/Boku_no_Hero_Academia_2nd_Season',\n",
       " '/anime/36456/Boku_no_Hero_Academia_3rd_Season']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_urls[:10] # We should convert these to absolute URLs in case we want to visit them in the future"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "39de72c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "base = 'https://myanimelist.net'\n",
    "\n",
    "# We only want to attach the base URL to the front if the href is indeed a relative URL (starting with '/')\n",
    "absolute_urls = [base+url if url.startswith('/') else url for url in anime_urls]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "4d516cc8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(absolute_urls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "2b15d871",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://myanimelist.net/anime/263/Hajime_no_Ippo',\n",
       " 'https://myanimelist.net/anime/47778/Kimetsu_no_Yaiba__Yuukaku-hen',\n",
       " 'https://myanimelist.net/anime/40834/Ousama_Ranking',\n",
       " 'https://myanimelist.net/anime/39551/Tensei_shitara_Slime_Datta_Ken_2nd_Season',\n",
       " 'https://myanimelist.net/anime/22199/Akame_ga_Kill',\n",
       " 'https://myanimelist.net/anime/36649/Banana_Fish',\n",
       " 'https://myanimelist.net/anime/34572/Black_Clover',\n",
       " 'https://myanimelist.net/anime/31964/Boku_no_Hero_Academia',\n",
       " 'https://myanimelist.net/anime/33486/Boku_no_Hero_Academia_2nd_Season',\n",
       " 'https://myanimelist.net/anime/36456/Boku_no_Hero_Academia_3rd_Season']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "absolute_urls[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "84399656",
   "metadata": {},
   "outputs": [],
   "source": [
    "#absolute_urls # Looks good"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f01a593c",
   "metadata": {},
   "source": [
    "## Getting all anime Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "24b646c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "cbd5c390",
   "metadata": {},
   "outputs": [],
   "source": [
    "anime_scores = [entry.find('span.score-label')[0].text for entry in list_entries]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "deb1e568",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(anime_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "3cedbce8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['9', '8', '9', '8', '7', '9', '10', '9', '9', '9']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_scores[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "0bfb0410",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'-', '10', '6', '7', '8', '9'}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(anime_scores)\n",
    "\n",
    "# We eventually want to convert these scores to some number data type so we need to handle the case of a score being '-'.\n",
    "# In MAL, if a score is '-' it means no score has been given yet (not that the score is necessarily a 0).\n",
    "# Therefore, we will convert the '-' scores to the special np.nan value indicating a missing value as to not cause inteference\n",
    "# when we recast the other scores to integer data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a6e53212",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting '-' scores to np.nan (missing value) and recasting all other scores as integer data types\n",
    "\n",
    "anime_scores = [np.nan if score=='-' else int(score) for score in anime_scores]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "2b81917c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(anime_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "234a9e20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[9, 8, 9, 8, 7, 9, 10, 9, 9, 9]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_scores[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "3e598c74",
   "metadata": {},
   "outputs": [],
   "source": [
    "#anime_scores # Looks good"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7cac6ff",
   "metadata": {},
   "source": [
    "## Getting all anime Statuses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "c015ce50",
   "metadata": {},
   "outputs": [],
   "source": [
    "anime_status = [entry.find('td.status')[0].attrs['class'][2] for entry in list_entries]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "2d0c6bf1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(anime_status)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "59f5f0f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['watching',\n",
       " 'watching',\n",
       " 'watching',\n",
       " 'watching',\n",
       " 'completed',\n",
       " 'completed',\n",
       " 'completed',\n",
       " 'completed',\n",
       " 'completed',\n",
       " 'completed']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_status[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "012ec61e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'completed', 'dropped', 'onhold', 'plantowatch', 'watching'}"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(anime_status) \n",
    "\n",
    "# These are already in an acceptable format for the purposes of this project, but we can perform minor cleaning later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "1eeec0aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "#anime_status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c4d79a3",
   "metadata": {},
   "source": [
    "## Getting all anime Types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "7cc2cdad",
   "metadata": {},
   "outputs": [],
   "source": [
    "anime_types = [entry.find('td.type')[0].text for entry in list_entries]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "dda1af74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(anime_types)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "269c4bf1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['TV', 'TV', 'TV', 'TV', 'TV', 'TV', 'TV', 'TV', 'TV', 'TV']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_types[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "3bef0a9c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Movie', 'ONA', 'Special', 'TV'}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(anime_types) # These are already in an acceptable format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "af4c7824",
   "metadata": {},
   "outputs": [],
   "source": [
    "#anime_types"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4157601d",
   "metadata": {},
   "source": [
    "## Getting anime Progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "211fbdb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The progress of each anime can be stored in one or two spans \n",
    "# (the progress of a Movie can only be 0 or 1 while that of a TV show is in the format [watched episodes] / [total episodes])\n",
    "\n",
    "progress_span_containers = [entry.find('td.progress')[0].find('span') for entry in list_entries]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "16566b57",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(progress_span_containers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "40171671",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element 'span' >, <Element 'span' >]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "progress_span_containers[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "e104279b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<span><a href=\"#\" class=\"link edit-disabled\">30</a> <!-- -->\\n              /\\n            </span>'"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "progress_span_containers[0][0].html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e9aa1ea4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<span>75</span>'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "progress_span_containers[0][1].html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "802bf449",
   "metadata": {},
   "source": [
    "**Plan:** In the case of an anime having two spans in their progress containers we know the final string should be in the format: (watched episodes) / (total episodes). If an anime only has one span, we will simply extract the text as we know it likely be just a 1 for completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "be723d45",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Joining the two spans together as strings or just extracting the text if only one span is present\n",
    "\n",
    "anime_progress = [\"\".join([span.text[:-2]+span.text[-1] if ('/' in span.text) else span.text for span in c]) for c in progress_span_containers]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "8a144102",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(anime_progress)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "79c68331",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['30/75', '3/-', '11/23', '2/12', '24', '24', '170', '13', '25', '25']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anime_progress[:10] # Looks good"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "e2b0c763",
   "metadata": {},
   "outputs": [],
   "source": [
    "#anime_progress # We will keep these as strings for now as we only want to collect the data not transform it too much"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "a0d79286",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Closing our Async session now that all the web scraping is done\n",
    "\n",
    "await session.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d151a1a5",
   "metadata": {},
   "source": [
    "# Combining lists into a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "043ff90e",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_animelist = pd.DataFrame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "f9788c98",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_animelist[\"Name\"] = anime_names\n",
    "user_animelist[\"User Score\"] = anime_scores\n",
    "user_animelist[\"Progress\"] = anime_progress\n",
    "user_animelist[\"Status\"] = anime_status\n",
    "user_animelist[\"Type\"] = anime_types\n",
    "user_animelist[\"URL\"] = absolute_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "ce00d2be",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_rows\", None, \"display.max_columns\", None, \"display.max_colwidth\", None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "88cf0693",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a function to clean up each anime's status as we mentioned above\n",
    "\n",
    "def convertStatus(item):\n",
    "    if item=='completed':\n",
    "        return \"Completed\"\n",
    "    elif item=='onhold':\n",
    "        return \"On Hold\"\n",
    "    elif item=='watching':\n",
    "        return \"Watching\"\n",
    "    elif item=='plantowatch':\n",
    "        return \"Plan To Watch\"\n",
    "    elif item=='dropped':\n",
    "        return \"Dropped\"\n",
    "    else:\n",
    "        return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "f5897546",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_animelist[\"Status\"] = user_animelist[\"Status\"].apply(convertStatus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "d269cbf8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(user_animelist) # Looks like we have all 90 entries intact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "617b220b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>User Score</th>\n",
       "      <th>Progress</th>\n",
       "      <th>Status</th>\n",
       "      <th>Type</th>\n",
       "      <th>URL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hajime no Ippo</td>\n",
       "      <td>9.0</td>\n",
       "      <td>30/75</td>\n",
       "      <td>Watching</td>\n",
       "      <td>TV</td>\n",
       "      <td>https://myanimelist.net/anime/263/Hajime_no_Ippo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Kimetsu no Yaiba: Yuukaku-hen</td>\n",
       "      <td>8.0</td>\n",
       "      <td>3/-</td>\n",
       "      <td>Watching</td>\n",
       "      <td>TV</td>\n",
       "      <td>https://myanimelist.net/anime/47778/Kimetsu_no_Yaiba__Yuukaku-hen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ousama Ranking</td>\n",
       "      <td>9.0</td>\n",
       "      <td>11/23</td>\n",
       "      <td>Watching</td>\n",
       "      <td>TV</td>\n",
       "      <td>https://myanimelist.net/anime/40834/Ousama_Ranking</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tensei shitara Slime Datta Ken 2nd Season</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2/12</td>\n",
       "      <td>Watching</td>\n",
       "      <td>TV</td>\n",
       "      <td>https://myanimelist.net/anime/39551/Tensei_shitara_Slime_Datta_Ken_2nd_Season</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Akame ga Kill!</td>\n",
       "      <td>7.0</td>\n",
       "      <td>24</td>\n",
       "      <td>Completed</td>\n",
       "      <td>TV</td>\n",
       "      <td>https://myanimelist.net/anime/22199/Akame_ga_Kill</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Name  User Score Progress     Status  \\\n",
       "0                             Hajime no Ippo         9.0    30/75   Watching   \n",
       "1              Kimetsu no Yaiba: Yuukaku-hen         8.0      3/-   Watching   \n",
       "2                             Ousama Ranking         9.0    11/23   Watching   \n",
       "3  Tensei shitara Slime Datta Ken 2nd Season         8.0     2/12   Watching   \n",
       "4                             Akame ga Kill!         7.0       24  Completed   \n",
       "\n",
       "  Type  \\\n",
       "0   TV   \n",
       "1   TV   \n",
       "2   TV   \n",
       "3   TV   \n",
       "4   TV   \n",
       "\n",
       "                                                                             URL  \n",
       "0                               https://myanimelist.net/anime/263/Hajime_no_Ippo  \n",
       "1              https://myanimelist.net/anime/47778/Kimetsu_no_Yaiba__Yuukaku-hen  \n",
       "2                             https://myanimelist.net/anime/40834/Ousama_Ranking  \n",
       "3  https://myanimelist.net/anime/39551/Tensei_shitara_Slime_Datta_Ken_2nd_Season  \n",
       "4                              https://myanimelist.net/anime/22199/Akame_ga_Kill  "
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_animelist.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "070a048d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving our final DataFrame as a CSV file for future use\n",
    "\n",
    "user_animelist.to_csv('Scraped_MAL_Data.csv', index=False, encoding='utf-8')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}