Web Scrapping for Data/Web Scrapping -- Batch Download CABI Species Distribution Data Automatically.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### This code shows how to download species distribution data from cabi.org automatically using python. The key python library used here is Selenium. Details on how to use selenium with python can be found here: https://selenium-python.readthedocs.io/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Another notebook documented the code to check whether cabi has the distribution map for species or not (can be found here https://github.com/wanwanliang/For-Open-Science/blob/master/Web%20Scrapping%20CABI%20to%20find%20information%20available%20for%20Species%20with%20Parallel.ipynb). This code was used to find all the species on cabi with distribution map/data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8157, 5)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Scientific.name</th>\n",
       "      <th>URL</th>\n",
       "      <th>Common.name</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Epuraea imperialis</td>\n",
       "      <td>https://www.cabi.org/isc/datasheet/52031793</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Basic</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mycoplasma putrefaciens</td>\n",
       "      <td>https://www.cabi.org/isc/datasheet/74521</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Basic</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Leptospira kirschneri</td>\n",
       "      <td>https://www.cabi.org/isc/datasheet/77190</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Basic</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Salmonella paratyphi</td>\n",
       "      <td>https://www.cabi.org/isc/datasheet/65300</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Basic</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cytospora parasitica</td>\n",
       "      <td>https://www.cabi.org/isc/datasheet/87416147</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Basic</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Scientific.name                                          URL  \\\n",
       "0       Epuraea imperialis  https://www.cabi.org/isc/datasheet/52031793   \n",
       "1  Mycoplasma putrefaciens     https://www.cabi.org/isc/datasheet/74521   \n",
       "2    Leptospira kirschneri     https://www.cabi.org/isc/datasheet/77190   \n",
       "3     Salmonella paratyphi     https://www.cabi.org/isc/datasheet/65300   \n",
       "4     Cytospora parasitica  https://www.cabi.org/isc/datasheet/87416147   \n",
       "\n",
       "  Common.name Coverage  map  \n",
       "0         NaN    Basic    1  \n",
       "1         NaN    Basic    1  \n",
       "2         NaN    Basic    1  \n",
       "3         NaN    Basic    1  \n",
       "4         NaN    Basic    1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'https://www.cabi.org/isc/datasheet/52031793'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.chdir(\"C:/Users/liang/Desktop/cabi\")\n",
    "\n",
    "dt=pd.read_csv(\"CABIwithmap.csv\",sep=\",\",header=0,encoding = \"ISO-8859-1\")\n",
    "display(dt.shape)\n",
    "display(dt.head())\n",
    "dt[\"URL\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### The URLs in the dataframe will be the websites to do the scrapping.  Here I'll first show an example, and then use a loop to batch download data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start webdriver and load the website to do the scrapping\n",
    "driver=webdriver.Chrome()\n",
    "url=dt[\"URL\"][0]\n",
    "driver.get(url)\n",
    "time.sleep(2)\n",
    "\n",
    "# find the button where we need to click in order to download data. the class of that button in cabi named \"Product_export-sml\"\n",
    "button=driver.find_elements_by_class_name(\"Product_export-sml\")\n",
    "# the function of the below code is to click that button\n",
    "# there are two buttons with the same class name, use button[0] to download kml file and button[1] to download csv file\n",
    "button[1].click()\n",
    "time.sleep(5) \n",
    "\n",
    "# in cabi, a new window pop up and need another click action. the class of this button is named “btn-primary”\n",
    "driver.find_elements_by_class_name(\"btn-primary\")[1].click()\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### And that's it. This simple code do the magic and check your folder for the downloaded zip file. I guess the above code doesn't work well in the jupyter notebook. The loop below works well to batch download the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver=webdriver.Chrome()\n",
    "\n",
    "for i in np.arange(10):\n",
    "\n",
    "    url=dt[\"URL\"][i]\n",
    "\n",
    "    driver.get(url)\n",
    "    time.sleep(2)\n",
    "\n",
    "    # find and click 1st button\n",
    "    driver.find_elements_by_class_name(\"Product_export-sml\")[1].click()\n",
    "    time.sleep(5)\n",
    "\n",
    "    # find and click 2nd button\n",
    "    driver.find_element_by_class_name(\"btn-primary\").click()\n",
    "   \n",
    "\n",
    "driver.quit()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}