CS-180-Final-Project.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e6661a62",
   "metadata": {},
   "source": [
    "GitHub repository Link: https://github.com/adzmco23/cs180-group23-project\n",
    "\n",
    "## Data Collection\n",
    "\n",
    "The dataset used was taken from Humanitarian Data Exchange (HDX). It is the World Food Programme Price Database which contains data regarding food prices in the Philippines. The data in this dataset spans from 2000 - 2023 garnering 184,714 entries (rows). The dataset is in the file named `wfp_food_prices_phl.csv` in the root folder but it was also made publicly available by HDX through the following:\n",
    "\n",
    "Link: https://data.humdata.org/dataset/wfp-food-prices-for-philippines?force_layout=desktop.\n",
    "\n",
    "We also manually added the yearly inflation rate from 2000 to 2023 which we got from WorldData.info\n",
    "\n",
    "Link: https://www.worlddata.info/asia/philippines/inflation-rates.php\n",
    "\n",
    "#### The dataset is imported as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "de9c923d",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>admin1</th>\n",
       "      <th>admin2</th>\n",
       "      <th>market</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>category</th>\n",
       "      <th>commodity</th>\n",
       "      <th>unit</th>\n",
       "      <th>priceflag</th>\n",
       "      <th>pricetype</th>\n",
       "      <th>currency</th>\n",
       "      <th>price</th>\n",
       "      <th>usdprice</th>\n",
       "      <th>inflation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/15/00</td>\n",
       "      <td>National Capital region</td>\n",
       "      <td>Metropolitan Manila</td>\n",
       "      <td>Metro Manila</td>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>cereals and tubers</td>\n",
       "      <td>Maize flour (yellow)</td>\n",
       "      <td>KG</td>\n",
       "      <td>actual</td>\n",
       "      <td>Retail</td>\n",
       "      <td>PHP</td>\n",
       "      <td>15.00</td>\n",
       "      <td>0.3717</td>\n",
       "      <td>3.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/15/00</td>\n",
       "      <td>National Capital region</td>\n",
       "      <td>Metropolitan Manila</td>\n",
       "      <td>Metro Manila</td>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>cereals and tubers</td>\n",
       "      <td>Rice (milled, superior)</td>\n",
       "      <td>KG</td>\n",
       "      <td>actual</td>\n",
       "      <td>Retail</td>\n",
       "      <td>PHP</td>\n",
       "      <td>20.00</td>\n",
       "      <td>0.4957</td>\n",
       "      <td>3.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/15/00</td>\n",
       "      <td>National Capital region</td>\n",
       "      <td>Metropolitan Manila</td>\n",
       "      <td>Metro Manila</td>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>cereals and tubers</td>\n",
       "      <td>Rice (milled, superior)</td>\n",
       "      <td>KG</td>\n",
       "      <td>actual</td>\n",
       "      <td>Wholesale</td>\n",
       "      <td>PHP</td>\n",
       "      <td>18.35</td>\n",
       "      <td>0.4548</td>\n",
       "      <td>3.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/15/00</td>\n",
       "      <td>National Capital region</td>\n",
       "      <td>Metropolitan Manila</td>\n",
       "      <td>Metro Manila</td>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>cereals and tubers</td>\n",
       "      <td>Rice (regular, milled)</td>\n",
       "      <td>KG</td>\n",
       "      <td>actual</td>\n",
       "      <td>Retail</td>\n",
       "      <td>PHP</td>\n",
       "      <td>18.00</td>\n",
       "      <td>0.4461</td>\n",
       "      <td>3.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1/15/00</td>\n",
       "      <td>National Capital region</td>\n",
       "      <td>Metropolitan Manila</td>\n",
       "      <td>Metro Manila</td>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>cereals and tubers</td>\n",
       "      <td>Rice (regular, milled)</td>\n",
       "      <td>KG</td>\n",
       "      <td>actual</td>\n",
       "      <td>Wholesale</td>\n",
       "      <td>PHP</td>\n",
       "      <td>16.35</td>\n",
       "      <td>0.4052</td>\n",
       "      <td>3.98</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      date                   admin1               admin2        market  \\\n",
       "0  1/15/00  National Capital region  Metropolitan Manila  Metro Manila   \n",
       "1  1/15/00  National Capital region  Metropolitan Manila  Metro Manila   \n",
       "2  1/15/00  National Capital region  Metropolitan Manila  Metro Manila   \n",
       "3  1/15/00  National Capital region  Metropolitan Manila  Metro Manila   \n",
       "4  1/15/00  National Capital region  Metropolitan Manila  Metro Manila   \n",
       "\n",
       "    latitude   longitude            category                commodity unit  \\\n",
       "0  14.604167  120.982222  cereals and tubers     Maize flour (yellow)   KG   \n",
       "1  14.604167  120.982222  cereals and tubers  Rice (milled, superior)   KG   \n",
       "2  14.604167  120.982222  cereals and tubers  Rice (milled, superior)   KG   \n",
       "3  14.604167  120.982222  cereals and tubers   Rice (regular, milled)   KG   \n",
       "4  14.604167  120.982222  cereals and tubers   Rice (regular, milled)   KG   \n",
       "\n",
       "  priceflag  pricetype currency  price  usdprice  inflation  \n",
       "0    actual     Retail      PHP  15.00    0.3717       3.98  \n",
       "1    actual     Retail      PHP  20.00    0.4957       3.98  \n",
       "2    actual  Wholesale      PHP  18.35    0.4548       3.98  \n",
       "3    actual     Retail      PHP  18.00    0.4461       3.98  \n",
       "4    actual  Wholesale      PHP  16.35    0.4052       3.98  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#import the dataset\n",
    "data = pd.read_csv('wfp_food_prices_phl.csv')\n",
    "data.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ef870532",
   "metadata": {},
   "source": [
    "#### Inspecting the columns of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "54bcfec5",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns:  Index(['date', 'admin1', 'admin2', 'market', 'latitude', 'longitude',\n",
      "       'category', 'commodity', 'unit', 'priceflag', 'pricetype', 'currency',\n",
      "       'price', 'usdprice', 'inflation'],\n",
      "      dtype='object')\n",
      "Data Shape:  (184714, 15)\n"
     ]
    }
   ],
   "source": [
    "#check the columns of the dataset\n",
    "print(\"Columns: \", data.columns)\n",
    "#check the shape of the dataset\n",
    "print(\"Data Shape: \",data.shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8e4f71c3",
   "metadata": {},
   "source": [
    "# The columns are the following:\n",
    "\n",
    "1) date - The date of the data entry\n",
    "2) admin1 - The region of the data entry\n",
    "3) admin2 - The province of the data entry\n",
    "4) market - The city of the data entry\n",
    "5) latitude - The latitude of the market\n",
    "6) longitude - The longitude of the market\n",
    "7) category - The category of the food item\n",
    "8) commodity - The food item\n",
    "9) unit - The unit of the food item\n",
    "10) priceflag - If it is the actual or projected or aggregate price.\n",
    "11) pricetype - The type of price (retail, wholesale, etc)\n",
    "12) currency - The currency of the price\n",
    "13) price - The price of the food item\n",
    "14) usdprice - The price of the food item in USD\n",
    "15) inflation - The yearly inflation rate\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c9284643",
   "metadata": {},
   "source": [
    "## Data preprocessing\n",
    "The data was preprocessed as follows, where the columns to be used in the features and target data were identified, as well as the train and test data:\n",
    "\n",
    "#### Inspecting the datatype of each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "88827659",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date          object\n",
       "admin1        object\n",
       "admin2        object\n",
       "market        object\n",
       "latitude     float64\n",
       "longitude    float64\n",
       "category      object\n",
       "commodity     object\n",
       "unit          object\n",
       "priceflag     object\n",
       "pricetype     object\n",
       "currency      object\n",
       "price        float64\n",
       "usdprice     float64\n",
       "inflation    float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check the datatypes of each column\n",
    "data.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ed257359",
   "metadata": {},
   "source": [
    "#### Converting data into appropriate types\n",
    "\n",
    "The features with object datatypes must be converted to their more appropriate datatypes. In this case, however, we will only convert (decompose) the <i>date</i> column into (<i>year, month</i>). For the other categorical columns (<i>commodity, pricetype</i>, etc.), we will just get dummy values in order to fit them to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ee8202bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>admin1</th>\n",
       "      <th>admin2</th>\n",
       "      <th>market</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>category</th>\n",
       "      <th>commodity</th>\n",
       "      <th>unit</th>\n",
       "      <th>priceflag</th>\n",
       "      <th>pricetype</th>\n",
       "      <th>currency</th>\n",
       "      <th>price</th>\n",
       "      <th>usdprice</th>\n",
       "      <th>inflation</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2000-01-15</td>\n",
       "      <td>National Capital region</td>\n",
       "      <td>Metropolitan Manila</td>\n",
       "      <td>Metro Manila</td>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>cereals and tubers</td>\n",
       "      <td>Maize flour (yellow)</td>\n",
       "      <td>KG</td>\n",
       "      <td>actual</td>\n",
       "      <td>Retail</td>\n",
       "      <td>PHP</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.3717</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000-01-15</td>\n",
       "      <td>National Capital region</td>\n",
       "      <td>Metropolitan Manila</td>\n",
       "      <td>Metro Manila</td>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>cereals and tubers</td>\n",
       "      <td>Rice (milled, superior)</td>\n",
       "      <td>KG</td>\n",
       "      <td>actual</td>\n",
       "      <td>Retail</td>\n",
       "      <td>PHP</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.4957</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date                   admin1               admin2        market  \\\n",
       "0 2000-01-15  National Capital region  Metropolitan Manila  Metro Manila   \n",
       "1 2000-01-15  National Capital region  Metropolitan Manila  Metro Manila   \n",
       "\n",
       "    latitude   longitude            category                commodity unit  \\\n",
       "0  14.604167  120.982222  cereals and tubers     Maize flour (yellow)   KG   \n",
       "1  14.604167  120.982222  cereals and tubers  Rice (milled, superior)   KG   \n",
       "\n",
       "  priceflag pricetype currency  price  usdprice  inflation  year  month  \n",
       "0    actual    Retail      PHP   15.0    0.3717       3.98  2000      1  \n",
       "1    actual    Retail      PHP   20.0    0.4957       3.98  2000      1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Change the datatype of date and get the year and month\n",
    "#Add a year and month column which will be used for our regression model\n",
    "data['date'] = data['date'].astype('datetime64[ns]')\n",
    "data['year'] = data['date'].dt.year\n",
    "data['month'] = data['date'].dt.month\n",
    "data.head(2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "88c2d0e6",
   "metadata": {},
   "source": [
    "#### Removing invalid entries:\n",
    "Next, the data must be cleaned by removing invalid entries. This was done by dropping the null values or rows with price == 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c7a21153",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Drop all the null values and the rows with price == 0 \n",
    "data.dropna(inplace= True)\n",
    "data.drop(data.loc[data['price']==0].index, inplace=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "715bf8d4",
   "metadata": {},
   "source": [
    "#### Removing unnecessary columns:\n",
    "The dataset was refined further by excluding columns that are either redundant or irrelevant to the precitive model (since including such features would only introduce noise during training). For example, <i>date</i> would redundant to <i>month</i> and <i>year</i> columns), while <i>admin1, admin2, </i>and <i>market</i> would be redundant to <i>latitude-longitude</i>. We chose to use latitude-longitude to also help estimate the price of food items in areas where there are no available data entries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ce201a83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>commodity</th>\n",
       "      <th>priceflag</th>\n",
       "      <th>pricetype</th>\n",
       "      <th>price</th>\n",
       "      <th>inflation</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>Maize flour (yellow)</td>\n",
       "      <td>actual</td>\n",
       "      <td>Retail</td>\n",
       "      <td>15.00</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>Rice (milled, superior)</td>\n",
       "      <td>actual</td>\n",
       "      <td>Retail</td>\n",
       "      <td>20.00</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>Rice (milled, superior)</td>\n",
       "      <td>actual</td>\n",
       "      <td>Wholesale</td>\n",
       "      <td>18.35</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    latitude   longitude                commodity priceflag  pricetype  price  \\\n",
       "0  14.604167  120.982222     Maize flour (yellow)    actual     Retail  15.00   \n",
       "1  14.604167  120.982222  Rice (milled, superior)    actual     Retail  20.00   \n",
       "2  14.604167  120.982222  Rice (milled, superior)    actual  Wholesale  18.35   \n",
       "\n",
       "   inflation  year  month  \n",
       "0       3.98  2000      1  \n",
       "1       3.98  2000      1  \n",
       "2       3.98  2000      1  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Drop the unnecessary columns from the dataset\n",
    "data = data.drop(['date', 'admin1','admin2','market','category', 'currency', 'unit'\n",
    "        ,'usdprice'], axis='columns')\n",
    "data.head(3)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "90fe25d2",
   "metadata": {},
   "source": [
    "#### Converting categorical columns:\n",
    "Some of the features in the dataset are categorical and cannot be properly represented (or evaluated) using integer label assignments within their columns, especially since the categories within the feature do not naturally have a relationship with each other (e.g., <i>commodity_banana</i> and <i>commodity_rice</i> under <i>commodity</i> are unrelated). Hence, the categories within each categorical feature (<i>commodity, priceflag,</i> and <i>pricetype</i>) are converted into their own columns using dummy values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "56be66e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>price</th>\n",
       "      <th>inflation</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>commodity_Anchovies</th>\n",
       "      <th>commodity_Bananas (lakatan)</th>\n",
       "      <th>commodity_Bananas (latundan)</th>\n",
       "      <th>commodity_Bananas (saba)</th>\n",
       "      <th>...</th>\n",
       "      <th>commodity_Sweet potatoes</th>\n",
       "      <th>commodity_Taro</th>\n",
       "      <th>commodity_Tomatoes</th>\n",
       "      <th>commodity_Water spinach</th>\n",
       "      <th>priceflag_actual</th>\n",
       "      <th>priceflag_actual,aggregate</th>\n",
       "      <th>priceflag_aggregate</th>\n",
       "      <th>pricetype_Farm Gate</th>\n",
       "      <th>pricetype_Retail</th>\n",
       "      <th>pricetype_Wholesale</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>15.0</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    latitude   longitude  price  inflation  year  month  commodity_Anchovies  \\\n",
       "0  14.604167  120.982222   15.0       3.98  2000      1                    0   \n",
       "1  14.604167  120.982222   20.0       3.98  2000      1                    0   \n",
       "\n",
       "   commodity_Bananas (lakatan)  commodity_Bananas (latundan)  \\\n",
       "0                            0                             0   \n",
       "1                            0                             0   \n",
       "\n",
       "   commodity_Bananas (saba)  ...  commodity_Sweet potatoes  commodity_Taro  \\\n",
       "0                         0  ...                         0               0   \n",
       "1                         0  ...                         0               0   \n",
       "\n",
       "   commodity_Tomatoes  commodity_Water spinach  priceflag_actual  \\\n",
       "0                   0                        0                 1   \n",
       "1                   0                        0                 1   \n",
       "\n",
       "   priceflag_actual,aggregate  priceflag_aggregate  pricetype_Farm Gate  \\\n",
       "0                           0                    0                    0   \n",
       "1                           0                    0                    0   \n",
       "\n",
       "   pricetype_Retail  pricetype_Wholesale  \n",
       "0                 1                    0  \n",
       "1                 1                    0  \n",
       "\n",
       "[2 rows x 85 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Get dummy values for the categorical columns\n",
    "data = pd.get_dummies(data=data)\n",
    "data.head(2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "122802d4",
   "metadata": {},
   "source": [
    "#### Preparing the train and test sets\n",
    "The train and test sets are divided into a random <i>80-20</i> partition  as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "792a5dc0",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(109930, 84)\n",
      "(27483, 84)\n",
      "(109930,)\n",
      "(27483,)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>inflation</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>commodity_Anchovies</th>\n",
       "      <th>commodity_Bananas (lakatan)</th>\n",
       "      <th>commodity_Bananas (latundan)</th>\n",
       "      <th>commodity_Bananas (saba)</th>\n",
       "      <th>commodity_Beans (green, fresh)</th>\n",
       "      <th>...</th>\n",
       "      <th>commodity_Sweet potatoes</th>\n",
       "      <th>commodity_Taro</th>\n",
       "      <th>commodity_Tomatoes</th>\n",
       "      <th>commodity_Water spinach</th>\n",
       "      <th>priceflag_actual</th>\n",
       "      <th>priceflag_actual,aggregate</th>\n",
       "      <th>priceflag_aggregate</th>\n",
       "      <th>pricetype_Farm Gate</th>\n",
       "      <th>pricetype_Retail</th>\n",
       "      <th>pricetype_Wholesale</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15312</th>\n",
       "      <td>16.016667</td>\n",
       "      <td>120.233333</td>\n",
       "      <td>3.03</td>\n",
       "      <td>2012</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51810</th>\n",
       "      <td>14.604167</td>\n",
       "      <td>120.982222</td>\n",
       "      <td>2.39</td>\n",
       "      <td>2020</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34435</th>\n",
       "      <td>16.486093</td>\n",
       "      <td>121.146518</td>\n",
       "      <td>2.39</td>\n",
       "      <td>2020</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35778</th>\n",
       "      <td>11.706772</td>\n",
       "      <td>122.370090</td>\n",
       "      <td>2.39</td>\n",
       "      <td>2020</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49991</th>\n",
       "      <td>8.040911</td>\n",
       "      <td>123.799419</td>\n",
       "      <td>2.39</td>\n",
       "      <td>2020</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73408</th>\n",
       "      <td>13.146926</td>\n",
       "      <td>123.750464</td>\n",
       "      <td>3.93</td>\n",
       "      <td>2021</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137475</th>\n",
       "      <td>10.667360</td>\n",
       "      <td>122.946930</td>\n",
       "      <td>5.80</td>\n",
       "      <td>2022</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77256</th>\n",
       "      <td>10.132101</td>\n",
       "      <td>124.834680</td>\n",
       "      <td>3.93</td>\n",
       "      <td>2021</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59971</th>\n",
       "      <td>13.137222</td>\n",
       "      <td>123.734444</td>\n",
       "      <td>2.39</td>\n",
       "      <td>2020</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55570</th>\n",
       "      <td>13.621347</td>\n",
       "      <td>123.183922</td>\n",
       "      <td>2.39</td>\n",
       "      <td>2020</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>109930 rows × 84 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         latitude   longitude  inflation  year  month  commodity_Anchovies  \\\n",
       "15312   16.016667  120.233333       3.03  2012     11                    0   \n",
       "51810   14.604167  120.982222       2.39  2020      9                    0   \n",
       "34435   16.486093  121.146518       2.39  2020      5                    0   \n",
       "35778   11.706772  122.370090       2.39  2020      5                    0   \n",
       "49991    8.040911  123.799419       2.39  2020      8                    0   \n",
       "...           ...         ...        ...   ...    ...                  ...   \n",
       "73408   13.146926  123.750464       3.93  2021      2                    0   \n",
       "137475  10.667360  122.946930       5.80  2022      7                    0   \n",
       "77256   10.132101  124.834680       3.93  2021      3                    0   \n",
       "59971   13.137222  123.734444       2.39  2020     10                    0   \n",
       "55570   13.621347  123.183922       2.39  2020      9                    0   \n",
       "\n",
       "        commodity_Bananas (lakatan)  commodity_Bananas (latundan)  \\\n",
       "15312                             0                             0   \n",
       "51810                             0                             0   \n",
       "34435                             0                             0   \n",
       "35778                             0                             0   \n",
       "49991                             0                             0   \n",
       "...                             ...                           ...   \n",
       "73408                             0                             0   \n",
       "137475                            0                             0   \n",
       "77256                             0                             0   \n",
       "59971                             0                             0   \n",
       "55570                             0                             0   \n",
       "\n",
       "        commodity_Bananas (saba)  commodity_Beans (green, fresh)  ...  \\\n",
       "15312                          0                               0  ...   \n",
       "51810                          0                               0  ...   \n",
       "34435                          0                               1  ...   \n",
       "35778                          0                               0  ...   \n",
       "49991                          0                               0  ...   \n",
       "...                          ...                             ...  ...   \n",
       "73408                          0                               0  ...   \n",
       "137475                         0                               0  ...   \n",
       "77256                          0                               0  ...   \n",
       "59971                          0                               0  ...   \n",
       "55570                          0                               0  ...   \n",
       "\n",
       "        commodity_Sweet potatoes  commodity_Taro  commodity_Tomatoes  \\\n",
       "15312                          0               0                   0   \n",
       "51810                          0               0                   0   \n",
       "34435                          0               0                   0   \n",
       "35778                          0               0                   0   \n",
       "49991                          0               0                   0   \n",
       "...                          ...             ...                 ...   \n",
       "73408                          0               0                   0   \n",
       "137475                         0               0                   0   \n",
       "77256                          0               0                   0   \n",
       "59971                          0               0                   0   \n",
       "55570                          0               0                   0   \n",
       "\n",
       "        commodity_Water spinach  priceflag_actual  priceflag_actual,aggregate  \\\n",
       "15312                         0                 1                           0   \n",
       "51810                         0                 1                           0   \n",
       "34435                         0                 1                           0   \n",
       "35778                         0                 1                           0   \n",
       "49991                         0                 1                           0   \n",
       "...                         ...               ...                         ...   \n",
       "73408                         0                 1                           0   \n",
       "137475                        0                 1                           0   \n",
       "77256                         0                 1                           0   \n",
       "59971                         0                 0                           0   \n",
       "55570                         0                 0                           0   \n",
       "\n",
       "        priceflag_aggregate  pricetype_Farm Gate  pricetype_Retail  \\\n",
       "15312                     0                    0                 1   \n",
       "51810                     0                    0                 1   \n",
       "34435                     0                    0                 1   \n",
       "35778                     0                    0                 1   \n",
       "49991                     0                    0                 1   \n",
       "...                     ...                  ...               ...   \n",
       "73408                     0                    0                 1   \n",
       "137475                    0                    0                 1   \n",
       "77256                     0                    0                 1   \n",
       "59971                     1                    0                 1   \n",
       "55570                     1                    0                 1   \n",
       "\n",
       "        pricetype_Wholesale  \n",
       "15312                     0  \n",
       "51810                     0  \n",
       "34435                     0  \n",
       "35778                     0  \n",
       "49991                     0  \n",
       "...                     ...  \n",
       "73408                     0  \n",
       "137475                    0  \n",
       "77256                     0  \n",
       "59971                     0  \n",
       "55570                     0  \n",
       "\n",
       "[109930 rows x 84 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Divide the data into test and training sets\n",
    "X = data.drop(['price'], axis='columns')\n",
    "y = data['price'].values\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=26)\n",
    "\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)\n",
    "display(X_train)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "02e2a0a9",
   "metadata": {},
   "source": [
    "#### Normalizing the data\n",
    "In order to keep the feature ranges similar, the non-dummy columns were normalized using MinMaxScaler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f4e0fe9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "sc = MinMaxScaler()\n",
    "X_train[['latitude','longitude','inflation','year','month']] = sc.fit_transform(X_train[['latitude','longitude','inflation','year','month']])\n",
    "X_test[['latitude','longitude','inflation','year','month']] = sc.transform(X_test[['latitude','longitude','inflation','year','month']])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f4ea9741",
   "metadata": {},
   "source": [
    "#### Summary\n",
    "The dataset was preprocessed as follows\n",
    "\n",
    "1. The following columns were removed:\n",
    "\n",
    "    1) <b>date</b> - since there is already a month and year column which were extracted from the date column.\n",
    "    2) <b>admin1, admin2, market</b> - longitude and latitude already represents the precise location\n",
    "    3) <b>category</b> - this decreases the accuracy of the model because there are many commodities in a category and all of them have different prices.\n",
    "    4) <b>unit, currency, usd price</b> - all of these columns are irrelevant for the model since unit is just the measurement, all currencies are Philippine peso, and USD price is just the converted price.\n",
    "\n",
    "2. The following categorical columns were converted into indicator/dummy variables:\n",
    "    1) <b>commodity</b> - 73 total categories\n",
    "    2) <b>priceflag</b> - 3 total categories\n",
    "    3) <b>pricetype</b> - 3 total categories\n",
    "\n",
    "3. Non-dummy columns were normalized: latitude, longitude, inflation, year, month\n",
    "\n",
    "##### The remaining features are then the independent variables used to predict the price."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fff74d34",
   "metadata": {},
   "source": [
    "## Model\n",
    "### Performing linear regression on the dataset.\n",
    "Since the aim is to predict a continuous value (price) from a given set of factors, a regression model would be the most appropriate for the dataset. \n",
    "\n",
    "Two linear regression models were tested (model reference: [sklearn](https://scikit-learn.org/stable/modules/linear_model.html))\n",
    "1. <b>sklearn's LinearRegression()</b> - ordinary least squares regression. This basic linear regression method would try to find the coefficients to fit the model with such that the residual sum of squares between the observed targets in the dataset would be minimized.\n",
    "\n",
    "2. <b>sklearn's RidgeCV()</b> - Ridge regression with cross validation. This uses the linear least squares model but builds upon it by incorporating a regularization method of imposing a penalty on the size of the coefficients. This regularization attempts to address the issue of multicollinearity (when multiple independent variables are highly correlated and thus muddles up determination of indiviual effects - this may be helpful for tight relationships between variables like commodity and location). Thus, this method is generally more stable. Finally, the built in cross validation allows the model to determine the best alpha parameter from multiple tests.\n",
    "\n",
    "#### Training and testing the two regression models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fe9198a5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Train the model using the training set\n",
    "from sklearn.linear_model import LinearRegression, RidgeCV\n",
    "regressor1 = LinearRegression()\n",
    "regressor2 = RidgeCV(alphas=np.logspace(1, 2, 50))\n",
    "\n",
    "regressor1.fit(X_train, y_train)\n",
    "regressor2.fit(X_train, y_train)\n",
    "\n",
    "#Use the model on the testing set\n",
    "test_predictions1 = regressor1.predict(X_test) # predictions using LinearRegression (OLS)\n",
    "test_predictions2 = regressor2.predict(X_test) # predictions using RidgeCV"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4b610437",
   "metadata": {},
   "source": [
    "### Results.\n",
    "The results of the two regression models in the following section were evaluated according to the following metrics: <b><i>MAE, MSE, RMSE, r<sup>2</sup></i></b>.\n",
    "\n",
    "#### Using LinearRegression (Ordinary Least Squares) and default parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "50598ae2",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Actual</th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>238.36</td>\n",
       "      <td>237.250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40.00</td>\n",
       "      <td>49.250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37.50</td>\n",
       "      <td>35.750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>42.77</td>\n",
       "      <td>24.250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>138.00</td>\n",
       "      <td>205.500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27478</th>\n",
       "      <td>62.00</td>\n",
       "      <td>48.750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27479</th>\n",
       "      <td>102.75</td>\n",
       "      <td>142.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27480</th>\n",
       "      <td>207.50</td>\n",
       "      <td>188.125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27481</th>\n",
       "      <td>36.74</td>\n",
       "      <td>28.250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27482</th>\n",
       "      <td>198.33</td>\n",
       "      <td>210.875</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27483 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Actual  Predicted\n",
       "0      238.36    237.250\n",
       "1       40.00     49.250\n",
       "2       37.50     35.750\n",
       "3       42.77     24.250\n",
       "4      138.00    205.500\n",
       "...       ...        ...\n",
       "27478   62.00     48.750\n",
       "27479  102.75    142.000\n",
       "27480  207.50    188.125\n",
       "27481   36.74     28.250\n",
       "27482  198.33    210.875\n",
       "\n",
       "[27483 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE = 24.167676199832623\n",
      "MSE = 1477.3576017856494\n",
      "RMSE = 38.436409845166985\n",
      "r2 = 0.8714324551306367\n"
     ]
    }
   ],
   "source": [
    "#get the MAE,MSE,RMSE, and R2 values to evaluate the model \n",
    "from sklearn.metrics import mean_absolute_error,mean_squared_error, r2_score\n",
    "\n",
    "MAE = mean_absolute_error(y_test,test_predictions1)\n",
    "MSE = mean_squared_error(y_test,test_predictions1)\n",
    "RMSE = np.sqrt(MSE)\n",
    "r2 = r2_score(y_test,test_predictions1)\n",
    "\n",
    "comparison_df = pd.DataFrame({\"Actual\":y_test,\"Predicted\":test_predictions1})\n",
    "display(comparison_df)\n",
    "\n",
    "print(f\"MAE = {MAE}\\nMSE = {MSE}\\nRMSE = {RMSE}\\nr2 = {r2}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f13a1af1",
   "metadata": {},
   "source": [
    "#### Using Ridge Regression (with built-in leave one out cross-validation) and alpha values in <i>np.logspace(-2, 2, 50)</i> :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8dc9ead0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alpha = 10.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Actual</th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>238.36</td>\n",
       "      <td>236.569427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40.00</td>\n",
       "      <td>49.483868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37.50</td>\n",
       "      <td>36.596485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>42.77</td>\n",
       "      <td>24.246054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>138.00</td>\n",
       "      <td>204.944898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27478</th>\n",
       "      <td>62.00</td>\n",
       "      <td>51.878985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27479</th>\n",
       "      <td>102.75</td>\n",
       "      <td>141.803221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27480</th>\n",
       "      <td>207.50</td>\n",
       "      <td>187.428533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27481</th>\n",
       "      <td>36.74</td>\n",
       "      <td>29.573565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27482</th>\n",
       "      <td>198.33</td>\n",
       "      <td>209.097832</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27483 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Actual   Predicted\n",
       "0      238.36  236.569427\n",
       "1       40.00   49.483868\n",
       "2       37.50   36.596485\n",
       "3       42.77   24.246054\n",
       "4      138.00  204.944898\n",
       "...       ...         ...\n",
       "27478   62.00   51.878985\n",
       "27479  102.75  141.803221\n",
       "27480  207.50  187.428533\n",
       "27481   36.74   29.573565\n",
       "27482  198.33  209.097832\n",
       "\n",
       "[27483 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE = 24.17642370350396\n",
      "MSE = 1476.4262153134666\n",
      "RMSE = 38.42429199495375\n",
      "r2 = 0.8715135093533303\n"
     ]
    }
   ],
   "source": [
    "print(f\"alpha = {regressor2.alpha_}\") # alpha value chosen by cross validation\n",
    "\n",
    "MAE = mean_absolute_error(y_test,test_predictions2)\n",
    "MSE = mean_squared_error(y_test,test_predictions2)\n",
    "RMSE = np.sqrt(MSE)\n",
    "r2 = r2_score(y_test,test_predictions2)\n",
    "\n",
    "comparison_df = pd.DataFrame({\"Actual\":y_test,\"Predicted\":test_predictions2})\n",
    "display(comparison_df)\n",
    "\n",
    "print(f\"MAE = {MAE}\\nMSE = {MSE}\\nRMSE = {RMSE}\\nr2 = {r2}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "45a9f68c",
   "metadata": {},
   "source": [
    "#### OLS vs Ridge\n",
    "The results of the two models yield an <i>RMSE</i> of around 38.43 (Pesos) and an <i>r<sup>2</sup></i> of around 0.8715. The model using Ridge regression is generally better since it is more stable. However, in this particular train-test split, both models performed with the same accuracy."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6d9e5be6",
   "metadata": {},
   "source": [
    "### Extending the Linear Model using Polynomial Features\n",
    "\n",
    "The weaknesses of ordinary linear regression include the assumption that the relationship between the predictors and response are (1) Additive, meaning a predictor X<sub>i</sub> and its association with response Y is independent of the associations of other predictors X<sub>j</sub>, and (2) Linear, meaning a change in the variable Y in response to a change in X is constant. \n",
    "\n",
    "When we extend the linear model using polynomial features, we are essentially allowing the model to be capable of fitting non-linear relationships between the dependent and independent variables, as well as of considering the relationships between the multiple predictors in the model.\n",
    "\n",
    "In this case, since we have a linear model that is trying to predict the price of food based on its previous prices and factors like location, we can extend the model by adding polynomial terms (or <i>interaction</i> terms) to represent the relationships between the independent variables. This would allow the model to capture the fact that the price of a food is not linearly related only to the sum individual effects of predictors like time and location, but can rather increase or decrease depending also on the combined effect of both (predictor variables would now include time_x_location instead of just time and location individually).\n",
    "\n",
    "Extending the linear model using polynomial features can be a very effective way to improve the model's performance, especially when the relationship between the dependent and independent variables is non-linear. However, it is important to note that adding too many polynomial terms can lead to overfitting, so it is important to choose the right number of terms carefully. \n",
    "\n",
    "### Preprocessing\n",
    "\n",
    "The added interaction features are as follows:\n",
    "1. <b>month2</b> - adds a degree of freedom to the month variable. This will allow the month factor to be represented non-linearly (like a parabola) so that seasonal changes can be modelled more dynamically (e.g., mango prices decreasing mid-year but staying high during the first and last quarters).\n",
    "\n",
    "2. <b>longlat</b> - represents the relationship between (and combined effect of) the latitude and longitude of a location, in addition to evaluating either, individually.\n",
    "\n",
    "3. <b>inflyr</b> - represents the relationship between (and combined effect of) the year and inflation during that year.\n",
    "\n",
    "4. <b>locyr</b> - represents the relationship between (and combined effect of) the year and location (using inflyr and longlat).\n",
    "\n",
    "5. <b>locmth</b> - represents the relationship between (and combined effect of) the month and location (using month, month2, and longlat)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1cbf187b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add polynomial features\n",
    "data['month2'] = data['month']**2 # add degree of freedom\n",
    "data['longlat'] = data['longitude']*data['latitude'] # long-lat relationship\n",
    "data['inflyr'] = data['year']*data['inflation'] # inflation-year relationship\n",
    "data['locyr'] = data['inflyr']*data['longlat'] # location-year relationship\n",
    "data['locmth'] = data['month']*data['month2']*data['longlat'] # location-month relationship"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2a5fa3cc",
   "metadata": {},
   "source": [
    "#### Commodity-specific relationships\n",
    "Some variables' effects on price are highly dependent on the commodity being specified. For example, <i>month</i>'s effect on price is highly dependent on the availability of the specified good (think mango or strawberry seasons) at a certain time. Thus, it might be better to model these commodities' relationiships with the other predictors individually (rather than generalize the month predictor to all other commodities). \n",
    "\n",
    "This was done as follows:\n",
    "1. For each commodity, add a new interaction feature with each of the non-commodity predictors in the dataset (inflation, year, inflyr, longlat, etc.).\n",
    "2. Remove commodity-dependent predictors that cannot be generalized to all commodities (month, month2, pricetype) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d9a5e48f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#divide the data into feature and target sets\n",
    "X = data.drop(['price'], axis='columns')\n",
    "y = data['price'].values\n",
    "\n",
    "# normalize non-dummy columns using MinMaxScaler\n",
    "X[['latitude','longitude','inflation','year','month','month2','longlat','inflyr','locyr','locmth']] = sc.fit_transform(X[['latitude','longitude','inflation','year','month','month2','longlat','inflyr','locyr','locmth']])\n",
    "\n",
    "# For each commodity, add a new interaction feature with each of the non-commodity predictors in the dataset\n",
    "for i in X.columns:\n",
    "    if i not in ['inflation','year','inflyr','longlat','latitude','longitude','locyr','locmth','month','month2','location','priceflag_actual','priceflag_actual,aggregate','priceflag_aggregate','pricetype_Farm Gate','pricetype_Retail','pricetype_Wholesale']:\n",
    "        for j in ['year','inflyr','locyr','locmth','latitude','longitude','longlat','month','month2','priceflag_actual','priceflag_actual,aggregate','priceflag_aggregate','pricetype_Farm Gate','pricetype_Retail','pricetype_Wholesale']:\n",
    "            #X.insert(len(X.columns),i+' x '+j,X[i]*X[j])\n",
    "            X=pd.concat((X,(X[i]*X[j]).rename(i+j)),axis=1)\n",
    "\n",
    "# remove commodity-dependent presictors\n",
    "X = X.drop(['month','month2','locmth','priceflag_actual','priceflag_actual,aggregate','priceflag_aggregate','pricetype_Farm Gate','pricetype_Retail','pricetype_Wholesale'], axis='columns')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "139c6bd1",
   "metadata": {},
   "source": [
    "The train set can be seen in the figure below after preprocessing. It is worth noting that although there are now a lot of columns in the feature set, most of the columns are commodity-specific and unrelated to most other columns, meaning there shouldn't be drastic increase in the complexity of the model that would be enough to risk overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "25747697",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(109930, 1175)\n",
      "(27483, 1175)\n",
      "(109930,)\n",
      "(27483,)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>inflation</th>\n",
       "      <th>year</th>\n",
       "      <th>commodity_Anchovies</th>\n",
       "      <th>commodity_Bananas (lakatan)</th>\n",
       "      <th>commodity_Bananas (latundan)</th>\n",
       "      <th>commodity_Bananas (saba)</th>\n",
       "      <th>commodity_Beans (green, fresh)</th>\n",
       "      <th>commodity_Beans (mung)</th>\n",
       "      <th>...</th>\n",
       "      <th>commodity_Water spinachlongitude</th>\n",
       "      <th>commodity_Water spinachlonglat</th>\n",
       "      <th>commodity_Water spinachmonth</th>\n",
       "      <th>commodity_Water spinachmonth2</th>\n",
       "      <th>commodity_Water spinachpriceflag_actual</th>\n",
       "      <th>commodity_Water spinachpriceflag_actual,aggregate</th>\n",
       "      <th>commodity_Water spinachpriceflag_aggregate</th>\n",
       "      <th>commodity_Water spinachpricetype_Farm Gate</th>\n",
       "      <th>commodity_Water spinachpricetype_Retail</th>\n",
       "      <th>commodity_Water spinachpricetype_Wholesale</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15312</th>\n",
       "      <td>0.834606</td>\n",
       "      <td>0.200479</td>\n",
       "      <td>0.310935</td>\n",
       "      <td>0.521739</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51810</th>\n",
       "      <td>0.727313</td>\n",
       "      <td>0.300700</td>\n",
       "      <td>0.226614</td>\n",
       "      <td>0.869565</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34435</th>\n",
       "      <td>0.870263</td>\n",
       "      <td>0.322688</td>\n",
       "      <td>0.226614</td>\n",
       "      <td>0.869565</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35778</th>\n",
       "      <td>0.507230</td>\n",
       "      <td>0.486434</td>\n",
       "      <td>0.226614</td>\n",
       "      <td>0.869565</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49991</th>\n",
       "      <td>0.228774</td>\n",
       "      <td>0.677716</td>\n",
       "      <td>0.226614</td>\n",
       "      <td>0.869565</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73408</th>\n",
       "      <td>0.616623</td>\n",
       "      <td>0.671164</td>\n",
       "      <td>0.429513</td>\n",
       "      <td>0.913043</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137475</th>\n",
       "      <td>0.428277</td>\n",
       "      <td>0.563630</td>\n",
       "      <td>0.675889</td>\n",
       "      <td>0.956522</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77256</th>\n",
       "      <td>0.387619</td>\n",
       "      <td>0.816261</td>\n",
       "      <td>0.429513</td>\n",
       "      <td>0.913043</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59971</th>\n",
       "      <td>0.615886</td>\n",
       "      <td>0.669020</td>\n",
       "      <td>0.226614</td>\n",
       "      <td>0.869565</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55570</th>\n",
       "      <td>0.652659</td>\n",
       "      <td>0.595346</td>\n",
       "      <td>0.226614</td>\n",
       "      <td>0.869565</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>109930 rows × 1175 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        latitude  longitude  inflation      year  commodity_Anchovies  \\\n",
       "15312   0.834606   0.200479   0.310935  0.521739                    0   \n",
       "51810   0.727313   0.300700   0.226614  0.869565                    0   \n",
       "34435   0.870263   0.322688   0.226614  0.869565                    0   \n",
       "35778   0.507230   0.486434   0.226614  0.869565                    0   \n",
       "49991   0.228774   0.677716   0.226614  0.869565                    0   \n",
       "...          ...        ...        ...       ...                  ...   \n",
       "73408   0.616623   0.671164   0.429513  0.913043                    0   \n",
       "137475  0.428277   0.563630   0.675889  0.956522                    0   \n",
       "77256   0.387619   0.816261   0.429513  0.913043                    0   \n",
       "59971   0.615886   0.669020   0.226614  0.869565                    0   \n",
       "55570   0.652659   0.595346   0.226614  0.869565                    0   \n",
       "\n",
       "        commodity_Bananas (lakatan)  commodity_Bananas (latundan)  \\\n",
       "15312                             0                             0   \n",
       "51810                             0                             0   \n",
       "34435                             0                             0   \n",
       "35778                             0                             0   \n",
       "49991                             0                             0   \n",
       "...                             ...                           ...   \n",
       "73408                             0                             0   \n",
       "137475                            0                             0   \n",
       "77256                             0                             0   \n",
       "59971                             0                             0   \n",
       "55570                             0                             0   \n",
       "\n",
       "        commodity_Bananas (saba)  commodity_Beans (green, fresh)  \\\n",
       "15312                          0                               0   \n",
       "51810                          0                               0   \n",
       "34435                          0                               1   \n",
       "35778                          0                               0   \n",
       "49991                          0                               0   \n",
       "...                          ...                             ...   \n",
       "73408                          0                               0   \n",
       "137475                         0                               0   \n",
       "77256                          0                               0   \n",
       "59971                          0                               0   \n",
       "55570                          0                               0   \n",
       "\n",
       "        commodity_Beans (mung)  ...  commodity_Water spinachlongitude  \\\n",
       "15312                        1  ...                               0.0   \n",
       "51810                        0  ...                               0.0   \n",
       "34435                        0  ...                               0.0   \n",
       "35778                        0  ...                               0.0   \n",
       "49991                        0  ...                               0.0   \n",
       "...                        ...  ...                               ...   \n",
       "73408                        0  ...                               0.0   \n",
       "137475                       0  ...                               0.0   \n",
       "77256                        0  ...                               0.0   \n",
       "59971                        0  ...                               0.0   \n",
       "55570                        0  ...                               0.0   \n",
       "\n",
       "        commodity_Water spinachlonglat  commodity_Water spinachmonth  \\\n",
       "15312                              0.0                           0.0   \n",
       "51810                              0.0                           0.0   \n",
       "34435                              0.0                           0.0   \n",
       "35778                              0.0                           0.0   \n",
       "49991                              0.0                           0.0   \n",
       "...                                ...                           ...   \n",
       "73408                              0.0                           0.0   \n",
       "137475                             0.0                           0.0   \n",
       "77256                              0.0                           0.0   \n",
       "59971                              0.0                           0.0   \n",
       "55570                              0.0                           0.0   \n",
       "\n",
       "        commodity_Water spinachmonth2  \\\n",
       "15312                             0.0   \n",
       "51810                             0.0   \n",
       "34435                             0.0   \n",
       "35778                             0.0   \n",
       "49991                             0.0   \n",
       "...                               ...   \n",
       "73408                             0.0   \n",
       "137475                            0.0   \n",
       "77256                             0.0   \n",
       "59971                             0.0   \n",
       "55570                             0.0   \n",
       "\n",
       "        commodity_Water spinachpriceflag_actual  \\\n",
       "15312                                         0   \n",
       "51810                                         0   \n",
       "34435                                         0   \n",
       "35778                                         0   \n",
       "49991                                         0   \n",
       "...                                         ...   \n",
       "73408                                         0   \n",
       "137475                                        0   \n",
       "77256                                         0   \n",
       "59971                                         0   \n",
       "55570                                         0   \n",
       "\n",
       "        commodity_Water spinachpriceflag_actual,aggregate  \\\n",
       "15312                                                   0   \n",
       "51810                                                   0   \n",
       "34435                                                   0   \n",
       "35778                                                   0   \n",
       "49991                                                   0   \n",
       "...                                                   ...   \n",
       "73408                                                   0   \n",
       "137475                                                  0   \n",
       "77256                                                   0   \n",
       "59971                                                   0   \n",
       "55570                                                   0   \n",
       "\n",
       "        commodity_Water spinachpriceflag_aggregate  \\\n",
       "15312                                            0   \n",
       "51810                                            0   \n",
       "34435                                            0   \n",
       "35778                                            0   \n",
       "49991                                            0   \n",
       "...                                            ...   \n",
       "73408                                            0   \n",
       "137475                                           0   \n",
       "77256                                            0   \n",
       "59971                                            0   \n",
       "55570                                            0   \n",
       "\n",
       "        commodity_Water spinachpricetype_Farm Gate  \\\n",
       "15312                                            0   \n",
       "51810                                            0   \n",
       "34435                                            0   \n",
       "35778                                            0   \n",
       "49991                                            0   \n",
       "...                                            ...   \n",
       "73408                                            0   \n",
       "137475                                           0   \n",
       "77256                                            0   \n",
       "59971                                            0   \n",
       "55570                                            0   \n",
       "\n",
       "        commodity_Water spinachpricetype_Retail  \\\n",
       "15312                                         0   \n",
       "51810                                         0   \n",
       "34435                                         0   \n",
       "35778                                         0   \n",
       "49991                                         0   \n",
       "...                                         ...   \n",
       "73408                                         0   \n",
       "137475                                        0   \n",
       "77256                                         0   \n",
       "59971                                         0   \n",
       "55570                                         0   \n",
       "\n",
       "        commodity_Water spinachpricetype_Wholesale  \n",
       "15312                                            0  \n",
       "51810                                            0  \n",
       "34435                                            0  \n",
       "35778                                            0  \n",
       "49991                                            0  \n",
       "...                                            ...  \n",
       "73408                                            0  \n",
       "137475                                           0  \n",
       "77256                                            0  \n",
       "59971                                            0  \n",
       "55570                                            0  \n",
       "\n",
       "[109930 rows x 1175 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# divide into train and test set \n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=26)\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)\n",
    "display(X_train)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2c1eb1bd",
   "metadata": {},
   "source": [
    "### Performing Linear Regression\n",
    "The new model will use the same two regression methods as before.\n",
    "\n",
    "#### Using LinearRegression (Ordinary Least Squares) and default parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ffadf4e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Actual</th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>238.36</td>\n",
       "      <td>214.796875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40.00</td>\n",
       "      <td>49.812500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37.50</td>\n",
       "      <td>37.109375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>42.77</td>\n",
       "      <td>30.140625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>138.00</td>\n",
       "      <td>160.171875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27478</th>\n",
       "      <td>62.00</td>\n",
       "      <td>56.304688</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27479</th>\n",
       "      <td>102.75</td>\n",
       "      <td>166.921875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27480</th>\n",
       "      <td>207.50</td>\n",
       "      <td>199.917969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27481</th>\n",
       "      <td>36.74</td>\n",
       "      <td>31.203125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27482</th>\n",
       "      <td>198.33</td>\n",
       "      <td>214.601562</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27483 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Actual   Predicted\n",
       "0      238.36  214.796875\n",
       "1       40.00   49.812500\n",
       "2       37.50   37.109375\n",
       "3       42.77   30.140625\n",
       "4      138.00  160.171875\n",
       "...       ...         ...\n",
       "27478   62.00   56.304688\n",
       "27479  102.75  166.921875\n",
       "27480  207.50  199.917969\n",
       "27481   36.74   31.203125\n",
       "27482  198.33  214.601562\n",
       "\n",
       "[27483 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE = 17.38388128570389\n",
      "MSE = 930.2825500686646\n",
      "RMSE = 30.500533603015285\n",
      "r2 = 0.9190418465017706\n"
     ]
    }
   ],
   "source": [
    "regressor = LinearRegression()\n",
    "regressor.fit(X_train, y_train)\n",
    "\n",
    "test_predictions = regressor.predict(X_test)\n",
    "\n",
    "#get the MAE,MSE,RMSE, and R2 values to evaluate the model \n",
    "MAE = mean_absolute_error(y_test,test_predictions)\n",
    "MSE = mean_squared_error(y_test,test_predictions)\n",
    "RMSE = np.sqrt(MSE)\n",
    "r2 = r2_score(y_test,test_predictions)\n",
    "\n",
    "comparison_df = pd.DataFrame({\"Actual\":y_test,\"Predicted\":test_predictions})\n",
    "display(comparison_df)\n",
    "\n",
    "print(f\"MAE = {MAE}\\nMSE = {MSE}\\nRMSE = {RMSE}\\nr2 = {r2}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "975838a4",
   "metadata": {},
   "source": [
    "#### Using Ridge Regression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bfb028cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alpha = 0.01\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Actual</th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>238.36</td>\n",
       "      <td>215.363156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40.00</td>\n",
       "      <td>50.784545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37.50</td>\n",
       "      <td>36.704489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>42.77</td>\n",
       "      <td>31.606012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>138.00</td>\n",
       "      <td>159.783664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27478</th>\n",
       "      <td>62.00</td>\n",
       "      <td>56.539222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27479</th>\n",
       "      <td>102.75</td>\n",
       "      <td>163.463086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27480</th>\n",
       "      <td>207.50</td>\n",
       "      <td>198.244000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27481</th>\n",
       "      <td>36.74</td>\n",
       "      <td>32.467403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27482</th>\n",
       "      <td>198.33</td>\n",
       "      <td>215.553517</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27483 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Actual   Predicted\n",
       "0      238.36  215.363156\n",
       "1       40.00   50.784545\n",
       "2       37.50   36.704489\n",
       "3       42.77   31.606012\n",
       "4      138.00  159.783664\n",
       "...       ...         ...\n",
       "27478   62.00   56.539222\n",
       "27479  102.75  163.463086\n",
       "27480  207.50  198.244000\n",
       "27481   36.74   32.467403\n",
       "27482  198.33  215.553517\n",
       "\n",
       "[27483 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE = 17.45419336766678\n",
      "MSE = 947.4384096443913\n",
      "RMSE = 30.780487482240943\n",
      "r2 = 0.9175488520208752\n"
     ]
    }
   ],
   "source": [
    "regressor = RidgeCV(alphas=np.logspace(-2,2,50))\n",
    "regressor.fit(X_train, y_train)\n",
    "print(f\"alpha = {regressor.alpha_}\")\n",
    "\n",
    "test_predictions = regressor.predict(X_test)\n",
    "\n",
    "#get the MAE,MSE,RMSE, and R2 values to evaluate the model \n",
    "MAE = mean_absolute_error(y_test,test_predictions)\n",
    "MSE = mean_squared_error(y_test,test_predictions)\n",
    "RMSE = np.sqrt(MSE)\n",
    "r2 = r2_score(y_test,test_predictions)\n",
    "\n",
    "comparison_df = pd.DataFrame({\"Actual\":y_test,\"Predicted\":test_predictions})\n",
    "display(comparison_df)\n",
    "\n",
    "print(f\"MAE = {MAE}\\nMSE = {MSE}\\nRMSE = {RMSE}\\nr2 = {r2}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5a666668",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "### Summary of metrics\n",
    "\n",
    "From the results above, it can be observed that the model with polynomial features <b>(r<sup>2</sup>≈0.92; RMSE≈30.48)</b> performs better than the previous model without polynomial features <b>(r<sup>2</sup>≈0.87; RMSE≈38.42)</b>. This is possibly because the model with polynomial features is able to represent the non-linear relationships between the dependent and independent variables. \n",
    "\n",
    "Between the two models with the polynomial features, LinearRegression and Ridge Regression have around the same <i>r<sup>2</sup></i> and <i>RMSE</i> score, with Linear being slightly better than Ridge, but considering the fact that Ridge  is more stable, it can be concluded that the latter is overall the better food price predictor.\n",
    "\n",
    "\n",
    "\n",
    "### Model Comparison\n",
    "A 2022 study by Rao et. al. used ARCH (autoregressive conditional heteroskedasticity) and GARCH (generalized autoregressive conditional heteroskedasticity) to predict the prices of staple food materials in India. The study used a 4392-row data from 2015-2018, which includes high price, thermal, and precipitation data. The models have an accuracy of more than 80%, with ARCH having up to 99.84% accuracy and GARCH having up to 96.57% accuracy.\n",
    "\n",
    "Asnhari et. al. (2019) used a similar data which includes precipitation and thermal data to predict the commodity prices in Indonesia. The study used linear regression and the Fourier model with ARIMA (Autoregressive Integrated Moving Average). The latter regression model predicts the prices for all commodities with above 80% accuracy, and reportedly produces better accuracy with data in higher fluctuations. Multiple linear regression with ARIMA produced predictions with high accuracy of up to 99.84%.\n",
    "\n",
    "The models created in the two studies took note of the correlation between the weather and the food prices, and thus included thermal and precipitation data in their models. While our models don't include these data, we included location data to take note of the variations of prices among different places. In contrast to the different time series models of these studies, which measures the volatility of the prices overtime, we used regression techniques which reduces data complexity.\n",
    "\n",
    "References:\n",
    "\n",
    "Asnhari, S. F., Gunawan, P. H., & Rusmawati, Y. (2019). \"Predicting Staple Food Materials Price Using Multivariables Factors (Regression and Fourier Models with ARIMA),\" 2019 7th International Conference on Information and Communication Technology (ICoICT), Kuala Lumpur, Malaysia, 2019, pp. 1-5, doi: 10.1109/ICoICT.2019.8835193\n",
    "\n",
    "Rao, K. V., Srilatha, D., Jagan Mohan Reddy, D., Desanamukula, V. S., Kejela, M. L. (2022). \"Regression Based Price Prediction of Staple Food Materials Using Multivariate Models\", Scientific Programming, vol. 2022, Article ID 4572064, 7 pages, 2022. https://doi.org/10.1155/2022/4572064"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}