Jupyter/Episode III – Revenge of the Outliers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Movie Wars\n",
    "## ~ Episode III – Revenge of the Outliers  ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we should set the notebook so that it outputs all results of each cell and not only the last one (and disable the warnings)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And import all the python libraries needed for this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from collections import Counter\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.utils import shuffle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we state where our data sources are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_folder_path = 'data\\\\'\n",
    "\n",
    "movies_file_path = data_folder_path + 'movies_with_genre_and_year.csv'\n",
    "users_file_path = data_folder_path + 'users_with_age_interval_and_occupation.csv'\n",
    "ratings_file_path = data_folder_path + 'ratings.csv'\n",
    "ratings_by_user_file_path  = data_folder_path + 'ratings_by_user_data.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_movies = pd.read_csv(movies_file_path, sep = ';', index_col = 'ID')\n",
    "all_users = pd.read_csv(users_file_path, sep = ';', index_col = 'Id')\n",
    "all_ratings = pd.read_csv(ratings_file_path, sep = ',')\n",
    "ratings_by_user = pd.read_csv(ratings_by_user_file_path, sep = ';')\n",
    "\n",
    "movie_genres = ['Action', 'Adventure', 'Animation', \"Children's\", 'Comedy', 'Crime', 'Documentary', 'Drama', 'Fantasy', 'Film-Noir', 'Horror', 'Musical', 'Mystery', 'Romance', 'Sci-Fi', 'Thriller', 'War', 'Western']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to start with the feature engineering process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transforming features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalizing Movie Year feature\n",
    "\n",
    "We split the 20th century into 5 epochs:\n",
    "\n",
    " - **(1900 - 1939)**: black and white movies and silent movies.\n",
    " - **(1939 - 1970)**: western movies and classic movies.\n",
    " - **(1970, 1985)**: first world famous actors, classic action movies, first blockbusters and color movies.\n",
    " - **(1985, 1995)**: future based topics, introduction of special effects\n",
    " - **(1995, 2000)**: sci-fi, new age movies and computer effects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def movie_year_normalization(year):\n",
    "    if year < 1939:\n",
    "        return 0\n",
    "    if year < 1970:\n",
    "        return 0.25\n",
    "    if year < 1985:\n",
    "        return 0.5\n",
    "    if year < 1995:\n",
    "        return 0.75\n",
    "    return 1\n",
    "\n",
    "all_movies['Year_normalized'] = all_movies['Year'].apply(movie_year_normalization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing the class mark for the users' age"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To improve the age feature we can set the **age** class mark as de midpoint of the interval rather than a extreme of the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "centered_user_ages = {1: (0 + 17)/2, 18: (18 + 24)/2, 25: (25 + 34)/2, 35: (35 + 44)/2, 45: (45 + 49)/2, 50: (50 + 55)/2, 56: (56 + 70)/2}\n",
    "\n",
    "all_users['Age'] = all_users['Age'].apply(lambda x: centered_user_ages[x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalizing the users' age values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the **min-max** normalization with users age."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minimun_age = min(all_users['Age'])\n",
    "maximun_age = max(all_users['Age'])\n",
    "\n",
    "all_users['Age_normalized'] = all_users['Age'].apply(lambda age: ((age - minimun_age)/(maximun_age - minimun_age)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouping the users' occupation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To improve the occupation feature we can regroup its classes in **fewer categories** and make them **more balanced**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "user_occupations_categories = {1: 'artist', 2: 'craftsmen', 3: 'engineer', 4: 'academic', 5: 'student', 6: 'customer-fancing',\n",
    "                       7: 'other', 8:'unemployed', 9: 'high-wage'}\n",
    "\n",
    "occupations_map = {20: 1, 2: 1, 18: 2, 8: 2, 9: 2, 12: 3, 17: 3, 15:4 ,1: 4, 10: 5, 4: 5, 16: 6, 14: 6, 5: 6, 0: 7, 3: 7, \n",
    "                  19: 8, 13: 8, 6: 9, 7: 9, 11: 9}\n",
    "\n",
    "all_users['Occupation_category'] = all_users['Occupation'].apply(lambda occupation: occupations_map[occupation])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how it looks now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "all_users['Occupation_category_decoded'] = all_users['Occupation_category'].apply(lambda category: user_occupations_categories[category])\n",
    "\n",
    "occupation_distribution = pd.DataFrame({\n",
    "    'Occupation': list(Counter(all_users.Occupation_category_decoded).keys()),\n",
    "    'Count': list(Counter(all_users.Occupation_category_decoded).values()),\n",
    "})\n",
    "\n",
    "occupations_order = occupation_distribution.sort_values(by = 'Count', ascending = False)['Occupation']\n",
    "\n",
    "occupation_barplot = sns.barplot(x = 'Occupation', y = 'Count', data = occupation_distribution, order = occupations_order)\n",
    "occupation_barplot.set_xticklabels(occupation_barplot.get_xticklabels(), rotation = 45, horizontalalignment = 'right')\n",
    "occupation_barplot.set_title(\"Users' occupation distribution\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transforming users gender feature into binary\n",
    "\n",
    "For model learning purposes, we transform the feature of users Gender from string type to binary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_users['Gender'] = all_users['Gender'].apply(lambda value: 1 if value == 'F' else 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improving the profiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Users profiles\n",
    "\n",
    "We can use information about the movies rated by each user for improving our models performance, for example we can: \n",
    "- Find the **user's average ratings** \n",
    "- Find the **user's affinity to each genre** using the average the ratings of movies corresponding to that genres.\n",
    "- Find the user's **favorite movie epoch**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "users_ids = list(all_ratings['user'].unique())\n",
    "movies_ids = list(all_ratings['movie'].unique())\n",
    "\n",
    "# Get the movies rated by each user\n",
    "movies_by_user = dict()\n",
    "for user in users_ids:\n",
    "    movies_by_user[user] = list(all_ratings[all_ratings['user'] == user]['movie'])\n",
    "\n",
    "# Get the movies by genre\n",
    "movies_by_genre = dict()\n",
    "for genre in movie_genres:\n",
    "    movies_by_genre[genre] = list(all_movies[all_movies[genre] > 0].index) \n",
    "\n",
    "# Compute the average rating of each movie\n",
    "mean_ratings = dict()\n",
    "for movie in movies_ids:\n",
    "    mean_ratings[movie] = round(all_ratings[all_ratings['movie'] == movie]['rating'].mean(), 3)\n",
    "\n",
    "# Compute the genre mean ratings and ratings counts for each user\n",
    "genre_means_per_user = dict()\n",
    "genre_count_per_user = dict()\n",
    "for genre in movie_genres:\n",
    "    selected_movies = list(map(lambda user_movies: set(user_movies) & set(movies_by_genre[genre]), movies_by_user.values()))\n",
    "    selected_ratings = list(map(lambda movies: [mean_ratings[movie] for movie in movies], selected_movies))\n",
    "    \n",
    "    genre_means_per_user[genre] = list(map(lambda ratings: round(np.array(ratings).mean(), 3), selected_ratings))\n",
    "    genre_count_per_user[genre] = list(map(lambda ratings: len(ratings), selected_ratings))\n",
    "\n",
    "# Create user profiles\n",
    "user_profiles = pd.DataFrame({\n",
    "    'User': users_ids,\n",
    "    'Age': [all_users.at[user_id, 'Age'] for user_id in users_ids],\n",
    "    'Gender': [all_users.at[user_id, 'Gender'] for user_id in users_ids],\n",
    "    'Occupation': [all_users.at[user_id, 'Occupation_category'] for user_id in users_ids],\n",
    "    'Favorite epoch': list(map(lambda movies: round(all_movies[all_movies.index.isin(list(movies))]['Year'].mean(), 3), movies_by_user.values())),\n",
    "    'Mean rating': list(map(lambda movies: round(np.array([mean_ratings[movie] for movie in movies]).mean(), 3), movies_by_user.values())),\n",
    "    'Ratings count': list(map(lambda movies: len(movies), movies_by_user.values()))\n",
    "}).set_index('User')\n",
    "\n",
    "for genre in movie_genres:\n",
    "    user_profiles[genre + '_affinity'] = [(mean - 1)/4 if (not np.isnan(mean)) else 0 for mean in genre_means_per_user[genre]]\n",
    "    user_profiles[genre + '_ratings_count'] = genre_count_per_user[genre]\n",
    "\n",
    "user_profiles.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Movies profiles\n",
    "\n",
    "We can use information about the users that rated each movie for improving our models performance, for example we can: \n",
    "- Find the **movie's average rating** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create movie profiles\n",
    "movie_profiles = pd.DataFrame({\n",
    "    'Movie': movies_ids,\n",
    "    'Year': [all_movies.at[movie_id, 'Year'] for movie_id in movies_ids],\n",
    "    'Mean rating': list(map(lambda movie: round(all_ratings[all_ratings['movie'] == movie]['rating'].mean(), 3), movies_ids)),\n",
    "    'Ratings count': list(map(lambda movie: len(all_ratings[all_ratings['movie'] == movie]['rating']), movies_ids)),\n",
    "    'Genre count': list(map(lambda movie: sum(all_movies.loc[movie][movie_genres].values), movies_ids))\n",
    "}).set_index('Movie')\n",
    "\n",
    "for genre in movie_genres:\n",
    "    movie_profiles[genre] = [all_movies.at[movie_id, genre] for movie_id in movies_ids]\n",
    "\n",
    "movie_profiles.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rating estimate\n",
    "\n",
    "Knowing the user's affinity to each genre and the genres of a movie, we can easily give an estimated approximation for the rating, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def estimator(movie, user):\n",
    "    user_genre_affinity = user_profiles.loc[user][[genre + \"_affinity\" for genre in movie_genres]].values\n",
    "    genres = movie_profiles.loc[movie][movie_genres].values\n",
    "    \n",
    "    affinities = np.array([x * y for x, y in zip(user_genre_affinity, genres)])\n",
    "    affinities = affinities[affinities != 0.]\n",
    "    \n",
    "    return round(affinities.mean() * 4 + 1, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is what most classic content-based recommender systems did. Now a days, it is used to further improve the models. \n",
    "\n",
    "***Note:*** *This can take a while, since it needs to compute it for half a million data points.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_ratings['estimate'] = all_ratings.apply(lambda rating: estimator(rating['movie'], rating['user']), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should not forget to add the other features that we want to use later on to our dataset, in our case we will use: **users (age, gender, occupation, favorite epoch)** and **movies (year, genres)**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_ratings['user_age'] = all_ratings['user'].apply(lambda user: user_profiles.at[user, 'Age'])\n",
    "all_ratings['user_gender'] = all_ratings['user'].apply(lambda user: user_profiles.at[user, 'Gender'])\n",
    "all_ratings['user_occupation_category'] = all_ratings['user'].apply(lambda user: user_profiles.at[user, 'Occupation'])\n",
    "all_ratings['user_movies_epoch'] = all_ratings['user'].apply(lambda user: user_profiles.at[user, 'Favorite epoch'])\n",
    "\n",
    "all_ratings['movie_year'] = all_ratings['movie'].apply(lambda movie: movie_profiles.at[movie, 'Year'])\n",
    "for genre in movie_genres:\n",
    "    all_ratings[genre] = all_ratings['movie'].apply(lambda movie: movie_profiles.at[movie, genre])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing noise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thankfully in a recommendation system based on ratings, neither missing values nor measurement noise exists. \n",
    "\n",
    "But there are still some things that we can do to improve the quality of the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outliers\n",
    "\n",
    "Outlier are a data points that differ significantly from other observations. In the case of our users, we can find two groups of them:\n",
    "- The *\"haters\"* that rate everything negatively\n",
    "- The *\"lovers\"* that rate everything as perfect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "haters = list(ratings_by_user[(ratings_by_user['Mean rating'] < 2.5) & (ratings_by_user['Rating deviation'] < 2)].index)\n",
    "lovers = list(ratings_by_user[(ratings_by_user['Mean rating'] > 4) & (ratings_by_user['Rating deviation'] < 0.6)].index)\n",
    "\n",
    "f\"Number of haters: {len(haters)} ({round(100*len(haters)/len(all_users), 2)}%)\"\n",
    "f\"Number of lovers: {len(lovers)} ({round(100*len(lovers)/len(all_users), 2)}%)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For models based on **user profiling**, we need to **remove** this anomalous users to improve the gereralization of the profiles. But for models based on **closest neighbors**, those anomalous users can be actually very helpful so is better to **keep** them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Lack of ratings\n",
    "\n",
    "For models based on user profiling, it is also important to **remove** users with very few ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_with_few_ratings = list(ratings_by_user[ratings_by_user['Rating count'] < 30].index)\n",
    "\n",
    "f\"People with few ratings: {len(users_with_few_ratings)} ({round(100*len(users_with_few_ratings)/len(all_users), 2)}%)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spliting the data\n",
    "\n",
    "We can consider two ways of spliting our data set into 80% training and 20% testing sets, the first option is just to do a **random sampling split**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classic_train_data, classic_test_data = train_test_split(all_ratings, test_size = 0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second one is to first add **at least one rating** for each film and one rating for each user to the training set and **fill the remaining capacity up to 80% randomly**, this warranties no movie nor user in our test set will not have a rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def minimum_random_sampling(dataset):\n",
    "    selected_users = set()\n",
    "    selected_movies = set()\n",
    "    selected_ratings = []\n",
    "    \n",
    "    shuffled_dataset = shuffle(dataset)\n",
    "    \n",
    "    for index, row in shuffled_dataset.iterrows():\n",
    "        if row.user not in selected_users or row.movie not in selected_movies :\n",
    "            selected_users |= {row.user}\n",
    "            selected_movies |= {row.movie}\n",
    "            selected_ratings.append(row.id)\n",
    "    \n",
    "    return (dataset[dataset.index.isin(selected_ratings)], dataset[~dataset.index.isin(selected_ratings)])\n",
    "\n",
    "def special_random_sampling(dataset, test_size = 0.2):   \n",
    "    (minimum, remaining) = minimum_random_sampling(dataset)\n",
    "    (train, test) = train_test_split(remaining, test_size = test_size * (len(dataset)/(len(dataset) - len(minimum))))\n",
    "    \n",
    "    return (train.append(minimum), test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This second approach must be taken when the model selected cannot handle unseen movies/users during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(special_train_data, special_test_data) = special_random_sampling(all_ratings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to remove the outliers and users with few ratings **only from the training set**, and only for ceirtaing models. If we were to remove them from the test set, our metrics wouldn't be valid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classic_train_data_reduced = classic_train_data[(~classic_train_data['user'].isin(haters)) & (~classic_train_data['user'].isin(lovers)) & (~classic_train_data['user'].isin(users_with_few_ratings))]\n",
    "special_train_data_reduced = special_train_data[(~special_train_data['user'].isin(haters)) & (~special_train_data['user'].isin(lovers)) & (~special_train_data['user'].isin(users_with_few_ratings))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we save the users and movies profiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_profiles.to_csv(data_folder_path + 'user_profiles.csv', sep = ';')\n",
    "movie_profiles.to_csv(data_folder_path + 'movie_profiles.csv', sep = ';')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the resulting training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['rating', 'user_age', 'user_gender', 'user_occupation_category', 'user_movies_epoch', 'movie_year', 'estimate'] + movie_genres\n",
    "\n",
    "classic_train_data[features].to_csv(data_folder_path + 'ratings_training_data_basic_split.csv', sep = ';')\n",
    "classic_train_data_reduced[features].to_csv(data_folder_path + 'ratings_training_data_basic_split_reduced.csv', sep = ';')\n",
    "classic_test_data[features].to_csv(data_folder_path + 'ratings_test_data_basic_split.csv', sep = ';')\n",
    "\n",
    "special_train_data[features].to_csv(data_folder_path +'ratings_training_data_special_split.csv', sep = ';')\n",
    "special_train_data_reduced[features].to_csv(data_folder_path +'ratings_training_data_special_split_reduced.csv', sep = ';')\n",
    "special_test_data[features].to_csv(data_folder_path +'ratings_test_data_special_split.csv', sep = ';')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}