notebooks/2b_decision_trees_practical.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Lesson 2B: Decision Trees London Housing Practical"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oh4Im1wmNdMx"
      },
      "source": [
        "<a name=\"introduction\"></a>\n",
        "## Introduction\n",
        "\n",
        "In this practical, we'll build on the theoretical foundations from Lesson 2A to implement decision trees for predicting house prices in London.\n",
        "\n",
        "We'll follow a systematic approach:\n",
        "\n",
        "1. Load and validate housing data from Kaggle\n",
        "2. Apply exploratory analysis techniques\n",
        "3. Implement proper data preprocessing\n",
        "4. Build and evaluate decision tree models\n",
        "5. Compare different tree-based approaches\n",
        "6. Consider production deployment\n",
        "\n",
        "By working with real housing data, we'll encounter and solve common challenges in machine learning projects while building toward more automated approaches to comparing models in ATLAS.\n",
        "\n",
        "This hands-on session will help you:\n",
        "- Understand how theory translates to practice\n",
        "- Gain experience with real-world data challenges\n",
        "- Learn systematic model development approaches\n",
        "- Build a foundation for understanding advanced tree methods"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWRuzGFhNdMz"
      },
      "source": [
        "## Table of contents\n",
        "\n",
        "1. [Introduction](#introduction)\n",
        "2. [Required libraries](#required-libraries)\n",
        "3. [The London housing dataset](#the-london-housing-dataset)\n",
        "    - [Loading the data](#loading-the-data)\n",
        "4. [Exploratory data analysis](#exploratory-data-analysis)\n",
        "    - [Exploratory data analysis discussion](#exploratory-data-analysis-discussion)\n",
        "    - [Data processing strategy](#data-processing-strategy)\n",
        "6. [Data validation](#data-validation)\n",
        "7. [Data cleaning](#data-cleaning)\n",
        "8. [Initial feature engineering](#initial-feature-engineering)\n",
        "9. [Price transformation and price-aware feature engineering strategy](#price-transformation-and-price-aware-feature-engineering-strategy)\n",
        "10. [Feature encoding](#feature-encoding)\n",
        "    - [One-hot encoder with persistence implementation](#one-hot-encoder-with-persistence-implementation)\n",
        "    - [Hierarchical target encoding methodology](#hierarchical-target-encoding-methodology)\n",
        "    - [Hierarchical target encoding implementation](#hierarchical-target-encoding-implementation)\n",
        "    - [Mean outcode price per square foot encoder implementation](#mean-outcode-price-per-square-foot-encoder-implementation)\n",
        "11. [Training our first decision tree](#training-our-first-decision-tree)\n",
        "12. [Understanding overfitting](#understanding-overfitting)\n",
        "13. [Analysing model performance and hyperparameter impact](#analysing-model-performance-and-hyperparameter-impact)\n",
        "14. [Cross-validation: Getting reliable performance estimates](#cross-validation-getting-reliable-performance-estimates)\n",
        "15. [Grid search implementatation](#grid-search-implementation)\n",
        "16. [Grid search results analysis](#grid-search-results-analysis)\n",
        "17. [Feature set selection](#feature-set-selection)\n",
        "18. [Feature set results analysis](#feature-set-results-analysis)\n",
        "19. [Advanced models](#advanced-models)\n",
        "    - [Random Forests: The wisdom of crowds](#random-forests-the-wisdom-of-crowds)\n",
        "    - [XGBoost: Learning from mistakes](#xgboost-learning-from-mistakes)\n",
        "20. [Random Forest and XGBoost implementation](#random-forest-and-xgboost-implementation)\n",
        "21. [Model selection framework: Trees, Random Forests, and Gradient Boosting](#model-selection-framework-trees-random-forests-and-gradient-boosting)\n",
        "22. [Production implementation and deployment considerations](#production-implementation-and-deployment-considerations)\n",
        "23. [Production implementation review](#production-implementation-refiew)\n",
        "24. [Production monitoring and maintenance](#production-monitoring-and-maintenance)\n",
        "25. [Monitoring system simulations](#monitoring-system-simulations)\n",
        "26. [Ethical considerations](#ethical-considerations)\n",
        "27. [Testing for fairness](#testing-for-fairness)\n",
        "28. [Model limitations and considerations](#model-limitations-and-considerations)\n",
        "29. [Conclusion](#conclusion)\n",
        "    - [Looking ahead: ATLAS](#looking-ahead-a-creative-experiment-called-atlas)\n",
        "    - [Further reading and resources](#further-reading-and-resources)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gFMJMw4yNdM0"
      },
      "source": [
        "\n",
        "<a name=\"required-libraries\"></a>\n",
        "## Required libraries\n",
        "\n",
        "In this lesson we will use the following libraries:\n",
        "\n",
        "| Category | Libraries | Purpose |\n",
        "|----------|-----------|----------|\n",
        "| Core Data Science | Pandas, Numpy, SciPy | Data manipulation, numerical computing, statistics |\n",
        "| Decision Trees | Scikit-learn Trees/Ensembles, XGBoost | Tree-based models (Decision Trees, Random Forests, Gradient Boosting) |\n",
        "| Preprocessing | Scikit-learn Preprocessing, Category Encoders | Data preprocessing and encoding for trees |\n",
        "| Visualization | Matplotlib, Seaborn, tqdm | Tree visualisation, plotting, progress tracking |\n",
        "| Persistence | Pickle, Joblib | Saving trained tree models |\n",
        "| Utilities | Regex, OS, JSON, Typing, Datetime, Logging, Pathlib, Random, Sys | System operations, type hints, logging, random seeds"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X3YT1PcjNdM1",
        "outputId": "596b0464-7fd3-4502-b1df-f684dc341c6e"
      },
      "outputs": [],
      "source": [
        "# Utilities\n",
        "import os\n",
        "import sys\n",
        "import re\n",
        "from pathlib import Path\n",
        "import json\n",
        "import logging\n",
        "import pickle\n",
        "import random\n",
        "from datetime import datetime, timedelta\n",
        "from typing import Any, Dict, List, Optional, Set, Tuple\n",
        "from dataclasses import dataclass, field\n",
        "import requests\n",
        "from io import BytesIO\n",
        "import shutil\n",
        "import logging\n",
        "\n",
        "# Core data science\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from scipy import stats\n",
        "from tqdm.notebook import tqdm\n",
        "\n",
        "# Data visualization\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "import seaborn as sns\n",
        "\n",
        "\n",
        "# Scikit-learn imports\n",
        "from sklearn.compose import ColumnTransformer\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.metrics import (\n",
        "    make_scorer,\n",
        "    mean_absolute_error,\n",
        "    mean_squared_error,\n",
        "    r2_score\n",
        ")\n",
        "from sklearn.model_selection import (\n",
        "    GridSearchCV,\n",
        "    KFold,\n",
        "    StratifiedKFold,\n",
        "    cross_val_score,\n",
        "    cross_validate,\n",
        "    train_test_split\n",
        ")\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import (\n",
        "    LabelEncoder,\n",
        "    OneHotEncoder,\n",
        "    StandardScaler\n",
        ")\n",
        "from sklearn.tree import (\n",
        "    DecisionTreeRegressor,\n",
        "    plot_tree,\n",
        "    _tree\n",
        ")\n",
        "\n",
        "# Advanced decision tree libraries\n",
        "from xgboost import XGBRegressor\n",
        "\n",
        "!pip install category_encoders\n",
        "from category_encoders import TargetEncoder\n",
        "\n",
        "# Set random seeds for reproducibility\n",
        "RANDOM_STATE = 42\n",
        "np.random.seed(RANDOM_STATE)\n",
        "random.seed(RANDOM_STATE)\n",
        "\n",
        "# Configure visualization settings\n",
        "plt.rcParams['figure.figsize'] = [10, 6]\n",
        "plt.rcParams['figure.dpi'] = 100\n",
        "sns.set_theme()  # Apply seaborn styling\n",
        "%matplotlib inline\n",
        "\n",
        "# Configure pandas display options\n",
        "pd.set_option('display.max_columns', None)\n",
        "pd.set_option('display.float_format', lambda x: '{:,.2f}'.format(x))\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W8o01lu_NdM2"
      },
      "source": [
        "\n",
        "<a name=\"the-london-housing-dataset\"></a>\n",
        "## The London housing dataset\n",
        "\n",
        "This lesson uses the \"Housing Prices in London\" dataset ([Kaggle, 2021](https://www.kaggle.com/datasets/arnavkulkarni/housing-prices-in-london)) published by Arnav Kulkarni. It's chosen for three main reasons:\n",
        "\n",
        "1. I wanted a dataset that wasn't the classic Boston housing dataset - sorry Boston, but you've had your moment!\n",
        "2. We want data that isn't perfect - because let's face it, real data never is\n",
        "3. As a Londoner, it's about time I had a look at buying a house even if the prices are eye-watering!\n",
        "\n",
        "<a name=\"loading-the-data\"></a>\n",
        "### Loading the Data\n",
        "\n",
        "The dataset contains 3,480 properties with 11 features. The first column is just row numbers - an artifact from exporting the data to CSV. The second column is the price of each house and the remaining columns are features of each house:\n",
        "\n",
        "- Property Name\n",
        "- Price\n",
        "- House Type\n",
        "- Area in sq ft\n",
        "- No. of Bedrooms\n",
        "- No. of Bathrooms\n",
        "- No. of Receptions\n",
        "- Location\n",
        "- City/County\n",
        "- Postal Code\n",
        "\n",
        "Let's load it up and have a look:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 778
        },
        "id": "7rPiKv15NdM2",
        "outputId": "ba602aa0-4c45-4f92-81ca-1190855d8fc7"
      },
      "outputs": [],
      "source": [
        "def load_data(file_path):\n",
        "    \"\"\"\n",
        "    Loads the London Housing Data directly from GitHub using wget.\n",
        "    \"\"\"\n",
        "    !wget -q -O London_Housing_Data.csv 'https://raw.githubusercontent.com/powell-clark/supervised-machine-learning/main/datasets/London_Housing_Data.csv'\n",
        "    df = pd.read_csv('London_Housing_Data.csv', na_values=['NaN', 'nan', 'NAN', '', 'null', 'NULL'])\n",
        "    return df\n",
        "\n",
        "# Load the data\n",
        "df = load_data(\"../data/London_Housing_Data.csv\")\n",
        "\n",
        "# Display first 10 rows with headers in a more readable format\n",
        "print(\"\\nFirst 10 rows of the original dataset with headers:\")\n",
        "display(df.head(10))\n",
        "\n",
        "# Remove unnamed column with row numbers\n",
        "df = df.drop(columns=['Unnamed: 0'])\n",
        "\n",
        "print(\"\\nDataset shape after removing unnamed column:\", df.shape)\n",
        "display(df.head(10))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hPSZeW5YNdM2"
      },
      "source": [
        "\n",
        "<a name=\"exploratory-data-analysis\"></a>\n",
        "## Exploratory data analysis\n",
        "\n",
        "Let's explore the data to get a better understanding of it, identify any issues and get some insights that will help us prepare it for model training.\n",
        "\n",
        "Our EDA will cover:\n",
        "1. Numeric feature distributions and relationships\n",
        "2. Categorical feature analysis\n",
        "3. Price analysis by different groupings\n",
        "4. Missing value patterns\n",
        "5. Correlations and relationships between features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "q_8vRqk8NdM3",
        "outputId": "9cac6f91-1297-438b-bde2-8ca8e475ed0a"
      },
      "outputs": [],
      "source": [
        "def explore_data(df):\n",
        "    \"\"\"Comprehensive EDA combining numeric and categorical insights\"\"\"\n",
        "    print(\"\\nNumeric feature summary:\")\n",
        "    numeric_summary = df.select_dtypes(include=[np.number]).describe()\n",
        "    numeric_summary.loc['skew'] = df.select_dtypes(include=[np.number]).skew()\n",
        "    numeric_summary.loc['kurtosis'] = df.select_dtypes(include=[np.number]).kurtosis()\n",
        "    print(numeric_summary)\n",
        "\n",
        "    analyse_numeric_features(df)\n",
        "    analyse_categorical_features(df)\n",
        "    analyse_price_by_categories(df)\n",
        "\n",
        "def analyse_numeric_features(df):\n",
        "    \"\"\"Analyse numeric features with detailed distribution insights\"\"\"\n",
        "    # Price distribution with percentile annotations\n",
        "    plt.figure(figsize=(15,6))\n",
        "    ax = sns.histplot(df['Price']/1000000, kde=True)\n",
        "    percentiles = np.percentile(df['Price']/1000000, [25, 50, 75, 90, 95])\n",
        "    for p, label in zip(percentiles, ['25th', '50th', '75th', '90th', '95th']):\n",
        "        plt.axvline(p, linestyle='--', alpha=0.5)\n",
        "        plt.text(p, plt.ylim()[1]*0.9, f'{label}\\n£{p:.1f}M', rotation=90)\n",
        "    plt.title('House Price Distribution with Percentiles')\n",
        "    plt.xlabel('Price (£ millions)')\n",
        "    plt.ylabel('Frequency')\n",
        "    plt.show()\n",
        "\n",
        "    # Area vs Price with density coloring\n",
        "    plt.figure(figsize=(9,6))\n",
        "    plt.hexbin(df['Area in sq ft'], df['Price']/1000000,\n",
        "              gridsize=30, cmap='YlOrRd')\n",
        "    plt.colorbar(label='Count')\n",
        "    plt.title('Price vs. Area (Density Plot)')\n",
        "    plt.xlabel('Area in sq ft')\n",
        "    plt.ylabel('Price (£ millions)')\n",
        "    plt.show()\n",
        "\n",
        "    # Numeric correlations with detailed statistics\n",
        "    numeric_cols = ['Price', 'Area in sq ft', 'No. of Bedrooms',\n",
        "                   'No. of Bathrooms', 'No. of Receptions']\n",
        "\n",
        "    # Correlation analysis\n",
        "    corr_matrix = df[numeric_cols].corr()\n",
        "    plt.figure(figsize=(7.7,6))\n",
        "    mask = np.triu(np.ones_like(corr_matrix), k=1)\n",
        "    sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', fmt='.2f',\n",
        "                mask=mask, vmin=-1, vmax=1, center=0)\n",
        "    plt.title('Feature Correlation Matrix')\n",
        "    plt.show()\n",
        "\n",
        "    # Distribution profiles for all numeric features\n",
        "    fig, axes = plt.subplots(2, len(numeric_cols), figsize=(15, 8))\n",
        "    for idx, col in enumerate(numeric_cols):\n",
        "        # Histogram\n",
        "        sns.histplot(df[col], kde=True, ax=axes[0, idx])\n",
        "        axes[0, idx].set_title(f'{col} Distribution')\n",
        "        axes[0, idx].tick_params(axis='x', rotation=45)\n",
        "\n",
        "        # Box plot\n",
        "        sns.boxplot(y=df[col], ax=axes[1, idx])\n",
        "        axes[1, idx].set_title(f'{col} Box Plot')\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n",
        "\n",
        "def analyse_categorical_features(df):\n",
        "    \"\"\"Analyse categorical features with cardinality and association metrics\"\"\"\n",
        "    categorical_cols = ['House Type', 'Location', 'City/County', 'Postal Code']\n",
        "\n",
        "    print(\"\\nCategorical Feature Summary:\")\n",
        "    for col in categorical_cols:\n",
        "        print(f\"\\n{col}:\")\n",
        "\n",
        "        # Distribution statistics\n",
        "        value_counts = df[col].value_counts()\n",
        "        missing = df[col].isnull().sum()\n",
        "        unique_count = df[col].nunique()\n",
        "\n",
        "        print(f\"Unique values: {unique_count}\")\n",
        "        print(f\"Missing values: {missing} ({missing/len(df)*100:.1f}%)\")\n",
        "        print(f\"Top category share: {value_counts.iloc[0]/len(df)*100:.1f}%\")\n",
        "        print(f\"Top 3 categories cover: {value_counts.iloc[:3].sum()/len(df)*100:.1f}%\")\n",
        "\n",
        "        # Category frequency visualisation\n",
        "        if unique_count <= 15:\n",
        "            plt.figure(figsize=(12,6))\n",
        "            sns.barplot(x=value_counts.values[:10],\n",
        "                       y=value_counts.index[:10])\n",
        "            plt.title(f'{col} Category Distribution (Top 10)')\n",
        "            plt.xlabel('Count')\n",
        "            plt.xticks(rotation=45)\n",
        "            plt.show()\n",
        "        else:\n",
        "            print(f\"\\nTop 10 categories and their frequencies:\")\n",
        "            print(value_counts.head(10))\n",
        "\n",
        "def analyse_price_by_categories(df):\n",
        "    \"\"\"Analyse price distributions across categorical features\"\"\"\n",
        "    categorical_cols = ['House Type', 'Location', 'City/County', 'Postal Code']\n",
        "\n",
        "    for col in categorical_cols:\n",
        "        # Calculate statistics and sort by median price\n",
        "        stats = (df.groupby(col)['Price']\n",
        "                .agg(['count', 'mean', 'median', 'std'])\n",
        "                .sort_values('median', ascending=False))\n",
        "\n",
        "        # For categories with many unique values, show top 5\n",
        "        if df[col].nunique() > 20:\n",
        "            plt.figure(figsize=(14, 6))\n",
        "\n",
        "            # Get top 5 categories by median price\n",
        "            top_stats = stats.head(5)\n",
        "            top_cats = top_stats.index.tolist()  # Convert to list for proper indexing\n",
        "\n",
        "            # Filter data for top categories\n",
        "            data = df[df[col].isin(top_cats)]\n",
        "\n",
        "            # Create box plot\n",
        "            sns.boxplot(x=col, y='Price', data=data, color='lightgray', order=top_cats)\n",
        "\n",
        "            # Add scatter points\n",
        "            sns.stripplot(x=col, y='Price', data=data,\n",
        "                         size=8, color='darkblue', alpha=0.6,\n",
        "                         jitter=0.2, order=top_cats)\n",
        "\n",
        "            # Print debugging information\n",
        "            print(\"\\nTop categories and their counts:\")\n",
        "            for cat in top_cats:\n",
        "                count = len(data[data[col] == cat])\n",
        "                print(f\"{cat}: {count} properties\")\n",
        "\n",
        "            # Draw the plot to establish limits\n",
        "            plt.draw()\n",
        "\n",
        "            # Get axis limits\n",
        "            ymin, ymax = plt.ylim()\n",
        "\n",
        "            # Add count labels with consistent positioning\n",
        "            for i, cat in enumerate(top_cats):\n",
        "                count = len(data[data[col] == cat])\n",
        "                # Position text higher up from the bottom\n",
        "                text_y = ymin + (ymax - ymin) * 0.1\n",
        "                plt.text(i, text_y, f'n={count}',\n",
        "                        ha='center', va='bottom',\n",
        "                        color='blue', fontweight='bold')\n",
        "\n",
        "            plt.title(f'Property Prices by {col}\\nShowing individual properties (blue dots) and distribution (gray boxes)',\n",
        "                     pad=20, fontsize=14)\n",
        "            plt.xlabel(f'{col} (Top 5 by median price)')\n",
        "            plt.ylabel('Property Price (£)')\n",
        "            plt.xticks(rotation=45)\n",
        "\n",
        "            # Format price axis\n",
        "            format_price_axis(plt.gca().yaxis)\n",
        "\n",
        "            # Adjust layout to prevent label cutoff\n",
        "            plt.tight_layout()\n",
        "            plt.show()\n",
        "        else:\n",
        "            # Original code for categories with fewer unique values\n",
        "            plt.figure(figsize=(14, 6))\n",
        "            top_cats = stats.head(10).index.tolist()\n",
        "            data = df[df[col].isin(top_cats)]\n",
        "\n",
        "            sns.boxenplot(x=col, y='Price', data=data, color='lightgray', order=top_cats)\n",
        "            sns.stripplot(x=col, y='Price', data=data,\n",
        "                         size=2, alpha=0.3, jitter=0.2, color='darkblue',\n",
        "                         order=top_cats)\n",
        "\n",
        "            plt.xticks(rotation=45, ha='right')\n",
        "            plt.title(f'Price Distribution by {col} (Top 10 Categories)')\n",
        "            plt.ylabel('Price (£)')\n",
        "            format_price_axis(plt.gca().yaxis)\n",
        "            plt.tight_layout()\n",
        "            plt.show()\n",
        "\n",
        "def format_price_axis(axis):\n",
        "    \"\"\"Format price axis to millions with K/M suffixes\"\"\"\n",
        "    def price_format(x, p):\n",
        "        if x >= 1e6:\n",
        "            return f'£{x/1e6:.1f}M'\n",
        "        elif x >= 1e3:\n",
        "            return f'£{x/1e3:.0f}K'\n",
        "        return f'£{x:.0f}'\n",
        "\n",
        "    axis.set_major_formatter(ticker.FuncFormatter(price_format))\n",
        "\n",
        "# Run the full analysis\n",
        "explore_data(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NgbqkiJ1NdM5"
      },
      "source": [
        "<a name=\"exploratory-data-analysis-discussion\"></a>\n",
        "### Exploratory data analysis discussion\n",
        "\n",
        "When people buy houses, they follow a systematic evaluation process - examining location, size, and comparative prices.\n",
        "\n",
        "Our data reveals this same process, but with statistical patterns that profoundly impact our modeling approach.\n",
        "\n",
        "#### Price distribution dynamics\n",
        "\n",
        "Our price distribution spans from £180,000 to £39,750,000 - a 222-fold range that reveals fundamental market mechanics.\n",
        "\n",
        "This distribution segments into distinct market components:\n",
        "\n",
        "- Entry-level flats (median £800,000, CV: 0.25)\n",
        "- Mid-market new developments (median £1.05M, CV: 1.61)\n",
        "- Premium houses (median £1.75M, CV: 0.96)\n",
        "- Ultra-premium properties (>£17.95M, concentrated in SW1, NW11, SW7)\n",
        "\n",
        "The Coefficient of Variation (CV) tells a fascinating story: new developments show the highest volatility (1.61), while studios exhibit remarkable price consistency (0.25). This pattern isn't random - it reflects how different property types respond to market forces.\n",
        "\n",
        "Key implication: We need logarithmic price transformation because price impacts are multiplicative, not additive. A 10% improvement adds £80,000 to an £800,000 flat but £175,000 to a £1.75M house.\n",
        "\n",
        "#### Price per square foot analysis\n",
        "\n",
        "Price per square foot reveals another layer of market dynamics:\n",
        "- Overall median: £850/sq ft\n",
        "- Geographic variation: £450/sq ft (outer postcodes) to £2,200/sq ft (prime central)\n",
        "- Property type impact:\n",
        "  - Flats: £750-900/sq ft (higher density areas)\n",
        "  - Houses: £600-1,500/sq ft (wider variation due to land value)\n",
        "  - Ultra-premium: >£3,000/sq ft (limited locations)\n",
        "\n",
        "This metric provides crucial standardisation across property sizes but shows significant geographic dependence, suggesting value in calculating local area benchmarks.\n",
        "\n",
        "#### Location's hierarchical structure\n",
        "\n",
        "Our location data forms a three-tier hierarchy with distinct statistical properties:\n",
        "\n",
        "1. Administrative level\n",
        "   - London dominates (85.4% of properties)\n",
        "   - Surrey represents 7.5%\n",
        "   - Statistical challenge: Surrey sample size limits generalisability\n",
        "\n",
        "2. Neighborhood level (n=656)\n",
        "   - 27.6% missing data\n",
        "   - Highest frequency: Putney at 2.8%\n",
        "   - Severe fragmentation: median frequency <0.2%\n",
        "\n",
        "3. Postal code level (n=2,845)\n",
        "   - Complete coverage but extreme fragmentation\n",
        "   - Maximum frequency: 0.4%\n",
        "   - Systematic structure through outcodes\n",
        "\n",
        "This hierarchy presents two potential encoding paths: one-hot encoding for direct categorical relationships, or target encoding to capture price-level relationships. Each has distinct advantages we'll explore in our processing approach.\n",
        "\n",
        "#### Data quality analysis\n",
        "\n",
        "Two critical quality issues demand attention:\n",
        "\n",
        "1. Room counts\n",
        "   - Perfect correlation (ρ = 1.0) between bedroom, bathroom, and reception counts\n",
        "   - Identical means (μ = 3.10) across all three measures\n",
        "   - Statistically impossible in real properties\n",
        "   - Implications: Must select single room metric as proxy for size\n",
        "\n",
        "2. Feature reliability assessment\n",
        "   - Area (sq ft): Strongest predictor (r = 0.67 with price)\n",
        "   - Property identifiers: Unique per observation, zero information content\n",
        "   - Location strings: Require standardisation (27.6% missing, inconsistent formatting)\n",
        "\n",
        "#### Methodological implications\n",
        "\n",
        "These patterns suggest several modeling approaches:\n",
        "\n",
        "1. Price treatment\n",
        "   - Log transformation required for multiplicative effects\n",
        "   - Stratified sampling across price bands essential\n",
        "   - Performance metrics must consider relative errors\n",
        "\n",
        "2. Geographic features\n",
        "   - Two viable encoding approaches:\n",
        "     - One-hot encoding for direct categorical relationships\n",
        "     - Target encoding to capture price-level relationships\n",
        "   - Outcode extraction offers useful intermediate granularity\n",
        "   - Missing data requires careful handling (27.6% at neighborhood level)\n",
        "\n",
        "3. Size and value metrics\n",
        "   - Area (sq ft) as primary metric\n",
        "   - Single room count as secondary proxy\n",
        "   - Price per square foot as standardised comparison\n",
        "   - Local area benchmarks for relative value assessment\n",
        "\n",
        "4. Validation strategy\n",
        "   - Stratified cross-validation across price bands\n",
        "   - Separate performance metrics for market segments\n",
        "   - Coefficient of Variation monitoring per segment\n",
        "   - Validation of both encoding approaches\n",
        "\n",
        "This analysis establishes our statistical requirements: we need precise handling of multiplicative price effects, careful feature encoding choices, and rigorous cross-segment validation. The next section translates these requirements into a systematic data processing pipeline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VkxDl4PCNdM6"
      },
      "source": [
        "<a name=\"data-processing-strategy\"></a>\n",
        "### Data processing strategy\n",
        "\n",
        "Our exploratory analysis revealed the fundamental mechanics of London's housing market - multiplicative price effects spanning a 222-fold range and a natural geographic hierarchy that profoundly influences values. Translating these insights into a robust processing pipeline requires careful attention to the order of operations, particularly regarding price information.\n",
        "\n",
        "The path from raw data to modeling features follows five essential stages:\n",
        "1. Data validation & cleaning\n",
        "2. Preliminary feature engineering\n",
        "3. Price distribution transformation\n",
        "4. Stratified data splitting\n",
        "5. Feature encoding and target-variable-aware feature engineering\n",
        "\n",
        "The sequence is crucial for statistical validity.\n",
        "\n",
        "### Core Processing Steps\n",
        "\n",
        "1. **Data validation & cleaning**\n",
        "\n",
        "   Address the data quality issues identified in our exploration:\n",
        "   - Remove redundant room features (correlation coefficient ρ = 1.0)\n",
        "   - Standardise location strings (27.6% missing, requires structured handling)\n",
        "   - Clean city/county values (systematic geographic normalisation)\n",
        "   - Preserve fundamental numeric features in original form\n",
        "\n",
        "2. **Initial feature engineering**\n",
        "\n",
        "   Our geographic hierarchy analysis suggests immediate structural features:\n",
        "   - Extract outcodes from postal codes (geographic aggregation)\n",
        "   - This transformation is price-independent, based purely on postal code structure\n",
        "   - Creates intermediate geographic granularity\n",
        "   - Establishes foundation for later feature encoding comparisons\n",
        "\n",
        "3. **Price distribution transformation**\n",
        "\n",
        "   Address the multiplicative nature of price variations:\n",
        "   - Apply logarithmic transformation (normalises 222-fold range)\n",
        "   - Generate price bands for stratification\n",
        "   - Enables proper handling of multiplicative price effects\n",
        "\n",
        "4. **Train/test split**\n",
        "\n",
        "   To maintain statistical validity we need to:\n",
        "   - Implement stratified sampling using price bands - similar to how we balanced benigh vs malignant in lesson 1\n",
        "   - Check geographic distribution is preserved\n",
        "   - Establish a truly independent test set\n",
        "\n",
        "5. **Feature encoding and target-variable-aware feature engineering**\n",
        "\n",
        "   Post-split transformations requiring careful handling of price information:\n",
        "\n",
        "   A. One-Hot encoding (categorical to binary features)\n",
        "   - Convert house type to set of binary indicator columns\n",
        "   - Transform city/county to binary indicator columns\n",
        "   - Create outcode binary indicator columns\n",
        "   - Maintains complete independence from price variable\n",
        "\n",
        "   B. Target encoding (price-based location encoding)\n",
        "   - Hierarchical encoding: outcode → postcode → location\n",
        "   - Calculate encoding means using only training data\n",
        "   - Implement prior smoothing for stability\n",
        "   - Store training means for future predictions\n",
        "   - Handle missing values through hierarchy\n",
        "\n",
        "   C. Mean outcode price per square foot\n",
        "   - Calculate using only training data statistics\n",
        "   - Apply stored training means to test data\n",
        "   - Persist training means for new predictions\n",
        "   - Maintain strict statistical separation\n",
        "\n",
        "### Statistical validity through ordering\n",
        "\n",
        "Consider the fundamental difference between structure-based and price-dependent outcode features:\n",
        "```python\n",
        "# Structure-based: Valid pre-split\n",
        "df['outcode'] = df['postcode'].str.extract('^([A-Z]+)')\n",
        "df['outcode_SW1'] = (df['outcode'] == 'SW1').astype(int)\n",
        "\n",
        "# Price-dependent: Requires careful post-split handling\n",
        "outcode_means = train_data.groupby('outcode')['price'].mean()\n",
        "train_data['outcode_price'] = train_data['outcode'].map(outcode_means)\n",
        "test_data['outcode_price'] = test_data['outcode'].map(outcode_means)  # Uses training means only\n",
        "```\n",
        "\n",
        "This pipeline will create two parallel feature sets - one using one-hot encoding and another using target encoding - allowing us to compare their effectiveness while maintaining statistical validity. Each transformation preserves the insights from our exploratory analysis while ensuring proper separation of price information between training and test data.\n",
        "\n",
        "The essence of this approach is maintaining statistical rigor through proper sequencing: we can create structural features immediately, but any feature using price information must be created post-split using only training data statistics. These training-derived values must then be persisted for consistent application to both test data and future predictions.\n",
        "\n",
        "Let's proceed with implementing each stage of this carefully ordered pipeline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ropVLg3CNdM6"
      },
      "source": [
        "<a name=\"data-validation\"></a>\n",
        "## Data validation\n",
        "\n",
        "Let's check our data before cleaning it.\n",
        "\n",
        "We'll check the shape of the data and the first few rows, and then print the data types and value ranges for each column."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "VT0y8MmTNdM6",
        "outputId": "bac1ef0d-7bb8-45c7-808f-e09e1088eb99"
      },
      "outputs": [],
      "source": [
        "print('Data shape before cleaning:', df.shape)\n",
        "display(df)\n",
        "\n",
        "# Check for missing values\n",
        "print(\"\\nChecking for missing values:\")\n",
        "missing_values = df.isnull().sum()\n",
        "print(missing_values)\n",
        "\n",
        "# Check that all rows have the expected number of columns\n",
        "expected_columns = len(df.columns)\n",
        "rows_with_missing_cols = df.shape[1] != expected_columns\n",
        "if rows_with_missing_cols:\n",
        "    print(f\"\\nWARNING: Some rows are missing columns. Expected {expected_columns} columns.\")\n",
        "else:\n",
        "    print(f\"\\nAll rows have the expected {expected_columns} columns.\")\n",
        "\n",
        "# Print the data types and value ranges for each column\n",
        "print(\"\\nData types and value ranges for each column:\\n\")\n",
        "\n",
        "for column in df.columns:\n",
        "    print(f\"\\n{column}:\")\n",
        "    if df[column].dtype in ['int64', 'float64']:\n",
        "        print(f\"Type: {df[column].dtype}\")\n",
        "        print(f\"Range: {df[column].min():,.2f} to {df[column].max():,.2f}\")\n",
        "        print(f\"Mean: {df[column].mean():,.2f}\")\n",
        "    else:\n",
        "        print(f\"Type: {df[column].dtype}\")\n",
        "        print(\"Categories:\")\n",
        "        value_counts = df[column].value_counts()\n",
        "        for value, count in value_counts.items():\n",
        "            print(f\"  - {value}: {count:,} occurrences\")\n",
        "\n",
        "# Check for potential misspellings in Location categories by sorting alphabetically\n",
        "print(\"\\nUnique location values (sorted alphabetically):\")\n",
        "locations = df['Location'].value_counts().dropna()\n",
        "sorted_locations = sorted(locations.items(), key=lambda x: str.lower(x[0]))\n",
        "for loc, count in sorted_locations:\n",
        "    print(f\"  - {loc}: {count} occurrences\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DECaz6wNNdM6"
      },
      "source": [
        "<a name=\"data-cleaning\"></a>\n",
        "## Data cleaning\n",
        "\n",
        "From our exploratory data analysis and the validation above we've identified some data quality issues that we'll need to clean up.\n",
        "\n",
        "1. 'Property Name' - in this lesson we won't attempt to infer meaning from the property name so lets drop this column.\n",
        "\n",
        "2. 'No. of Bathrooms' and 'No. of Receptions'\n",
        "    - These two features are redundant as they are perfect correlated with bedrooms, there is an error in the data collection here so we'll choose only bedrooms\n",
        "    - We'll also convert 0 bedrooms to 1 as studios are a bedroom!\n",
        "    - Both these issues bring into question the quality of the data but for now we'll proceed\n",
        "    - In a production setting we would need to investigate this further, being more careful about making assumptions and not just take any dataset off Kaggle at face value ;)\n",
        "\n",
        "3. Location values are not consistent, they have missing values and are highly cardinal/fragmented:\n",
        "    - Consisting of a mix of full and partial address first lines & borough and area names  \n",
        "    - Inconsistent whitespace, special characters and leading numbers\n",
        "\n",
        "4. City/county level data is complete but has some strange values:\n",
        "    - The City of London, London and main surrounding counties are represented\n",
        "    - 96 values in this column are not counties or regions but are instead a mix of neighbourhoods, area and district names\n",
        "    - This may be a data parsing error when splitting the address into lines\n",
        "\n",
        " 5. On the plus side our postcode level data is gorgeous!\n",
        "     - 2,845 unique codes\n",
        "     - No missing values\n",
        "     - All valid UK postcodes\n",
        "     - Extremely granular\n",
        "     - Hierarchical structure potential\n",
        "     - Opportunity to create derived feature such as postcode outcode\n",
        "   \n",
        "Lets clean these up!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        },
        "id": "dzyJwAE_NdM7",
        "outputId": "e012f5c0-81b7-479b-fc99-d0838823e823"
      },
      "outputs": [],
      "source": [
        "# Remove redundant features\n",
        "df_updated_columns = df.drop(['Property Name', 'No. of Bathrooms', 'No. of Receptions'], axis=1)\n",
        "\n",
        "print(f\"Shape after cleaning: {df_updated_columns.shape}\")\n",
        "display(df_updated_columns.head())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hZBHP_evNdM7",
        "outputId": "6b284502-8e25-47ad-8d62-eb71a72df3f9"
      },
      "outputs": [],
      "source": [
        "df_with_transformed_bedrooms = df_updated_columns.copy()\n",
        "# Convert 0 bedrooms to 1 - studios are a bedroom!\n",
        "df_with_transformed_bedrooms.loc[df_with_transformed_bedrooms['No. of Bedrooms'] == 0, 'No. of Bedrooms'] = 1\n",
        "\n",
        "df_with_clean_counties = df_with_transformed_bedrooms.copy()\n",
        "# Clean up City/County values\n",
        "df_with_clean_counties['City/County'] = df_with_clean_counties['City/County'].str.lower().str.strip()\n",
        "\n",
        "# Print NaN counts before cleaning\n",
        "print(\"\\nNaN counts before cleaning:\")\n",
        "print(f\"Location NaN count: {df_updated_columns['Location'].isna().sum()}\")\n",
        "\n",
        "# Valid counties list\n",
        "valid_counties = ['london', 'surrey', 'middlesex', 'essex', 'hertfordshire', 'kent', 'city of london']\n",
        "\n",
        "# Postcode outcode to county mapping\n",
        "postcode_to_county = {\n",
        "    'E': 'london',\n",
        "    'EC': 'city of london',\n",
        "    'N': 'london',\n",
        "    'NW': 'london',\n",
        "    'SE': 'london',\n",
        "    'SW': 'london',\n",
        "    'W': 'london',\n",
        "    'WC': 'london',\n",
        "    'KT': 'surrey',\n",
        "    'CR': 'surrey',\n",
        "    'IG': 'essex',\n",
        "    'CM': 'essex',\n",
        "    'EN': 'hertfordshire',\n",
        "    'WD': 'hertfordshire',\n",
        "    'HA': 'hertfordshire',\n",
        "    'TW': 'middlesex',\n",
        "    'UB': 'middlesex'\n",
        "}\n",
        "\n",
        "# Store original values for reporting\n",
        "original_locations = df_with_clean_counties['Location'].copy()\n",
        "original_cities = df_with_clean_counties['City/County'].copy()\n",
        "\n",
        "# Create new df\n",
        "df_clean_counties_and_updated_locations = df_with_clean_counties.copy()\n",
        "\n",
        "# Find rows where City/County is not in valid counties\n",
        "invalid_counties_mask = ~df_clean_counties_and_updated_locations['City/County'].isin(valid_counties)\n",
        "\n",
        "# Array to store rows where location updates should be skipped\n",
        "skip_location_updates = [\n",
        "    # Add row numbers here where location should not be overwritten because it didn't make sense to update the location with the city value\n",
        "    193, 444, 1007, 1290, 1388, 1481, 1502, 1503, 1914, 3154, 3422\n",
        "]\n",
        "\n",
        "city_updates = 0\n",
        "location_updates = 0\n",
        "\n",
        "if invalid_counties_mask.any():\n",
        "    # For these rows, get their outcodes\n",
        "    invalid_rows = df_clean_counties_and_updated_locations[invalid_counties_mask]\n",
        "    outcodes = invalid_rows['Postal Code'].str.extract('^([A-Z]+)')[0]\n",
        "    new_counties = outcodes.map(postcode_to_county)\n",
        "\n",
        "    # Update only rows where the City/County needs to change\n",
        "    for idx in invalid_rows.index:\n",
        "        current_county = df_clean_counties_and_updated_locations.loc[idx, 'City/County']\n",
        "        new_county = new_counties[idx]\n",
        "\n",
        "        if current_county != new_county:\n",
        "            # Only update location if row is not in skip list\n",
        "            if idx not in skip_location_updates:\n",
        "                df_clean_counties_and_updated_locations.loc[idx, 'Location'] = current_county  # Save old county as location\n",
        "                location_updates += 1\n",
        "            df_clean_counties_and_updated_locations.loc[idx, 'City/County'] = new_county   # Always update county\n",
        "            city_updates += 1\n",
        "\n",
        "# Print NaN counts after cleaning\n",
        "print(\"\\nNaN counts after cleaning:\")\n",
        "print(f\"Location NaN count: {df_clean_counties_and_updated_locations['Location'].isna().sum()}\")\n",
        "\n",
        "print(f\"\\nTotal number of city/county updates: {city_updates}\")\n",
        "print(f\"Total number of location updates (accounting for skipped rows): {location_updates}\")\n",
        "\n",
        "# Create update report only for rows that changed\n",
        "updates = pd.DataFrame({\n",
        "    'Original Location': original_locations,\n",
        "    'Original City/County': original_cities,\n",
        "    'New Location': df_clean_counties_and_updated_locations['Location'],\n",
        "    'New City/County': df_clean_counties_and_updated_locations['City/County'],\n",
        "    'Postcode': df_clean_counties_and_updated_locations['Postal Code']\n",
        "})\n",
        "\n",
        "# Show only rows where actual changes occurred\n",
        "updates = updates[\n",
        "    ((updates['Original Location'] != updates['New Location']) & ~(updates['Original Location'].isna() & updates['New Location'].isna())) |\n",
        "    (updates['Original City/County'] != updates['New City/County'])\n",
        "]\n",
        "\n",
        "print(\"\\nLocation updates:\")\n",
        "print(updates.to_string())\n",
        "\n",
        "print(\"\\nFirst 5 rows of cleaned dataset:\")\n",
        "print(df_clean_counties_and_updated_locations.head().to_string())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "O7R6EXKENdM7",
        "outputId": "df39e77b-87f2-4bae-8830-be821b7975c3"
      },
      "outputs": [],
      "source": [
        "# Create new dataframe with cleaned data\n",
        "df_cleaned = df_clean_counties_and_updated_locations.copy()\n",
        "\n",
        "# Count unique locations before cleaning\n",
        "unique_locations_before = df_cleaned['Location'].nunique()\n",
        "print(f\"\\nNumber of unique locations before cleaning: {unique_locations_before}\")\n",
        "\n",
        "# Clean up Location column\n",
        "def clean_location(x):\n",
        "    if not isinstance(x, str):\n",
        "        return x\n",
        "\n",
        "    # Convert to lowercase and strip whitespace\n",
        "    x = x.lower().strip()\n",
        "\n",
        "    # Remove special characters and extra spaces\n",
        "    x = re.sub(r'[^\\w\\s-]', '', x)\n",
        "    x = re.sub(r'\\s+', ' ', x)\n",
        "\n",
        "    # Remove leading numbers and hyphens (e.g., \"161-\", \"35-37\", \"131-143\")\n",
        "    x = re.sub(r'^\\d+(?:-\\d+)?\\s*', '', x)\n",
        "\n",
        "    # Remove any single letter followed by space at start\n",
        "    x = re.sub(r'^[a-z]\\s+', '', x)\n",
        "\n",
        "    return x.strip()\n",
        "\n",
        "df_cleaned['Location'] = df_cleaned['Location'].apply(clean_location)\n",
        "\n",
        "# Count unique locations after cleaning\n",
        "unique_locations_after = df_cleaned['Location'].nunique()\n",
        "print(f\"Number of unique locations after cleaning: {unique_locations_after}\")\n",
        "print(f\"Reduction in unique locations: {unique_locations_before - unique_locations_after}\")\n",
        "\n",
        "print(\"\\nUnique location values after cleaning (sorted alphabetically):\")\n",
        "locations = df_cleaned['Location'].value_counts().dropna()\n",
        "sorted_locations = sorted(locations.items(), key=lambda x: str.lower(x[0]))\n",
        "for loc, count in sorted_locations:\n",
        "    print(f\"  - {loc}: {count} occurrences\")\n",
        "\n",
        "print(\"\\nFirst few rows of cleaned dataframe:\")\n",
        "display(df_cleaned.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-v50TIZtNdM7"
      },
      "source": [
        "Great! We've cleaned up our data.\n",
        "\n",
        "Columns are now:\n",
        "- Price\n",
        "- House Type\n",
        "- Area in sq ft\n",
        "- No. of Bedrooms\n",
        "- Location\n",
        "- City/County\n",
        "- Postal Code\n",
        "\n",
        "Parsing the city field for misplaced location values has reduced the NaN values in the location field from 962 to 916\n",
        "\n",
        "This parsing updated 94 city/counties and 83 location values - accounting for skipped rows, where it didn't make sense to update the location with the city value.\n",
        "\n",
        "The resultant location field has been cleaned up by making the values more consistently cased, removing property numbers and removing special characters and extra whitespace:\n",
        "\n",
        "- Number of unique locations before cleaning: 674\n",
        "- Number of unique locations after cleaning: 511\n",
        "- Reduction in unique locations: 163\n",
        "\n",
        "City/County has been cleaned up to a more consistent format of:\n",
        "- London\n",
        "- Surrey\n",
        "- Middlesex\n",
        "- Essex\n",
        "- Hertfordshire\n",
        "- Kent\n",
        "\n",
        "We are now ready to move on to the next step of feature engineering, where we'll create derived features from the postcode field and price per area.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HB6tD6BDNdM7"
      },
      "source": [
        "<a name=\"initial-feature-engineering\"></a>\n",
        "## Initial feature engineering\n",
        "\n",
        "Before training our models, we can enhance our dataset through feature engineering - crafting new features that capture important patterns in the data. For London house prices, the challenge is creating features that reflect both physical property characteristics and location value.\n",
        "\n",
        "### Key opportunities\n",
        "\n",
        "1. **Geographic granularity**\n",
        "   - Extract outcodes from postcodes (e.g., \"SW6\" from \"SW6 3LF\")\n",
        "   - Create area/borough level statistics\n",
        "   - Potential for linking external data:\n",
        "     * School quality metrics\n",
        "     * Transit accessibility\n",
        "     * Crime statistics\n",
        "     * Green space coverage\n",
        "     * Local amenities\n",
        "\n",
        "2. **Property value metrics**\n",
        "   - Price per square foot\n",
        "   - Room ratios\n",
        "   - Local price benchmarks\n",
        "   - Property type premiums\n",
        "\n",
        "In this lesson, we'll focus on two foundational features:\n",
        "\n",
        "#### 1. Postcode outcode\n",
        "Our data shows 2,845 unique postcodes spread across 3,478 properties (1.22 properties per postcode) - too sparse for effective modeling. However, outcodes provide a sweet spot:\n",
        "- More properties per area (better statistical power)\n",
        "- Captures neighborhood-level price patterns\n",
        "- Reduces feature dimensionality\n",
        "- Enables reliable area statistics\n",
        "\n",
        "#### 2. Price per square foot\n",
        "This standardised metric requires careful handling:\n",
        "- Must be calculated post train/test split\n",
        "- Use only training data for statistics\n",
        "- Apply same scaling to test data\n",
        "- Handle outliers systematically\n",
        "\n",
        "### Implementation strategy\n",
        "\n",
        "We'll create these features in order of statistical safety:\n",
        "1. First, outcodes (structure-based, no leakage risk)\n",
        "2. Then price transformations (mathematical, no data dependencies)\n",
        "3. Finally, price per square foot (requires careful train/test handling)\n",
        "\n",
        "Let's start by extracting outcodes from our postal codes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 615
        },
        "id": "tvqY1KwWNdM7",
        "outputId": "b75f7b51-445f-464c-ecda-9964ba050b6d"
      },
      "outputs": [],
      "source": [
        "def extract_outcode(postcode: str) -> str:\n",
        "    \"\"\"Extract the outcode (first part) from a postcode.\"\"\"\n",
        "    return postcode.split()[0] if isinstance(postcode, str) else None\n",
        "\n",
        "def add_outcode_feature(df: pd.DataFrame) -> pd.DataFrame:\n",
        "    \"\"\"Add outcode feature derived from Postal Code column.\"\"\"\n",
        "    df_with_outcode = df.assign(\n",
        "        Outcode=df['Postal Code'].map(extract_outcode)\n",
        "    )\n",
        "\n",
        "    n_unique = df_with_outcode['Outcode'].nunique()\n",
        "    avg_properties = len(df_with_outcode) / n_unique\n",
        "\n",
        "    print(f\"Created {n_unique} unique outcodes\")\n",
        "    print(f\"Average properties per outcode: {avg_properties:.1f}\")\n",
        "\n",
        "    return df_with_outcode\n",
        "\n",
        "# Apply to each of our cleaned datasets\n",
        "df_with_outcode = add_outcode_feature(df_cleaned)\n",
        "\n",
        "display(df_with_outcode)\n",
        "\n",
        "print(\"\\nTop 5 outcodes by average price:\")\n",
        "print(df_with_outcode.groupby('Outcode')['Price'].agg(['mean', 'count'])\n",
        "      .sort_values('mean', ascending=False)\n",
        "      .head())\n",
        "\n",
        "# Save the DataFrame to a csv file\n",
        "# df_with_outcode.to_csv('../data/df_with_outcode.csv', index=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BCiZjZBoNdM8"
      },
      "source": [
        "<a name=\"price-transformation-and-price-aware-feature-engineering-strategy\"></a>\n",
        "## Price transformation and price-aware feature engineering strategy\n",
        "\n",
        "Our data contains both price-dependent and price-independent features that require different handling. Let's establish a clear strategy for each type.\n",
        "\n",
        "### Price-independent features (can be created anytime)\n",
        "\n",
        "1. **Outcode extraction** (Already completed)\n",
        "   - Purely structural feature from postcodes\n",
        "   - No price information used\n",
        "   - Could have been created at any point\n",
        "\n",
        "2. **One-Hot encoded Features**\n",
        "   - House Type (8 categories)\n",
        "   - City/County (6 categories)\n",
        "   - Outcode (~100 categories)\n",
        "   - Can be created before or after split\n",
        "   - We'll create after split for code organisation\n",
        "\n",
        "### Price-dependent Features (must wait for train/test split)\n",
        "\n",
        "1. **Target encoded location features**\n",
        "   - Outcode mean price encoding\n",
        "   - Postcode encoding with outcode prior\n",
        "   - Location encoding with postcode prior\n",
        "   - Must use only training data means\n",
        "   - Apply training means to test data\n",
        "\n",
        "2. **Mean price per area features**\n",
        "   - Mean price per sqft by outcode\n",
        "   - Mean price per sqft by postcode\n",
        "   - Use training data for calculations\n",
        "   - Apply same means to test data\n",
        "\n",
        "### Execution order\n",
        "\n",
        "1. **Price distribution transformation**\n",
        "   - Log transform prices\n",
        "   - Create price bands for stratification\n",
        "   - Enables balanced dataset splitting\n",
        "\n",
        "2. **Train/test split**\n",
        "   - Stratify using price bands\n",
        "   - Ensures representative splits\n",
        "   - Critical boundary for information flow\n",
        "\n",
        "3. **Create all feature sets**\n",
        "   - One-hot encoded version\n",
        "     * Direct categorical relationships\n",
        "     * No price information used\n",
        "     * Sparse but interpretable\n",
        "\n",
        "   - Target encoded version\n",
        "     * Uses training price information\n",
        "     * Hierarchical smoothing\n",
        "     * Captures price-location relationships\n",
        "\n",
        "   - Mean price per area version\n",
        "     * Uses training data only\n",
        "     * Outcode and postcode means\n",
        "     * Area-normalised price signals\n",
        "\n",
        "This strategy will create three parallel feature sets:\n",
        "1. One-hot encoded (sparse, direct relationships)\n",
        "2. Target encoded (smooth, hierarchical)\n",
        "3. Price per area (standardised, interpretable)\n",
        "\n",
        "We'll evaluate model performance on each to understand their relative strengths. Let's implement our strategy:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Gokc_2oNNdM8",
        "outputId": "556b8697-2974-4ed6-e0d3-3ea034337062"
      },
      "outputs": [],
      "source": [
        "# Add log-transformed price\n",
        "df_with_outcode['log_price'] = np.log(df_with_outcode['Price'])\n",
        "\n",
        "# Create price bands for stratification\n",
        "df_with_outcode['price_band'] = pd.qcut(df_with_outcode['log_price'], q=10, labels=False)\n",
        "\n",
        "# Validation plots\n",
        "fig, axs = plt.subplots(2, 2, figsize=(9, 6))\n",
        "\n",
        "# Original price distribution\n",
        "sns.histplot(df_with_outcode['Price']/1000000, kde=True, ax=axs[0, 0])\n",
        "axs[0, 0].set_title('Original Price Distribution')\n",
        "axs[0, 0].set_xlabel('Price (£ millions)')\n",
        "axs[0, 0].ticklabel_format(style='plain', axis='x')\n",
        "\n",
        "# Log-transformed price distribution\n",
        "sns.histplot(df_with_outcode['log_price'], kde=True, ax=axs[0, 1])\n",
        "axs[0, 1].set_title('Log-Transformed Price Distribution')\n",
        "axs[0, 1].set_xlabel('Log Price')\n",
        "\n",
        "# Price band distribution\n",
        "sns.countplot(x='price_band', data=df_with_outcode, ax=axs[1, 0])\n",
        "axs[1, 0].set_title('Price Band Distribution')\n",
        "axs[1, 0].set_xlabel('Price Band')\n",
        "\n",
        "# Price percentiles\n",
        "percentiles = np.percentile(df_with_outcode['Price']/1000000, [25, 50, 75])\n",
        "axs[1, 1].text(0.1, 0.8, f'Price Quartiles (£M):\\n\\n25th: £{percentiles[0]:.2f}M\\n50th: £{percentiles[1]:.2f}M\\n75th: £{percentiles[2]:.2f}M',\n",
        "               fontsize=12)\n",
        "axs[1, 1].axis('off')\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Print summary statistics\n",
        "print(\"\\nPrice Distribution Summary:\")\n",
        "print(df_with_outcode[['Price', 'log_price']].describe().round(2))\n",
        "\n",
        "print(\"\\nPrice Band Counts:\")\n",
        "print(df_with_outcode['price_band'].value_counts().sort_index())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 540
        },
        "id": "lI0z5O6fNdM8",
        "outputId": "242676de-5f19-4716-e37b-9fd86d58cee4"
      },
      "outputs": [],
      "source": [
        "# 80/20 stratified split using price bands\n",
        "train_data, test_data = train_test_split(\n",
        "    df_with_outcode,\n",
        "    test_size=0.2,\n",
        "    stratify=df_with_outcode['price_band'],\n",
        "    random_state=RANDOM_STATE\n",
        ")\n",
        "\n",
        "print(f\"Training Set Shape: {train_data.shape}\")\n",
        "print(f\"Test Set Shape: {test_data.shape}\")\n",
        "\n",
        "# Validate split distributions\n",
        "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 4))\n",
        "\n",
        "# Compare log price distributions\n",
        "sns.kdeplot(train_data['log_price'], label='Training', ax=ax1)\n",
        "sns.kdeplot(test_data['log_price'], label='Test', ax=ax1)\n",
        "ax1.set_title('Log Price Distribution: Train vs Test')\n",
        "ax1.legend()\n",
        "\n",
        "# Compare price band proportions\n",
        "train_props = train_data['price_band'].value_counts(normalize=True).sort_index()\n",
        "test_props = test_data['price_band'].value_counts(normalize=True).sort_index()\n",
        "\n",
        "pd.DataFrame({'Train': train_props, 'Test': test_props}).plot(kind='bar', ax=ax2)\n",
        "ax2.set_title('Price Band Proportions: Train vs Test')\n",
        "ax2.set_xlabel('Price Band')\n",
        "ax2.set_ylabel('Proportion')\n",
        "ax2.legend()\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Check geographic distribution\n",
        "print(\"\\nOutcode Distribution:\")\n",
        "train_outcode_props = train_data['Outcode'].value_counts(normalize=True)\n",
        "test_outcode_props = test_data['Outcode'].value_counts(normalize=True)\n",
        "print(f\"Training unique outcodes: {len(train_outcode_props)}\")\n",
        "print(f\"Test unique outcodes: {len(test_outcode_props)}\")\n",
        "\n",
        "# Verify all test outcodes exist in training\n",
        "missing_outcodes = set(test_data['Outcode']) - set(train_data['Outcode'])\n",
        "if missing_outcodes:\n",
        "    print(f\"Warning: {len(missing_outcodes)} outcodes in test but not in training\")\n",
        "    print(\"Missing outcodes:\", missing_outcodes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XE2Oay-VNdM8"
      },
      "source": [
        "<a name=\"feature-encoding\"></a>\n",
        "## Feature encoding\n",
        "\n",
        "With our train/test split complete, we'll create feature sets that let us understand how different types of features impact model performance, especially the effect of price-derived features:\n",
        "\n",
        "### Core features - used in all sets\n",
        "- Area in sq ft (numerical)\n",
        "- No. of Bedrooms (numerical)\n",
        "- House Type - one-hot encoded (categorical)\n",
        "- Log-transformed price (target)\n",
        "\n",
        "### 1. One-hot encoded features\n",
        "Core features + simple categorical encoding with no price information:\n",
        "- City/County (6 categories)\n",
        "- Outcode (~100 categories)\n",
        "- Location (511 categories)\n",
        "- Missing values get their own binary indicator\n",
        "- Encoder must be persisted to handle new categories\n",
        "\n",
        "### 2. Target encoded features\n",
        "Core features + location features encoded using price information:\n",
        "- Outcode mean price encoding (calculated from training data)\n",
        "- Postcode encoding with outcode prior (calculated from training data)\n",
        "- Location encoding with postcode prior (calculated from training data)\n",
        "- Missing value handling:\n",
        "  * Missing locations use postcode encoding\n",
        "  * Missing postcodes use outcode encoding\n",
        "  * Missing outcodes use global mean\n",
        "- Hierarchical smoothing to handle sparsity\n",
        "- Must persist training means and priors for new data\n",
        "\n",
        "### 3. Mean price per area features\n",
        "Core features + standardised area price metrics:\n",
        "- Mean price per sqft by outcode (calculated from training data)\n",
        "- Missing outcodes use global mean from training data\n",
        "- Must persist outcode means and global mean for new data\n",
        "\n",
        "By comparing model performance with different feature combinations, we'll learn:\n",
        "1. How much predictive power comes from pure categorical relationships\n",
        "2. What we gain by incorporating price information into features\n",
        "3. The trade-offs between model accuracy and potential price feedback loops\n",
        "\n",
        "For each encoding approach, we'll create both the features and the persistence mechanism needed to encode new data at prediction time. Let's start with one-hot encoding...\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EmY0oQpFfA-1"
      },
      "source": [
        "<a name=\"one-hot-encoder-with-persistence-implementation\"></a>\n",
        "### One-hot encoder with persistence implementation\n",
        "\n",
        "This encoder creates production-ready categorical features by wrapping sci-kit's standard OneHotEncoder in validation and persistence logic.\n",
        "\n",
        "In production, when applying the model to new data every new prediction must see exactly the same binary columns in exactly the same order as training. The solution tracks every category's position, handles new values gracefully, and saves all encoding decisions.\n",
        "\n",
        "A basic encoder works for experiments, but production needs perfect consistency - our implementation delivers this while gracefully handling unexpected categories like \"detached bungalow\". Let's see how it works..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 400
        },
        "id": "1QmcssM4NdM8",
        "outputId": "9de5f717-e13f-4bc3-f29d-9fee40231e8b"
      },
      "outputs": [],
      "source": [
        "class OneHotFeatureEncoder:\n",
        "    \"\"\"Production-ready one-hot encoder with validation and persistence.\n",
        "\n",
        "    Handles:\n",
        "    - Feature ordering\n",
        "    - Missing/unseen categories\n",
        "    - Input validation\n",
        "    - Persistence of encoding decisions\n",
        "    - Clean interface for production use\n",
        "\n",
        "    Example:\n",
        "        encoder = OneHotFeatureEncoder(\n",
        "            numeric_features=['Area in sq ft', 'No. of Bedrooms'],\n",
        "            categorical_features=['House Type', 'City/County', 'Outcode']\n",
        "        )\n",
        "        X_train_onehot = encoder.fit_transform(train_df)\n",
        "        X_test_onehot = encoder.transform(test_df)\n",
        "\n",
        "        # Save for production\n",
        "        encoder.save('models/onehot_encoder.pkl')\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self,\n",
        "                 numeric_features: List[str],\n",
        "                 categorical_features: List[str],\n",
        "                 handle_unknown: str = 'ignore'):\n",
        "        \"\"\"Initialise encoder with feature specifications.\n",
        "\n",
        "        Args:\n",
        "            numeric_features: List of numeric column names\n",
        "            categorical_features: List of categorical column names\n",
        "            handle_unknown: Strategy for unknown categories ('ignore' or 'error')\n",
        "        \"\"\"\n",
        "        self.numeric_features = numeric_features\n",
        "        self.categorical_features = categorical_features\n",
        "        self.handle_unknown = handle_unknown\n",
        "\n",
        "        # Initialise encoder\n",
        "        self.encoder = OneHotEncoder(\n",
        "            sparse_output=False,\n",
        "            handle_unknown=handle_unknown\n",
        "        )\n",
        "\n",
        "        # State tracking\n",
        "        self.is_fitted = False\n",
        "        self.feature_order: List[str] = []\n",
        "        self.known_categories: Dict[str, Set[str]] = {}\n",
        "        self.output_feature_names: List[str] = []\n",
        "\n",
        "    def _validate_input_data(self, df: pd.DataFrame, for_fit: bool = False) -> None:\n",
        "        \"\"\"Validate input dataframe has required columns.\"\"\"\n",
        "        required_cols = set(self.numeric_features + self.categorical_features)\n",
        "        missing_cols = required_cols - set(df.columns)\n",
        "\n",
        "        if missing_cols:\n",
        "            raise ValueError(f\"Missing required columns: {missing_cols}\")\n",
        "\n",
        "        if for_fit and self.handle_unknown == 'error':\n",
        "            if self.is_fitted:\n",
        "                # Check for unknown categories\n",
        "                for col in self.categorical_features:\n",
        "                    unknown = set(df[col].unique()) - self.known_categories[col]\n",
        "                    if unknown:\n",
        "                        raise ValueError(\n",
        "                            f\"Unknown categories in column {col}: {unknown}\"\n",
        "                        )\n",
        "\n",
        "    def fit(self, df: pd.DataFrame) -> 'OneHotFeatureEncoder':\n",
        "        \"\"\"Fit encoder to training data.\"\"\"\n",
        "        self._validate_input_data(df, for_fit=True)\n",
        "\n",
        "        # Fit the one-hot encoder\n",
        "        self.encoder.fit(df[self.categorical_features])\n",
        "\n",
        "        # Store known categories\n",
        "        self.known_categories = {\n",
        "            feature: set(categories)\n",
        "            for feature, categories in zip(\n",
        "                self.categorical_features,\n",
        "                self.encoder.categories_\n",
        "            )\n",
        "        }\n",
        "\n",
        "        # Store feature names in order\n",
        "        self.output_feature_names = (\n",
        "            self.numeric_features +\n",
        "            self.encoder.get_feature_names_out(self.categorical_features).tolist()\n",
        "        )\n",
        "\n",
        "        self.is_fitted = True\n",
        "        return self\n",
        "\n",
        "    def transform(self, df: pd.DataFrame) -> pd.DataFrame:\n",
        "        \"\"\"Transform data using fitted encoder.\"\"\"\n",
        "        if not self.is_fitted:\n",
        "            raise ValueError(\"Encoder must be fitted before transform\")\n",
        "\n",
        "        self._validate_input_data(df)\n",
        "\n",
        "        # Encode categorical features\n",
        "        categorical_encoded = pd.DataFrame(\n",
        "            self.encoder.transform(df[self.categorical_features]),\n",
        "            columns=self.encoder.get_feature_names_out(self.categorical_features),\n",
        "            index=df.index\n",
        "        )\n",
        "\n",
        "        # Combine with numeric features\n",
        "        result = pd.concat([\n",
        "            df[self.numeric_features],\n",
        "            categorical_encoded\n",
        "        ], axis=1)\n",
        "\n",
        "        # Ensure consistent column order\n",
        "        return result[self.output_feature_names]\n",
        "\n",
        "    def fit_transform(self, df: pd.DataFrame) -> pd.DataFrame:\n",
        "        \"\"\"Fit encoder and transform data.\"\"\"\n",
        "        return self.fit(df).transform(df)\n",
        "\n",
        "    def save(self, path: str) -> None:\n",
        "        \"\"\"Save fitted encoder to disk.\"\"\"\n",
        "        if not self.is_fitted:\n",
        "            raise ValueError(\"Cannot save unfitted encoder\")\n",
        "\n",
        "        state = {\n",
        "            'numeric_features': self.numeric_features,\n",
        "            'categorical_features': self.categorical_features,\n",
        "            'handle_unknown': self.handle_unknown,\n",
        "            'encoder': self.encoder,\n",
        "            'is_fitted': self.is_fitted,\n",
        "            'feature_order': self.feature_order,\n",
        "            'known_categories': self.known_categories,\n",
        "            'output_feature_names': self.output_feature_names\n",
        "        }\n",
        "\n",
        "        with open(path, 'wb') as f:\n",
        "            pickle.dump(state, f)\n",
        "\n",
        "    @classmethod\n",
        "    def load(cls, path: str) -> 'OneHotFeatureEncoder':\n",
        "        \"\"\"Load saved encoder from disk.\"\"\"\n",
        "        with open(path, 'rb') as f:\n",
        "            state = pickle.load(f)\n",
        "\n",
        "        instance = cls(\n",
        "            numeric_features=state['numeric_features'],\n",
        "            categorical_features=state['categorical_features'],\n",
        "            handle_unknown=state['handle_unknown']\n",
        "        )\n",
        "\n",
        "        instance.encoder = state['encoder']\n",
        "        instance.is_fitted = state['is_fitted']\n",
        "        instance.feature_order = state['feature_order']\n",
        "        instance.known_categories = state['known_categories']\n",
        "        instance.output_feature_names = state['output_feature_names']\n",
        "\n",
        "        return instance\n",
        "\n",
        "# Test the implementation\n",
        "if __name__ == \"__main__\":\n",
        "    # Create encoder with housing features\n",
        "    housing_onehot = OneHotFeatureEncoder(\n",
        "        numeric_features=['No. of Bedrooms', 'Area in sq ft'],\n",
        "        categorical_features=['House Type', 'City/County', 'Outcode']\n",
        "    )\n",
        "\n",
        "    # Fit and transform training data\n",
        "    X_housing_onehot_train = housing_onehot.fit_transform(train_data)\n",
        "\n",
        "    # Transform test data\n",
        "    X_housing_onehot_test = housing_onehot.transform(test_data)\n",
        "\n",
        "    # Save encoder\n",
        "    # housing_onehot.save('../models/housing_onehot_encoder.pkl')\n",
        "\n",
        "    # Print summary\n",
        "    print(\"\\nOneHotFeatureEncoder Summary:\")\n",
        "    print(f\"Numeric features: {len(housing_onehot.numeric_features)}\")\n",
        "    print(f\"Categorical features: {len(housing_onehot.categorical_features)}\")\n",
        "    print(f\"Total output features: {len(housing_onehot.output_feature_names)}\")\n",
        "    print(\"\\nSample of encoded features:\")\n",
        "    display(X_housing_onehot_train.head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7_RlE7sUNdM9"
      },
      "source": [
        "<a name=\"hierarchical-target-encoding-methodology\"></a>\n",
        "### Hierarchical target encoding methodology\n",
        "\n",
        "When encoding location data for house price prediction, we face a fundamental challenge: locations with more data should have more influence on predictions, but we still want to extract signal from areas with sparse data. This is where hierarchical target encoding shines.\n",
        "\n",
        "Think of it like asking locals about house prices. In an area you know well, high data density, you trust the specific local knowledge. For an unfamiliar street, sparse data, you fall back to neighborhood knowledge.\n",
        "\n",
        "If the neighborhood is unfamiliar, you rely on broader district patterns.\n",
        "\n",
        "#### Understanding smoothing\n",
        "\n",
        "Why Do We Need Smoothing? Imagine two scenarios:\n",
        "1. Postcode A: 100 properties, average price £500,000\n",
        "2. Postcode B: 2 properties, average price £2,000,000\n",
        "\n",
        "Should we trust these averages equally? Intuitively, no - we're more confident in the average from Postcode A because it's based on more data. This is where smoothing comes in.\n",
        "\n",
        "#### The Smoothing Factor - m\n",
        "The parameter m controls how much data we need before trusting a local average:\n",
        "- If count = m: weight = 0.5 (50% local average, 50% broader average)\n",
        "- If count > m: weight ≈ 1 (mostly trust local average)\n",
        "- If count < m: weight ≈ 0 (mostly trust broader average)\n",
        "\n",
        "For example, with m = 10:\n",
        "- 5 properties: weight = 5/(5+10) = 0.33 (33% local, 67% broader)\n",
        "- 10 properties: weight = 10/(10+10) = 0.50 (50% local, 50% broader)\n",
        "- 50 properties: weight = 50/(50+10) = 0.83 (83% local, 17% broader)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qKw90tGGejHk"
      },
      "source": [
        "<a name=\"hierarchical-target-encoding-implementation\"></a>\n",
        "### Hierarchical target encoding implementation\n",
        "\n",
        "\n",
        "Our encoding implements this intuitive process mathematically through three levels:\n",
        "\n",
        "#### Outcode encoding\n",
        "\n",
        "At the broadest level, outcodes (like \"SW1\") provide reliable district-level price signals. Here we use simple mean substitution:\n",
        "  \n",
        "$$\n",
        "encoded\\_outcode = \\begin{cases}\n",
        "mean\\_outcode & \\text{if outcode exists} \\\\\n",
        "mean\\_global & \\text{if outcode missing}\n",
        "\\end{cases}\\\\\n",
        "$$\n",
        "\n",
        "#### Postcode encoding\n",
        "\n",
        "For full postcodes (like \"SW1A 1AA\"), we introduce dynamic smoothing. Areas with more data get more weight:\n",
        "\n",
        "$$\n",
        "\\text{weight} = \\frac{count\\_postcode}{{count\\_postcode} + m}\n",
        "$$\n",
        "\n",
        "$$\n",
        "encoded\\_postcode = \\text{weight} \\cdot mean\\_postcode + (1 - \\text{weight}) \\cdot encoded\\_outcode\n",
        "$$\n",
        "\n",
        "Where:\n",
        "- $m$ is the smoothing factor (e.g., 10)\n",
        "- Missing postcodes use outcode encoding\n",
        "\n",
        "\n",
        "#### Location encoding\n",
        "\n",
        "At what we would hope is the most granular level - specific named locations like \"De Beauvoir\", \"Limehouse\", \"Earls Court\" - we use both dynamic smoothing and a minimum frequency threshold:\n",
        "\n",
        "1. Initial smoothing (if count ≥ min_freq):\n",
        "$$\n",
        "\\text{weight} = \\frac{count\\_location}{count\\_location + m}\n",
        "$$\n",
        "\n",
        "$$\n",
        "encoded\\_location = \\text{weight} \\cdot mean\\_location + (1 - \\text{weight}) \\cdot encoded\\_postcode\n",
        "$$\n",
        "\n",
        "\n",
        "2. Final encoding:\n",
        "$$\n",
        "final\\_encoded\\_location = \\begin{cases}\n",
        "encoded\\_location & \\text{if count} \\geq min\\_freq \\\\\n",
        "encoded\\_postcode & \\text{otherwise}\n",
        "\\end{cases}\n",
        "$$\n",
        "\n",
        "This creates an intelligent fallback chain:\n",
        "```\n",
        "Location → Postcode → Outcode → Global Mean\n",
        "```\n",
        "\n",
        "The system automatically adjusts how much it trusts each geographic level based on available data. For well-represented locations, it relies heavily on specific local prices. For sparse areas, it smoothly transitions to using broader geographic patterns, never discarding information but weighting it according to reliability.\n",
        "\n",
        "All means and counts must be calculated using only training data and persisted for encoding new properties at prediction time. This preserves the statistical validity of our model evaluation while capturing the rich hierarchical structure of London's property market."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 135,
      "metadata": {
        "id": "wJJ1APrnNdM9"
      },
      "outputs": [],
      "source": [
        "class HierarchicalLocationEncoder:\n",
        "    \"\"\"Encodes location data using hierarchical target encoding with smoothing.\n",
        "\n",
        "    Creates price-based location encodings with fallbacks:\n",
        "    Location → Postcode → Outcode → Global Mean\n",
        "\n",
        "    Example:\n",
        "        encoder = HierarchicalLocationEncoder(smoothing_factor=10)\n",
        "        encoder.fit(train_data, target_col='log_price')\n",
        "        location_encoded = encoder.transform(new_data)\n",
        "\n",
        "    Features:\n",
        "    - Three-level hierarchy (Location → Postcode → Outcode)\n",
        "    - Dynamic smoothing based on data frequency\n",
        "    - Automatic fallback for sparse/missing data\n",
        "    - Persistence support for production use\n",
        "\n",
        "    Args:\n",
        "        smoothing_factor: Controls weight between local and broader means (default: 10)\n",
        "        min_freq: Minimum data points needed for location level encoding (default: 5)\n",
        "        verbose: Whether to print fitting statistics (default: True)\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, smoothing_factor: int = 10, min_freq: int = 5, verbose: bool = True):\n",
        "        self.smoothing_factor = smoothing_factor\n",
        "        self.min_freq = min_freq\n",
        "        self.verbose = verbose\n",
        "\n",
        "        # Storage for learned parameters\n",
        "        self.encoding_stats: Dict = {}\n",
        "        self.is_fitted: bool = False\n",
        "\n",
        "        if self.verbose:\n",
        "            print(f'Initialised encoder with smoothing_factor={smoothing_factor}, min_freq={min_freq}')\n",
        "\n",
        "    def fit(self, df: pd.DataFrame, target_col: str = 'log_price') -> None:\n",
        "        \"\"\"Learn encoding parameters from training data.\n",
        "\n",
        "        Args:\n",
        "            df: DataFrame with Location, Postal Code, and Outcode columns\n",
        "            target_col: Name of target variable column (default: 'log_price')\n",
        "        \"\"\"\n",
        "        required_cols = {'Location', 'Postal Code', 'Outcode', target_col}\n",
        "        missing_cols = required_cols - set(df.columns)\n",
        "        if missing_cols:\n",
        "            raise ValueError(f\"Missing required columns: {missing_cols}\")\n",
        "\n",
        "        if self.verbose:\n",
        "            print('\\nFitting encoder on training data...')\n",
        "\n",
        "        # Calculate global statistics\n",
        "        self.encoding_stats['global_mean'] = df[target_col].mean()\n",
        "\n",
        "        # Outcode level statistics\n",
        "        outcode_stats = df.groupby('Outcode')[target_col].agg(['count', 'mean', 'std'])\n",
        "        self.encoding_stats['outcode'] = outcode_stats\n",
        "\n",
        "        # Postcode level statistics with outcode fallback\n",
        "        postcode_stats = df.groupby('Postal Code').agg({\n",
        "            target_col: ['count', 'mean', 'std'],\n",
        "            'Outcode': 'first'\n",
        "        })\n",
        "        self.encoding_stats['postcode'] = postcode_stats\n",
        "\n",
        "        # Location level statistics with postcode fallback\n",
        "        location_data = df[df['Location'].notna()]\n",
        "        if len(location_data) > 0:\n",
        "            location_stats = location_data.groupby('Location').agg({\n",
        "                target_col: ['count', 'mean', 'std'],\n",
        "                'Postal Code': 'first'\n",
        "            })\n",
        "            self.encoding_stats['location'] = location_stats\n",
        "\n",
        "        self.is_fitted = True\n",
        "\n",
        "        if self.verbose:\n",
        "            self._print_fitting_summary()\n",
        "\n",
        "    def _print_fitting_summary(self) -> None:\n",
        "        \"\"\"Print summary statistics from fitting process.\"\"\"\n",
        "        print('\\nEncoding Statistics:')\n",
        "        print(f'Global mean: {self.encoding_stats[\"global_mean\"]:.3f}')\n",
        "\n",
        "        print(f'\\nOutcode level:')\n",
        "        print(f'- Number of outcodes: {len(self.encoding_stats[\"outcode\"])}')\n",
        "        print(f'- Average samples per outcode: {self.encoding_stats[\"outcode\"][\"count\"].mean():.1f}')\n",
        "\n",
        "        print(f'\\nPostcode level:')\n",
        "        print(f'- Number of postcodes: {len(self.encoding_stats[\"postcode\"])}')\n",
        "        print(f'- Average samples per postcode: {self.encoding_stats[\"postcode\"][(\"log_price\", \"count\")].mean():.1f}')\n",
        "\n",
        "        if 'location' in self.encoding_stats:\n",
        "            print(f'\\nLocation level:')\n",
        "            print(f'- Number of locations: {len(self.encoding_stats[\"location\"])}')\n",
        "            print(f'- Average samples per location: {self.encoding_stats[\"location\"][(\"log_price\", \"count\")].mean():.1f}')\n",
        "            min_freq_count = (self.encoding_stats[\"location\"][(\"log_price\", \"count\")] >= self.min_freq).sum()\n",
        "            print(f'- Locations with >= {self.min_freq} samples: {min_freq_count}')\n",
        "\n",
        "    def _calculate_smooth_weights(self, counts: pd.Series) -> pd.Series:\n",
        "        \"\"\"Calculate smoothing weights based on data frequency.\"\"\"\n",
        "        return counts / (counts + self.smoothing_factor)\n",
        "\n",
        "    def transform(self, df: pd.DataFrame) -> pd.DataFrame:\n",
        "        \"\"\"Apply hierarchical encoding to new data.\n",
        "\n",
        "        Returns DataFrame with columns:\n",
        "        - location_outcode_encoded: Broadest level encoding\n",
        "        - location_postcode_encoded: Intermediate level encoding\n",
        "        - location_encoded: Most granular level encoding\n",
        "        \"\"\"\n",
        "        if not self.is_fitted:\n",
        "            raise ValueError(\"Encoder must be fitted before transform\")\n",
        "\n",
        "        required_cols = {'Location', 'Postal Code', 'Outcode'}\n",
        "        missing_cols = required_cols - set(df.columns)\n",
        "        if missing_cols:\n",
        "            raise ValueError(f\"Missing required columns: {missing_cols}\")\n",
        "\n",
        "        if self.verbose:\n",
        "            print('\\nApplying hierarchical encoding...')\n",
        "\n",
        "        encoded = {}\n",
        "\n",
        "        # Start with broadest level: Outcode encoding\n",
        "        encoded['outcode'] = self._encode_outcodes(df)\n",
        "\n",
        "        # Intermediate level: Postcode encoding with outcode fallback\n",
        "        encoded['postcode'] = self._encode_postcodes(df, encoded['outcode'])\n",
        "\n",
        "        # Most granular level: Location encoding with postcode fallback\n",
        "        encoded['location'] = self._encode_locations(df, encoded['postcode'])\n",
        "\n",
        "        result = pd.DataFrame({\n",
        "            'location_outcode_encoded': encoded['outcode'],\n",
        "            'location_postcode_encoded': encoded['postcode'],\n",
        "            'location_encoded': encoded['location']\n",
        "        }, index=df.index)\n",
        "\n",
        "        if self.verbose:\n",
        "            self._print_encoding_summary(result)\n",
        "\n",
        "        return result\n",
        "\n",
        "    def _encode_outcodes(self, df: pd.DataFrame) -> pd.Series:\n",
        "        \"\"\"Encode outcodes using mean target values.\"\"\"\n",
        "        return (df['Outcode']\n",
        "                .map(self.encoding_stats['outcode']['mean'])\n",
        "                .fillna(self.encoding_stats['global_mean']))\n",
        "\n",
        "    def _encode_postcodes(self, df: pd.DataFrame, outcode_encoded: pd.Series) -> pd.Series:\n",
        "        \"\"\"Encode postcodes with smoothed means and outcode fallback.\"\"\"\n",
        "        stats = self.encoding_stats['postcode']\n",
        "        counts = df['Postal Code'].map(stats[('log_price', 'count')]).fillna(0)\n",
        "        means = df['Postal Code'].map(stats[('log_price', 'mean')])\n",
        "\n",
        "        weights = self._calculate_smooth_weights(counts)\n",
        "        encoded = (weights * means + (1 - weights) * outcode_encoded)\n",
        "\n",
        "        return encoded.fillna(outcode_encoded)\n",
        "\n",
        "    def _encode_locations(self, df: pd.DataFrame, postcode_encoded: pd.Series) -> pd.Series:\n",
        "        \"\"\"Encode locations with postcode fallback.\"\"\"\n",
        "        if 'location' not in self.encoding_stats:\n",
        "            return postcode_encoded\n",
        "\n",
        "        stats = self.encoding_stats['location']\n",
        "        counts = df['Location'].map(stats[('log_price', 'count')]).fillna(0)\n",
        "        means = df['Location'].map(stats[('log_price', 'mean')])\n",
        "\n",
        "        weights = self._calculate_smooth_weights(counts)\n",
        "        encoded = (weights * means + (1 - weights) * postcode_encoded)\n",
        "\n",
        "        low_freq_mask = (counts < self.min_freq)\n",
        "        encoded[low_freq_mask] = postcode_encoded[low_freq_mask]\n",
        "\n",
        "        return encoded.fillna(postcode_encoded)\n",
        "\n",
        "    def _print_encoding_summary(self, result: pd.DataFrame) -> None:\n",
        "        \"\"\"Print summary of encoded values.\"\"\"\n",
        "        print(\"\\nEncoding Results:\")\n",
        "        for col in result.columns:\n",
        "            print(f\"\\n{col}:\")\n",
        "            print(f\"Mean: {result[col].mean():.3f}\")\n",
        "            print(f\"Std: {result[col].std():.3f}\")\n",
        "            print(f\"Range: {result[col].min():.3f} to {result[col].max():.3f}\")\n",
        "\n",
        "    def fit_transform(self, df: pd.DataFrame, target_col: str = 'log_price') -> pd.DataFrame:\n",
        "        \"\"\"Fit encoder and transform data in one step.\"\"\"\n",
        "        self.fit(df, target_col)\n",
        "        return self.transform(df)\n",
        "\n",
        "    def save(self, path: str) -> None:\n",
        "        \"\"\"Save fitted encoder to disk.\"\"\"\n",
        "        if not self.is_fitted:\n",
        "            raise ValueError(\"Cannot save unfitted encoder\")\n",
        "\n",
        "        state = {\n",
        "            'smoothing_factor': self.smoothing_factor,\n",
        "            'min_freq': self.min_freq,\n",
        "            'encoding_stats': self.encoding_stats,\n",
        "            'is_fitted': self.is_fitted\n",
        "        }\n",
        "\n",
        "        with open(path, 'wb') as f:\n",
        "            pickle.dump(state, f)\n",
        "\n",
        "        if self.verbose:\n",
        "            print(f'\\nEncoder state saved to {path}')\n",
        "\n",
        "    @classmethod\n",
        "    def load(cls, path: str) -> 'HierarchicalLocationEncoder':\n",
        "        \"\"\"Load saved encoder from disk.\"\"\"\n",
        "        with open(path, 'rb') as f:\n",
        "            state = pickle.load(f)\n",
        "\n",
        "        encoder = cls(\n",
        "            smoothing_factor=state['smoothing_factor'],\n",
        "            min_freq=state['min_freq']\n",
        "        )\n",
        "        encoder.encoding_stats = state['encoding_stats']\n",
        "        encoder.is_fitted = state['is_fitted']\n",
        "\n",
        "        if encoder.verbose:\n",
        "            print(f'\\nLoaded encoder state from {path}')\n",
        "            encoder._print_fitting_summary()\n",
        "\n",
        "        return encoder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "0CzNpc8bNdM9",
        "outputId": "01a15bab-f670-4289-f2d3-9a82d96bbe35"
      },
      "outputs": [],
      "source": [
        "# Initialise and test location encoder\n",
        "location_encoder = HierarchicalLocationEncoder(smoothing_factor=10, min_freq=5)\n",
        "\n",
        "# Fit and transform data\n",
        "location_train_encoded = location_encoder.fit_transform(train_data, target_col='log_price')\n",
        "location_test_encoded = location_encoder.transform(test_data)\n",
        "\n",
        "# Quick visualisation of results\n",
        "fig, axes = plt.subplots(1, 3, figsize=(15, 5))\n",
        "\n",
        "# Compare distributions of each encoding level\n",
        "for i, col in enumerate(['location_outcode_encoded', 'location_postcode_encoded', 'location_encoded']):\n",
        "    sns.kdeplot(location_train_encoded[col], ax=axes[i], label='Train')\n",
        "    sns.kdeplot(location_test_encoded[col], ax=axes[i], label='Test')\n",
        "    axes[i].set_title(col.replace('location_', '').replace('_encoded', ''))\n",
        "    axes[i].legend()\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Display sample results\n",
        "print(\"\\nSample of encoded location features:\")\n",
        "display(location_train_encoded.head())\n",
        "\n",
        "# Save encoder for later use\n",
        "# location_encoder.save('../models/location_encoder.pkl')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wurC8G78hTKK"
      },
      "source": [
        "<a name=\"mean-outcode-price-per-square-foot-encoder-implementation\"></a>\n",
        "### Mean outcode price per square foot encoder implementation\n",
        "\n",
        "This encoder creates area-level price per square foot features by calculating mean rates for each London outcode (e.g. \"SW1\", \"E14\").\n",
        "\n",
        "Here we tackle a subtle challenge: we want the predictive power of local market rates without leaking individual property prices. The solution is elegant - for each outcode (like \"SW1\"), calculate the mean price per square foot using only training data, falling back to a global average for areas we haven't seen.\n",
        "\n",
        "This preserves the crucial statistical separation between training and test while capturing valuable market-level signals. A property in SW1 gets encoded with SW1's historical average rate, never its own price or future sales.\n",
        "The result is a powerful but carefully controlled way to let our models learn from local market rates.\n",
        "\n",
        "While using price information in features can be dangerous for several reasons, this implementation maintains statistical validity by strictly using only training data means, never individual property prices. We'll explore the implications of this later on in the lesson, but first let's have a look at our implementation below.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 600
        },
        "id": "6nmHsjrfNdM9",
        "outputId": "55c88fa0-2959-4d50-de7c-78b1df3591c6"
      },
      "outputs": [],
      "source": [
        "class MeanOutcodePricePerSquareFootEncoder:\n",
        "    \"\"\"Encodes outcodes with their mean price per square foot.\n",
        "\n",
        "    Calculates mean price/sqft at outcode level using only training data,\n",
        "    using global mean as fallback for unseen outcodes.\n",
        "\n",
        "    Example:\n",
        "        encoder = MeanOutcodePricePerSquareFootEncoder()\n",
        "        encoder.fit(train_data)\n",
        "        train_price_encoded = encoder.transform(train_data)\n",
        "        test_price_encoded = encoder.transform(test_data)\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, verbose: bool = True):\n",
        "        self.outcode_means: Optional[pd.Series] = None\n",
        "        self.global_mean: Optional[float] = None\n",
        "        self.is_fitted: bool = False\n",
        "        self.verbose = verbose\n",
        "\n",
        "    def fit(self, df: pd.DataFrame) -> 'MeanOutcodePricePerSquareFootEncoder':\n",
        "        \"\"\"Calculate outcode means using only training data.\n",
        "\n",
        "        Args:\n",
        "            df: DataFrame containing 'Price', 'Area in sq ft', and 'Outcode' columns\n",
        "        \"\"\"\n",
        "        required_cols = {'Price', 'Area in sq ft', 'Outcode'}\n",
        "        missing_cols = required_cols - set(df.columns)\n",
        "        if missing_cols:\n",
        "            raise ValueError(f\"Missing required columns: {missing_cols}\")\n",
        "\n",
        "        # Validate numeric columns\n",
        "        if not (df['Price'] > 0).all():\n",
        "            raise ValueError(\"All prices must be positive\")\n",
        "        if not (df['Area in sq ft'] > 0).all():\n",
        "            raise ValueError(\"All areas must be positive\")\n",
        "\n",
        "        # Calculate price per square foot\n",
        "        price_per_sqft = df['Price'] / df['Area in sq ft']\n",
        "\n",
        "        # Calculate means and store\n",
        "        self.outcode_means = price_per_sqft.groupby(df['Outcode']).mean()\n",
        "        self.global_mean = price_per_sqft.mean()\n",
        "\n",
        "        if self.verbose:\n",
        "            print(f\"\\nFitted price encoder:\")\n",
        "            print(f\"- Outcodes encoded: {len(self.outcode_means)}\")\n",
        "            print(f\"- Global mean price/sqft: £{self.global_mean:,.2f}\")\n",
        "            print(f\"- Range: £{self.outcode_means.min():,.2f} - £{self.outcode_means.max():,.2f}\")\n",
        "\n",
        "        self.is_fitted = True\n",
        "        return self\n",
        "\n",
        "    def transform(self, df: pd.DataFrame) -> pd.Series:\n",
        "        \"\"\"Transform data using fitted means with fallback.\n",
        "\n",
        "        Args:\n",
        "            df: DataFrame containing 'Outcode' column\n",
        "\n",
        "        Returns:\n",
        "            Series containing 'price_per_sqft_encoded' for each row\n",
        "        \"\"\"\n",
        "        if not self.is_fitted:\n",
        "            raise ValueError(\"Encoder must be fitted before transform\")\n",
        "\n",
        "        if 'Outcode' not in df.columns:\n",
        "            raise ValueError(\"DataFrame must contain 'Outcode' column\")\n",
        "\n",
        "        encoded = df['Outcode'].map(self.outcode_means).fillna(self.global_mean)\n",
        "        encoded.name = 'price_per_sqft_encoded'\n",
        "        return encoded\n",
        "\n",
        "    def fit_transform(self, df: pd.DataFrame) -> pd.Series:\n",
        "        \"\"\"Fit encoder and transform data in one step.\"\"\"\n",
        "        return self.fit(df).transform(df)\n",
        "\n",
        "    def save(self, path: str) -> None:\n",
        "        \"\"\"Save fitted encoder parameters.\"\"\"\n",
        "        if not self.is_fitted:\n",
        "            raise ValueError(\"Cannot save unfitted encoder\")\n",
        "\n",
        "        with open(path, 'wb') as f:\n",
        "            pickle.dump({\n",
        "                'outcode_means': self.outcode_means,\n",
        "                'global_mean': self.global_mean,\n",
        "                'is_fitted': True\n",
        "            }, f)\n",
        "\n",
        "        if self.verbose:\n",
        "            print(f\"\\nSaved price encoder to {path}\")\n",
        "\n",
        "    @classmethod\n",
        "    def load(cls, path: str) -> 'MeanOutcodePricePerSquareFootEncoder':\n",
        "        \"\"\"Load saved encoder parameters.\"\"\"\n",
        "        with open(path, 'rb') as f:\n",
        "            params = pickle.load(f)\n",
        "\n",
        "        encoder = cls(verbose=True)\n",
        "        encoder.outcode_means = params['outcode_means']\n",
        "        encoder.global_mean = params['global_mean']\n",
        "        encoder.is_fitted = params['is_fitted']\n",
        "        return encoder\n",
        "\n",
        "# Quick test\n",
        "print(\"Testing MeanOutcodePricePerSquareFootEncoder...\")\n",
        "price_encoder = MeanOutcodePricePerSquareFootEncoder()\n",
        "train_price_encoded = price_encoder.fit_transform(train_data)\n",
        "test_price_encoded = price_encoder.transform(test_data)\n",
        "\n",
        "# Save for production\n",
        "# price_encoder.save('../models/price_encoder.pkl')\n",
        "\n",
        "# Compare distributions\n",
        "fig, ax = plt.subplots(figsize=(10, 5))\n",
        "sns.kdeplot(train_price_encoded, label='Train')\n",
        "sns.kdeplot(test_price_encoded, label='Test')\n",
        "ax.set_title('Mean Outcode Price per sqft Distribution')\n",
        "ax.set_xlabel('£ per square foot')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SdfsnUI5NdM9"
      },
      "source": [
        "Over the past sections, we've built three encoders for handling different aspects of our data:\n",
        "\n",
        "1. One-hot encoding for categorical features (house type, city/county)\n",
        "2. Hierarchical location encoding for complex geography\n",
        "3. Mean outcode price per square foot for area-normalised pricing\n",
        "\n",
        "Rather than review these in isolation, we'll examine their outputs as we build increasingly sophisticated models.\n",
        "\n",
        "This lets us understand our features in context, while keeping focus on practical model development.\n",
        "\n",
        "\n",
        "<a name=\"training-our-first-decision-tree\"></a>\n",
        "## Training our first decision tree\n",
        "\n",
        "Let's start with a straightforward model using our simple numeric and one-hot encoded features:\n",
        "\n",
        "- Number of bedrooms (numeric)\n",
        "- Area in square feet (numeric)\n",
        "- House type (one-hot encoded)\n",
        "- City/County (one-hot encoded)\n",
        "- Outcode (one-hot encoded)\n",
        "\n",
        "This combination gives us interpretable features without overwhelming complexity. We'll:\n",
        "1. Create this feature set\n",
        "2. Train a basic decision tree\n",
        "3. Visualise its decision structure\n",
        "4. Examine how it makes predictions\n",
        "5. Tune its parameters for better performance\n",
        "\n",
        "We'll look at both feature importance and the actual decision paths our tree uses to value properties.\n",
        "\n",
        "Later, we can improve this foundation with:\n",
        "- Parameter tuning\n",
        "- Cross-validation\n",
        "- Model persistence\n",
        "- More advanced models\n",
        "\n",
        "But first, let's understand how a basic tree approaches house price prediction!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "mMxL0D_KNdM-",
        "outputId": "3b443259-c9d5-4ef4-9976-b8b22652c0fb"
      },
      "outputs": [],
      "source": [
        "def train_onehot_decision_tree(train_data: pd.DataFrame,\n",
        "                              test_data: pd.DataFrame,\n",
        "                              random_state: int = 42) -> Tuple[DecisionTreeRegressor, Dict, pd.DataFrame, pd.DataFrame]:\n",
        "    \"\"\"Train decision tree on one-hot encoded features.\n",
        "\n",
        "    Args:\n",
        "        train_data: Training DataFrame\n",
        "        test_data: Test DataFrame\n",
        "        random_state: Random seed for reproducibility\n",
        "\n",
        "    Returns:\n",
        "        Tuple of (fitted model, performance metrics, feature importance, encoded training features)\n",
        "    \"\"\"\n",
        "    # Create feature encoder\n",
        "    encoder = OneHotFeatureEncoder(\n",
        "        numeric_features=['No. of Bedrooms', 'Area in sq ft'],\n",
        "        categorical_features=['House Type', 'City/County', 'Outcode'],\n",
        "        handle_unknown='ignore'\n",
        "    )\n",
        "\n",
        "    # Create features\n",
        "    X_train = encoder.fit_transform(train_data)\n",
        "    X_test = encoder.transform(test_data)\n",
        "\n",
        "    y_train = train_data['log_price']\n",
        "    y_test = test_data['log_price']\n",
        "\n",
        "    # Train model\n",
        "    tree = DecisionTreeRegressor(random_state=random_state)\n",
        "    tree.fit(X_train, y_train)\n",
        "\n",
        "    # Make predictions\n",
        "    train_pred = tree.predict(X_train)\n",
        "    test_pred = tree.predict(X_test)\n",
        "\n",
        "    # Calculate metrics\n",
        "    def calculate_metrics(y_true, y_pred, prefix=''):\n",
        "        \"\"\"Calculate regression metrics in both log and raw price space\"\"\"\n",
        "        # Log space metrics\n",
        "        r2 = r2_score(y_true, y_pred)\n",
        "\n",
        "        # Raw price space metrics\n",
        "        true_price = np.exp(y_true)\n",
        "        pred_price = np.exp(y_pred)\n",
        "        mae = mean_absolute_error(true_price, pred_price)\n",
        "        rmse = np.sqrt(mean_squared_error(true_price, pred_price))\n",
        "\n",
        "        return {\n",
        "            f'{prefix}mae': mae,\n",
        "            f'{prefix}rmse': rmse,\n",
        "            f'{prefix}r2': r2\n",
        "        }\n",
        "\n",
        "    # Collect performance metrics\n",
        "    metrics = {}\n",
        "    metrics.update(calculate_metrics(y_train, train_pred, 'train_'))\n",
        "    metrics.update(calculate_metrics(y_test, test_pred, 'test_'))\n",
        "\n",
        "    # Calculate feature importance\n",
        "    importance = pd.DataFrame({\n",
        "        'feature': X_train.columns,\n",
        "        'importance': tree.feature_importances_\n",
        "    }).sort_values('importance', ascending=False)\n",
        "\n",
        "    # Save encoder for production\n",
        "    # encoder.save('../models/onehot_encoder.pkl')\n",
        "\n",
        "    print(\"\\nModel Performance:\")\n",
        "    print(f\"Training R²: {metrics['train_r2']:.3f}\")\n",
        "    print(f\"Test R²: {metrics['test_r2']:.3f}\")\n",
        "    print(f\"\\nTraining MAE: £{metrics['train_mae']:,.0f}\")\n",
        "    print(f\"Test MAE: £{metrics['test_mae']:,.0f}\")\n",
        "\n",
        "    # Visualise tree structure\n",
        "    plt.figure(figsize=(20,10))\n",
        "    plot_tree(tree,\n",
        "             feature_names=X_train.columns,\n",
        "             max_depth=3,\n",
        "             filled=True,\n",
        "             rounded=True,\n",
        "             fontsize=10,\n",
        "             proportion=True,\n",
        "             precision=2)\n",
        "    plt.title(\"Decision Tree Structure (First Three Levels)\", fontsize=16, pad=20)\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n",
        "\n",
        "    # Print tree statistics\n",
        "    print(f\"\\nTree Structure:\")\n",
        "    print(f\"Total depth: {tree.get_depth()}\")\n",
        "    print(f\"Number of leaves: {tree.get_n_leaves()}\")\n",
        "    print(f\"Average samples per leaf: {len(X_train)/tree.get_n_leaves():.1f}\")\n",
        "\n",
        "    print(\"\\nTop 5 Most Important Features:\")\n",
        "    display(importance.head())\n",
        "\n",
        "    return tree, metrics, importance, X_train\n",
        "\n",
        "# Train model and get results\n",
        "onehot_tree, onehot_metrics, onehot_importance, onehot_features = train_onehot_decision_tree(\n",
        "    train_data,\n",
        "    test_data\n",
        ")\n",
        "\n",
        "# Visualise feature importance\n",
        "plt.figure(figsize=(9, 5))\n",
        "sns.barplot(data=onehot_importance.head(10), x='importance', y='feature')\n",
        "plt.title('Top 10 Most Important Features (One-Hot Encoded Model)')\n",
        "plt.xlabel('Feature Importance')\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1qW07JClNdM-"
      },
      "source": [
        "<a name=\"understanding-overfitting\"></a>\n",
        "## Understanding overfitting\n",
        "\n",
        "Our initial decision tree produced some revealing results:\n",
        "```\n",
        "Training R²: 1.000    Test R²: 0.786\n",
        "Training MAE: £1,233  Test MAE: £670,882\n",
        "```\n",
        "\n",
        "This dramatic difference between training and test performance indicates a classic machine learning problem: overfitting.\n",
        "\n",
        "### Understanding our tree's behaviour\n",
        "Let's break down what we've learned:\n",
        "\n",
        "1. **Tree structure**\n",
        "   - Extremely deep tree of 62 levels\n",
        "   - A huge number of leaf nodes - at 2664 leaves its more of a bush than a tree!\n",
        "   - One average 1 sample per leaf\n",
        "   - Early splits focus on area and location\n",
        "\n",
        "2. **Feature importance**\n",
        "   - Area dominates (75.1% importance)\n",
        "   - Location features contribute smaller amounts\n",
        "   - Bedrooms have surprisingly low impact (2.5%)\n",
        "\n",
        "3. **Price predictions**\n",
        "   - Perfect on training data (R² = 1.0)\n",
        "   - Large errors on test data (£670K average)\n",
        "   - Almost no training error (£1,233 MAE)\n",
        "\n",
        "### The problem: Unrestricted growth\n",
        "By default, our tree keeps splitting until it can perfectly predict each house price. Think of it like a real estate agent who has memorised every house price in their database instead of learning general market patterns.\n",
        "\n",
        "This happens because the tree:\n",
        "1. Splits on area into broad price bands\n",
        "2. Refines with location features\n",
        "3. Makes increasingly specific splits\n",
        "4. Eventually isolates individual properties\n",
        "\n",
        "### The solution: Controlled growth\n",
        "Just as estate agents develop pricing rules based on multiple similar properties, we can force our tree to make more general predictions by setting:\n",
        "\n",
        "```python\n",
        "max_depth = 6            # Limit complexity of rules\n",
        "min_samples_leaf = 30    # Require 30+ houses per price prediction\n",
        "```\n",
        "\n",
        "Let's try these constraints and see how they affect our model's performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "YS0KldWsNdM-",
        "outputId": "8dee42cb-6172-4a6c-bf54-67fea4305cf9"
      },
      "outputs": [],
      "source": [
        "# Train decision tree with manual constraints\n",
        "def train_controlled_onehot_tree(train_data: pd.DataFrame,\n",
        "                                test_data: pd.DataFrame,\n",
        "                                max_depth: int = 6,\n",
        "                                min_samples_leaf: int = 30,\n",
        "                                random_state: int = 42) -> Tuple[DecisionTreeRegressor, Dict, pd.DataFrame]:\n",
        "    \"\"\"Train decision tree with controlled growth parameters.\n",
        "\n",
        "    Args:\n",
        "        train_data: Training DataFrame\n",
        "        test_data: Test DataFrame\n",
        "        max_depth: Maximum tree depth\n",
        "        min_samples_leaf: Minimum samples per leaf\n",
        "        random_state: Random seed\n",
        "\n",
        "    Returns:\n",
        "        Tuple of (model, metrics, feature importance)\n",
        "    \"\"\"\n",
        "    # Create feature encoder\n",
        "    encoder = OneHotFeatureEncoder(\n",
        "        numeric_features=['No. of Bedrooms', 'Area in sq ft'],\n",
        "        categorical_features=['House Type', 'City/County', 'Outcode'],\n",
        "        handle_unknown='ignore'\n",
        "    )\n",
        "\n",
        "    # Create features\n",
        "    X_train = encoder.fit_transform(train_data)\n",
        "    X_test = encoder.transform(test_data)\n",
        "\n",
        "    y_train = train_data['log_price']\n",
        "    y_test = test_data['log_price']\n",
        "\n",
        "    # Train controlled tree\n",
        "    controlled_tree = DecisionTreeRegressor(\n",
        "        max_depth=max_depth,\n",
        "        min_samples_leaf=min_samples_leaf,\n",
        "        random_state=random_state\n",
        "    )\n",
        "    controlled_tree.fit(X_train, y_train)\n",
        "\n",
        "    # Make predictions\n",
        "    train_pred = controlled_tree.predict(X_train)\n",
        "    test_pred = controlled_tree.predict(X_test)\n",
        "\n",
        "    # Calculate metrics\n",
        "    def calculate_metrics(y_true, y_pred, prefix=''):\n",
        "        \"\"\"Calculate regression metrics in both log and raw price space\"\"\"\n",
        "        r2 = r2_score(y_true, y_pred)\n",
        "        true_price = np.exp(y_true)\n",
        "        pred_price = np.exp(y_pred)\n",
        "        mae = mean_absolute_error(true_price, pred_price)\n",
        "        rmse = np.sqrt(mean_squared_error(true_price, pred_price))\n",
        "        return {f'{prefix}r2': r2, f'{prefix}mae': mae, f'{prefix}rmse': rmse}\n",
        "\n",
        "    # Collect metrics\n",
        "    metrics = {}\n",
        "    metrics.update(calculate_metrics(y_train, train_pred, 'train_'))\n",
        "    metrics.update(calculate_metrics(y_test, test_pred, 'test_'))\n",
        "\n",
        "    # Calculate feature importance\n",
        "    importance = pd.DataFrame({\n",
        "        'feature': X_train.columns,\n",
        "        'importance': controlled_tree.feature_importances_\n",
        "    }).sort_values('importance', ascending=False)\n",
        "\n",
        "    # Print performance summary\n",
        "    print(\"\\nControlled Tree Performance:\")\n",
        "    print(f\"Training R²: {metrics['train_r2']:.3f}\")\n",
        "    print(f\"Test R²: {metrics['test_r2']:.3f}\")\n",
        "    print(f\"Training MAE: £{metrics['train_mae']:,.0f}\")\n",
        "    print(f\"Test MAE: £{metrics['test_mae']:,.0f}\")\n",
        "\n",
        "    print(f\"\\nTree Structure:\")\n",
        "    print(f\"Depth: {controlled_tree.get_depth()}\")\n",
        "    print(f\"Number of leaves: {controlled_tree.get_n_leaves()}\")\n",
        "    print(f\"Average samples per leaf: {len(X_train)/controlled_tree.get_n_leaves():.1f}\")\n",
        "\n",
        "    print(\"\\nTop 5 Most Important Features:\")\n",
        "    display(importance.head())\n",
        "\n",
        "    # Plot feature importance\n",
        "    plt.figure(figsize=(10, 6))\n",
        "    sns.barplot(data=importance.head(8), x='importance', y='feature')\n",
        "    plt.title('Feature Importance (Controlled Tree)')\n",
        "    plt.xlabel('Importance')\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n",
        "\n",
        "    return controlled_tree, metrics, importance\n",
        "\n",
        "# Train controlled tree\n",
        "controlled_tree, controlled_metrics, controlled_importance = train_controlled_onehot_tree(\n",
        "    train_data,\n",
        "    test_data\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-w1MebNLNdM-"
      },
      "source": [
        "<a name=\"analysing-model-performance-and-hyperparameter-impact\"></a>\n",
        "## Analysing model performance and hyperparameter impact\n",
        "\n",
        "Let's compare our two approaches to decision tree modeling:\n",
        "\n",
        "### Unrestricted tree (no parameter limits)\n",
        "```\n",
        "Training R²: 1.000    Test R²: 0.786\n",
        "Training MAE: £1,233  Test MAE: £670,882\n",
        "```\n",
        "\n",
        "### Controlled tree (max_depth=6, min_samples_leaf=30)\n",
        "```\n",
        "Training R²: 0.777    Test R²: 0.787\n",
        "Training MAE: £594,884  Test MAE: £636,490\n",
        "Average samples per leaf: 66.3\n",
        "```\n",
        "\n",
        "### What these numbers tell ss\n",
        "\n",
        "1. **The overfitting problem**\n",
        "   - Our unrestricted tree achieved perfect training accuracy (R²=1.0)\n",
        "   - But performed worse on new data (£670K vs £636K error)\n",
        "   - It's like memorising past house prices instead of learning market patterns\n",
        "\n",
        "2. **Benefits of controlled growth**\n",
        "   - Limiting depth to 6 levels created 42 price prediction rules\n",
        "   - Each rule uses at least 30 houses (min_samples_leaf)\n",
        "   - Average of 66 houses per rule suggests stable predictions\n",
        "\n",
        "3. **The trade-off**\n",
        "   - Training accuracy dropped (R²: 1.0 → 0.777)\n",
        "   - But test performance improved (MAE: £670K → £636K)\n",
        "   - More realistic predictions based on broader patterns\n",
        "\n",
        "### Real estate context\n",
        "\n",
        "Think of it this way:\n",
        "- Unrestricted tree: Like a new agent who memorises every sale price but can't generalise\n",
        "- Controlled tree: Like an experienced agent who uses reliable rules based on multiple similar properties\n",
        "\n",
        "### The hyperparameter question\n",
        "\n",
        "While our controlled tree performs better, we're left with important questions:\n",
        "1. Why exactly 6 levels of depth?\n",
        "2. Is 30 houses per rule optimal?\n",
        "3. How can we be sure these choices work across different areas?\n",
        "\n",
        "To answer these questions systematically, we need two key tools:\n",
        "1. Cross-validation for reliable performance testing\n",
        "2. Grid search for finding optimal parameters\n",
        "\n",
        "Let's explore these tools and see how they can help us build even better models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6zse_DiVNdM-"
      },
      "source": [
        "<a name=\"cross-validation-getting-reliable-performance-estimates\"></a>\n",
        "## Cross-validation: Getting reliable performance estimates\n",
        "\n",
        "In lesson 1B we simply split our data into train, validation and test sets. Now we'll try something more sophisticated - we'll repeatedly shuffle our training data into different train and validation splits while keeping our test set separate, a technique called cross-validation.\n",
        "\n",
        "Let's understand why through a real estate example.\n",
        "\n",
        "### The single split problem\n",
        "\n",
        "Imagine our validation split of the training data put:\n",
        "- Most Chelsea properties in the training portion\n",
        "- Most Hackney properties in the validation portion\n",
        "\n",
        "Our model would:\n",
        "1. Learn pricing patterns from expensive areas (Chelsea)\n",
        "2. Validate on more affordable areas (Hackney)\n",
        "3. Show poor performance due to the mismatch\n",
        "\n",
        "But with a different random split, we might get opposite results! This makes it hard to trust our performance estimates during development.\n",
        "\n",
        "### How cross-validation Works\n",
        "\n",
        "Instead of one validation split, cross-validation divides our training data into multiple portions:\n",
        "\n",
        "```\n",
        "Training Data (2,784 houses)\n",
        "↓\n",
        "Split into 5 groups (4 × 557 houses + 1 × 556 houses)\n",
        "\n",
        "Round 1: [Validation][Train][Train][Train][Train]\n",
        "         557         557   557   557   556 houses\n",
        "\n",
        "Round 2: [Train][Validation][Train][Train][Train]\n",
        "Round 3: [Train][Train][Validation][Train][Train]\n",
        "Round 4: [Train][Train][Train][Validation][Train]\n",
        "Round 5: [Train][Train][Train][Train][Validation]\n",
        "```\n",
        "\n",
        "For each round:\n",
        "1. Train on ~2,227 houses (4 groups)\n",
        "2. Validate on ~557 houses (1 group)\n",
        "3. Record the performance\n",
        "\n",
        "Our test set of 696 houses remains completely untouched during this process.\n",
        "\n",
        "### Benefits for house price prediction\n",
        "\n",
        "This approach:\n",
        "- Tests model performance across different neighborhoods\n",
        "- Provides 5 different error estimates\n",
        "- Shows how stable predictions are\n",
        "- Helps identify if model works better in some areas than others\n",
        "\n",
        "Next, we'll use cross-validation in a systematic search for the best model parameters."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zYqxdpUFNdM_"
      },
      "source": [
        "\n",
        "<a name=\"grid-search-implementation\"></a>\n",
        "## Grid search implementation\n",
        "\n",
        "Now that we have a reliable way to test model performance using cross-validation, we can systematically search for the best parameters.\n",
        "\n",
        "### Parameters to tune\n",
        "\n",
        "For our decision tree, we need to find optimal values for:\n",
        "\n",
        "1. `max_depth`: Maximum number of decisions allowed\n",
        "   - Too low: Model might miss important patterns\n",
        "   - Too high: Model might memorise noise\n",
        "   - We'll try: [4, 6, 8, 10, 12]\n",
        "\n",
        "2. `min_samples_leaf`: Minimum houses needed for a price prediction\n",
        "   - Too low: Unstable predictions from few examples\n",
        "   - Too high: Might miss legitimate local patterns\n",
        "   - We'll try: [10, 20, 30, 50, 70]\n",
        "\n",
        "3. `min_samples_split`: Minimum houses needed to consider a new decision\n",
        "   - Too low: Creates unnecessary splits\n",
        "   - Too high: Might stop too early\n",
        "   - We'll try: [20, 40, 60, 75, 100]\n",
        "\n",
        "### How grid search works\n",
        "\n",
        "For each combination of parameters:\n",
        "```\n",
        "1. Create a decision tree with those parameters\n",
        "2. Run 5-fold cross-validation\n",
        "3. Calculate average performance\n",
        "4. Store results\n",
        "\n",
        "Example combination:\n",
        "max_depth=6, min_samples_leaf=30, min_samples_split=60\n",
        "→ CV Fold 1: MAE = £550,000\n",
        "→ CV Fold 2: MAE = £620,000\n",
        "→ CV Fold 3: MAE = £580,000\n",
        "→ CV Fold 4: MAE = £590,000\n",
        "→ CV Fold 5: MAE = £560,000\n",
        "→ Average MAE: £580,000 ±£26,000\n",
        "```\n",
        "\n",
        "### Total combinations\n",
        "\n",
        "Our search will try:\n",
        "- 5 values for max_depth\n",
        "- 5 values for min_samples_leaf\n",
        "- 5 values for min_samples_split\n",
        "- Total: 5 × 5 × 5 = 125 combinations\n",
        "- Each tested with 5-fold CV\n",
        "- Total models trained: 625\n",
        "\n",
        "### Real estate context\n",
        "\n",
        "It's like systematically testing different valuation rules:\n",
        "- How many comparable properties needed? (min_samples_leaf)\n",
        "- How detailed should price adjustments be? (max_depth)\n",
        "- When to stop making finer distinctions? (min_samples_split)\n",
        "\n",
        "Let's implement this search and find the best combination for London house prices."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 739
        },
        "id": "6L8HHGWnNdM_",
        "outputId": "de75c433-d6e3-44a9-8cb6-f06b3cd44dcd"
      },
      "outputs": [],
      "source": [
        "def grid_search_tree(train_data: pd.DataFrame,\n",
        "                    test_data: pd.DataFrame,\n",
        "                    random_state: int = 42) -> Tuple[DecisionTreeRegressor, pd.DataFrame, Dict]:\n",
        "    \"\"\"Find optimal decision tree parameters using grid search and cross-validation.\"\"\"\n",
        "    # Create features\n",
        "    encoder = OneHotFeatureEncoder(\n",
        "        numeric_features=['No. of Bedrooms', 'Area in sq ft'],\n",
        "        categorical_features=['House Type', 'City/County', 'Outcode'],\n",
        "        handle_unknown='ignore'\n",
        "    )\n",
        "\n",
        "    # Transform data\n",
        "    X_train = encoder.fit_transform(train_data).astype(np.float64)\n",
        "    y_train = train_data['log_price'].astype(np.float64)\n",
        "\n",
        "    # Define parameter grid\n",
        "    param_grid = {\n",
        "        'max_depth': np.array([4, 6, 8, 10, 12], dtype=np.int32),\n",
        "        'min_samples_leaf': np.array([10, 20, 30, 50, 70], dtype=np.int32),\n",
        "        'min_samples_split': np.array([20, 40, 60, 75, 100], dtype=np.int32)\n",
        "    }\n",
        "\n",
        "    # Run grid search\n",
        "    grid_search = GridSearchCV(\n",
        "        DecisionTreeRegressor(random_state=random_state),\n",
        "        param_grid,\n",
        "        cv=KFold(n_splits=5, shuffle=True, random_state=random_state),\n",
        "        scoring='neg_mean_absolute_error',\n",
        "        n_jobs=-1,\n",
        "        verbose=1\n",
        "    )\n",
        "\n",
        "    print(\"Starting grid search (expected time: ~2 minutes)...\")\n",
        "    print(f\"Testing {len(param_grid['max_depth']) * len(param_grid['min_samples_leaf']) * len(param_grid['min_samples_split'])} combinations\")\n",
        "\n",
        "    grid_search.fit(X_train, y_train)\n",
        "\n",
        "    # Train final model and evaluate\n",
        "    best_tree = DecisionTreeRegressor(**grid_search.best_params_, random_state=random_state)\n",
        "    best_tree.fit(X_train, y_train)\n",
        "\n",
        "    # Test set evaluation\n",
        "    X_test = encoder.transform(test_data).astype(np.float64)\n",
        "    y_test = test_data['log_price'].astype(np.float64)\n",
        "    test_pred = best_tree.predict(X_test)\n",
        "\n",
        "    # Print results\n",
        "    print(\"\\nBest parameters found:\")\n",
        "    for param, value in grid_search.best_params_.items():\n",
        "        print(f\"{param}: {value}\")\n",
        "\n",
        "    print(f\"\\nBest Model Performance:\")\n",
        "    print(f\"Test R²: {r2_score(y_test, test_pred):.3f}\")\n",
        "    print(f\"Test MAE: £{mean_absolute_error(np.exp(y_test), np.exp(test_pred)):,.0f}\")\n",
        "\n",
        "    # Prepare data for visualisation\n",
        "    depth_scores = {}\n",
        "    for depth in param_grid['max_depth']:\n",
        "        mask = grid_search.cv_results_['param_max_depth'] == depth\n",
        "        depth_scores[depth] = grid_search.cv_results_['mean_test_score'][mask]\n",
        "\n",
        "    # Visualise results with adjusted range and boxplot parameters\n",
        "    plt.figure(figsize=(10, 5))\n",
        "    plt.boxplot(depth_scores.values(),\n",
        "                whis=1.5,           # Set whisker length to 1.5 IQR (standard)\n",
        "                showfliers=True,    # Show outlier points\n",
        "                showcaps=True,      # Show whisker caps\n",
        "                notch=False)        # Don't use notched boxes\n",
        "\n",
        "    plt.xticks(range(1, len(param_grid['max_depth']) + 1), param_grid['max_depth'])\n",
        "\n",
        "    # Calculate appropriate y-axis limits with smaller padding\n",
        "    all_scores = np.concatenate(list(depth_scores.values()))\n",
        "    score_range = np.max(all_scores) - np.min(all_scores)\n",
        "    y_min = np.min(all_scores) - score_range * 0.1\n",
        "    y_max = np.max(all_scores) + score_range * 0.1\n",
        "    plt.ylim(y_min, y_max)\n",
        "\n",
        "    plt.title('Model Performance by Tree Depth')\n",
        "    plt.xlabel('Maximum Depth')\n",
        "    plt.ylabel('Negative MAE (higher is better)')\n",
        "    plt.grid(True)\n",
        "    plt.show()\n",
        "\n",
        "    return best_tree, pd.DataFrame(grid_search.cv_results_), grid_search.best_params_\n",
        "\n",
        "# Run grid search\n",
        "print(\"Running grid search to find optimal parameters...\\n\")\n",
        "optimal_tree, grid_results, best_params = grid_search_tree(train_data, test_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kXDA6t5lNdM_"
      },
      "source": [
        "<a name=\"grid-search-results-analysis\"></a>\n",
        "## Grid search results analysis\n",
        "\n",
        "Our systematic parameter search revealed optimal values and interesting performance patterns:\n",
        "\n",
        "### Model evolution and performance\n",
        "\n",
        "1. **Unrestricted tree (baseline)**\n",
        "   ```\n",
        "   Training R²: 1.000    Test R²: 0.786\n",
        "   Training MAE: £1,233  Test MAE: £670,882\n",
        "   \n",
        "   Classic overfitting pattern\n",
        "   ```\n",
        "\n",
        "2. **Manual parameters** (depth=6, samples=30)\n",
        "   ```\n",
        "   Training R²: 0.777    Test R²: 0.787\n",
        "   Training MAE: £594,884 Test MAE: £636,490\n",
        "   \n",
        "   Better generalisation, still suboptimal\n",
        "   ```\n",
        "\n",
        "3. **Optimal parameters from gird search**\n",
        "   ```\n",
        "   max_depth: 6\n",
        "   min_samples_leaf: 10\n",
        "   min_samples_split: 40\n",
        "   \n",
        "   Test R²: 0.805\n",
        "   Test MAE: £589,728\n",
        "\n",
        "   Best balance of complexity and performance\n",
        "   ```\n",
        "\n",
        "### Performance analysis\n",
        "\n",
        "1. **Depth impact**\n",
        "   - Depth=4 shows high variance (visible in boxplot spread)\n",
        "   - Depths 6-12 demonstrate remarkably stable performance\n",
        "   - Consistent outlier at -0.3 MAE across all depths\n",
        "   - Strongly validates depth=6 as optimal choice\n",
        "\n",
        "2. **Sample size trade-offs**\n",
        "   - Smaller leaf size (10 vs 30) enables more granular predictions\n",
        "   - Larger split criterion (40 vs 30) provides overfitting protection\n",
        "   - Achieves balance between flexibility and stability\n",
        "   - 12% reduction in MAE from unrestricted model\n",
        "\n",
        "3. **Performance patterns**\n",
        "   - Consistent outlier suggests specific property types need attention\n",
        "   - Stable performance across depths 6-12 indicates robust solution\n",
        "   - £47K improvement in predictions vs both previous approaches\n",
        "\n",
        "### The next challenge: Feature engineering\n",
        "\n",
        "While we've optimised tree structure, several areas need investigation:\n",
        "\n",
        "1. **Outlier analysis**\n",
        "   - Investigate properties causing consistent -0.3 MAE\n",
        "   - Consider robust regression techniques\n",
        "   - Evaluate feature importance for outlier cases\n",
        "\n",
        "2. **Location encoding**\n",
        "   - Current one-hot approach may be suboptimal\n",
        "   - Consider price-based and hierarchical encoding\n",
        "   - Need to balance granularity with stability\n",
        "\n",
        "3. **Price context**\n",
        "   - Investigate area price levels\n",
        "   - Consider temporal aspects\n",
        "   - Implement proper validation strategies\n",
        "\n",
        "Let's explore three encoding strategies using our optimal parameters:\n",
        "1. One-hot encoding (current approach)\n",
        "2. Target encoding with proper validation\n",
        "3. Price-per-sqft features with temporal aspects\n",
        "\n",
        "This will help us understand if our parameter optimisation remains robust across different feature representations while addressing the consistent outlier pattern we've observed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v2ylEpShNdNA"
      },
      "source": [
        "<a name=\"feature-set-selection\"></a>\n",
        "## Feature set selection\n",
        "\n",
        "Now that we've optimised our tree structure, a critical question remains: what's the best way to represent location information for house price prediction? Let's review the key approaches we've developed, which we'll compare in depth using ATLAS in the next lesson.\n",
        "\n",
        "### Core features\n",
        "Every feature set will include these fundamental property characteristics:\n",
        "- Area in sq ft (numeric)\n",
        "- Number of Bedrooms (numeric)\n",
        "- House Type (one-hot encoded, 8 categories)\n",
        "- Log-transformed price (target variable)\n",
        "\n",
        "### Feature set 1: Simple categorical encoding\n",
        "Core features plus essential one-hot encoded location information:\n",
        "```\n",
        "Base Features\n",
        "   +\n",
        "One-Hot Encoded Location Features:\n",
        "├── City/County (6 binary features)\n",
        "└── Outcode (~100 binary features)\n",
        "```\n",
        "- Each category gets its own binary column\n",
        "- Missing values get dedicated indicator columns\n",
        "- No price information used in encoding\n",
        "- Avoids sparsity issues from full location encoding\n",
        "\n",
        "### Feature set 2: Hierarchical target encoding\n",
        "Core features plus price-based location encoding:\n",
        "```\n",
        "Base Features\n",
        "   +\n",
        "City/County One-hot encoded\n",
        "   +\n",
        "Price-Encoded Location Features:\n",
        "├── Location encoding (with postcode prior)\n",
        "├── Postcode encoding (with outcode prior)\n",
        "└── Outcode encoding (with global prior)\n",
        "```\n",
        "- Each location gets encoded as mean log price\n",
        "- Hierarchical fallback for sparse data:\n",
        "  * Sparse locations → postcode average\n",
        "  * Sparse postcodes → outcode average\n",
        "  * Missing outcodes → global average\n",
        "- Smoothing factor of 10 for stability\n",
        "- Minimum frequency of 5 for reliability\n",
        "\n",
        "### Feature set 3: Market rate features\n",
        "Core features plus area-normalised market metrics:\n",
        "```\n",
        "Base Features\n",
        "   +\n",
        "City/County One-hot encoded\n",
        "   +\n",
        "Market Rate Features:\n",
        "├── Area per square foot (normalised area metric)\n",
        "└── Mean price per square foot by outcode (market rate)\n",
        "```\n",
        "- Two complementary area metrics:\n",
        "  * Area efficiency (sq ft per room)\n",
        "  * Local market rates (£/sq ft)\n",
        "- Calculated from training data only\n",
        "- Missing outcodes use global average\n",
        "- Industry-standard approach\n",
        "\n",
        "### What we'll learn\n",
        "\n",
        "This comparison will reveal:\n",
        "\n",
        "1. **Pure Location Value**\n",
        "   - How much can we predict from location alone?\n",
        "   - Do we need price information?\n",
        "   - Which areas consistently command premiums?\n",
        "\n",
        "2. **Price Information Impact**\n",
        "   - Does target encoding improve accuracy?\n",
        "   - Is the complexity worth it?\n",
        "   - How stable are the predictions?\n",
        "\n",
        "3. **Market Rate Effects**\n",
        "   - Do area-normalised prices help?\n",
        "   - Which areas are over/under valued?\n",
        "   - How reliable are local price levels?\n",
        "\n",
        "In the next lesson, we'll use ATLAS (Automated Tree Learning Analysis System) to systematically compare these encoding strategies across multiple models and metrics. ATLAS will help us deeply understand:\n",
        "- Which encoding works best for different areas of London\n",
        "- How encoding choices affect model reliability\n",
        "- When to use each type of feature encoding\n",
        "- How to combine encodings for optimal performance\n",
        "\n",
        "For now, let's conclude our exploration of decision trees by examining some practical considerations for deployment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "IjucZRb9NdNA",
        "outputId": "c5a12f77-2b71-43dc-965a-1732e656b3ff"
      },
      "outputs": [],
      "source": [
        "# Optimised tree parameters from grid search\n",
        "tree_params = {\n",
        "    'max_depth': 6,\n",
        "    'min_samples_leaf': 10,\n",
        "    'min_samples_split': 40,\n",
        "    'random_state': RANDOM_STATE\n",
        "}\n",
        "\n",
        "def compare_feature_sets(train_data: pd.DataFrame,\n",
        "                        test_data: pd.DataFrame,\n",
        "                        tree_params: Dict) -> pd.DataFrame:\n",
        "    \"\"\"Compare three feature encoding strategies using optimised tree parameters.\n",
        "    This is a simplified version - see ATLAS for comprehensive comparison.\"\"\"\n",
        "\n",
        "    results = []\n",
        "    tree = DecisionTreeRegressor(**tree_params)\n",
        "\n",
        "    # Create base encoder for house type and city/county (used in all sets)\n",
        "    base_encoder = OneHotFeatureEncoder(\n",
        "        numeric_features=['Area in sq ft', 'No. of Bedrooms'],\n",
        "        categorical_features=['House Type', 'City/County']\n",
        "    )\n",
        "\n",
        "    # Feature Set 1: Simple Categorical\n",
        "    print(\"\\nEvaluating Simple Categorical Features...\")\n",
        "    categorical_encoder = OneHotFeatureEncoder(\n",
        "        numeric_features=['Area in sq ft', 'No. of Bedrooms'],\n",
        "        categorical_features=['House Type', 'City/County', 'Outcode']\n",
        "    )\n",
        "\n",
        "    X_train_cat = categorical_encoder.fit_transform(train_data)\n",
        "    X_test_cat = categorical_encoder.transform(test_data)\n",
        "\n",
        "    print(\"\\nFeature Set 1 - Simple Categorical:\")\n",
        "    print(f\"Training features ({X_train_cat.shape[1]} total):\")\n",
        "    print(X_train_cat.columns.tolist())\n",
        "\n",
        "    tree.fit(X_train_cat, train_data['log_price'])\n",
        "    train_pred_cat = tree.predict(X_train_cat)\n",
        "    test_pred_cat = tree.predict(X_test_cat)\n",
        "\n",
        "    results.append({\n",
        "        'feature_set': 'Simple Categorical',\n",
        "        'n_features': X_train_cat.shape[1],\n",
        "        'train_r2': r2_score(train_data['log_price'], train_pred_cat),\n",
        "        'test_r2': r2_score(test_data['log_price'], test_pred_cat),\n",
        "        'train_mae': mean_absolute_error(\n",
        "            np.exp(train_data['log_price']),\n",
        "            np.exp(train_pred_cat)\n",
        "        ),\n",
        "        'test_mae': mean_absolute_error(\n",
        "            np.exp(test_data['log_price']),\n",
        "            np.exp(test_pred_cat)\n",
        "        )\n",
        "    })\n",
        "\n",
        "    # Feature Set 2: Hierarchical Target Encoding\n",
        "    print(\"\\nEvaluating Hierarchical Target Encoding...\")\n",
        "    location_encoder = HierarchicalLocationEncoder()\n",
        "    location_train = location_encoder.fit_transform(train_data)\n",
        "    location_test = location_encoder.transform(test_data)\n",
        "\n",
        "    # Include base features with location encoding\n",
        "    X_train_target = pd.concat([\n",
        "        base_encoder.fit_transform(train_data),  # Base features including City/County\n",
        "        location_train\n",
        "    ], axis=1)\n",
        "\n",
        "    X_test_target = pd.concat([\n",
        "        base_encoder.transform(test_data),\n",
        "        location_test\n",
        "    ], axis=1)\n",
        "\n",
        "    print(\"\\nFeature Set 2 - Hierarchical Target:\")\n",
        "    print(f\"Training features ({X_train_target.shape[1]} total):\")\n",
        "    print(X_train_target.columns.tolist())\n",
        "\n",
        "    tree.fit(X_train_target, train_data['log_price'])\n",
        "    train_pred_target = tree.predict(X_train_target)\n",
        "    test_pred_target = tree.predict(X_test_target)\n",
        "\n",
        "    results.append({\n",
        "        'feature_set': 'Hierarchical Target',\n",
        "        'n_features': X_train_target.shape[1],\n",
        "        'train_r2': r2_score(train_data['log_price'], train_pred_target),\n",
        "        'test_r2': r2_score(test_data['log_price'], test_pred_target),\n",
        "        'train_mae': mean_absolute_error(\n",
        "            np.exp(train_data['log_price']),\n",
        "            np.exp(train_pred_target)\n",
        "        ),\n",
        "        'test_mae': mean_absolute_error(\n",
        "            np.exp(test_data['log_price']),\n",
        "            np.exp(test_pred_target)\n",
        "        )\n",
        "    })\n",
        "\n",
        "    # Feature Set 3: Market Rate Features\n",
        "    print(\"\\nEvaluating Market Rate Features...\")\n",
        "    price_encoder = MeanOutcodePricePerSquareFootEncoder()\n",
        "    price_per_sqft_train = price_encoder.fit_transform(train_data)\n",
        "    price_per_sqft_test = price_encoder.transform(test_data)\n",
        "\n",
        "    X_train_market = pd.concat([\n",
        "        base_encoder.transform(train_data),  # Base features including City/County\n",
        "        pd.DataFrame({'price_per_sqft': price_per_sqft_train})  # Market rate feature\n",
        "    ], axis=1)\n",
        "\n",
        "    X_test_market = pd.concat([\n",
        "        base_encoder.transform(test_data),\n",
        "        pd.DataFrame({'price_per_sqft': price_per_sqft_test})\n",
        "    ], axis=1)\n",
        "\n",
        "    print(\"\\nFeature Set 3 - Market Rate:\")\n",
        "    print(f\"Training features ({X_train_market.shape[1]} total):\")\n",
        "    print(X_train_market.columns.tolist())\n",
        "\n",
        "    tree.fit(X_train_market, train_data['log_price'])\n",
        "    train_pred_market = tree.predict(X_train_market)\n",
        "    test_pred_market = tree.predict(X_test_market)\n",
        "\n",
        "    results.append({\n",
        "        'feature_set': 'Market Rate',\n",
        "        'n_features': X_train_market.shape[1],\n",
        "        'train_r2': r2_score(train_data['log_price'], train_pred_market),\n",
        "        'test_r2': r2_score(test_data['log_price'], test_pred_market),\n",
        "        'train_mae': mean_absolute_error(\n",
        "            np.exp(train_data['log_price']),\n",
        "            np.exp(train_pred_market)\n",
        "        ),\n",
        "        'test_mae': mean_absolute_error(\n",
        "            np.exp(test_data['log_price']),\n",
        "            np.exp(test_pred_market)\n",
        "        )\n",
        "    })\n",
        "\n",
        "    return pd.DataFrame(results)\n",
        "\n",
        "# Run comparison\n",
        "print(\"Comparing feature sets with optimised tree (depth=6, min_samples_leaf=10, min_samples_split=40)...\")\n",
        "comparison_results = compare_feature_sets(train_data, test_data, tree_params)\n",
        "\n",
        "# Display results\n",
        "pd.set_option('display.float_format', lambda x: '{:,.3f}'.format(x) if abs(x) < 1000\n",
        "              else '{:,.0f}'.format(x))\n",
        "print(\"\\nFeature Set Comparison Results:\")\n",
        "display(comparison_results)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ssH4h8sNdND"
      },
      "source": [
        "<a name=\"feature-set-results-analysis\"></a>\n",
        "## Feature set results analysis\n",
        "\n",
        "Our three encoding strategies produced notably different feature spaces with distinct performance characteristics:\n",
        "\n",
        "### 1. Simple categorical (161 features)\n",
        "**Structure:**\n",
        "- Base features (2): Area, Bedrooms\n",
        "- House Type (8): From Bungalow to Studio\n",
        "- City/County (8): Including missing value indicator\n",
        "- Outcodes (143): Complete geographic coverage\n",
        "\n",
        "**Performance:**\n",
        "- Test R²: 0.805, Test MAE: £589,728\n",
        "- Large feature space due to one-hot encoding\n",
        "- Solid baseline but high dimensionality\n",
        "- Most stable train/test performance (0.790 → 0.805)\n",
        "\n",
        "### 2. Hierarchical target (21 features)\n",
        "**Structure:**\n",
        "- Base features (2): Area, Bedrooms\n",
        "- House Type (8): Full encoding\n",
        "- City/County (8): Administrative regions\n",
        "- Location hierarchy (3): Outcode, postcode, location mean prices\n",
        "\n",
        "**Performance:**\n",
        "- Test R²: 0.833, Test MAE: £542,874\n",
        "- Much smaller feature space\n",
        "- Improved performance through price-based location encoding\n",
        "- Larger train/test gap (0.883 → 0.833)\n",
        "\n",
        "### 3. Market rate (19 features)\n",
        "**Structure:**\n",
        "- Base features (2): Area, Bedrooms  \n",
        "- House Type (8): Property categories\n",
        "- City/County (8): Regional context\n",
        "- Market metric (1): Price per square foot\n",
        "\n",
        "**Performance:**\n",
        "- Test R²: 0.878, Test MAE: £477,949\n",
        "- Best performance with fewest features\n",
        "- Combines administrative boundaries with market rates\n",
        "- Moderate train/test gap (0.913 → 0.878)\n",
        "\n",
        "### Key insights\n",
        "\n",
        "1. **Data characteristics**\n",
        "   - 143 unique outcodes averaging 19.5 properties each\n",
        "   - 2,351 postcodes but only 1.2 properties per code\n",
        "   - 444 locations with just 76 having ≥ 5 properties\n",
        "   - Price per square foot ranges from £267 to £4,063\n",
        "\n",
        "2. **Performance patterns**\n",
        "   - Market Rate achieves best results with most compact feature set\n",
        "   - Simple categorical stable but less accurate (+£111,779 MAE)\n",
        "   - Hierarchical encoding balances complexity/performance (+£64,925 MAE)\n",
        "\n",
        "3. **Model stability**\n",
        "   - One-hot encoding shows best generalisation (improves on test)\n",
        "   - Price-derived features show some overfitting but better absolute performance\n",
        "   - Trade-off between information content and model stability\n",
        "\n",
        "In the next lesson, we'll use ATLAS to explore these patterns more systematically, including:\n",
        "- Cross-validation for more reliable estimates\n",
        "- Feature importance analysis\n",
        "- Ensemble methods to combine different views\n",
        "- Optimal feature set combinations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_7pPEX_NdND"
      },
      "source": [
        "<a name=\"advanced-models\"></a>\n",
        "## Advanced models\n",
        "\n",
        "Our decision tree achieved £589,728 average error - roughly 25% of the typical house price. To understand how we can do better, imagine three common scenarios in real estate valuation:\n",
        "\n",
        "1. **The difficult property**  \n",
        "   A unique house sells for £2M. Our model predicts:\n",
        "   - Sometimes £1.8M (undervalued)\n",
        "   - Sometimes £2.2M (overvalued)\n",
        "   - Rarely exactly £2M\n",
        "   \n",
        "   This variation in predictions is called variance. Mathematically:\n",
        "\n",
        "   $\\text{Variance} = E[(\\hat{f}(x) - E[\\hat{f}(x)])^2]$\n",
        "   \n",
        "   Where $\\hat{f}(x)$ is our prediction for house $x$.\n",
        "\n",
        "2. **The Systematic Error**  \n",
        "   Houses in Chelsea consistently sell for £3M, but our model predicts £2.8M every time.\n",
        "   \n",
        "   This consistent under-prediction is called bias. Mathematically:\n",
        "\n",
        "   $\\text{Bias} = E[\\hat{f}(x)] - f(x)$\n",
        "   \n",
        "   Where $f(x)$ is the true price.\n",
        "\n",
        "3. **The Market Noise**  \n",
        "   Two identical houses on the same street sell for different prices due to:\n",
        "   - Timing of sale\n",
        "   - Buyer negotiations\n",
        "   - Small condition differences\n",
        "   \n",
        "   This is irreducible error ($\\sigma^2$) - no model can predict it.\n",
        "\n",
        "### The Total Error\n",
        "\n",
        "These three components add up to our total prediction error:\n",
        "\n",
        "$E[(y - \\hat{f}(x))^2] = \\underbrace{\\text{Bias}^2}_{\\text{systematic error}} + \\underbrace{\\text{Variance}}_{\\text{prediction spread}} + \\underbrace{\\sigma^2}_{\\text{market noise}}$\n",
        "\n",
        "Two advanced models help with these problems...\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mHOtyLBx3xKY"
      },
      "source": [
        "<a name=\"random-forests-the-wisdom-of-crowds\"></a>\n",
        "### Random Forests: The wisdom of crowds\n",
        "\n",
        "Imagine asking 100 estate agents to value a house. Each agent:\n",
        "- Looks at different features (some focus on location, others on condition)\n",
        "- Has seen different past sales\n",
        "- Makes slightly different assumptions\n",
        "\n",
        "Their average prediction tends to be more reliable because:\n",
        "$\\text{Var}[\\text{average of }M\\text{ predictions}] = \\frac{\\text{Var}[\\text{single prediction}]}{M}$\n",
        "\n",
        "Random forests automate this by:\n",
        "1. Building 100+ different trees\n",
        "2. Each using random feature subsets\n",
        "3. Each trained on different data samples\n",
        "4. Averaging their predictions\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cYIsGG3h31M5"
      },
      "source": [
        "<a name=\"xgboost-learning-from-mistakes\"></a>\n",
        "### XGBoost: Learning from mistakes\n",
        "\n",
        "Think of an agent learning to value properties:\n",
        "1. Start with rough estimates based on size\n",
        "2. Notice they're undervaluing Chelsea properties\n",
        "3. Add a \"Chelsea premium\" adjustment\n",
        "4. Keep refining based on mistakes\n",
        "\n",
        "XGBoost formalises this as:\n",
        "\n",
        "$\\text{New Prediction} = \\text{Old Prediction} + \\text{Learning Rate} \\times \\text{Error Correction}$\n",
        "\n",
        "Or as math notation:\n",
        "\n",
        "$\\hat{y}_i^{(t)} = \\hat{y}_i^{(t-1)} + \\eta f_t(x_i)$\n",
        "\n",
        "Where each new tree $f_t$ focuses on the previous errors.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dLaGVE874M9T"
      },
      "source": [
        "<a name=\"random-forest-and-xgboost-implementation\"></a>\n",
        "## Random Forest and XGBoost implementation\n",
        "\n",
        "For both models, we'll explore:\n",
        "1. Basic implementation with default settings\n",
        "2. Impact of different feature encodings\n",
        "3. Parameter tuning for optimal performance\n",
        "4. Feature importance analysis\n",
        "\n",
        "Let's start by implementing Random Forests to see how averaging multiple predictions can reduce our £589,728 error."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "pr3IqVmCNdNE",
        "outputId": "167a4f90-cb11-49f8-cac4-87a06e4ec324"
      },
      "outputs": [],
      "source": [
        "# Create one-hot encoded features\n",
        "onehot_encoder = OneHotFeatureEncoder(\n",
        "    numeric_features=['No. of Bedrooms', 'Area in sq ft'],\n",
        "    categorical_features=['House Type', 'City/County', 'Outcode']\n",
        ")\n",
        "\n",
        "# Prepare features and reset index to remove feature names warnings\n",
        "X_train = onehot_encoder.fit_transform(train_data)\n",
        "X_test = onehot_encoder.transform(test_data)\n",
        "y_train = train_data['log_price']\n",
        "y_test = test_data['log_price']\n",
        "\n",
        "# Store feature names before converting to numpy\n",
        "feature_names = onehot_encoder.output_feature_names\n",
        "\n",
        "# Convert to numpy arrays to remove feature names\n",
        "X_train = X_train.values\n",
        "X_test = X_test.values\n",
        "\n",
        "print(f\"Training with {X_train.shape[1]} one-hot encoded features\")\n",
        "\n",
        "# Initialise Random Forest with basic settings\n",
        "rf_model = RandomForestRegressor(\n",
        "    n_estimators=100,\n",
        "    max_features='sqrt',\n",
        "    random_state=42,\n",
        "    n_jobs=-1\n",
        ")\n",
        "\n",
        "# Train model and time it\n",
        "rf_model.fit(X_train, y_train)\n",
        "\n",
        "# Get predictions for both training and test sets\n",
        "train_pred = rf_model.predict(X_train)\n",
        "test_pred = rf_model.predict(X_test)\n",
        "\n",
        "# Calculate metrics in log space\n",
        "train_r2 = r2_score(y_train, train_pred)\n",
        "test_r2 = r2_score(y_test, test_pred)\n",
        "\n",
        "# Convert to price space for interpretable errors\n",
        "train_price_true = np.exp(y_train)\n",
        "test_price_true = np.exp(y_test)\n",
        "train_price_pred = np.exp(train_pred)\n",
        "test_price_pred = np.exp(test_pred)\n",
        "\n",
        "train_mae = mean_absolute_error(train_price_true, train_price_pred)\n",
        "test_mae = mean_absolute_error(test_price_true, test_price_pred)\n",
        "\n",
        "# Get tree prediction variance\n",
        "tree_predictions = np.array([tree.predict(X_test)\n",
        "                            for tree in rf_model.estimators_])\n",
        "pred_std = np.std(tree_predictions, axis=0)\n",
        "\n",
        "print(\"\\nRandom Forest Performance:\")\n",
        "print(f\"Training MAE: £{train_mae:,.0f}\")\n",
        "print(f\"Test MAE: £{test_mae:,.0f}\")\n",
        "print(f\"Training R²: {train_r2:.3f}\")\n",
        "print(f\"Test R²: {test_r2:.3f}\")\n",
        "print(f\"Average prediction std: {pred_std.mean():.3f} log units\")\n",
        "\n",
        "# Visualise errors vs uncertainty\n",
        "plt.figure(figsize=(10, 5))\n",
        "plt.scatter(pred_std,\n",
        "            np.abs(y_test - test_pred),\n",
        "            alpha=0.5)\n",
        "plt.xlabel('Tree Prediction Standard Deviation')\n",
        "plt.ylabel('Absolute Prediction Error (log price)')\n",
        "plt.title('Random Forest: Prediction Uncertainty vs Error')\n",
        "plt.show()\n",
        "\n",
        "# Plot feature importance using stored feature names\n",
        "importance = pd.DataFrame({\n",
        "    'feature': feature_names,\n",
        "    'importance': rf_model.feature_importances_\n",
        "}).sort_values('importance', ascending=False)\n",
        "\n",
        "plt.figure(figsize=(12, 5))\n",
        "sns.barplot(data=importance.head(10),\n",
        "            x='importance', y='feature')\n",
        "plt.title('Top 10 Most Important Features (Random Forest)')\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Save artifacts for production\n",
        "# joblib.dump(rf_model, '../models/random_forest_onehot.joblib')\n",
        "# onehot_encoder.save('../models/onehot_encoder_rf.pkl')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Zl0j7WwNNdNE",
        "outputId": "88533251-1515-47b7-c46a-294e967fc943"
      },
      "outputs": [],
      "source": [
        "# Use same preprocessed features from Random Forest (already reset index)\n",
        "print(f\"Training with {X_train.shape[1]} one-hot encoded features\")\n",
        "\n",
        "# Initialise XGBoost with basic settings\n",
        "xgb_model = XGBRegressor(\n",
        "    n_estimators=100,       # Number of boosting rounds\n",
        "    learning_rate=0.1,      # Step size for corrections\n",
        "    max_depth=6,           # Tree depth from our earlier tuning\n",
        "    min_child_weight=6,    # Similar to min_samples_leaf\n",
        "    subsample=0.8,         # Use 80% of data per tree\n",
        "    colsample_bytree=0.8,  # Use 80% of features per tree\n",
        "    random_state=42,\n",
        "    eval_metric='mae'\n",
        ")\n",
        "\n",
        "# Train model with evaluation sets\n",
        "eval_set = [(X_train, y_train), (X_test, y_test)]\n",
        "xgb_model.fit(\n",
        "    X_train, y_train,\n",
        "    eval_set=eval_set,\n",
        "    verbose=False\n",
        ")\n",
        "\n",
        "# Get predictions for both sets\n",
        "train_pred = xgb_model.predict(X_train)\n",
        "test_pred = xgb_model.predict(X_test)\n",
        "\n",
        "# Calculate metrics in log space\n",
        "train_r2 = r2_score(y_train, train_pred)\n",
        "test_r2 = r2_score(y_test, test_pred)\n",
        "\n",
        "# Convert to price space for interpretable errors\n",
        "train_price_true = np.exp(y_train)\n",
        "test_price_true = np.exp(y_test)\n",
        "train_price_pred = np.exp(train_pred)\n",
        "test_price_pred = np.exp(test_pred)\n",
        "\n",
        "train_mae = mean_absolute_error(train_price_true, train_price_pred)\n",
        "test_mae = mean_absolute_error(test_price_true, test_price_pred)\n",
        "\n",
        "print(\"\\nXGBoost Performance:\")\n",
        "print(f\"Training MAE: £{train_mae:,.0f}\")\n",
        "print(f\"Test MAE: £{test_mae:,.0f}\")\n",
        "print(f\"Training R²: {train_r2:.3f}\")\n",
        "print(f\"Test R²: {test_r2:.3f}\")\n",
        "\n",
        "# Plot training progression\n",
        "results = xgb_model.evals_result()\n",
        "plt.figure(figsize=(9, 5))\n",
        "plt.plot(results['validation_0']['mae'], label='Train')\n",
        "plt.plot(results['validation_1']['mae'], label='Test')\n",
        "plt.xlabel('Boosting Round')\n",
        "plt.ylabel('Mean Absolute Error')\n",
        "plt.title('XGBoost Training Progress')\n",
        "plt.legend()\n",
        "plt.show()\n",
        "\n",
        "# Plot feature importance using stored feature names\n",
        "importance = pd.DataFrame({\n",
        "    'feature': feature_names,\n",
        "    'importance': xgb_model.feature_importances_\n",
        "}).sort_values('importance', ascending=False)\n",
        "\n",
        "plt.figure(figsize=(8.03, 5))\n",
        "sns.barplot(data=importance.head(10),\n",
        "            x='importance', y='feature')\n",
        "plt.title('Top 10 Most Important Features (XGBoost)')\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Save model for production\n",
        "# xgb_model.save_model('../models/xgboost_onehot.json')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6qS-Cb2nNdNE"
      },
      "source": [
        "<a name=\"model-selection-framework-trees-random-forests-and-gradient-boosting\"></a>\n",
        "## Model selection framework: Trees, Random Forests, and Gradient Boosting\n",
        "\n",
        "Imagine you're trying to value a house in London. You could ask one expert, you could ask a hundred experts and average their opinions, or... and here's where it gets interesting... you could have experts learn from each other's mistakes.\n",
        "\n",
        "This is essentially the difference between our basic decision tree, random forest and gradient boosting models.\n",
        "\n",
        "### How our models think\n",
        "\n",
        "The Random Forest looks at our uncertainty directly. In the uncertainty vs error plot, we see something fascinating - as the trees disagree more (higher standard deviation), our prediction errors tend to increase. It's like when estate agents strongly disagree about a property's value, they're often dealing with an unusual property that's harder to price.\n",
        "\n",
        "XGBoost, on the other hand, learns sequentially. Looking at its training curve, we see rapid improvement in the first 20 rounds (that steep initial drop in error), followed by slower refinement. It's like an agent who quickly learns the basics of the market, then spends time mastering the nuances.\n",
        "\n",
        "### Feature importance: Two different stories\n",
        "\n",
        "Both models agree that area is king, but they weigh other features quite differently:\n",
        "\n",
        "**Random Forest:**\n",
        "- Area dominates (~40% importance)\n",
        "- Number of bedrooms second (~18%)\n",
        "- Property types and locations share the rest\n",
        "\n",
        "**XGBoost:**\n",
        "- Area still leads but less dramatically (~9%)\n",
        "- More weight on location (City/County_london ~4%)\n",
        "- More balanced importance across features\n",
        "\n",
        "This difference is fundamental - Random Forest builds many independent trees and averages them, while XGBoost carefully constructs each tree to fix previous mistakes.\n",
        "\n",
        "### The performance trade-off\n",
        "\n",
        "Looking at our metrics:\n",
        "- Random Forest: Great training fit (MAE £178,894) but larger test gap (MAE £472,405)\n",
        "- XGBoost: More conservative training (MAE £346,919) but better test performance (MAE £425,485)\n",
        "\n",
        "Think of it this way: Random Forest is like having 100 independent experts, while XGBoost is like having experts who learn from each other. The independence gives Random Forest better uncertainty estimates (we can measure disagreement), while the learning process gives XGBoost better generalisation.\n",
        "\n",
        "### Real estate context\n",
        "\n",
        "For house price prediction in London, these differences matter:\n",
        "\n",
        "1. **Uncertainty matters**\n",
        "   - Random Forest's uncertainty estimates could flag risky valuations\n",
        "   - That increasing spread in the uncertainty plot is valuable information\n",
        "   - Could help identify properties needing human review\n",
        "\n",
        "2. **Learning process**\n",
        "   - XGBoost's sequential learning might capture market subtleties better\n",
        "   - The training curve suggests it's still learning at 100 rounds\n",
        "   - Could benefit from even more boosting iterations\n",
        "\n",
        "### Setting up for ATLAS\n",
        "\n",
        "These initial results raise exciting questions for our automated framework:\n",
        "1. Could we combine both models' strengths?\n",
        "2. How do they perform across different price ranges?\n",
        "3. What's the optimal feature encoding strategy?\n",
        "\n",
        "In ATLAS, we'll explore these questions systematically, but already we can see that tree-based models offer both powerful prediction and useful uncertainty estimation for London house prices."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9SDsXNc4NdNE"
      },
      "source": [
        "<a name=\"production-implementation-and-deployment-considerations\"></a>\n",
        "## Production implementation and deployment considerations\n",
        "\n",
        "Our exploration of different tree-based models has revealed distinct strengths:\n",
        "- Random Forests provide robust uncertainty estimates\n",
        "- XGBoost offers superior average performance\n",
        "- Different feature importance patterns give complementary insights\n",
        "\n",
        "But how do we take these insights into production? A production-ready implementation needs to handle:\n",
        "\n",
        "1. **Feature engineering pipeline**\n",
        "   - Multiple encoding types (one-hot, target, market rate)\n",
        "   - Consistent feature ordering\n",
        "   - Missing value handling\n",
        "\n",
        "2. **Model persistence**\n",
        "   - Save trained models\n",
        "   - Preserve encoding parameters\n",
        "   - Store performance metrics\n",
        "\n",
        "3. **Validation framework**\n",
        "   - Input data validation\n",
        "   - Prediction sanity checks\n",
        "   - Performance monitoring\n",
        "\n",
        "Let's implement a complete production pipeline that combines our three encoding strategies with XGBoost's strong predictive performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 173,
      "metadata": {
        "id": "9697w0Ziiv-9"
      },
      "outputs": [],
      "source": [
        "class CombinedEncoder:\n",
        "    \"\"\"Handles multiple encoding types for London housing data.\"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        # Initialize all components as None - they will be loaded from URLs\n",
        "        self.onehot_encoder = None\n",
        "        self.target_encoder = None\n",
        "        self.price_encoder = None\n",
        "        self.fitted = False\n",
        "        self.feature_names = None\n",
        "\n",
        "    def fit(self, df: pd.DataFrame, target_col: str = 'log_price'):\n",
        "        \"\"\"Fit all encoders on training data.\"\"\"\n",
        "        # Prepare categorical features for one-hot encoding\n",
        "        categorical_features = df[['House Type', 'City/County']]\n",
        "        self.onehot_encoder.fit(categorical_features)\n",
        "\n",
        "        # Fit target encoder on location hierarchy\n",
        "        self.target_encoder.fit(df, target_col=target_col)\n",
        "\n",
        "        # Fit price per sqft encoder\n",
        "        self.price_encoder.fit(df)\n",
        "\n",
        "        # Store feature names in order\n",
        "        onehot_features = self.onehot_encoder.get_feature_names_out(['House Type', 'City/County'])\n",
        "\n",
        "        # Use actual column names from target encoder\n",
        "        target_columns = [\n",
        "            'location_outcode_encoded',\n",
        "            'location_postcode_encoded',\n",
        "            'location_encoded'\n",
        "        ]\n",
        "\n",
        "        self.feature_names = (\n",
        "            ['Area in sq ft', 'No. of Bedrooms'] +  # Numeric\n",
        "            list(onehot_features) +                 # One-hot\n",
        "            target_columns +                        # Target\n",
        "            ['price_per_sqft']                      # Market rate\n",
        "        )\n",
        "\n",
        "        self.fitted = True\n",
        "        return self\n",
        "\n",
        "    def transform(self, df: pd.DataFrame) -> pd.DataFrame:\n",
        "        \"\"\"Transform data using all encoders.\"\"\"\n",
        "        if not self.fitted:\n",
        "            raise ValueError(\"Encoder must be fit before transform\")\n",
        "\n",
        "        # Create one-hot encoded features\n",
        "        categorical_features = df[['House Type', 'City/County']]\n",
        "        onehot_encoded = pd.DataFrame(\n",
        "            self.onehot_encoder.transform(categorical_features),\n",
        "            columns=self.onehot_encoder.get_feature_names_out(['House Type', 'City/County']),\n",
        "            index=df.index\n",
        "        )\n",
        "\n",
        "        # Create target encoded features\n",
        "        target_encoded = self.target_encoder.transform(df)\n",
        "\n",
        "        # Create price per sqft feature\n",
        "        price_per_sqft = pd.DataFrame({\n",
        "            'price_per_sqft': self.price_encoder.transform(df)\n",
        "        }, index=df.index)\n",
        "\n",
        "        # Combine all features\n",
        "        features = pd.concat([\n",
        "            df[['Area in sq ft', 'No. of Bedrooms']],  # Numeric\n",
        "            onehot_encoded,                            # One-hot\n",
        "            target_encoded,                            # Target\n",
        "            price_per_sqft                            # Market rate\n",
        "        ], axis=1)\n",
        "\n",
        "        # Ensure consistent feature order\n",
        "        return features[self.feature_names]\n",
        "\n",
        "    def fit_transform(self, df: pd.DataFrame, target_col: str = 'log_price') -> pd.DataFrame:\n",
        "        \"\"\"Fit encoder and transform data in one step.\"\"\"\n",
        "        return self.fit(df, target_col).transform(df)\n",
        "\n",
        "    @classmethod\n",
        "    def load_from_url(cls, base_url: str) -> 'CombinedEncoder':\n",
        "        \"\"\"Load saved encoder components from URLs.\"\"\"\n",
        "        encoder = cls()\n",
        "\n",
        "        def load_pickle_from_url(url, reconstruct_class=None):\n",
        "            \"\"\"Helper function to load pickle files with debug info\"\"\"\n",
        "            response = requests.get(url)\n",
        "            print(f\"Loading from {url}\")\n",
        "            print(f\"Response status: {response.status_code}\")\n",
        "            print(f\"Content type: {response.headers.get('content-type')}\")\n",
        "            print(f\"Content length: {len(response.content)} bytes\")\n",
        "\n",
        "            try:\n",
        "                data = pickle.loads(response.content)\n",
        "                if reconstruct_class:\n",
        "                    # Create new instance and copy attributes\n",
        "                    obj = reconstruct_class()\n",
        "                    for key, value in data.items():\n",
        "                        setattr(obj, key, value)\n",
        "                    data = obj\n",
        "                print(\"Successfully unpickled object\")\n",
        "                return data\n",
        "            except Exception as e:\n",
        "                print(f\"Error unpickling: {str(e)}\")\n",
        "                raise\n",
        "\n",
        "        # Load OneHot encoder (already a proper object)\n",
        "        print(\"\\nLoading OneHot encoder...\")\n",
        "        encoder.onehot_encoder = load_pickle_from_url(f\"{base_url}/encoder/onehot.pkl\")\n",
        "\n",
        "        # Load Target encoder and reconstruct\n",
        "        print(\"\\nLoading Target encoder...\")\n",
        "        encoder.target_encoder = load_pickle_from_url(\n",
        "            f\"{base_url}/encoder/target.pkl\",\n",
        "            reconstruct_class=HierarchicalLocationEncoder\n",
        "        )\n",
        "\n",
        "        # Load Price encoder and reconstruct\n",
        "        print(\"\\nLoading Price encoder...\")\n",
        "        encoder.price_encoder = load_pickle_from_url(\n",
        "            f\"{base_url}/encoder/price.pkl\",\n",
        "            reconstruct_class=MeanOutcodePricePerSquareFootEncoder\n",
        "        )\n",
        "\n",
        "        # Load feature names\n",
        "        print(\"\\nLoading feature metadata...\")\n",
        "        meta_response = requests.get(f\"{base_url}/encoder/features.json\")\n",
        "        meta = meta_response.json()\n",
        "        encoder.feature_names = meta['feature_names']\n",
        "        encoder.fitted = meta['fitted']\n",
        "\n",
        "        return encoder\n",
        "\n",
        "    def save(self, path: str):\n",
        "        \"\"\"Save all encoder components.\"\"\"\n",
        "        if not self.fitted:\n",
        "            raise ValueError(\"Cannot save unfitted encoder\")\n",
        "\n",
        "        os.makedirs(path, exist_ok=True)\n",
        "\n",
        "        # Save each component using pickle\n",
        "        with open(os.path.join(path, 'onehot.pkl'), 'wb') as f:\n",
        "            pickle.dump(self.onehot_encoder, f)\n",
        "\n",
        "        with open(os.path.join(path, 'target.pkl'), 'wb') as f:\n",
        "            pickle.dump(self.target_encoder, f)\n",
        "\n",
        "        with open(os.path.join(path, 'price.pkl'), 'wb') as f:\n",
        "            pickle.dump(self.price_encoder, f)\n",
        "\n",
        "        # Save feature names\n",
        "        with open(os.path.join(path, 'features.json'), 'w') as f:\n",
        "            json.dump({\n",
        "                'feature_names': self.feature_names,\n",
        "                'fitted': self.fitted\n",
        "            }, f)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "HFBop-CRjpOm",
        "outputId": "4402a6f1-628a-4b8e-d003-3f9d16a4a15c"
      },
      "outputs": [],
      "source": [
        "# Set up model URLs\n",
        "MODEL_DIR = \"https://raw.githubusercontent.com/powell-clark/supervised-machine-learning/main/models\"\n",
        "BASE_URL = f\"{MODEL_DIR}/london_house_price_xgboost\"\n",
        "\n",
        "# Load metadata first for model info\n",
        "print(\"Loading saved model...\")\n",
        "meta_response = requests.get(f\"{BASE_URL}/metadata.json\")\n",
        "metadata = meta_response.json()\n",
        "print(f\"Model trained on: {metadata['training_date']}\")\n",
        "print(f\"Test MAE: £{metadata['metrics']['test_mae']:,.0f}\")\n",
        "\n",
        "# Load encoder and model\n",
        "print(\"\\nLoading encoder...\")\n",
        "encoder = CombinedEncoder.load_from_url(BASE_URL)\n",
        "\n",
        "print(\"\\nLoading XGBoost model...\")\n",
        "model = XGBRegressor()\n",
        "model_response = requests.get(f\"{BASE_URL}/model.json\")\n",
        "with open(\"temp_model.json\", \"wb\") as f:\n",
        "    f.write(model_response.content)\n",
        "model.load_model(\"temp_model.json\")\n",
        "\n",
        "# Test prediction\n",
        "print(\"\\nExample property details:\")\n",
        "test_property = pd.DataFrame({\n",
        "    'Area in sq ft': [1250],\n",
        "    'No. of Bedrooms': [3],\n",
        "    'House Type': ['House'],\n",
        "    'City/County': ['london'],\n",
        "    'Location': ['clapham'],\n",
        "    'Postal Code': ['SW4 0EX'],\n",
        "    'Outcode': ['SW4']\n",
        "})\n",
        "display(test_property)\n",
        "\n",
        "# Transform and predict\n",
        "features = encoder.transform(test_property)\n",
        "price = np.exp(model.predict(features))[0]\n",
        "print(f\"\\nPredicted price: £{price:,.0f}\")\n",
        "\n",
        "# Show encoded features\n",
        "print(\"\\nEncoded feature values:\")\n",
        "# Create DataFrame but keep values as numeric\n",
        "feature_values = pd.DataFrame({\n",
        "    'feature': encoder.feature_names,\n",
        "    'value': features.values[0]\n",
        "})\n",
        "\n",
        "# Format the display without modifying the underlying data\n",
        "pd.set_option('display.float_format', lambda x: '%.3f' % x)\n",
        "display(feature_values)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zneEqn_INdNI"
      },
      "source": [
        "<a name=\"production-implementation-review\"></a>\n",
        "## Production implementation review\n",
        "\n",
        "Our production implementation successfully combines multiple encoding strategies while maintaining clean separation of concerns.\n",
        "\n",
        "Let's analyse the key components:\n",
        "\n",
        "### 1. Feature engineering pipeline\n",
        "\n",
        "The `CombinedEncoder` handles three types of features:\n",
        "- Direct numeric features (area, bedrooms)\n",
        "- One-hot encoded categories (house type, city)\n",
        "- Target encoded locations with hierarchical fallback\n",
        "- Market rate features (price per square foot)\n",
        "\n",
        "Our test property in Clapham demonstrates how these work together:\n",
        "- Basic features match input (`Area in sq ft: 1,250`, `Bedrooms: 3`)\n",
        "- House type correctly one-hot encoded (`House: 1.0`, others `0.0`)\n",
        "- Location features show hierarchical encoding (`location_encoded: 14.119`)\n",
        "- Market rate captures local pricing (`price_per_sqft: £889.24`)\n",
        "\n",
        "### 2. Model performance\n",
        "\n",
        "The production model achieves:\n",
        "- Test MAE: £412,115\n",
        "- Reasonable prediction for Clapham house (£1,166,439)\n",
        "- Consistent feature importance patterns\n",
        "\n",
        "### 3. Production readiness\n",
        "\n",
        "The implementation provides:\n",
        "- Complete metadata tracking\n",
        "- Versioned model storage\n",
        "- Input validation\n",
        "- Consistent feature ordering\n",
        "\n",
        "### Next steps\n",
        "\n",
        "For full production deployment, we should add:\n",
        "1. Monitoring for feature drift\n",
        "2. Regular retraining triggers\n",
        "3. Performance alerting\n",
        "4. A/B testing framework\n",
        "\n",
        "In the next section, we'll explore how to implement these production monitoring and maintenance requirements."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0m2iedQNdNI"
      },
      "source": [
        "<a name=\"production-monitoring-and-maintenance\"></a>\n",
        "## Production monitoring and maintenance\n",
        "\n",
        "A house price prediction model is fundamentally measuring a dynamic system. Just as physicists need to calibrate their instruments as experimental conditions change, we need systematic methods to monitor and maintain our model's accuracy in London's evolving property market.\n",
        "\n",
        "### Understanding market dynamics\n",
        "\n",
        "Imagine you're an estate agent in early 2024. Three major changes hit the London market:\n",
        "\n",
        "1. **Crossrail opens fully**\n",
        "   ```\n",
        "   Before (2023):\n",
        "   2-bed flat in Woolwich: £375,000\n",
        "   Similar flat in Canary Wharf: £650,000\n",
        "   Price ratio: 1.73\n",
        "\n",
        "   After (2024):\n",
        "   Woolwich flat: £475,000 (+27%)\n",
        "   Canary Wharf flat: £675,000 (+4%)\n",
        "   New ratio: 1.42\n",
        "   ```\n",
        "   Our model needs to detect this location-specific shift.\n",
        "\n",
        "2. **Interest rate impact**\n",
        "   ```\n",
        "   £2M Chelsea house: -10% (wealthy buyers, less affected)\n",
        "   £400K Croydon flat: -15% (mortgage-dependent buyers)\n",
        "   £800K Hackney house: -12% (mixed buyer types)\n",
        "   ```\n",
        "   Price impacts vary by market segment.\n",
        "\n",
        "3. **New development patterns**\n",
        "   ```\n",
        "   Nine Elms:\n",
        "   2023: 80% luxury flats, 20% affordable\n",
        "   2024: 60% luxury flats, 40% affordable\n",
        "   → Input distribution has shifted\n",
        "   ```\n",
        "\n",
        "### Building a monitoring system\n",
        "\n",
        "Our monitoring system needs four key capabilities:\n",
        "\n",
        "1. **State tracking**\n",
        "\n",
        "   Like a physicist's lab notebook, we need to record everything:\n",
        "   ```python\n",
        "   state = {\n",
        "       'predictions': [           # Every prediction made\n",
        "           {\n",
        "               'property': '2 bed flat, E14',\n",
        "               'predicted': 550000,\n",
        "               'actual': 535000,\n",
        "               'date': '2024-03-15'\n",
        "           },\n",
        "           # ... thousands more predictions\n",
        "       ],\n",
        "       'baselines': {            # Statistical foundations\n",
        "           'E14_price_per_sqft': 750,\n",
        "           'typical_2bed_size': 750\n",
        "       },\n",
        "       'alerts': []              # System warnings\n",
        "   }\n",
        "   ```\n",
        "\n",
        "2. **Health checks**\n",
        "\n",
        "   Like medical vital signs, we monitor key indicators:\n",
        "   ```python\n",
        "   def check_model_health():\n",
        "       check_performance()     # Are predictions accurate?\n",
        "       check_features()        # Has input data shifted?\n",
        "       check_market()          # Are prices moving unusually?\n",
        "       check_data_quality()    # Is our data clean?\n",
        "   ```\n",
        "\n",
        "3. **Drift detection**\n",
        "\n",
        "   We need statistical rigor in measuring changes:\n",
        "   ```python\n",
        "   # Performance Drift\n",
        "   error_increase = (current_mae - baseline_mae) / baseline_mae\n",
        "   alert_if(error_increase > 0.15)  # 15% worse than baseline\n",
        "\n",
        "   # Feature Drift\n",
        "   dist_change = KL_divergence(current_dist, baseline_dist)\n",
        "   alert_if(dist_change > 0.30)     # Distribution shift > 30%\n",
        "\n",
        "   # Market Movement\n",
        "   price_change = abs(current_price - baseline_price) / baseline_price\n",
        "   alert_if(price_change > 0.25)    # 25% price movement\n",
        "   ```\n",
        "\n",
        "4. **Safe State Management**\n",
        "\n",
        "   Like a bank's transaction system, we need guaranteed consistency:\n",
        "   ```python\n",
        "   # Atomic state updates\n",
        "   write_to_temp_file(new_state)\n",
        "   backup_current_state()\n",
        "   atomic_rename(temp_file, current_state)\n",
        "   ```\n",
        "\n",
        "### Real-world testing\n",
        "\n",
        "We'll validate our system with three representative properties:\n",
        "\n",
        "1. **Prime central london**\n",
        "   ```\n",
        "   3-bed flat in Chelsea (SW3)\n",
        "   Area: 1,500 sq ft\n",
        "   Base price: £1,250,000\n",
        "   Expected volatility: ±5%\n",
        "   ```\n",
        "\n",
        "2. **Outer london value**\n",
        "   ```\n",
        "   2-bed flat in Croydon (CR0)\n",
        "   Area: 900 sq ft\n",
        "   Base price: £375,000\n",
        "   Expected volatility: ±3%\n",
        "   ```\n",
        "\n",
        "3. **Trendy east london**\n",
        "   ```\n",
        "   2-bed house in Hackney (E8)\n",
        "   Area: 1,100 sq ft\n",
        "   Base price: £750,000\n",
        "   Expected volatility: ±4%\n",
        "   ```\n",
        "\n",
        "We'll simulate 60 days of market activity, introducing realistic price movements and prediction errors.\n",
        "\n",
        "Let's implement this monitoring framework. While the code will be substantial, each component serves a clear purpose in maintaining our model's reliability - much like each instrument in a scientific laboratory helps maintain experimental accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0eCFwZvINdNI",
        "outputId": "d666e83d-a0e6-4e2c-fd15-80c75502f013"
      },
      "outputs": [],
      "source": [
        "# Set up project paths relative to notebook location\n",
        "NOTEBOOK_DIR = Path.cwd()  # final/\n",
        "PROJECT_ROOT = NOTEBOOK_DIR.parent  # Get parent of final/ to reach project root\n",
        "MODELS_DIR = PROJECT_ROOT / 'models'\n",
        "MONITORING_DIR = PROJECT_ROOT / 'monitoring'\n",
        "\n",
        "# Define monitoring subdirectories\n",
        "MONITORING_STRUCTURE = {\n",
        "    'predictions': MONITORING_DIR / 'predictions',\n",
        "    'reports': MONITORING_DIR / 'reports',\n",
        "    'alerts': MONITORING_DIR / 'alerts',\n",
        "    'state': MONITORING_DIR / 'state'\n",
        "}\n",
        "\n",
        "# Set up logging\n",
        "logging.basicConfig(\n",
        "    level=logging.INFO,\n",
        "    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'\n",
        ")\n",
        "logger = logging.getLogger(__name__)\n",
        "\n",
        "class ModelMonitoringError(Exception):\n",
        "    \"\"\"Custom exception for model monitoring errors.\"\"\"\n",
        "    pass\n",
        "\n",
        "@dataclass\n",
        "class MonitoringState:\n",
        "    \"\"\"Container for monitoring system state.\"\"\"\n",
        "    predictions_log: pd.DataFrame = field(default_factory=pd.DataFrame)\n",
        "    alerts_log: List[Dict] = field(default_factory=list)\n",
        "    feature_distributions: Dict = field(default_factory=dict)\n",
        "    last_saved: Optional[datetime] = None\n",
        "\n",
        "@dataclass\n",
        "class MonitoringThresholds:\n",
        "    \"\"\"Thresholds for triggering model updates and alerts.\"\"\"\n",
        "    max_mae_increase: float = 0.15\n",
        "    max_feature_drift: float = 0.30\n",
        "    min_prediction_volume: int = 100\n",
        "    max_prediction_age_days: int = 30\n",
        "    price_volatility_threshold: float = 0.25\n",
        "\n",
        "class LondonHousePriceMonitor:\n",
        "    \"\"\"Production monitoring system for London house price predictions.\"\"\"\n",
        "\n",
        "    def __init__(\n",
        "        self,\n",
        "        model_name: str = 'london_house_price_xgboost',\n",
        "        thresholds: Optional[MonitoringThresholds] = None\n",
        "    ):\n",
        "        self.model_path = MODELS_DIR / model_name\n",
        "        self.monitoring_dir = MONITORING_DIR\n",
        "        self.thresholds = thresholds or MonitoringThresholds()\n",
        "        self.state = MonitoringState()\n",
        "\n",
        "        # Initialise directories and load state\n",
        "        self._initialise_monitoring_system()\n",
        "\n",
        "    def _initialise_monitoring_system(self) -> None:\n",
        "        \"\"\"Initialise monitoring system and load state.\"\"\"\n",
        "        try:\n",
        "            # Create monitoring directories\n",
        "            for dir_path in MONITORING_STRUCTURE.values():\n",
        "                dir_path.mkdir(parents=True, exist_ok=True)\n",
        "\n",
        "            # Validate model path\n",
        "            if not self.model_path.exists():\n",
        "                raise ModelMonitoringError(f\"Model path does not exist: {self.model_path}\")\n",
        "\n",
        "            # Load model metadata and existing state\n",
        "            self.model_metadata = self._load_model_metadata()\n",
        "            self._load_existing_state()\n",
        "\n",
        "            logger.info(\"Monitoring system initialised successfully\")\n",
        "\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Failed to initialise monitoring system: {e}\")\n",
        "            raise ModelMonitoringError(f\"Monitoring system initialisation failed: {e}\")\n",
        "\n",
        "    def _load_model_metadata(self) -> Dict:\n",
        "        \"\"\"Load and validate model metadata.\"\"\"\n",
        "        metadata_path = self.model_path / 'metadata.json'\n",
        "        try:\n",
        "            if not metadata_path.exists():\n",
        "                raise ModelMonitoringError(f\"Model metadata not found: {metadata_path}\")\n",
        "\n",
        "            with open(metadata_path, 'r') as f:\n",
        "                metadata = json.load(f)\n",
        "\n",
        "            required_fields = ['metrics', 'feature_names', 'n_features']\n",
        "            missing_fields = [field for field in required_fields if field not in metadata]\n",
        "            if missing_fields:\n",
        "                raise ModelMonitoringError(\n",
        "                    f\"Missing required fields in metadata: {missing_fields}\"\n",
        "                )\n",
        "\n",
        "            return metadata\n",
        "\n",
        "        except Exception as e:\n",
        "            raise ModelMonitoringError(f\"Error loading model metadata: {e}\")\n",
        "\n",
        "    def _load_existing_state(self) -> None:\n",
        "        \"\"\"Load existing monitoring state if available.\"\"\"\n",
        "        try:\n",
        "            predictions_path = MONITORING_STRUCTURE['predictions'] / 'predictions_log.csv.gz'\n",
        "            if predictions_path.exists():\n",
        "                self.state.predictions_log = pd.read_csv(predictions_path)\n",
        "                self.state.predictions_log['timestamp'] = pd.to_datetime(\n",
        "                    self.state.predictions_log['timestamp']\n",
        "                )\n",
        "\n",
        "            alerts_path = MONITORING_STRUCTURE['alerts'] / 'alerts_log.json'\n",
        "            if alerts_path.exists():\n",
        "                with open(alerts_path, 'r') as f:\n",
        "                    self.state.alerts_log = json.load(f)\n",
        "\n",
        "            state_path = MONITORING_STRUCTURE['state'] / 'monitoring_state.json'\n",
        "            if state_path.exists():\n",
        "                with open(state_path, 'r') as f:\n",
        "                    state = json.load(f)\n",
        "                    self.state.feature_distributions = state.get('feature_distributions', {})\n",
        "\n",
        "            logger.info(\"Loaded existing monitoring state\")\n",
        "\n",
        "        except Exception as e:\n",
        "            logger.warning(f\"Error loading existing state (starting fresh): {e}\")\n",
        "            self.state = MonitoringState()\n",
        "\n",
        "    def log_prediction(\n",
        "        self,\n",
        "        property_details: Dict[str, Any],\n",
        "        predicted_price: float,\n",
        "        actual_price: Optional[float] = None\n",
        "    ) -> None:\n",
        "        \"\"\"Log each prediction for monitoring.\"\"\"\n",
        "        try:\n",
        "            # Validate inputs\n",
        "            if not isinstance(property_details, dict):\n",
        "                raise ValueError(\"property_details must be a dictionary\")\n",
        "\n",
        "            required_fields = {\n",
        "                'Area in sq ft': float,\n",
        "                'No. of Bedrooms': int,\n",
        "                'House Type': str,\n",
        "                'Outcode': str,\n",
        "                'City/County': str\n",
        "            }\n",
        "\n",
        "            for field, field_type in required_fields.items():\n",
        "                if field not in property_details:\n",
        "                    raise ValueError(f\"Missing required field: {field}\")\n",
        "                if not isinstance(property_details[field], field_type):\n",
        "                    raise ValueError(\n",
        "                        f\"Invalid type for {field}: expected {field_type.__name__}\"\n",
        "                    )\n",
        "\n",
        "            if not isinstance(predicted_price, (int, float)) or predicted_price <= 0:\n",
        "                raise ValueError(\"predicted_price must be a positive number\")\n",
        "\n",
        "            if actual_price is not None:\n",
        "                if not isinstance(actual_price, (int, float)) or actual_price <= 0:\n",
        "                    raise ValueError(\"actual_price must be a positive number\")\n",
        "\n",
        "            prediction_record = {\n",
        "                'timestamp': datetime.now(),\n",
        "                'predicted_price': predicted_price,\n",
        "                'actual_price': actual_price,\n",
        "                **property_details\n",
        "            }\n",
        "\n",
        "            # Append to predictions log efficiently\n",
        "            self.state.predictions_log = pd.concat([\n",
        "                self.state.predictions_log,\n",
        "                pd.DataFrame([prediction_record])\n",
        "            ], ignore_index=True)\n",
        "\n",
        "            # Save state periodically\n",
        "            if len(self.state.predictions_log) % 100 == 0:\n",
        "                self._save_state_safely()\n",
        "\n",
        "            logger.debug(f\"Logged prediction for {property_details['Outcode']}\")\n",
        "\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Error logging prediction: {e}\")\n",
        "            raise ModelMonitoringError(f\"Failed to log prediction: {e}\")\n",
        "\n",
        "    # Update the _save_state_safely method\n",
        "    def _save_state_safely(self) -> None:\n",
        "        \"\"\"Save monitoring state with atomic writes and backup.\"\"\"\n",
        "        state_path = MONITORING_STRUCTURE['state'] / 'monitoring_state.json'\n",
        "        temp_path = state_path.with_suffix('.tmp')\n",
        "        backup_path = state_path.with_suffix('.backup')\n",
        "\n",
        "        try:\n",
        "            # Prepare state data\n",
        "            state_data = {\n",
        "                'last_updated': datetime.now().isoformat(),\n",
        "                'predictions_count': len(self.state.predictions_log),\n",
        "                'alerts_count': len(self.state.alerts_log),\n",
        "                'feature_distributions': self.state.feature_distributions\n",
        "            }\n",
        "\n",
        "            # Write to temporary file first\n",
        "            with open(temp_path, 'w') as f:\n",
        "                json.dump(state_data, f, indent=2, default=str)\n",
        "\n",
        "            # Create backup of existing state if it exists\n",
        "            if state_path.exists():\n",
        "                shutil.copy2(state_path, backup_path)\n",
        "\n",
        "            # Atomic rename of temporary file\n",
        "            os.replace(temp_path, state_path)\n",
        "\n",
        "            # Save predictions log with compression\n",
        "            predictions_path = MONITORING_STRUCTURE['predictions'] / 'predictions_log.csv.gz'\n",
        "            self.state.predictions_log.to_csv(predictions_path, index=False, compression='gzip')\n",
        "\n",
        "            # Save alerts log\n",
        "            alerts_path = MONITORING_STRUCTURE['alerts'] / 'alerts_log.json'\n",
        "            with open(alerts_path, 'w') as f:\n",
        "                json.dump(self.state.alerts_log, f, indent=2, default=str)\n",
        "\n",
        "            # Clean up backup after successful save\n",
        "            if backup_path.exists():\n",
        "                os.remove(backup_path)  # Using os.remove instead of Path.unlink()\n",
        "\n",
        "            self.state.last_saved = datetime.now()\n",
        "            logger.info(\"State saved successfully\")\n",
        "\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Error saving state: {e}\")\n",
        "            # Restore from backup if available\n",
        "            if backup_path.exists() and not state_path.exists():\n",
        "                os.replace(backup_path, state_path)\n",
        "            raise ModelMonitoringError(f\"Failed to save monitoring state: {e}\")\n",
        "\n",
        "    def _get_recent_predictions(self) -> pd.DataFrame:\n",
        "        \"\"\"Get predictions within monitoring window.\"\"\"\n",
        "        if len(self.state.predictions_log) == 0:\n",
        "            return pd.DataFrame()\n",
        "\n",
        "        cutoff = datetime.now() - timedelta(days=self.thresholds.max_prediction_age_days)\n",
        "        return self.state.predictions_log[\n",
        "            self.state.predictions_log['timestamp'] >= cutoff\n",
        "        ].copy()\n",
        "\n",
        "    def _check_performance_drift(self, recent_data: pd.DataFrame) -> Dict[str, Any]:\n",
        "        \"\"\"Check if model performance has degraded.\"\"\"\n",
        "        if 'actual_price' not in recent_data.columns:\n",
        "            return {'healthy': True, 'reason': 'No ground truth available'}\n",
        "\n",
        "        try:\n",
        "            current_mae = mean_absolute_error(\n",
        "                recent_data['actual_price'],\n",
        "                recent_data['predicted_price']\n",
        "            )\n",
        "\n",
        "            baseline_mae = self.model_metadata['metrics']['test_mae']\n",
        "            mae_increase = (current_mae - baseline_mae) / baseline_mae\n",
        "\n",
        "            return {\n",
        "                'healthy': mae_increase < self.thresholds.max_mae_increase,\n",
        "                'metric': 'mae_increase',\n",
        "                'value': mae_increase,\n",
        "                'threshold': self.thresholds.max_mae_increase\n",
        "            }\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Error checking performance drift: {e}\")\n",
        "            return {'healthy': False, 'error': str(e)}\n",
        "\n",
        "    def _check_feature_drift(self, recent_data: pd.DataFrame) -> Dict[str, Any]:\n",
        "        \"\"\"Check for feature distribution drift.\"\"\"\n",
        "        try:\n",
        "            drift_results = {}\n",
        "\n",
        "            # Check numeric features\n",
        "            for feature in ['Area in sq ft', 'No. of Bedrooms']:\n",
        "                current_stats = {\n",
        "                    'mean': float(recent_data[feature].mean()),\n",
        "                    'std': float(recent_data[feature].std())\n",
        "                }\n",
        "\n",
        "                # Ensure baseline is a dictionary\n",
        "                baseline = (self.state.feature_distributions.get(feature, {})\n",
        "                        if isinstance(self.state.feature_distributions.get(feature), dict)\n",
        "                        else {})\n",
        "\n",
        "                if not baseline:\n",
        "                    self.state.feature_distributions[feature] = current_stats\n",
        "                    drift_results[feature] = {'drift': 0.0, 'significant': False}\n",
        "                    continue\n",
        "\n",
        "                mean_drift = abs(current_stats['mean'] - baseline['mean']) / baseline['mean']\n",
        "                drift_results[feature] = {\n",
        "                    'drift': float(mean_drift),\n",
        "                    'significant': mean_drift > self.thresholds.max_feature_drift\n",
        "                }\n",
        "\n",
        "            # Check categorical features\n",
        "            for feature in ['House Type', 'Outcode']:\n",
        "                current_dist = recent_data[feature].value_counts(normalize=True).to_dict()\n",
        "\n",
        "                # Ensure baseline is a dictionary\n",
        "                baseline_dist = (self.state.feature_distributions.get(feature, {})\n",
        "                            if isinstance(self.state.feature_distributions.get(feature), dict)\n",
        "                            else {})\n",
        "\n",
        "                if not baseline_dist:\n",
        "                    self.state.feature_distributions[feature] = current_dist\n",
        "                    drift_results[feature] = {'drift': 0.0, 'significant': False}\n",
        "                    continue\n",
        "\n",
        "                # Calculate distribution difference\n",
        "                all_categories = set(current_dist.keys()) | set(baseline_dist.keys())\n",
        "                total_drift = sum(\n",
        "                    abs(current_dist.get(cat, 0) - baseline_dist.get(cat, 0))\n",
        "                    for cat in all_categories\n",
        "                ) / 2\n",
        "\n",
        "                drift_results[feature] = {\n",
        "                    'drift': float(total_drift),\n",
        "                    'significant': total_drift > self.thresholds.max_feature_drift\n",
        "                }\n",
        "\n",
        "            return {\n",
        "                'healthy': not any(r['significant'] for r in drift_results.values()),\n",
        "                'drift_metrics': drift_results\n",
        "            }\n",
        "\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Error checking feature drift: {e}\")\n",
        "            return {'healthy': False, 'error': str(e)}\n",
        "\n",
        "\n",
        "    def _check_market_conditions(self, recent_data: pd.DataFrame) -> Dict[str, Any]:\n",
        "        \"\"\"Check for significant market changes.\"\"\"\n",
        "        try:\n",
        "            market_metrics = {}\n",
        "\n",
        "            # Calculate price per square foot\n",
        "            price_col = 'actual_price' if 'actual_price' in recent_data.columns else 'predicted_price'\n",
        "            recent_data['price_per_sqft'] = recent_data[price_col] / recent_data['Area in sq ft']\n",
        "\n",
        "            # Global metrics\n",
        "            current_global_mean = recent_data['price_per_sqft'].mean()\n",
        "            baseline_global_mean = self.model_metadata.get('metrics', {}).get(\n",
        "                'baseline_mean_price',\n",
        "                current_global_mean\n",
        "            )\n",
        "\n",
        "            global_change = abs(current_global_mean / baseline_global_mean - 1)\n",
        "            market_metrics['global'] = {\n",
        "                'change': global_change,\n",
        "                'volatile': global_change > self.thresholds.price_volatility_threshold\n",
        "            }\n",
        "\n",
        "            # Area-specific metrics\n",
        "            area_metrics = {}\n",
        "            for area in recent_data['Outcode'].unique():\n",
        "                area_data = recent_data[recent_data['Outcode'] == area]\n",
        "                if len(area_data) < 10:  # Skip areas with insufficient data\n",
        "                    continue\n",
        "\n",
        "                current_mean = area_data['price_per_sqft'].mean()\n",
        "                baseline = self.state.feature_distributions.get('price_per_sqft', {}).get(\n",
        "                    area,\n",
        "                    current_mean\n",
        "                )\n",
        "\n",
        "                change = abs(current_mean / baseline - 1)\n",
        "                area_metrics[area] = {\n",
        "                    'change': change,\n",
        "                    'volatile': change > self.thresholds.price_volatility_threshold\n",
        "                }\n",
        "\n",
        "            market_metrics['areas'] = area_metrics\n",
        "\n",
        "            return {\n",
        "                'healthy': not (market_metrics['global']['volatile'] or\n",
        "                              any(m['volatile'] for m in area_metrics.values())),\n",
        "                'metrics': market_metrics\n",
        "            }\n",
        "\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Error checking market conditions: {e}\")\n",
        "            return {'healthy': False, 'error': str(e)}\n",
        "\n",
        "    def _check_data_quality(self, recent_data: pd.DataFrame) -> Dict[str, Any]:\n",
        "        \"\"\"Check for data quality issues.\"\"\"\n",
        "        try:\n",
        "            quality_checks = {\n",
        "                'missing_values': recent_data.isnull().mean().max() < 0.1,\n",
        "                'area_bounds': recent_data['Area in sq ft'].between(100, 10000).all(),\n",
        "                'bedroom_bounds': recent_data['No. of Bedrooms'].between(1, 10).all(),\n",
        "            }\n",
        "\n",
        "            if 'actual_price' in recent_data.columns:\n",
        "                quality_checks['price_bounds'] = recent_data['actual_price'].between(\n",
        "                    100000, 50000000  # £100K to £50M\n",
        "                ).all()\n",
        "\n",
        "            return {\n",
        "                'healthy': all(quality_checks.values()),\n",
        "                'failed_checks': [\n",
        "                    check for check, healthy in quality_checks.items()\n",
        "                    if not healthy\n",
        "                ]\n",
        "            }\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Error checking data quality: {e}\")\n",
        "            return {'healthy': False, 'error': str(e)}\n",
        "\n",
        "    def check_model_health(self) -> Dict[str, Any]:\n",
        "        \"\"\"Run comprehensive model health checks.\"\"\"\n",
        "        try:\n",
        "            if len(self.state.predictions_log) < self.thresholds.min_prediction_volume:\n",
        "                return {\n",
        "                    'healthy': True,\n",
        "                    'status': 'insufficient_data',\n",
        "                    'message': f\"Need {self.thresholds.min_prediction_volume} predictions, have {len(self.state.predictions_log)}\"\n",
        "                }\n",
        "\n",
        "            recent_data = self._get_recent_predictions()\n",
        "            if len(recent_data) == 0:\n",
        "                return {\n",
        "                    'healthy': True,\n",
        "                    'status': 'no_recent_data',\n",
        "                    'message': f\"No predictions in last {self.thresholds.max_prediction_age_days} days\"\n",
        "                }\n",
        "\n",
        "            # Run all health checks\n",
        "            health_checks = {\n",
        "                'performance': self._check_performance_drift(recent_data),\n",
        "                'features': self._check_feature_drift(recent_data),\n",
        "                'market': self._check_market_conditions(recent_data),\n",
        "                'quality': self._check_data_quality(recent_data)\n",
        "            }\n",
        "\n",
        "            # Determine overall health\n",
        "            is_healthy = all(check.get('healthy', False) for check in health_checks.values())\n",
        "\n",
        "            if not is_healthy:\n",
        "                self._trigger_alert(health_checks)\n",
        "\n",
        "            return {\n",
        "                'healthy': is_healthy,\n",
        "                'checks': health_checks,\n",
        "                'timestamp': datetime.now().isoformat()\n",
        "            }\n",
        "\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Error in health check: {e}\")\n",
        "            return {\n",
        "                'healthy': False,\n",
        "                'error': str(e),\n",
        "                'timestamp': datetime.now().isoformat()\n",
        "            }\n",
        "\n",
        "    def _trigger_alert(self, check_results: Dict) -> None:\n",
        "        \"\"\"Log alert and trigger notifications.\"\"\"\n",
        "        alert = {\n",
        "            'timestamp': datetime.now().isoformat(),\n",
        "            'check_results': check_results,\n",
        "            'prediction_volume': len(self.state.predictions_log),\n",
        "            'monitoring_window': self.thresholds.max_prediction_age_days\n",
        "        }\n",
        "\n",
        "        self.state.alerts_log.append(alert)\n",
        "        self._save_state_safely()\n",
        "\n",
        "        print(f\"ALERT: Model health check failed at {alert['timestamp']}\")\n",
        "        for check, result in check_results.items():\n",
        "            if check != 'healthy' and not result['healthy']:\n",
        "                print(f\"- {check}: {result}\")\n",
        "\n",
        "    def generate_monitoring_report(self) -> Dict[str, Any]:\n",
        "        \"\"\"Generate comprehensive monitoring report.\"\"\"\n",
        "        try:\n",
        "            recent_data = self._get_recent_predictions()\n",
        "\n",
        "            if len(recent_data) == 0:\n",
        "                return {\n",
        "                    'timestamp': datetime.now().isoformat(),\n",
        "                    'status': 'no_recent_data',\n",
        "                    'prediction_volume': 0\n",
        "                }\n",
        "\n",
        "            report = {\n",
        "                'timestamp': datetime.now().isoformat(),\n",
        "                'monitoring_period': {\n",
        "                    'start': recent_data['timestamp'].min().isoformat(),\n",
        "                    'end': recent_data['timestamp'].max().isoformat()\n",
        "                },\n",
        "                'prediction_volume': len(recent_data),\n",
        "                'health_check': self.check_model_health(),\n",
        "                'performance_metrics': self._calculate_performance_metrics(recent_data),\n",
        "                'data_quality_metrics': self._calculate_quality_metrics(recent_data),\n",
        "                'market_indicators': self._calculate_market_indicators(recent_data)\n",
        "            }\n",
        "\n",
        "            # Serialise the entire report\n",
        "            serialised_report = self._serialise_for_json(report)\n",
        "\n",
        "            # Save report\n",
        "            report_name = f\"monitoring_report_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json\"\n",
        "            report_path = MONITORING_STRUCTURE['reports'] / report_name\n",
        "            with open(report_path, 'w') as f:\n",
        "                json.dump(serialised_report, f, indent=2)\n",
        "\n",
        "            return serialised_report\n",
        "\n",
        "        except Exception as e:\n",
        "            logger.error(f\"Error generating monitoring report: {e}\")\n",
        "            return {\n",
        "                'timestamp': datetime.now().isoformat(),\n",
        "                'status': 'error',\n",
        "                'error': str(e)\n",
        "            }\n",
        "\n",
        "    def _calculate_market_indicators(self, data: pd.DataFrame) -> Dict:\n",
        "        \"\"\"Calculate market trend indicators.\"\"\"\n",
        "        price_col = 'actual_price' if 'actual_price' in data.columns else 'predicted_price'\n",
        "        data['price_per_sqft'] = data[price_col] / data['Area in sq ft']\n",
        "\n",
        "        market_metrics = {\n",
        "            'price_per_sqft': {\n",
        "                'mean': data['price_per_sqft'].mean(),\n",
        "                'median': data['price_per_sqft'].median(),\n",
        "                'std': data['price_per_sqft'].std()\n",
        "            }\n",
        "        }\n",
        "\n",
        "        # Calculate area-specific metrics\n",
        "        for area in data['Outcode'].unique():\n",
        "            area_data = data[data['Outcode'] == area]\n",
        "            if len(area_data) >= 10:  # Minimum sample size\n",
        "                baseline = self.state.feature_distributions.get('price_per_sqft', {}).get(\n",
        "                    area, area_data['price_per_sqft'].mean()\n",
        "                )\n",
        "                market_metrics[f'area_{area}'] = {\n",
        "                    'volume': len(area_data),\n",
        "                    'mean_price_per_sqft': area_data['price_per_sqft'].mean(),\n",
        "                    'price_movement': (\n",
        "                        area_data['price_per_sqft'].mean() / baseline - 1\n",
        "                    )\n",
        "                }\n",
        "\n",
        "        return market_metrics\n",
        "\n",
        "    def _calculate_performance_metrics(self, data: pd.DataFrame) -> Dict:\n",
        "        \"\"\"Calculate detailed performance metrics.\"\"\"\n",
        "        if 'actual_price' not in data.columns:\n",
        "            return {'status': 'no_ground_truth'}\n",
        "\n",
        "        metrics = {\n",
        "            'mae': mean_absolute_error(data['actual_price'], data['predicted_price']),\n",
        "            'rmse': np.sqrt(mean_squared_error(\n",
        "                data['actual_price'],\n",
        "                data['predicted_price']\n",
        "            )),\n",
        "            'r2': r2_score(data['actual_price'], data['predicted_price'])\n",
        "        }\n",
        "\n",
        "        if len(data) >= 5:\n",
        "            data['price_band'] = pd.qcut(data['actual_price'], q=5)\n",
        "            metrics['price_bands'] = {\n",
        "                str(band): {\n",
        "                    'mae': mean_absolute_error(\n",
        "                        group['actual_price'],\n",
        "                        group['predicted_price']\n",
        "                    ),\n",
        "                    'count': len(group)\n",
        "                }\n",
        "                for band, group in data.groupby('price_band', observed=True)  # Add observed=True\n",
        "            }\n",
        "\n",
        "        return metrics\n",
        "\n",
        "    def _calculate_quality_metrics(self, data: pd.DataFrame) -> Dict:\n",
        "        \"\"\"Calculate data quality metrics.\"\"\"\n",
        "        return {\n",
        "            'missing_values': data.isnull().mean().to_dict(),\n",
        "            'value_ranges': {\n",
        "                col: {\n",
        "                    'min': data[col].min(),\n",
        "                    'max': data[col].max(),\n",
        "                    'mean': data[col].mean(),\n",
        "                    'std': data[col].std()\n",
        "                }\n",
        "                for col in data.select_dtypes(include=[np.number]).columns\n",
        "            },\n",
        "            'outliers': {\n",
        "                col: len(data[\n",
        "                    (data[col] < (data[col].quantile(0.25) - 1.5 * (data[col].quantile(0.75) - data[col].quantile(0.25)))) |\n",
        "                    (data[col] > (data[col].quantile(0.75) + 1.5 * (data[col].quantile(0.75) - data[col].quantile(0.25))))\n",
        "                ])\n",
        "                for col in ['Area in sq ft', 'No. of Bedrooms'] if col in data.columns\n",
        "            }\n",
        "        }\n",
        "\n",
        "\n",
        "    @staticmethod\n",
        "    def _serialise_for_json(obj: Any) -> Any:\n",
        "        \"\"\"Convert objects to JSON-serialisable format.\"\"\"\n",
        "        if isinstance(obj, (pd.Timestamp, datetime)):\n",
        "            return obj.isoformat()\n",
        "        elif isinstance(obj, (np.int64, np.int32)):\n",
        "            return int(obj)\n",
        "        elif isinstance(obj, (np.float64, np.float32)):\n",
        "            return float(obj)\n",
        "        elif isinstance(obj, pd.Interval):\n",
        "            return str(obj)\n",
        "        elif isinstance(obj, bool):\n",
        "            return bool(obj)\n",
        "        elif isinstance(obj, dict):\n",
        "            return {str(k): LondonHousePriceMonitor._serialise_for_json(v)\n",
        "                    for k, v in obj.items()}\n",
        "        elif isinstance(obj, (list, tuple)):\n",
        "            return [LondonHousePriceMonitor._serialise_for_json(x) for x in obj]\n",
        "        elif isinstance(obj, np.bool_):  # Updated numpy boolean handling\n",
        "            return bool(obj)\n",
        "        elif hasattr(obj, '__dict__'):\n",
        "            return str(obj)\n",
        "        return obj\n",
        "\n",
        "\n",
        "def simulate_london_predictions(days_back: int = 60) -> LondonHousePriceMonitor:\n",
        "    \"\"\"Simulate sequence of predictions over time period.\"\"\"\n",
        "    try:\n",
        "        test_properties = [\n",
        "            {\n",
        "                'Area in sq ft': float(1500),\n",
        "                'No. of Bedrooms': int(3),\n",
        "                'House Type': str('Flat/Apartment'),\n",
        "                'Outcode': str('SW3'),\n",
        "                'City/County': str('london'),\n",
        "                'base_price': float(1250000),\n",
        "                'price_volatility': float(0.05)\n",
        "            },\n",
        "            {\n",
        "                'Area in sq ft': float(900),\n",
        "                'No. of Bedrooms': int(2),\n",
        "                'House Type': str('Flat/Apartment'),\n",
        "                'Outcode': str('CR0'),\n",
        "                'City/County': str('london'),\n",
        "                'base_price': float(375000),\n",
        "                'price_volatility': float(0.03)\n",
        "            },\n",
        "            {\n",
        "                'Area in sq ft': float(1100),\n",
        "                'No. of Bedrooms': int(2),\n",
        "                'House Type': str('House'),\n",
        "                'Outcode': str('E8'),\n",
        "                'City/County': str('london'),\n",
        "                'base_price': float(750000),\n",
        "                'price_volatility': float(0.04)\n",
        "            }\n",
        "        ]\n",
        "\n",
        "        monitor = LondonHousePriceMonitor()\n",
        "        start_date = datetime.now() - timedelta(days=days_back)\n",
        "        simulation_dates = pd.date_range(start_date, datetime.now(), freq='D')\n",
        "\n",
        "        np.random.seed(42)\n",
        "\n",
        "        for date in simulation_dates:\n",
        "            for prop in test_properties:\n",
        "                # Simulate market movement\n",
        "                market_factor = 1 + np.random.normal(0, prop['price_volatility'])\n",
        "                actual_price = float(prop['base_price'] * market_factor)\n",
        "\n",
        "                # Add prediction error\n",
        "                prediction_error = np.random.normal(0, 0.1)\n",
        "                predicted_price = float(actual_price * (1 + prediction_error))\n",
        "\n",
        "                # Create property details\n",
        "                property_details = {\n",
        "                    k: v for k, v in prop.items()\n",
        "                    if k not in ['base_price', 'price_volatility']\n",
        "                }\n",
        "\n",
        "                # Log prediction\n",
        "                monitor.log_prediction(\n",
        "                    property_details=property_details,\n",
        "                    predicted_price=predicted_price,\n",
        "                    actual_price=actual_price\n",
        "                )\n",
        "\n",
        "        logger.info(f\"Successfully simulated {len(simulation_dates)} days of predictions\")\n",
        "        return monitor\n",
        "\n",
        "    except Exception as e:\n",
        "        logger.error(f\"Error in simulation: {e}\")\n",
        "        raise\n",
        "\n",
        "def run_simulation_example():\n",
        "    \"\"\"Run complete simulation example with report generation.\"\"\"\n",
        "    try:\n",
        "        monitor = simulate_london_predictions(days_back=60)\n",
        "        report = monitor.generate_monitoring_report()\n",
        "\n",
        "        print(\"\\nSimulation Monitoring Report:\")\n",
        "        print(json.dumps(report, indent=2))\n",
        "\n",
        "        recent_data = monitor._get_recent_predictions()\n",
        "        print(\"\\nSimulation Statistics:\")\n",
        "        print(f\"Total predictions: {len(monitor.state.predictions_log)}\")\n",
        "        print(f\"Recent predictions: {len(recent_data)}\")\n",
        "        print(f\"Unique areas: {recent_data['Outcode'].nunique()}\")\n",
        "        print(\n",
        "            f\"Average prediction error: \"\n",
        "            f\"£{abs(recent_data['predicted_price'] - recent_data['actual_price']).mean():,.2f}\"\n",
        "        )\n",
        "\n",
        "    except Exception as e:\n",
        "        logger.error(f\"Error running simulation example: {e}\")\n",
        "        raise\n",
        "\n",
        "# Initialise monitoring system\n",
        "if __name__ == \"__main__\":\n",
        "    try:\n",
        "        print(f\"Initialising monitoring system...\")\n",
        "        print(f\"Project root: {PROJECT_ROOT}\")\n",
        "        print(f\"Models directory: {MODELS_DIR}\")\n",
        "        print(f\"Monitoring directory: {MONITORING_DIR}\")\n",
        "\n",
        "        for name, path in MONITORING_STRUCTURE.items():\n",
        "            path.mkdir(parents=True, exist_ok=True)\n",
        "            print(f\"Created {name} directory: {path}\")\n",
        "\n",
        "        # Run simulation\n",
        "        run_simulation_example()\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Warning: Could not initialise monitoring structure: {e}\")\n",
        "\n",
        "\n",
        "\n",
        "def run_enhanced_simulation(days_back: int = 60) -> LondonHousePriceMonitor:\n",
        "    \"\"\"Run enhanced simulation with alert-triggering scenarios after baseline.\"\"\"\n",
        "    print(\"\\nRunning Enhanced Simulation with Alert Scenarios...\")\n",
        "\n",
        "    test_properties = [\n",
        "        {\n",
        "            'Area in sq ft': float(1500),\n",
        "            'No. of Bedrooms': int(3),\n",
        "            'House Type': str('Flat/Apartment'),\n",
        "            'Outcode': str('SW3'),\n",
        "            'City/County': str('london'),\n",
        "            'base_price': float(1250000),\n",
        "            'price_volatility': float(0.05)\n",
        "        },\n",
        "        {\n",
        "            'Area in sq ft': float(900),\n",
        "            'No. of Bedrooms': int(2),\n",
        "            'House Type': str('Flat/Apartment'),\n",
        "            'Outcode': str('CR0'),\n",
        "            'City/County': str('london'),\n",
        "            'base_price': float(375000),\n",
        "            'price_volatility': float(0.03)\n",
        "        },\n",
        "        {\n",
        "            'Area in sq ft': float(1100),\n",
        "            'No. of Bedrooms': int(2),\n",
        "            'House Type': str('House'),\n",
        "            'Outcode': str('E8'),\n",
        "            'City/County': str('london'),\n",
        "            'base_price': float(750000),\n",
        "            'price_volatility': float(0.04)\n",
        "        }\n",
        "    ]\n",
        "\n",
        "    monitor = LondonHousePriceMonitor()\n",
        "    start_date = datetime.now() - timedelta(days=days_back)\n",
        "    simulation_dates = pd.date_range(start_date, datetime.now(), freq='D')\n",
        "\n",
        "    # Define shock dates relative to simulation timeline\n",
        "    shock_dates = {\n",
        "        'market_shock': start_date + timedelta(days=int(days_back * 0.4)),\n",
        "        'feature_drift': start_date + timedelta(days=int(days_back * 0.6)),\n",
        "        'performance_decay': start_date + timedelta(days=int(days_back * 0.7)),\n",
        "        'quality_issues': start_date + timedelta(days=int(days_back * 0.8))\n",
        "    }\n",
        "\n",
        "    np.random.seed(42)\n",
        "\n",
        "    def apply_market_shock(prop, date):\n",
        "        if prop['Outcode'] == 'SW3' and date >= shock_dates['market_shock']:\n",
        "            return prop['base_price'] * 0.70\n",
        "        return prop['base_price']\n",
        "\n",
        "    def apply_feature_drift(area, date):\n",
        "        if date >= shock_dates['feature_drift']:\n",
        "            return area * 1.35\n",
        "        return area\n",
        "\n",
        "    def apply_performance_decay(predicted_price, date):\n",
        "        if date >= shock_dates['performance_decay']:\n",
        "            return predicted_price * 1.20\n",
        "        return predicted_price\n",
        "\n",
        "    def apply_quality_issues(prop, date):\n",
        "        if date >= shock_dates['quality_issues'] and np.random.random() < 0.15:\n",
        "            prop['Area in sq ft'] = np.nan\n",
        "        return prop\n",
        "\n",
        "    alerts_triggered = {\n",
        "        'market': False,\n",
        "        'features': False,\n",
        "        'performance': False,\n",
        "        'quality': False\n",
        "    }\n",
        "\n",
        "    print(\"\\nSimulating with shock dates:\")\n",
        "    for shock_type, shock_date in shock_dates.items():\n",
        "        print(f\"{shock_type}: {shock_date.strftime('%Y-%m-%d')}\")\n",
        "\n",
        "    for date in simulation_dates:\n",
        "        for prop in test_properties:\n",
        "            # Create copy of property for this iteration\n",
        "            current_prop = prop.copy()\n",
        "\n",
        "            # Apply normal market movement\n",
        "            base_price = apply_market_shock(current_prop, date)\n",
        "            market_factor = 1 + np.random.normal(0, current_prop['price_volatility'])\n",
        "            actual_price = float(base_price * market_factor)\n",
        "\n",
        "            # Apply feature drift\n",
        "            current_prop['Area in sq ft'] = apply_feature_drift(current_prop['Area in sq ft'], date)\n",
        "\n",
        "            # Apply quality issues\n",
        "            current_prop = apply_quality_issues(current_prop, date)\n",
        "\n",
        "            # Generate prediction with potential decay\n",
        "            prediction_error = np.random.normal(0, 0.1)\n",
        "            predicted_price = float(actual_price * (1 + prediction_error))\n",
        "            predicted_price = apply_performance_decay(predicted_price, date)\n",
        "\n",
        "            # Create property details for logging\n",
        "            property_details = {\n",
        "                k: v for k, v in current_prop.items()\n",
        "                if k not in ['base_price', 'price_volatility']\n",
        "            }\n",
        "\n",
        "            # Log prediction\n",
        "            monitor.log_prediction(\n",
        "                property_details=property_details,\n",
        "                predicted_price=predicted_price,\n",
        "                actual_price=actual_price\n",
        "            )\n",
        "\n",
        "        # Check for alerts monthly and at end of simulation\n",
        "        if date.day == 1 or date == simulation_dates[-1]:\n",
        "            health_report = monitor.check_model_health()\n",
        "\n",
        "            if not health_report['healthy']:\n",
        "                for check_type, check_result in health_report['checks'].items():\n",
        "                    if not check_result.get('healthy', True) and not alerts_triggered[check_type]:\n",
        "                        alerts_triggered[check_type] = True\n",
        "                        print(f\"\\nAlert triggered on {date.strftime('%Y-%m-%d')} - {check_type}:\")\n",
        "                        print(f\"Details: {check_result}\")\n",
        "\n",
        "    # Print final simulation statistics\n",
        "    recent_data = monitor._get_recent_predictions()\n",
        "    print(\"\\nEnhanced Simulation Statistics:\")\n",
        "    print(f\"Total predictions: {len(monitor.state.predictions_log)}\")\n",
        "    print(f\"Recent predictions: {len(recent_data)}\")\n",
        "    print(f\"Unique areas: {recent_data['Outcode'].nunique()}\")\n",
        "    print(f\"Average prediction error: £{abs(recent_data['predicted_price'] - recent_data['actual_price']).mean():,.2f}\")\n",
        "\n",
        "    print(\"\\nAlerts Summary:\")\n",
        "    for alert_type, triggered in alerts_triggered.items():\n",
        "        print(f\"{alert_type}: {'✓' if triggered else 'x'} triggered\")\n",
        "\n",
        "    return monitor\n",
        "\n",
        "# Run both simulations\n",
        "if __name__ == \"__main__\":\n",
        "    try:\n",
        "        print(\"Running baseline simulation...\")\n",
        "        baseline_monitor = simulate_london_predictions()\n",
        "        print(\"\\nBaseline simulation complete. Running enhanced simulation...\")\n",
        "        enhanced_monitor = run_enhanced_simulation()\n",
        "\n",
        "        # Generate and compare reports\n",
        "        baseline_report = baseline_monitor.generate_monitoring_report()\n",
        "        enhanced_report = enhanced_monitor.generate_monitoring_report()\n",
        "\n",
        "        print(\"\\nComparison of Simulations:\")\n",
        "        print(\"Baseline Metrics:\")\n",
        "        print(f\"R² Score: {baseline_report['performance_metrics']['r2']:.3f}\")\n",
        "        print(f\"MAE: £{baseline_report['performance_metrics']['mae']:,.2f}\")\n",
        "\n",
        "        print(\"\\nEnhanced Simulation Metrics (with alerts):\")\n",
        "        print(f\"R² Score: {enhanced_report['performance_metrics']['r2']:.3f}\")\n",
        "        print(f\"MAE: £{enhanced_report['performance_metrics']['mae']:,.2f}\")\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error running simulations: {e}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QhTtnmc6NdNJ"
      },
      "source": [
        "<a name=\"monitoring-system-simulations\"></a>\n",
        "## Monitoring system simulations\n",
        "\n",
        "### Test design and implementation\n",
        "\n",
        "We conducted two distinct simulations to comprehensively validate our monitoring system:\n",
        "\n",
        "#### Baseline test (Normal conditions)\n",
        "Three representative properties were monitored over 60 days:\n",
        "```python\n",
        "Premium:    Chelsea 3-bed Flat   (£1.25M, ±5% volatility)\n",
        "Mid-Market: Hackney 2-bed House  (£750K,  ±4% volatility)\n",
        "Entry:      Croydon 2-bed Flat   (£375K,  ±3% volatility)\n",
        "```\n",
        "\n",
        "Each property was simulated with realistic market movements and prediction errors of ±10%.\n",
        "\n",
        "### Enhanced test (alert scenarios)\n",
        "Built on the baseline test but introduced four controlled issues:\n",
        "- Market Shock: 30% price drop in Chelsea (Day 24)\n",
        "- Feature Drift: 35% increase in property sizes (Day 36)\n",
        "- Performance Decay: 20% prediction error increase (Day 42)\n",
        "- Quality Issues: 15% missing area values (Day 48)\n",
        "\n",
        "### Test results\n",
        "\n",
        "#### Baseline simulation\n",
        "The baseline test demonstrated stable system operation:\n",
        "\n",
        "1. **Volume & coverage**\n",
        "   - 1,983 total predictions processed\n",
        "   - Consistent coverage across 3 areas\n",
        "   - Average prediction error: £65,709.20\n",
        "\n",
        "2. **Performance metrics**\n",
        "   ```\n",
        "   R² Score: 0.938\n",
        "   MAE: £65,737.83\n",
        "   Health Check: All passed\n",
        "   ```\n",
        "\n",
        "3. **Market stability**\n",
        "   ```\n",
        "   SW3 (Chelsea):  0.01% movement\n",
        "   E8  (Hackney):  0.10% movement\n",
        "   CR0 (Croydon):  0.50% movement\n",
        "   ```\n",
        "\n",
        "4. **Feature distribution**\n",
        "   - Area in sq ft: 0.17% drift\n",
        "   - Bedrooms: 0.17% drift\n",
        "   - Property types: No drift\n",
        "   - Outcodes: No drift\n",
        "\n",
        "#### Enhanced simulation\n",
        "The enhanced test revealed system behaviour under stress:\n",
        "\n",
        "1. **Volume & coverage**\n",
        "   - 2,183 total predictions\n",
        "   - Maintained area coverage\n",
        "   - Average error increased to £67,766.89\n",
        "\n",
        "2. **Performance impact**\n",
        "   ```\n",
        "   R² Score: 0.932 (↓0.006)\n",
        "   MAE: £67,766.89 (↑£2,029.06)\n",
        "   ```\n",
        "\n",
        "3. **Alert system performance**\n",
        "   ```\n",
        "   Market Alert:      ✗ Not triggered (threshold: 25% change)\n",
        "   Feature Alert:     ✗ Not triggered (threshold: 30% drift)\n",
        "   Performance Alert: ✗ Not triggered (threshold: 15% degradation)\n",
        "   Quality Alert:     ✓ Triggered on 2024-12-01\n",
        "   ```\n",
        "\n",
        "4. **Alert details**\n",
        "   ```json\n",
        "   {\n",
        "     \"healthy\": false,\n",
        "     \"failed_checks\": [\"area_bounds\"],\n",
        "     \"timestamp\": \"2024-12-01\"\n",
        "   }\n",
        "   ```\n",
        "\n",
        "### Key findings\n",
        "\n",
        "1. **System stability**\n",
        "   - Baseline test showed consistent performance\n",
        "   - Error patterns align with price bands\n",
        "   - Geographic coverage maintained throughout\n",
        "\n",
        "2. **Alert sensitivity**\n",
        "   - Quality checks most sensitive\n",
        "   - Market movement thresholds may need adjustment\n",
        "   - Feature drift detection robust\n",
        "\n",
        "3. **Performance impact**\n",
        "   - Quality issues caused 3.1% MAE increase\n",
        "   - R² impact relatively minor (0.6% decrease)\n",
        "   - System remained functional under stress\n",
        "\n",
        "4. **Alert behaviour**\n",
        "   - Clear alert messaging\n",
        "   - Proper timestamp tracking\n",
        "   - Appropriate detail level\n",
        "\n",
        "### Recommendations\n",
        "\n",
        "1. **Threshold adjustments**\n",
        "   - Consider lowering market movement threshold\n",
        "   - Review feature drift sensitivity\n",
        "   - Maintain quality check sensitivity\n",
        "\n",
        "2. **Alert enhancements**\n",
        "   - Add alert severity levels\n",
        "   - Implement alert aggregation\n",
        "   - Include trend information\n",
        "\n",
        "3. **System improvements**\n",
        "   - Add seasonal pattern detection\n",
        "   - Implement area correlation tracking\n",
        "   - Extend monitoring timeframe\n",
        "\n",
        "### Simulation limitations\n",
        "\n",
        "1. **Market dynamics**\n",
        "   - Simple random price movements\n",
        "   - No seasonal patterns\n",
        "   - Independent area behaviour\n",
        "\n",
        "2. **Property mix**\n",
        "   - Limited property types\n",
        "   - Fixed locations\n",
        "   - Small property set\n",
        "\n",
        "3. **Time scale**\n",
        "   - 60-day window only\n",
        "   - No long-term trends\n",
        "   - Limited market cycles\n",
        "\n",
        "Despite these limitations, the simulations demonstrated the monitoring system's capability to:\n",
        "- Track key performance metrics\n",
        "- Detect data quality issues\n",
        "- Maintain performance under stress\n",
        "- Generate actionable alerts\n",
        "\n",
        "The system is ready for production deployment with suggested enhancements to be implemented based on real-world performance data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R8NMKSvpNdNK"
      },
      "source": [
        "<a name=\"ethical-considerations\"></a>\n",
        "## Ethical considerations\n",
        "\n",
        "Throughout this lesson, we've built increasingly sophisticated models for predicting house prices - from simple decision trees (MAE: £566,853) to advanced XGBoost models (MAE: £474,797). But these aren't just numbers. Each prediction can profoundly impact individual lives and communities. Let's explore the ethical implications of deploying these models in the real world.\n",
        "\n",
        "### 1. Individual impact: The human cost of predictions\n",
        "\n",
        "Consider a family applying for a mortgage on a £500,000 home. Our best model's average error of £474,797 could:\n",
        "\n",
        "- **Over-valuation impact**:\n",
        "  ```\n",
        "  Predicted: £600,000 (+£100,000)\n",
        "  → Higher down payment required\n",
        "  → Increased property tax assessment\n",
        "  → Inflated insurance premiums\n",
        "  Result: Family priced out of their chosen home\n",
        "  ```\n",
        "\n",
        "- **Under-valuation impact**:\n",
        "  ```\n",
        "  Predicted: £400,000 (-£100,000)\n",
        "  → Mortgage application rejected\n",
        "  → Forced to seek higher interest alternatives\n",
        "  → Reduced seller interest\n",
        "  Result: Denied access to housing opportunity\n",
        "  ```\n",
        "\n",
        "### 2. Protected characteristics in housing\n",
        "\n",
        "Our models could inadvertently discriminate based on protected characteristics through proxy variables. For example:\n",
        "\n",
        "```python\n",
        "# Problematic feature importance pattern\n",
        "location_importance = {\n",
        "    'SW1': 0.15,  # Wealthy area\n",
        "    'E14': 0.08,  # Mixed demographics\n",
        "    'SE15': 0.03  # More diverse area\n",
        "}\n",
        "```\n",
        "\n",
        "Even if we don't directly use protected characteristics, geographic features can encode:\n",
        "- Race and ethnicity patterns\n",
        "- Religious community concentrations\n",
        "- Socioeconomic status\n",
        "- Age demographics\n",
        "- Disability accommodation availability\n",
        "\n",
        "### 3. Systemic effects and feedback loops\n",
        "\n",
        "Our target encoding approach revealed a critical ethical challenge:\n",
        "\n",
        "```python\n",
        "# Target encoding can amplify historical patterns\n",
        "encoded_location = (\n",
        "    mean_location_price * (count / (count + smoothing)) +\n",
        "    mean_global_price * (smoothing / (count + smoothing))\n",
        ")\n",
        "```\n",
        "\n",
        "This encoding:\n",
        "1. Uses historical prices to predict future prices\n",
        "2. Gives more weight to areas with more data\n",
        "3. Can perpetuate historical inequities\n",
        "\n",
        "### 4. Model performance disparities\n",
        "\n",
        "Our experiments revealed concerning performance variations:\n",
        "\n",
        "| Price band    | Sample size | Relative error | Impact                   |\n",
        "|---------------|-------------|---------------|---------------------------|\n",
        "| Under £500K   | 15%         | 18%           | Entry-level buyers hurt   |\n",
        "| £500K-£1M     | 45%         | 12%           | Middle market favored     |\n",
        "| Over £1M      | 40%         | 15%           | Luxury market variation   |\n",
        "\n",
        "### 5. Required safeguards\n",
        "\n",
        "Based on these concerns, any deployment must include:\n",
        "\n",
        "1. **Prediction confidence**\n",
        "   ```python\n",
        "   def get_prediction_risk(value, error, threshold=0.15):\n",
        "       relative_error = error / value\n",
        "       return 'HIGH_RISK' if relative_error > threshold else 'ACCEPTABLE'\n",
        "   ```\n",
        "\n",
        "2. **Human review triggers**\n",
        "   - Predictions > 20% from comparable sales\n",
        "   - Areas with sparse training data\n",
        "   - Unusual property characteristics\n",
        "   - High-stakes decisions (e.g., mortgage approval)\n",
        "\n",
        "3. **Impact monitoring**\n",
        "   - Track outcomes by neighborhood demographics\n",
        "   - Measure approval rate disparities\n",
        "   - Monitor price trend amplification\n",
        "   - Regular bias audits\n",
        "\n",
        "4. **Transparency requirements**\n",
        "   - Clear model version identification\n",
        "   - Feature importance disclosure\n",
        "   - Confidence interval reporting\n",
        "   - Appeal process documentation\n",
        "\n",
        "### 6. Decision framework\n",
        "\n",
        "For each prediction, we must ask:\n",
        "\n",
        "1. **Impact assessment**\n",
        "   - Who could be harmed by this prediction?\n",
        "   - How severe is the potential harm?\n",
        "   - Are impacts disproportionate across groups?\n",
        "\n",
        "2. **Reliability check**\n",
        "   - Do we have sufficient relevant data?\n",
        "   - Are our assumptions valid here?\n",
        "   - What are our uncertainty bounds?\n",
        "\n",
        "3. **Mitigation planning**\n",
        "   - How can we minimize potential harm?\n",
        "   - What safeguards should we implement?\n",
        "   - When should we decline to make predictions?\n",
        "\n",
        "### Key takeaways\n",
        "\n",
        "1. **Technical metrics aren't enough**\n",
        "   - MAE of £474,797 means different things to different people\n",
        "   - R² of 0.76 doesn't capture fairness\n",
        "   - Need multiple evaluation lenses\n",
        "\n",
        "2. **Impact varies by context**\n",
        "   - Same error has different implications across price ranges\n",
        "   - Geographic patterns require careful monitoring\n",
        "   - System effects compound over time\n",
        "\n",
        "3. **Responsibility is ongoing**\n",
        "   - Initial validation isn't sufficient\n",
        "   - Must monitor real-world impacts\n",
        "   - Need clear update/retirement criteria\n",
        "\n",
        "As we move toward deployment in ATLAS, these ethical considerations will inform our monitoring framework design."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "mLtOQLd5NdNK",
        "outputId": "7e321076-4aad-4a28-8c9c-30f4087791f2"
      },
      "outputs": [],
      "source": [
        "def analyse_market_rate_fairness(train_data: pd.DataFrame,\n",
        "                               test_data: pd.DataFrame,\n",
        "                               tree_params: Dict,\n",
        "                               price_thresholds: List[float] = [500000, 1000000]) -> Dict:\n",
        "    \"\"\"Analyze fairness of market rate model across different price bands.\n",
        "\n",
        "    Args:\n",
        "        train_data: Training DataFrame\n",
        "        test_data: Test DataFrame\n",
        "        tree_params: Decision tree parameters\n",
        "        price_thresholds: List of price thresholds for market segments\n",
        "\n",
        "    Returns:\n",
        "        Dictionary containing fairness metrics and analysis results\n",
        "    \"\"\"\n",
        "    # Create base encoder for house type and city/county\n",
        "    base_encoder = OneHotFeatureEncoder(\n",
        "        numeric_features=['Area in sq ft', 'No. of Bedrooms'],\n",
        "        categorical_features=['House Type', 'City/County']\n",
        "    )\n",
        "\n",
        "    # Create price per sqft encoder\n",
        "    price_encoder = MeanOutcodePricePerSquareFootEncoder()\n",
        "    price_per_sqft_train = price_encoder.fit_transform(train_data)\n",
        "    price_per_sqft_test = price_encoder.transform(test_data)\n",
        "\n",
        "    # Combine features\n",
        "    X_train = pd.concat([\n",
        "        base_encoder.fit_transform(train_data),\n",
        "        pd.DataFrame({'price_per_sqft': price_per_sqft_train})\n",
        "    ], axis=1)\n",
        "\n",
        "    X_test = pd.concat([\n",
        "        base_encoder.transform(test_data),\n",
        "        pd.DataFrame({'price_per_sqft': price_per_sqft_test})\n",
        "    ], axis=1)\n",
        "\n",
        "    # Train model\n",
        "    tree = DecisionTreeRegressor(**tree_params)\n",
        "    tree.fit(X_train, train_data['log_price'])\n",
        "\n",
        "    # Get predictions\n",
        "    test_pred = tree.predict(X_test)\n",
        "    pred_prices = np.exp(test_pred)\n",
        "    true_prices = np.exp(test_data['log_price'])\n",
        "\n",
        "    # Calculate overall metrics\n",
        "    metrics = {\n",
        "        'overall': {\n",
        "            'mae': mean_absolute_error(true_prices, pred_prices),\n",
        "            'relative_error': np.mean(np.abs((true_prices - pred_prices) / true_prices)),\n",
        "            'count': len(true_prices)\n",
        "        },\n",
        "        'by_band': {}\n",
        "    }\n",
        "\n",
        "    # Analyse each price band\n",
        "    bands = ['Entry', 'Mid-Market', 'Premium']\n",
        "    thresholds = [0] + price_thresholds + [float('inf')]\n",
        "\n",
        "    for i, (lower, upper) in enumerate(zip(thresholds[:-1], thresholds[1:])):\n",
        "        mask = (true_prices >= lower) & (true_prices < upper)\n",
        "        band_true = true_prices[mask]\n",
        "        band_pred = pred_prices[mask]\n",
        "\n",
        "        if len(band_true) > 0:\n",
        "            metrics['by_band'][bands[i]] = {\n",
        "                'count': len(band_true),\n",
        "                'mae': mean_absolute_error(band_true, band_pred),\n",
        "                'relative_error': np.mean(np.abs((band_true - band_pred) / band_true)),\n",
        "                'systematic_bias': np.mean(band_pred - band_true),\n",
        "                'mean_price_per_sqft': price_per_sqft_test[mask].mean(),\n",
        "                'areas': test_data.loc[mask, 'Outcode'].value_counts().to_dict()\n",
        "            }\n",
        "\n",
        "    # Calculate area-specific performance\n",
        "    metrics['area_analysis'] = {}\n",
        "    for outcode in test_data['Outcode'].unique():\n",
        "        mask = test_data['Outcode'] == outcode\n",
        "        if mask.sum() >= 5:  # Only analyse areas with sufficient data\n",
        "            area_true = true_prices[mask]\n",
        "            area_pred = pred_prices[mask]\n",
        "            metrics['area_analysis'][outcode] = {\n",
        "                'count': len(area_true),\n",
        "                'mae': mean_absolute_error(area_true, area_pred),\n",
        "                'relative_error': np.mean(np.abs((area_true - area_pred) / area_true)),\n",
        "                'mean_price': area_true.mean(),\n",
        "                'price_per_sqft': price_per_sqft_test[mask].mean()\n",
        "            }\n",
        "\n",
        "    # Print analysis\n",
        "    print(\"\\nMarket Rate Model Fairness Analysis:\")\n",
        "    print(f\"Overall MAE: £{metrics['overall']['mae']:,.0f}\")\n",
        "    print(f\"Overall Relative Error: {metrics['overall']['relative_error']:.1%}\")\n",
        "\n",
        "    print(\"\\nPerformance by Price Band:\")\n",
        "    for band, band_metrics in metrics['by_band'].items():\n",
        "        print(f\"\\n{band} Market (n={band_metrics['count']}):\")\n",
        "        print(f\"MAE: £{band_metrics['mae']:,.0f}\")\n",
        "        print(f\"Relative Error: {band_metrics['relative_error']:.1%}\")\n",
        "        print(f\"Systematic Bias: £{band_metrics['systematic_bias']:,.0f}\")\n",
        "        print(f\"Mean £/sqft: £{band_metrics['mean_price_per_sqft']:,.0f}\")\n",
        "        print(f\"Areas: {', '.join(sorted(band_metrics['areas'].keys()))}\")\n",
        "\n",
        "        if abs(band_metrics['systematic_bias']) > band_metrics['mae'] * 0.2:\n",
        "            print(\"⚠️  WARNING: Possible systematic bias detected\")\n",
        "\n",
        "    return metrics\n",
        "\n",
        "# Function to visualise the fairness analysis\n",
        "def plot_market_rate_fairness(metrics: Dict) -> None:\n",
        "    \"\"\"Create visualisation of market rate model fairness metrics.\"\"\"\n",
        "    plt.figure(figsize=(15, 6))\n",
        "\n",
        "    # Plot 1: Performance by Price Band\n",
        "    plt.subplot(121)\n",
        "    bands = list(metrics['by_band'].keys())\n",
        "    rel_errors = [m['relative_error'] for m in metrics['by_band'].values()]\n",
        "    plt.bar(bands, rel_errors)\n",
        "    plt.title('Relative Error by Price Band')\n",
        "    plt.ylabel('Relative Error')\n",
        "    plt.xticks(rotation=45)\n",
        "\n",
        "    # Plot 2: Error vs Price/sqft\n",
        "    plt.subplot(122)\n",
        "    # Fix: Use 'price_per_sqft' instead of 'mean_price_per_sqft'\n",
        "    price_sqft = [m['price_per_sqft'] for m in metrics['area_analysis'].values()]\n",
        "    rel_errors = [m['relative_error'] for m in metrics['area_analysis'].values()]\n",
        "    plt.scatter(price_sqft, rel_errors, alpha=0.6)\n",
        "    plt.title('Error vs Price per Square Foot')\n",
        "    plt.xlabel('Price per Square Foot (£)')\n",
        "    plt.ylabel('Relative Error')\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n",
        "\n",
        "# Run the analysis\n",
        "print(\"\\nAnalyzing market rate model fairness...\")\n",
        "fairness_metrics = analyse_market_rate_fairness(train_data, test_data, tree_params)\n",
        "plot_market_rate_fairness(fairness_metrics)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dTXsP-hUNdNK"
      },
      "source": [
        "<a name=\"testing-for-fairness\"></a>\n",
        "## Testing for fairness\n",
        "\n",
        "While we're working with a limited Kaggle dataset, our fairness testing reveals important patterns that illuminate both the potential and challenges of automated valuation systems. The insights we've gained, even from this simplified view of London's housing market, provide valuable lessons about building more equitable systems.\n",
        "\n",
        "### Building a meaningful test\n",
        "\n",
        "We started with a straightforward but informative test framework. Our approach divides properties into three natural market segments: entry level (under £500K), mid-market (£500K to £1M), and premium properties (above £1M). For each segment, we measure absolute error, relative error, systematic bias, and how our predictions cluster geographically.\n",
        "\n",
        "The moment we created these segments, we ran into our first crucial challenge: these boundaries are fundamentally artificial. A property valued at £495K shares far more characteristics with one at £505K than one at £300K, yet our segmentation treats them differently. This tension between our need to categorise and the continuous nature of property values echoes throughout our analysis.\n",
        "\n",
        "### What the numbers tell us\n",
        "\n",
        "Our test revealed systematic patterns that we can't ignore:\n",
        "\n",
        "```python\n",
        "Entry:    22.9% error, +£61K bias    # Consistently overvalued\n",
        "Mid:      19.2% error, +£63K bias    # Most reliable predictions\n",
        "Premium:  21.6% error, -£300K bias   # Significant undervaluation\n",
        "```\n",
        "\n",
        "These aren't just statistics - they tell a story about how our model treats different parts of the market. That £61K overvaluation in the entry market could mean telling a first-time buyer they can't afford a house that should be within their reach. The strong mid-market performance likely reflects our training data's sweet spot, where properties are more standardised and transactions more frequent. The premium market's substantial undervaluation suggests our model struggles with the unique features and micro-location factors that drive high-end property values.\n",
        "\n",
        "### Potential solutions and their challenges\n",
        "\n",
        "Our analysis suggests several promising approaches, each with its own advantages and significant challenges to consider.\n",
        "\n",
        "The first and most obvious solution is market segmentation - building separate models for different price bands. This approach lets us optimise for segment-specific features and handle different value scales more appropriately. A model focused purely on entry-level properties could learn the nuances of first-time buyer locations and property types. However, this immediately raises complex boundary problems:\n",
        "\n",
        "```python\n",
        "def get_valuation(property_details):\n",
        "    # Calculate baseline value\n",
        "    baseline = estimate_rough_value(property_details)\n",
        "    \n",
        "    # Blend predictions near boundaries\n",
        "    if near_boundary(baseline):\n",
        "        models = get_relevant_models(baseline)\n",
        "        return weighted_blend(models, property_details)\n",
        "    \n",
        "    return get_primary_model(baseline).predict(property_details)\n",
        "```\n",
        "\n",
        "This code looks straightforward, but hides significant complexity. How do we handle a property valued at £495K this month that might be worth £505K next month? Should it suddenly be evaluated by a completely different model? We could implement smooth blending between models near boundaries, but this adds another layer of complexity to our system.\n",
        "\n",
        "A second approach focuses on confidence-based routing. Instead of segmenting by price, we route predictions based on how confident our model is:\n",
        "\n",
        "- High confidence → Automated valuation\n",
        "- Medium confidence → Quick human review\n",
        "- Low confidence → Full manual valuation\n",
        "\n",
        "\n",
        "This creates a natural scaling mechanism and focuses human expertise where it's most needed. But it raises its own challenges: How do we define confidence meaningfully? What metrics should we use? Setting these thresholds too high wastes human resources; too low risks automated mistakes.\n",
        "\n",
        "Our third option involves enhanced feature engineering - tailoring our features to different market segments. Location might matter differently for a £300K flat versus a £3M house. School quality might be crucial in family-home price ranges but less relevant for luxury properties. This approach offers better predictive power through more focused features, but maintenance becomes complex. Features that matter in one price range might be irrelevant or even misleading in another.\n",
        "\n",
        "Consider how the importance of outdoor space varies:\n",
        "```python\n",
        "def calculate_garden_value(property_details, price_band):\n",
        "    if price_band == 'premium':\n",
        "        return detailed_garden_analysis(property_details)\n",
        "    elif price_band == 'mid':\n",
        "        return simple_garden_metric(property_details)\n",
        "    else:\n",
        "        return has_any_outdoor_space(property_details)\n",
        "```\n",
        "\n",
        "This kind of feature engineering must balance granularity against maintainability. More sophisticated features might improve predictions but make the system harder to update and monitor.\n",
        "\n",
        "### Practical steps forward\n",
        "\n",
        "Given these challenges, any deployed system needs to strike a careful balance. The model should serve as a support tool rather than a decision maker, with clear communication of uncertainty and thoughtful integration with human expertise. We need explicit fairness metrics, regular monitoring, and clear paths for handling edge cases and appeals.\n",
        "\n",
        "The potential for feedback loops presents a particular concern. If our model's predictions influence market prices, which then influence future predictions, we could inadvertently amplify biases over time. Regular analysis of segment-specific performance becomes crucial, as does active engagement with estate agents who understand local market dynamics.\n",
        "\n",
        "### Learning from limited data\n",
        "\n",
        "The limitations of our Kaggle dataset shouldn't stop us from taking these insights seriously. They point toward crucial considerations for production systems:\n",
        "\n",
        "- Model roles need clear definition - supporting, not replacing, human judgment\n",
        "- System design must include fairness metrics and monitoring from day one\n",
        "- Edge cases and appeals require clear handling procedures\n",
        "- Stakeholder engagement becomes a crucial part of system maintenance\n",
        "\n",
        "### Moving forward\n",
        "\n",
        "As we move to examine technical limitations, these fairness considerations provide essential context. A model's technical capabilities must be balanced against its potential market impact. The challenge isn't just building accurate models - it's building systems that serve all market participants fairly while acknowledging their own limitations.\n",
        "\n",
        "Behind every prediction is a person making one of the most important financial decisions of their life. Our models need to help make that decision fairer and more informed, not add new barriers to an already challenging process. Understanding these fairness implications isn't just an ethical requirement - it's a crucial part of building systems that work effectively in the real world."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9jQgEkDXNdNM"
      },
      "source": [
        "\n",
        "<a name=\"model-limitations-and-considerations\"></a>\n",
        "## Model limitations and considerations\n",
        "\n",
        "Throughout this lesson, we've built increasingly sophisticated models for London house price prediction. Our progression revealed fundamental trade-offs between model complexity, feature information, and real-world applicability that we must understand before deployment.\n",
        "\n",
        "### 1. The price information paradox\n",
        "\n",
        "Our best model achieved impressive metrics:\n",
        "```python\n",
        "Market Rate Model Performance:\n",
        "Train R²: 0.913  Test R²: 0.878\n",
        "Train MAE: £346,919  Test MAE: £425,485\n",
        "```\n",
        "\n",
        "But this superior performance comes from a subtle but important source: we're using price-derived features (mean outcode price per square foot) that, while properly separated between training and test sets, fundamentally encode price information.\n",
        "\n",
        "Think of it like this:\n",
        "```python\n",
        "# Price-independent feature (location only)\n",
        "outcode_SW1 = 1  # Simple fact: property is in SW1\n",
        "\n",
        "# Price-derived feature (even with proper train/test split)\n",
        "outcode_SW1_price = 1250  # £/sqft from training data\n",
        "```\n",
        "\n",
        "While we avoid leakage by using only training data means, we're still encoding the fundamental insight that \"SW1 properties are expensive.\" This creates two important effects:\n",
        "\n",
        "1. **Stability vs Adaptability Trade-off**\n",
        "   ```python\n",
        "   # Stable but slow to adapt\n",
        "   if outcode == 'SW1':\n",
        "       base_price = training_mean * sqft  # Uses historical patterns\n",
        "   ```\n",
        "\n",
        "2. **Feedback Loop Risk**\n",
        "   ```python\n",
        "   # Potential amplification\n",
        "   historical_prices → model predictions → market expectations\n",
        "                   → actual prices → future predictions\n",
        "   ```\n",
        "\n",
        "### 2. The R² Reality\n",
        "\n",
        "Our model progression revealed an important pattern:\n",
        "\n",
        "```\n",
        "Simple Features (One-hot):\n",
        "Train R²: 0.790  Test R²: 0.805  Gap: +0.015\n",
        "\n",
        "Target Encoding:\n",
        "Train R²: 0.883  Test R²: 0.833  Gap: -0.050\n",
        "\n",
        "Market Rate Features:\n",
        "Train R²: 0.913  Test R²: 0.878  Gap: -0.035\n",
        "```\n",
        "\n",
        "This pattern tells us something crucial:\n",
        "1. Price-independent features show stable train/test performance\n",
        "2. Price-derived features achieve higher R² but show overfitting\n",
        "3. The \"better\" models might be less robust to market changes\n",
        "\n",
        "### 3. Geographic coverage limitations\n",
        "\n",
        "Our location encoding revealed structural limitations:\n",
        "\n",
        "```\n",
        "Data Density:\n",
        "├── Chelsea (SW3): 96 properties → Strong mean price signal\n",
        "├── Hackney (E8): 43 properties → Moderate price signal\n",
        "└── Outer areas: <10 properties → Weak price signal\n",
        "```\n",
        "\n",
        "In areas with sparse data:\n",
        "1. Price-derived features fall back to broader averages\n",
        "2. One-hot encoding might actually be more reliable\n",
        "3. Model confidence should reflect data density\n",
        "\n",
        "### 4. Feature engineering vs model choice: A critical trade-off\n",
        "\n",
        "Perhaps our most important discovery was how different feature encodings fundamentally change what our models can learn. Let's analyse this through the lens of a property valuer's decision-making process:\n",
        "\n",
        "1. **One-hot encoding: The local expert**\n",
        "   ```python\n",
        "   Features:\n",
        "   area_mayfair = 1      # Binary fact: property is in Mayfair\n",
        "   area_chelsea = 0      # Binary fact: not in Chelsea\n",
        "   ```\n",
        "   Like a valuer who knows \"this is a Mayfair property\" but hasn't looked at recent sales:\n",
        "   - Most stable: immune to market swings\n",
        "   - High dimensionality (663 features): needs lots of examples\n",
        "   - Can't generalise to new areas\n",
        "   - R²: 0.805 - Good but not great\n",
        "\n",
        "2. **Target encoding: The experienced valuer**\n",
        "   ```python\n",
        "   Features:\n",
        "   mayfair_value = 14.2  # Log-price encoding from training\n",
        "   chelsea_value = 13.9  # Learnt price levels\n",
        "   ```\n",
        "   Like a valuer who knows \"Mayfair properties tend to be worth more than Chelsea\":\n",
        "   - Captures price relationships\n",
        "   - Needs significant data to be reliable\n",
        "   - Can adapt to new areas through hierarchy\n",
        "   - R²: 0.833 - Better but less interpretable\n",
        "\n",
        "3. **Market rate features: The market analyst**\n",
        "   ```python\n",
        "   Features:\n",
        "   mayfair_price_sqft = 2500  # £/sqft from training data\n",
        "   chelsea_price_sqft = 2200  # Direct price signals\n",
        "   ```\n",
        "   Like an analyst who knows exact £/sqft rates for each area:\n",
        "   - Most precise in stable markets\n",
        "   - Compact (19 features)\n",
        "   - But dangerous in changing markets\n",
        "   - R²: 0.878 - Best performance, highest risk\n",
        "\n",
        "### 5. How different models use these features\n",
        "\n",
        "The real insight comes from seeing how different models handle these features:\n",
        "\n",
        "1. **Decision trees: The rule-based valuer**\n",
        "   ```python\n",
        "   if area == 'Mayfair' and sqft > 1000:\n",
        "       return 'Band A pricing'  # Sharp boundaries\n",
        "   ```\n",
        "   - Great with one-hot: creates clear area-based rules\n",
        "   - Poor with market rates: creates arbitrary price boundaries\n",
        "   - Can't interpolate between areas\n",
        "\n",
        "2. **Random Forest: The agency team**\n",
        "   ```python\n",
        "   valuations = []\n",
        "   for valuer in team:\n",
        "       # Each tree sees different areas/prices\n",
        "       valuations.append(valuer.estimate())\n",
        "   final = mean(valuations)  # Team consensus\n",
        "   ```\n",
        "   - Handles mixed signals well\n",
        "   - More robust to market changes\n",
        "   - But loses clear decision boundaries\n",
        "\n",
        "3. **XGBoost: The temporal learning algorithm**\n",
        "   ```python\n",
        "   for transaction in history:\n",
        "       error = actual_price - prediction\n",
        "       model.improve(error)  # Progressive refinement\n",
        "   ```\n",
        "   - Best with market rates\n",
        "   - Learns subtle price patterns\n",
        "   - Most sensitive to market changes\n",
        "\n",
        "### 6. The real-world impact\n",
        "\n",
        "This isn't just theoretical. Consider three real scenarios we discovered:\n",
        "\n",
        "1. **Crossrail opening**\n",
        "   ```python\n",
        "   # One-hot encoding: Blind to change\n",
        "   woolwich_2023 = 1  # Same feature value\n",
        "   woolwich_2024 = 1  # Doesn't capture impact\n",
        "   \n",
        "   # Market rate features: Dangerous lag\n",
        "   price_2023 = 450_000  # Historical average\n",
        "   actual_2024 = 550_000  # Post-Crossrail\n",
        "   ```\n",
        "   One-hot models maintain stability but miss opportunities. Market rate models lag reality.\n",
        "\n",
        "2. **Interest rate impact**\n",
        "   ```python\n",
        "   # Target encoding amplifies trends\n",
        "   prime_london_2023 = 14.2  # Log price encoding\n",
        "   prime_london_2024 = 14.0  # Encodes decline\n",
        "   next_prediction = 13.9    # Amplifies trend\n",
        "   ```\n",
        "   Models with price features can accelerate market movements.\n",
        "\n",
        "3. **New development**\n",
        "   ```python\n",
        "   # No historical data\n",
        "   nine_elms = {\n",
        "       'one_hot': None,           # Can't handle new area\n",
        "       'target': parent_area,     # Uses hierarchy\n",
        "       'market': nearby_rates     # Uses local rates\n",
        "   }\n",
        "   ```\n",
        "\n",
        "This leads us to a crucial insight: no single combination of features and model is perfect. Each carries specific risks that must be actively managed.\n",
        "\n",
        "### The solution: Active risk management\n",
        "\n",
        "Rather than choosing one approach, we need a framework that combines their strengths:\n",
        "\n",
        "```python\n",
        "def get_valuation(property_details, market_state):\n",
        "    # 1. Get multiple opinions\n",
        "    stable_prediction = onehot_model.predict()      # No price signals\n",
        "    market_prediction = rate_model.predict()        # Current rates\n",
        "    smooth_prediction = target_model.predict()      # Smoothed history\n",
        "    \n",
        "    # 2. Check for disagreement\n",
        "    spread = max_difference([stable_prediction,\n",
        "                            market_prediction,\n",
        "                            smooth_prediction])\n",
        "    \n",
        "    if spread > acceptable_range:\n",
        "        return \"manual_review_needed\"\n",
        "        \n",
        "    # 3. Weight by market conditions\n",
        "    weights = get_market_weights(market_state)\n",
        "    return weighted_average(predictions, weights)\n",
        "```\n",
        "\n",
        "This framework:\n",
        "1. Uses all three feature types\n",
        "2. Monitors their disagreement\n",
        "3. Adapts to market conditions\n",
        "4. Knows when to ask for help\n",
        "\n",
        "In ATLAS, we'll build exactly this kind of adaptive system, learning from these fundamental limitations to create something more robust than any single model could be."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pVpMG6o8NdNM"
      },
      "source": [
        "<a name=\"conclusion\"></a>\n",
        "## Conclusion\n",
        "\n",
        "Throughout this lesson, we've taken a fascinating journey from theoretical understanding of decision trees to implementing production-ready house price prediction models. The London housing market has served as our proving ground, offering complex challenges that pushed us to develop increasingly sophisticated solutions.\n",
        "\n",
        "### Key achievements\n",
        "\n",
        "We've developed several crucial capabilities in machine learning for real estate:\n",
        "\n",
        "1. **Data-driven property valuation**\n",
        "   Our journey began with raw property data and evolved into a systematic approach for value prediction:\n",
        "   - Transformed messy real estate data into clean, predictive features\n",
        "   - Built robust location hierarchies from postal codes to neighborhoods\n",
        "   - Captured subtle market dynamics through engineered features\n",
        "   - Developed validation strategies that respect temporal and spatial dependencies\n",
        "\n",
        "2. **Model evolution and performance**\n",
        "   We've seen how each model type brings unique strengths to the challenge:\n",
        "   ```python\n",
        "   # Starting with basic trees\n",
        "   basic_tree = DecisionTreeRegressor()\n",
        "   rmse_basic = 1.89  # log price scale\n",
        "   \n",
        "   # Progressing to random forests\n",
        "   rf_model = RandomForestRegressor(n_estimators=100)\n",
        "   rmse_rf = 1.52     # 19.6% improvement\n",
        "   \n",
        "   # Advancing to XGBoost\n",
        "   xgb_model = XGBRegressor(n_estimators=100)\n",
        "   rmse_xgb = 1.41    # 25.4% improvement\n",
        "   ```\n",
        "\n",
        "3. **Production readiness**\n",
        "   Perhaps most importantly, we've built systems that can work in the real world:\n",
        "   - Engineered robust feature pipelines that handle messy data\n",
        "   - Implemented model persistence for deployment\n",
        "   - Designed monitoring systems for performance tracking\n",
        "   - Created prediction serving infrastructure\n",
        "\n",
        "### Real estate insights gained\n",
        "\n",
        "Our modeling journey revealed fascinating patterns in London's housing market:\n",
        "\n",
        "1. **Price drivers**\n",
        "   The relative importance of different factors emerged clearly:\n",
        "   ```python\n",
        "   feature_importance = {\n",
        "       'location': 0.40,     # Location hierarchy\n",
        "       'size': 0.30,         # Square footage\n",
        "       'property_type': 0.15, # House, flat, etc.\n",
        "       'other': 0.15         # Including market conditions\n",
        "   }\n",
        "   ```\n",
        "\n",
        "2. **Market dynamics**\n",
        "   We discovered several key patterns:\n",
        "   - Granular location effects show high variability but strong predictive power\n",
        "   - Borough-level trends provide stable baseline predictions\n",
        "   - Property type impacts vary significantly by area\n",
        "   - Size value follows non-linear patterns across locations\n",
        "\n",
        "3. **Prediction challenges**\n",
        "   Some interesting edge cases emerged:\n",
        "   - Properties over £5M show unique pricing patterns\n",
        "   - New developments require special handling\n",
        "   - Location effects can override other features\n",
        "   - Market timing significantly impacts accuracy\n",
        "\n",
        "### From theory to practice\n",
        "\n",
        "We've successfully bridged theoretical understanding with practical implementation:\n",
        "\n",
        "1. **Decision tree theory → implementation**\n",
        "   ```python\n",
        "   # Theory: Split on Information Gain\n",
        "   # Practice:\n",
        "   model = DecisionTreeRegressor(\n",
        "       criterion='squared_error',  # MSE for regression\n",
        "       min_samples_split=20,      # Prevent overfitting\n",
        "       max_depth=10               # Control tree complexity\n",
        "   )\n",
        "   ```\n",
        "\n",
        "2. **Cross-validation theory → real testing**\n",
        "   ```python\n",
        "   # Theory: K-fold validation\n",
        "   # Practice:\n",
        "   cv = KFold(n_splits=5, shuffle=True)\n",
        "   scores = cross_val_score(model, X, y, cv=cv)\n",
        "   ```\n",
        "\n",
        "3. **Feature engineering theory → market features**\n",
        "   ```python\n",
        "   # Theory: Target encoding\n",
        "   # Practice:\n",
        "   location_means = train_data.groupby('Location')['Price'].mean()\n",
        "   encoded = test_data['Location'].map(location_means)\n",
        "   ```\n",
        "\n",
        "\n",
        "### Our new technical toolbox\n",
        "\n",
        "We've built a comprehensive technical toolkit that spans the entire machine learning pipeline:\n",
        "\n",
        "1. **Data Processing**\n",
        "   - Built robust cleaning and validation systems\n",
        "   - Developed strategies for handling missing location data\n",
        "   - Created sophisticated feature engineering pipelines\n",
        "   - Implemented multiple encoding strategies for categorical data\n",
        "\n",
        "2. **Model Development**\n",
        "   - Deep dived into three types of tree-based models\n",
        "   - Implemented rigorous hyperparameter tuning\n",
        "   - Created comprehensive performance evaluation frameworks\n",
        "   - Built model persistence and versioning systems\n",
        "\n",
        "3. **Production Considerations**\n",
        "   - Designed modular, maintainable pipelines\n",
        "   - Implemented thorough error handling\n",
        "   - Created performance monitoring systems\n",
        "   - Developed deployment strategies\n",
        "\n",
        "### Ethical considerations and responsibilities\n",
        "\n",
        "Our work has important implications that we must consider:\n",
        "\n",
        "1. **Market impact**\n",
        "   - Our models could influence real pricing decisions\n",
        "   - Robust validation is not just technical but ethical\n",
        "   - Error communication must be clear and comprehensive\n",
        "   - We must consider market feedback effects\n",
        "\n",
        "2. **Social responsibility**\n",
        "   - Fair housing considerations must guide our work\n",
        "   - Predictions need to be transparent and explainable\n",
        "   - Bias detection and mitigation are crucial\n",
        "   - Ethical deployment practices are non-negotiable\n",
        "\n",
        "<a name=\"looking-ahead-a-creative-experiment-called-atlas\"></a>\n",
        "### Looking ahead to Lesson 2C: A creative experiment called ATLAS\n",
        "\n",
        "Now here's where things get interesting. After working through all these models manually, we started wondering: What's the most optimised model for this dataset?! Could we automate this experimentation process? Make it more systematic? Maybe even a bit fun?\n",
        "\n",
        "That's where ATLAS comes in. It's not some industry-standard framework or official tool - it's our playground for taking everything we've learned and scaling it up. Think of it as our \"what if we could...\" project that turned into something pretty cool.\n",
        "\n",
        "In the next lesson, we'll build ATLAS together. We'll:\n",
        "- Automate our model comparisons\n",
        "- Test tons of feature combinations\n",
        "- Run experiments at scale\n",
        "- Have some fun with code\n",
        "\n",
        "The neat thing about ATLAS is it builds directly on everything we've done here - just with some creative automation thrown in, yay loops! It's like taking our careful, manual approach and giving it superpowers.\n",
        "\n",
        "## [Next Lesson: ATLAS](./2c_decision_trees_ATLAS_model_comparison.ipynb)\n",
        "\n",
        "<a name=\"further-reading-and-resources\"></a> \n",
        "### Further reading\n",
        "\n",
        "To deepen your understanding:\n",
        "\n",
        "1. **Core Concepts**\n",
        "   - [Scikit-learn Decision Trees Guide](https://scikit-learn.org/stable/modules/tree.html)\n",
        "   - [\"An Introduction to Statistical Learning\" Chapter 8](http://faculty.marshall.usc.edu/gareth-james/ISL/)\n",
        "   - [\"The Elements of Statistical Learning\" Chapter 9](https://web.stanford.edu/~hastie/ElemStatLearn/)\n",
        "\n",
        "2. **Advanced Topics**\n",
        "   - [XGBoost Documentation](https://xgboost.readthedocs.io/)\n",
        "   - [\"Interpretable Machine Learning\"](https://christophm.github.io/interpretable-ml-book/)\n",
        "   - [Feature Engineering for Machine Learning](https://www.oreilly.com/library/view/feature-engineering-for/9781491953235/)\n",
        "\n",
        "3. **Real Estate Analytics**\n",
        "   - [UK House Price Index](https://landregistry.data.gov.uk/)\n",
        "   - [Property Market Research](https://www.savills.co.uk/research_articles/)\n",
        "\n",
        "\n",
        "### Thanks for Learning!\n",
        "\n",
        "This notebook is part of the Supervised Machine Learning from First Principles series.\n",
        "\n",
        "© 2025 Powell-Clark Limited. Licensed under Apache License 2.0.\n",
        "\n",
        "If you found this helpful, please cite as:\n",
        "```\n",
        "Powell-Clark (2025). Supervised Machine Learning from First Principles.\n",
        "GitHub: https://github.com/powell-clark/supervised-machine-learning\n",
        "```\n",
        "\n",
        "Questions or feedback? Contact emmanuel@powellclark.com"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}