model_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# From Notebook to ModelOp Center:\n",
    "\n",
    "## Training, Evaluating, and Conforming a Model for Deployment\n",
    "In this notebook, we demonstrate the process of\n",
    "1. simple cleaning, feature engineering, and feature selection of a dataset,\n",
    "2. training a model,\n",
    "3. evaluating model performance,\n",
    "4. saving (pickling) the model (and other assets) for later use,\n",
    "5. and conforming the code into MOC standard\n",
    "\n",
    "More specifically, we will train a linear regression predictor on the Ames Housing Data dataset."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's load in the necessary libraries. We will be using `sklearn` to train the model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import pickle\n",
    "import pandas\n",
    "import numpy\n",
    "import copy\n",
    "import seaborn\n",
    "\n",
    "from sklearn import set_config\n",
    "from sklearn.linear_model import LassoCV, LinearRegression\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error\n",
    "\n",
    "#set_config(display='diagram')\n",
    "pandas.options.mode.chained_assignment = None"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**I - Data Cleaning, Feature Engineering, and Feature Selection**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The **Ames Housing Data** dataset can be found [at this link](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data). Download the train dataset (we will be using it exclusively as those have all have an actual SalePrice value, our ground truth to use with the monitoring capabilities of ModelOp Center) and load it into a Pandas DataFrame."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "df = pandas.read_csv('./house_price_data.csv')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "df.columns.values"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "array(['Id', 'MSSubClass', 'MSZoning', 'LotFrontage', 'LotArea', 'Street',\n",
       "       'Alley', 'LotShape', 'LandContour', 'Utilities', 'LotConfig',\n",
       "       'LandSlope', 'Neighborhood', 'Condition1', 'Condition2',\n",
       "       'BldgType', 'HouseStyle', 'OverallQual', 'OverallCond',\n",
       "       'YearBuilt', 'YearRemodAdd', 'RoofStyle', 'RoofMatl',\n",
       "       'Exterior1st', 'Exterior2nd', 'MasVnrType', 'MasVnrArea',\n",
       "       'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual', 'BsmtCond',\n",
       "       'BsmtExposure', 'BsmtFinType1', 'BsmtFinSF1', 'BsmtFinType2',\n",
       "       'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'Heating', 'HeatingQC',\n",
       "       'CentralAir', 'Electrical', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF',\n",
       "       'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath',\n",
       "       'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'KitchenQual',\n",
       "       'TotRmsAbvGrd', 'Functional', 'Fireplaces', 'FireplaceQu',\n",
       "       'GarageType', 'GarageYrBlt', 'GarageFinish', 'GarageCars',\n",
       "       'GarageArea', 'GarageQual', 'GarageCond', 'PavedDrive',\n",
       "       'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch',\n",
       "       'ScreenPorch', 'PoolArea', 'PoolQC', 'Fence', 'MiscFeature',\n",
       "       'MiscVal', 'MoSold', 'YrSold', 'SaleType', 'SaleCondition',\n",
       "       'SalePrice'], dtype=object)"
      ]
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's look at the top of the data:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "df.head()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "3   4          70       RL         60.0     9550   Pave   NaN      IR1   \n",
       "4   5          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities  ... PoolArea PoolQC Fence MiscFeature MiscVal MoSold  \\\n",
       "0         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      2   \n",
       "1         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      5   \n",
       "2         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      9   \n",
       "3         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      2   \n",
       "4         Lvl    AllPub  ...        0    NaN   NaN         NaN       0     12   \n",
       "\n",
       "  YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0   2008        WD         Normal     208500  \n",
       "1   2007        WD         Normal     181500  \n",
       "2   2008        WD         Normal     223500  \n",
       "3   2006        WD        Abnorml     140000  \n",
       "4   2008        WD         Normal     250000  \n",
       "\n",
       "[5 rows x 81 columns]"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 81 columns</p>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Before proceeding, let's split the original dataset into two sets: a **baseline** set which will be used as a reference set, and a **sample** set which will mimic input data to the model once the model is in use.\n",
    "\n",
    "We'l also prepare a `_scored` version of the dataframes for later use during our MOC monitoring phase. In that DataFrame, we'll mainly be using the `ground_truth`, which in our case is `SalePrice`, and `predictions` (later to be added) to compare drift, bias, and other metrics that we will want to monitor in the lifecycle of the model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "df_baseline, df_sample = train_test_split(df, train_size=0.8, random_state=777)\n",
    "\n",
    "df_baseline_scored = df_baseline.copy(deep=True)\n",
    "df_sample_scored = df_sample.copy(deep=True)\n",
    "\n",
    "df_baseline.to_json('df_baseline.json', orient='records', lines=True)\n",
    "df_sample.to_json('df_sample.json', orient='records', lines=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's **clean up** the data. There are quite a few null values in the dataset and we'll fill them in with appropriate values. Note that these steps will also be necessary once we want to write code that conforms to ModelOp standards."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "print('Numerical features that contain 1 or more missing values:')\n",
    "for col, dtype in df.dtypes.items():\n",
    "    if dtype != 'object' and df[col].isna().sum() > 0:\n",
    "        print('- ', col)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Numerical features that contain 1 or more missing values:\n",
      "-  LotFrontage\n",
      "-  MasVnrArea\n",
      "-  GarageYrBlt\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "# imputing missing GarageYrBlt values with corresponding YrBlt values\n",
    "df_baseline.loc[:,'GarageYrBlt'] = df_baseline.loc[:, 'GarageYrBlt'].fillna(df_baseline['YearBuilt'])\n",
    "df_sample.loc[:,'GarageYrBlt'] = df_sample.loc[:, 'GarageYrBlt'].fillna(df_sample['YearBuilt'])"
   ],
   "outputs": [],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "# imputing `MasVnrArea` missing values with 0 and \n",
    "df_baseline.loc[:, 'MasVnrArea'] = df_baseline.loc[:, 'MasVnrArea'].fillna(0)\n",
    "df_sample.loc[:, 'MasVnrArea'] = df_sample.loc[:, 'MasVnrArea'].fillna(0)"
   ],
   "outputs": [],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "# `LotFrontage` missing values with the median value of the neighborhood\n",
    "df_baseline['LotFrontage'] = df_baseline.groupby('Neighborhood')['LotFrontage'].transform(lambda x: x.fillna(x.median()))\n",
    "df_baseline['LotFrontage'] = df_baseline['LotFrontage'].fillna(0)\n",
    "df_sample['LotFrontage'] = df_sample.groupby('Neighborhood')['LotFrontage'].transform(lambda x: x.fillna(x.median()))\n",
    "df_sample['LotFrontage'] = df_sample['LotFrontage'].fillna(0)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "/Users/smerhi/anaconda3/envs/modelop/lib/python3.6/site-packages/numpy/lib/nanfunctions.py:1111: RuntimeWarning: Mean of empty slice\n",
      "  return np.nanmean(a, axis, out=out, keepdims=keepdims)\n"
     ]
    }
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "# imputing the rest of the missing values in categorical features with 'None'\n",
    "for col in list(df.columns):\n",
    "    df_baseline.loc[:, col] = df_baseline.loc[:, col].fillna('None')\n",
    "    df_sample.loc[:, col] = df_sample.loc[:, col].fillna('None')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Next, let's write up some functions that will convert some categorical features into numerical features, then apply those functions to our data."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "# Many columns use this generic scale\n",
    "generic = {\"Ex\": 4, \"Gd\": 3, \"TA\": 2, \"Fa\": 1, \"None\": 0}\n",
    "fireplace_quality = {\"Ex\": 5, \"Gd\": 4, \"TA\": 3, \"Fa\": 2, \"Po\": 1, \"None\": 0}\n",
    "garage_finish = {'Fin': 3, 'RFn': 2, 'Unf': 1, 'None': 0}\n",
    "fin_type = {'GLQ': 6, 'ALQ': 5, 'BLQ': 4, 'Rec': 3, 'LwQ': 2, 'Unf': 1, 'None': 0}\n",
    "functional = {'Typ': 7, 'Min1': 6, 'Min2': 5, 'Mod': 4, 'Maj1': 3, 'Maj2': 2, 'Sev': 1, 'None': 0}"
   ],
   "outputs": [],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "# Generic conversions\n",
    "generic_columns = ['ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'HeatingQC', 'KitchenQual', 'GarageQual', 'GarageCond', 'PoolQC']\n",
    "for col in generic_columns:\n",
    "    df_baseline.loc[:, col] = df_baseline[col].map(generic)\n",
    "    df_sample.loc[:, col] = df_sample[col].map(generic)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "# FireplaceQu convervsions\n",
    "df_baseline.loc[:, 'FireplaceQu'] = df_baseline['FireplaceQu'].map(fireplace_quality)\n",
    "df_sample.loc[:, 'FireplaceQu'] = df_sample['FireplaceQu'].map(fireplace_quality)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "# FinType conversions\n",
    "fintype_columns = ['BsmtFinType1', 'BsmtFinType2']\n",
    "for col in fintype_columns:\n",
    "    df_baseline.loc[:, col] = df_baseline[col].map(fin_type)\n",
    "    df_sample.loc[:, col] = df_sample[col].map(fin_type)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "# Functional conversion\n",
    "df_baseline.loc[:, 'Functional'] = df_baseline['Functional'].map(functional)\n",
    "df_sample.loc[:, 'Functional'] = df_sample['Functional'].map(functional)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "# GarageFinish conversion\n",
    "df_baseline.loc[:, 'GarageFinish'] = df_baseline['GarageFinish'].map(garage_finish)\n",
    "df_sample.loc[:, 'GarageFinish'] = df_sample['GarageFinish'].map(garage_finish)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We also have the `MSSubClass` numerical column which should be treated as a categorical column. We will convert this into an onject column to later be one-hot encoded."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "df_baseline['MSSubClass'] = df_baseline['MSSubClass'].astype(str)\n",
    "df_sample['MSSubClass'] = df_sample['MSSubClass'].astype(str)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Our data still contains non-predictive features, such as `Id` and `SalePrice`. We will set these aside and remove them the final list of predictive features."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "df_baseline_ids = df_baseline['Id']\n",
    "df_sample_ids = df_sample['Id']\n",
    "\n",
    "predictive_features = [\n",
    "    f for f in list(df.columns.values)\n",
    "    if f not in ['Id', 'SalePrice']\n",
    "]"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Everything looks good; we'll proceed with model training. We need to specify **predictive** and **responsive** variables for each of the training and test sets. We'll set those by filtering the baseline and sample sets."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "X_train = df_baseline[predictive_features]\n",
    "X_test = df_sample[predictive_features]\n",
    "\n",
    "y_train = df_baseline['SalePrice']\n",
    "y_test = df_sample['SalePrice']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's take a look at which columns contain muerical data and which contain categorical data. Some categorical features can easily be converted to numerical, as they might hold some order of value. Other categorical features will have to be one-hot encoded."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "source": [
    "numerical_features = []\n",
    "categorical_features = []\n",
    "for i,j in zip(X_train.dtypes.index, X_train.dtypes.values):\n",
    "    if j=='object':\n",
    "        categorical_features.append(i)\n",
    "    else:\n",
    "        numerical_features.append(i)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Categorical Features**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "print(categorical_features)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['MSSubClass', 'MSZoning', 'Street', 'Alley', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'Foundation', 'BsmtExposure', 'Heating', 'CentralAir', 'Electrical', 'GarageType', 'PavedDrive', 'Fence', 'MiscFeature', 'SaleType', 'SaleCondition']\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Numerical Features**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "print(numerical_features)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['LotFrontage', 'LotArea', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtFinType1', 'BsmtFinSF1', 'BsmtFinType2', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'HeatingQC', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'KitchenQual', 'TotRmsAbvGrd', 'Functional', 'Fireplaces', 'FireplaceQu', 'GarageYrBlt', 'GarageFinish', 'GarageCars', 'GarageArea', 'GarageQual', 'GarageCond', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'PoolQC', 'MiscVal', 'MoSold', 'YrSold']\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "source": [
    "# One hot encoding with pandas.get_dummies()\n",
    "X_train = pandas.get_dummies(X_train, columns=categorical_features)\n",
    "X_test = pandas.get_dummies(X_test, columns=categorical_features)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's do some **feature engineering**. While this portion of a data science project can be lengthy, we'll be doing some basic feature engineering in this example. A few ones that might be helpful are boolean features like `HasGarage` or `HasBsmt` and additive features like `TotalSF` or `TotalBathrooms`. We'll be adding `e` as a prefix to these columns to show that they have been engineered."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "source": [
    "for data in [X_train, X_test]:\n",
    "    #  computing total square footage as a new feature\n",
    "    data['eTotalSF'] = data['TotalBsmtSF'] + data['1stFlrSF'] + data['2ndFlrSF']\n",
    "\n",
    "    #  Computing total 'porch' square-footage as a new feature\n",
    "    data['eTotalPorchSF'] = (data['OpenPorchSF'] + data['3SsnPorch'] + data['EnclosedPorch'] + data['ScreenPorch'] + data['WoodDeckSF'])\n",
    "    \n",
    "    #  Computing total bathrooms as a new feature\n",
    "    data['eTotalBathrooms'] = (data['FullBath'] + (0.5 * data['HalfBath']) + data['BsmtFullBath'] + (0.5 * data['BsmtHalfBath']))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "source": [
    "# Engineering some features into Booleans\n",
    "f = lambda x: bool(1) if x > 0 else bool(0)\n",
    "\n",
    "for data in [X_train, X_test]:\n",
    "    data['eHasPool'] = data['PoolArea'].apply(f)\n",
    "    data['eHasGarage'] = data['GarageArea'].apply(f)\n",
    "    data['eHasBsmt'] = data['TotalBsmtSF'].apply(f)\n",
    "    data['eHasFireplace'] = data['Fireplaces'].apply(f)\n",
    "    # feature to determine if a house had a remodeling\n",
    "    data['eHasRemodeling'] = data['YearRemodAdd'] - data['YearBuilt'] > 0"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's engineer one more feature, `OverallQual_TotalSF`. This feature will be the multiplication of the `OverallQual` and `TotalSF` features. This specific combination of features, in fact, has the highest correlation to `SalePrice`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "source": [
    "for data in [X_train, X_test]:\n",
    "    data['eOverallQual_TotalSF'] = data['OverallQual'] * data['eTotalSF']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's move on to **feature selection**. We want our final model to only have a limited number of features, as too many features will cause the model to overfit, though too little will not allow our model to perform well. We'll take a look at the correlation of `SalePrice` to all other columns to determine which ones have the most predictive power, and choose 10 features to predict with."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "source": [
    "X_train['SalePrice'] = df['SalePrice']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "source": [
    "strong_cols = [(col, round(corr, 2)) for col, corr in X_train.corr()['SalePrice'].items() if abs(corr) > 0.4]\n",
    "strong_cols.sort(key=lambda x:x[1])\n",
    "strong_cols"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[('Neighborhood_NridgHt', 0.4),\n",
       " ('BsmtFinSF1', 0.41),\n",
       " ('HeatingQC', 0.43),\n",
       " ('Fireplaces', 0.46),\n",
       " ('eHasFireplace', 0.47),\n",
       " ('MasVnrArea', 0.48),\n",
       " ('Foundation_PConc', 0.5),\n",
       " ('FireplaceQu', 0.51),\n",
       " ('YearRemodAdd', 0.52),\n",
       " ('TotRmsAbvGrd', 0.52),\n",
       " ('GarageYrBlt', 0.52),\n",
       " ('YearBuilt', 0.53),\n",
       " ('FullBath', 0.55),\n",
       " ('GarageFinish', 0.55),\n",
       " ('1stFlrSF', 0.6),\n",
       " ('TotalBsmtSF', 0.61),\n",
       " ('GarageArea', 0.62),\n",
       " ('BsmtQual', 0.63),\n",
       " ('eTotalBathrooms', 0.63),\n",
       " ('GarageCars', 0.64),\n",
       " ('KitchenQual', 0.66),\n",
       " ('ExterQual', 0.69),\n",
       " ('GrLivArea', 0.7),\n",
       " ('eTotalSF', 0.77),\n",
       " ('OverallQual', 0.78),\n",
       " ('eOverallQual_TotalSF', 0.84),\n",
       " ('SalePrice', 1.0)]"
      ]
     },
     "metadata": {},
     "execution_count": 28
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "source": [
    "# the final list of encoded columns\n",
    "train_encoded_columns = ['eOverallQual_TotalSF', 'OverallQual', 'eTotalSF', 'GrLivArea',\n",
    "                         'ExterQual', 'KitchenQual', 'GarageCars', 'eTotalBathrooms', 'BsmtQual', \n",
    "                         'GarageArea', 'TotalBsmtSF', 'GarageFinish', 'YearBuilt', 'eHasGarage', 'TotRmsAbvGrd', \n",
    "                         'eHasRemodeling', 'FireplaceQu', 'MasVnrArea', 'eHasFireplace', 'eHasBsmt']"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**II - Training the model**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's move on to training the model. Let's first restrict our feature space to only be that of the final list of columns. We'll be saving this list, as we'll need it when we deploy the model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "source": [
    "# restricting X_train and X_test columns to only be final list of columns\n",
    "X_train = X_train[train_encoded_columns]\n",
    "X_test = X_test[train_encoded_columns]\n",
    "\n",
    "# Saving the final list of encoded columns\n",
    "pickle.dump(train_encoded_columns, open('train_encoded_columns.pickle', 'wb'))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Another thing we can do to make our model a bit more performant, as well as explainable, is to standardize the features. This will allow us to interpret the weights on each feature directly against each other."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "source": [
    "X_test.dtypes"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "eOverallQual_TotalSF      int64\n",
       "OverallQual               int64\n",
       "eTotalSF                  int64\n",
       "GrLivArea                 int64\n",
       "ExterQual                 int64\n",
       "KitchenQual               int64\n",
       "GarageCars                int64\n",
       "eTotalBathrooms         float64\n",
       "BsmtQual                  int64\n",
       "GarageArea                int64\n",
       "TotalBsmtSF               int64\n",
       "GarageFinish              int64\n",
       "YearBuilt                 int64\n",
       "eHasGarage                 bool\n",
       "TotRmsAbvGrd              int64\n",
       "eHasRemodeling             bool\n",
       "FireplaceQu               int64\n",
       "MasVnrArea              float64\n",
       "eHasFireplace              bool\n",
       "eHasBsmt                   bool\n",
       "dtype: object"
      ]
     },
     "metadata": {},
     "execution_count": 31
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "source": [
    "ss = StandardScaler()\n",
    "X_train_ss = ss.fit_transform(numpy.array(X_train))\n",
    "X_test_ss = ss.transform(numpy.array(X_test))\n",
    "pickle.dump(ss, open('standard_scaler.pickle', 'wb'))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "One last thing we can do is to apply a logarithm to `SalePrice`. If we look at the distribution of values for `SalePrice`, we can notice that there is a right skew, with values at the top of the distribution quite distant from the average value of the distribution."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "source": [
    "y_train.hist();"
   ],
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ],
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"251.721024pt\" version=\"1.1\" viewBox=\"0 0 375.2875 251.721024\" width=\"375.2875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <metadata>\n  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2021-08-27T10:07:41.843965</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.3.4, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 251.721024 \nL 375.2875 251.721024 \nL 375.2875 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 33.2875 227.842899 \nL 368.0875 227.842899 \nL 368.0875 10.402899 \nL 33.2875 10.402899 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 48.505682 227.842899 \nL 78.942045 227.842899 \nL 78.942045 183.75135 \nL 48.505682 183.75135 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 78.942045 227.842899 \nL 109.378409 227.842899 \nL 109.378409 20.757184 \nL 78.942045 20.757184 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 109.378409 227.842899 \nL 139.814773 227.842899 \nL 139.814773 122.312308 \nL 109.378409 122.312308 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 139.814773 227.842899 \nL 170.251136 227.842899 \nL 170.251136 187.365412 \nL 139.814773 187.365412 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_7\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 170.251136 227.842899 \nL 200.6875 227.842899 \nL 200.6875 212.302435 \nL 170.251136 212.302435 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_8\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 200.6875 227.842899 \nL 231.123864 227.842899 \nL 231.123864 222.421807 \nL 200.6875 222.421807 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_9\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 231.123864 227.842899 \nL 261.560227 227.842899 \nL 261.560227 226.397274 \nL 231.123864 226.397274 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_10\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 261.560227 227.842899 \nL 291.996591 227.842899 \nL 291.996591 226.75868 \nL 261.560227 226.75868 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_11\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 291.996591 227.842899 \nL 322.432955 227.842899 \nL 322.432955 227.120086 \nL 291.996591 227.120086 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_12\">\n    <path clip-path=\"url(#pe899fec5cd)\" d=\"M 322.432955 227.842899 \nL 352.869318 227.842899 \nL 352.869318 227.120086 \nL 322.432955 227.120086 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 33.754549 227.842899 \nL 33.754549 10.402899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m68672d9af1\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.754549\" xlink:href=\"#m68672d9af1\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <g transform=\"translate(30.573299 242.441336)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 76.021406 227.842899 \nL 76.021406 10.402899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"76.021406\" xlink:href=\"#m68672d9af1\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 100000 -->\n      <g transform=\"translate(56.933906 242.441336)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"254.492188\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"318.115234\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 118.288263 227.842899 \nL 118.288263 10.402899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"118.288263\" xlink:href=\"#m68672d9af1\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 200000 -->\n      <g transform=\"translate(99.200763 242.441336)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"254.492188\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"318.115234\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 160.555119 227.842899 \nL 160.555119 10.402899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"160.555119\" xlink:href=\"#m68672d9af1\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 300000 -->\n      <g transform=\"translate(141.467619 242.441336)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"254.492188\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"318.115234\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 202.821976 227.842899 \nL 202.821976 10.402899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"202.821976\" xlink:href=\"#m68672d9af1\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 400000 -->\n      <g transform=\"translate(183.734476 242.441336)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"254.492188\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"318.115234\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 245.088833 227.842899 \nL 245.088833 10.402899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"245.088833\" xlink:href=\"#m68672d9af1\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 500000 -->\n      <g transform=\"translate(226.001333 242.441336)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"254.492188\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"318.115234\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_7\">\n     <g id=\"line2d_13\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 287.35569 227.842899 \nL 287.35569 10.402899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"287.35569\" xlink:href=\"#m68672d9af1\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 600000 -->\n      <g transform=\"translate(268.26819 242.441336)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 33.015625 40.375 \nQ 26.375 40.375 22.484375 35.828125 \nQ 18.609375 31.296875 18.609375 23.390625 \nQ 18.609375 15.53125 22.484375 10.953125 \nQ 26.375 6.390625 33.015625 6.390625 \nQ 39.65625 6.390625 43.53125 10.953125 \nQ 47.40625 15.53125 47.40625 23.390625 \nQ 47.40625 31.296875 43.53125 35.828125 \nQ 39.65625 40.375 33.015625 40.375 \nz\nM 52.59375 71.296875 \nL 52.59375 62.3125 \nQ 48.875 64.0625 45.09375 64.984375 \nQ 41.3125 65.921875 37.59375 65.921875 \nQ 27.828125 65.921875 22.671875 59.328125 \nQ 17.53125 52.734375 16.796875 39.40625 \nQ 19.671875 43.65625 24.015625 45.921875 \nQ 28.375 48.1875 33.59375 48.1875 \nQ 44.578125 48.1875 50.953125 41.515625 \nQ 57.328125 34.859375 57.328125 23.390625 \nQ 57.328125 12.15625 50.6875 5.359375 \nQ 44.046875 -1.421875 33.015625 -1.421875 \nQ 20.359375 -1.421875 13.671875 8.265625 \nQ 6.984375 17.96875 6.984375 36.375 \nQ 6.984375 53.65625 15.1875 63.9375 \nQ 23.390625 74.21875 37.203125 74.21875 \nQ 40.921875 74.21875 44.703125 73.484375 \nQ 48.484375 72.75 52.59375 71.296875 \nz\n\" id=\"DejaVuSans-54\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-54\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"254.492188\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"318.115234\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_8\">\n     <g id=\"line2d_15\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 329.622547 227.842899 \nL 329.622547 10.402899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"329.622547\" xlink:href=\"#m68672d9af1\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 700000 -->\n      <g transform=\"translate(310.535047 242.441336)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 8.203125 72.90625 \nL 55.078125 72.90625 \nL 55.078125 68.703125 \nL 28.609375 0 \nL 18.3125 0 \nL 43.21875 64.59375 \nL 8.203125 64.59375 \nz\n\" id=\"DejaVuSans-55\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-55\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"254.492188\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"318.115234\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_17\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 33.2875 227.842899 \nL 368.0875 227.842899 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"ma093e09126\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#ma093e09126\" y=\"227.842899\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0 -->\n      <g transform=\"translate(19.925 231.642117)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_19\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 33.2875 191.702285 \nL 368.0875 191.702285 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#ma093e09126\" y=\"191.702285\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 100 -->\n      <g transform=\"translate(7.2 195.501504)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 33.2875 155.561672 \nL 368.0875 155.561672 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#ma093e09126\" y=\"155.561672\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 200 -->\n      <g transform=\"translate(7.2 159.360891)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_23\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 33.2875 119.421059 \nL 368.0875 119.421059 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_24\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#ma093e09126\" y=\"119.421059\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 300 -->\n      <g transform=\"translate(7.2 123.220277)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_25\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 33.2875 83.280445 \nL 368.0875 83.280445 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_26\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#ma093e09126\" y=\"83.280445\"/>\n      </g>\n     </g>\n     <g id=\"text_13\">\n      <!-- 400 -->\n      <g transform=\"translate(7.2 87.079664)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_27\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 33.2875 47.139832 \nL 368.0875 47.139832 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_28\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#ma093e09126\" y=\"47.139832\"/>\n      </g>\n     </g>\n     <g id=\"text_14\">\n      <!-- 500 -->\n      <g transform=\"translate(7.2 50.939051)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_29\">\n      <path clip-path=\"url(#pe899fec5cd)\" d=\"M 33.2875 10.999219 \nL 368.0875 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_30\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#ma093e09126\" y=\"10.999219\"/>\n      </g>\n     </g>\n     <g id=\"text_15\">\n      <!-- 600 -->\n      <g transform=\"translate(7.2 14.798437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-54\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"patch_13\">\n    <path d=\"M 33.2875 227.842899 \nL 33.2875 10.402899 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_14\">\n    <path d=\"M 368.0875 227.842899 \nL 368.0875 10.402899 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_15\">\n    <path d=\"M 33.2875 227.842899 \nL 368.0875 227.842899 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_16\">\n    <path d=\"M 33.2875 10.402899 \nL 368.0875 10.402899 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pe899fec5cd\">\n   <rect height=\"217.44\" width=\"334.8\" x=\"33.2875\" y=\"10.402899\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAT7UlEQVR4nO3dbYxcZ3nG8f+NDSFkILGbZGVsCwfVonVw87ZyQVRoTVpiCML5EskoRQ4N8oe6UVAjUbtIbflgyYBAUEFaLAfqKoBxgTQWCS+Wy7a0hZgYAokTXC+Jmyw2NgQCXVSlONz9MI/j8WbXM17P7M48/H/Sas555jnnXLvevfbsmRdHZiJJqssL5jqAJKn7LHdJqpDlLkkVstwlqUKWuyRVyHKXpAp1VO4RcVFEfC4ivh8Rj0bEayNiYUTsiYhD5XZBy/zNETEWEQcj4rrexZckTaXTM/ePAF/OzN8BrgAeBTYBezNzObC3rBMRK4B1wOXAGuCOiJjX7eCSpOm1LfeIeBnweuBOgMz8v8x8GlgL7CjTdgA3lOW1wM7MfCYzHwfGgFXdjS1JOpP5Hcx5JfBj4JMRcQWwH7gNGMrMowCZeTQiLi3zFwPfbNl+vIxN6+KLL85LLrmECy644Gzzz7pf/vKX5uyyQclqzu4alJzQv1n379//k8y8ZKr7Oin3+cDVwK2ZeX9EfIRyCWYaMcXY897jICI2ABsAhoaG+MAHPkCj0eggztyamJgwZ5cNSlZzdteg5IT+zbp69er/nu6+Tsp9HBjPzPvL+udolvuxiFhUztoXAcdb5i9t2X4JcGTyTjNzG7ANYHh4OBuNBiMjIx3EmVujo6Pm7LJByWrO7hqUnDBYWU9qe809M38EPBkRrypD1wKPALuB9WVsPXBPWd4NrIuI8yLiMmA5sK+rqSVJZ9TJmTvArcCnIuJFwGPAO2j+YtgVEbcATwA3AmTmgYjYRfMXwAlgY2Y+2/XkkqRpdVTumfkgMDzFXddOM38LsGXmsSRJ58JXqEpShSx3SaqQ5S5JFbLcJalClrskVajTp0JqCss23Tsnxz289fo5Oa6kweGZuyRVyHKXpApZ7pJUIctdkipkuUtShSx3SaqQ5S5JFbLcJalClrskVchyl6QKWe6SVCHLXZIqZLlLUoUsd0mqkOUuSRWy3CWpQpa7JFXIcpekClnuklQhy12SKmS5S1KFOir3iDgcEQ9FxIMR8UAZWxgReyLiULld0DJ/c0SMRcTBiLiuV+ElSVM7mzP31Zl5ZWYOl/VNwN7MXA7sLetExApgHXA5sAa4IyLmdTGzJKmNc7kssxbYUZZ3ADe0jO/MzGcy83FgDFh1DseRJJ2lTss9ga9GxP6I2FDGhjLzKEC5vbSMLwaebNl2vIxJkmZJZGb7SREvz8wjEXEpsAe4FdidmRe1zPlZZi6IiI8B38jMu8r4ncB9mfn5SfvcAGwAGBoaumb79u00Go1ufV49MzEx8VzOh3748znJsHLxhW3ntObsd4OS1ZzdNSg5oX+zrl69en/LpfLTzO9kB5l5pNwej4i7aV5mORYRizLzaEQsAo6X6ePA0pbNlwBHptjnNmAbwPDwcDYaDUZGRjr8lObO6Ojoczlv3nTvnGQ4fNNI2zmtOfvdoGQ1Z3cNSk4YrKwntb0sExEXRMRLTy4DbwQeBnYD68u09cA9ZXk3sC4izouIy4DlwL5uB5ckTa+TM/ch4O6IODn/05n55Yj4FrArIm4BngBuBMjMAxGxC3gEOAFszMxne5JekjSltuWemY8BV0wx/hRw7TTbbAG2nHM6SdKM+ApVSaqQ5S5JFbLcJalClrskVchyl6QKWe6SVCHLXZIqZLlLUoUsd0mqkOUuSRWy3CWpQpa7JFXIcpekClnuklQhy12SKmS5S1KFLHdJqpDlLkkVstwlqUKWuyRVyHKXpApZ7pJUIctdkipkuUtShSx3SaqQ5S5JFbLcJalCHZd7RMyLiO9ExBfL+sKI2BMRh8rtgpa5myNiLCIORsR1vQguSZre2Zy53wY82rK+CdibmcuBvWWdiFgBrAMuB9YAd0TEvO7ElSR1oqNyj4glwPXA9pbhtcCOsrwDuKFlfGdmPpOZjwNjwKqupJUkdaTTM/cPA+8Gft0yNpSZRwHK7aVlfDHwZMu88TImSZol89tNiIi3AMczc39EjHSwz5hiLKfY7wZgA8DQ0BATExOMjo52sPu51Zrz9pUn5iRDJ1+nQfl6wuBkNWd3DUpOGKysJ7Utd+B1wFsj4s3Ai4GXRcRdwLGIWJSZRyNiEXC8zB8HlrZsvwQ4MnmnmbkN2AYwPDycjUaDkZGRmX8ms2R0dPS5nDdvundOMhy+aaTtnNac/W5QspqzuwYlJwxW1pPaXpbJzM2ZuSQzl9F8oPRfMvOPgd3A+jJtPXBPWd4NrIuI8yLiMmA5sK/rySVJ0+rkzH06W4FdEXEL8ARwI0BmHoiIXcAjwAlgY2Y+e85JJUkdO6tyz8xRYLQsPwVcO828LcCWc8wmSZohX6EqSRWy3CWpQpa7JFXIcpekClnuklQhy12SKmS5S1KFLHdJqpDlLkkVstwlqUKWuyRVyHKXpApZ7pJUIctdkipkuUtShSx3SaqQ5S5JFbLcJalClrskVchyl6QKWe6SVCHLXZIqZLlLUoUsd0mqkOUuSRWy3CWpQpa7JFXIcpekCrUt94h4cUTsi4jvRsSBiHhvGV8YEXsi4lC5XdCyzeaIGIuIgxFxXS8/AUnS83Vy5v4M8IbMvAK4ElgTEa8BNgF7M3M5sLesExErgHXA5cAa4I6ImNeD7JKkabQt92yaKKsvLB8JrAV2lPEdwA1leS2wMzOfyczHgTFgVTdDS5LOLDKz/aTmmfd+4LeBj2XmX0TE05l5Ucucn2Xmgoj4KPDNzLyrjN8JfCkzPzdpnxuADQBDQ0PXbN++nUaj0a3Pq2cmJiaey/nQD38+JxlWLr6w7ZzWnP1uULKas7sGJSf0b9bVq1fvz8zhqe6b38kOMvNZ4MqIuAi4OyJefYbpMdUuptjnNmAbwPDwcDYaDUZGRjqJM6dGR0efy3nzpnvnJMPhm0bazmnN2e8GJas5u2tQcsJgZT3prJ4tk5lPA6M0r6Ufi4hFAOX2eJk2Dixt2WwJcORcg0qSOtf2zD0iLgF+lZlPR8T5wB8C7wN2A+uBreX2nrLJbuDTEfEh4OXAcmBfD7L/xlrWwV8Mt6880ZO/LA5vvb7r+5TUfZ1cllkE7CjX3V8A7MrML0bEN4BdEXEL8ARwI0BmHoiIXcAjwAlgY7msI0maJW3LPTO/B1w1xfhTwLXTbLMF2HLO6SRJM+IrVCWpQpa7JFXIcpekClnuklQhy12SKmS5S1KFLHdJqpDlLkkVstwlqUKWuyRVyHKXpApZ7pJUIctdkipkuUtShSx3SaqQ5S5JFbLcJalClrskVchyl6QKWe6SVCHLXZIqZLlLUoUsd0mqkOUuSRWy3CWpQpa7JFXIcpekCrUt94hYGhFfi4hHI+JARNxWxhdGxJ6IOFRuF7RsszkixiLiYERc18tPQJL0fJ2cuZ8Abs/M3wVeA2yMiBXAJmBvZi4H9pZ1yn3rgMuBNcAdETGvF+ElSVNrW+6ZeTQzv12W/wd4FFgMrAV2lGk7gBvK8lpgZ2Y+k5mPA2PAqi7nliSdwVldc4+IZcBVwP3AUGYeheYvAODSMm0x8GTLZuNlTJI0SyIzO5sY0QD+FdiSmV+IiKcz86KW+3+WmQsi4mPANzLzrjJ+J3BfZn5+0v42ABsAhoaGrtm+fTuNRqMrn1QvTUxMPJfzoR/+fI7TTG/ofDj2v93f78rFF3Z9n61f035mzu4alJzQv1lXr169PzOHp7pvfic7iIgXAp8HPpWZXyjDxyJiUWYejYhFwPEyPg4sbdl8CXBk8j4zcxuwDWB4eDgbjQYjIyOdxJlTo6Ojz+W8edO9cxvmDG5feYIPPtTRP+9ZOXzTSNf32fo17Wfm7K5ByQmDlfWkTp4tE8CdwKOZ+aGWu3YD68vyeuCelvF1EXFeRFwGLAf2dS+yJKmdTk7tXge8HXgoIh4sY38JbAV2RcQtwBPAjQCZeSAidgGP0HymzcbMfLbbwSVJ02tb7pn570BMc/e102yzBdhyDrkkSefAV6hKUoUsd0mqkOUuSRWy3CWpQpa7JFXIcpekClnuklQhy12SKmS5S1KFLHdJqpDlLkkVstwlqUKWuyRVyHKXpApZ7pJUIctdkipkuUtShSx3SaqQ5S5JFbLcJalClrskVchyl6QKWe6SVKH5cx2gG5ZtunfWjnX7yhPcPIvHk6SZ8MxdkipkuUtShaq4LKPZ04tLYJ1c6jq89fquH1eqWdsz94j4REQcj4iHW8YWRsSeiDhUbhe03Lc5IsYi4mBEXNer4JKk6XVyWeYfgDWTxjYBezNzObC3rBMRK4B1wOVlmzsiYl7X0kqSOtK23DPz34CfThpeC+woyzuAG1rGd2bmM5n5ODAGrOpOVElSp2b6gOpQZh4FKLeXlvHFwJMt88bLmCRpFnX7AdWYYiynnBixAdgAMDQ0xMTEBKOjozM66O0rT8xou5kYOn92jzdTg5ITOss60++NbjqX79HZZM7uG6SsJ8203I9FxKLMPBoRi4DjZXwcWNoybwlwZKodZOY2YBvA8PBwNhoNRkZGZhRmNl9UdPvKE3zwof5/ktGg5ITOsh6+aWR2wpzB6OjojL9HZ5M5u2+Qsp4008syu4H1ZXk9cE/L+LqIOC8iLgOWA/vOLaIk6Wy1PbWLiM8AI8DFETEO/DWwFdgVEbcATwA3AmTmgYjYBTwCnAA2ZuazPcouSZpG23LPzLdNc9e108zfAmw5l1CSpHPj2w9IUoUsd0mqkOUuSRWy3CWpQpa7JFXIcpekClnuklQhy12SKmS5S1KFLHdJqpDlLkkVstwlqUKWuyRVyHKXpApZ7pJUocH4f9j0G2/ZLP5XipMd3nr9nB1bminP3CWpQpa7JFXIcpekClnuklQhy12SKmS5S1KFLHdJqpDlLkkVstwlqUK+QlVq4+SrY29feYKbZ/GVsr4yVufCM3dJqlDPyj0i1kTEwYgYi4hNvTqOJOn5elLuETEP+BjwJmAF8LaIWNGLY0mSnq9X19xXAWOZ+RhAROwE1gKP9Oh4UnVm+k6Y3XhswOv9g69X5b4YeLJlfRz4/R4dS1KXzcZbLM/2A9TnopdZe/WLNDKz+zuNuBG4LjPfWdbfDqzKzFtb5mwANpTVVwFPAT/pepjuuxhzdtugZDVndw1KTujfrK/IzEumuqNXZ+7jwNKW9SXAkdYJmbkN2HZyPSIeyMzhHuXpGnN236BkNWd3DUpOGKysJ/Xq2TLfApZHxGUR8SJgHbC7R8eSJE3SkzP3zDwREX8GfAWYB3wiMw/04liSpOfr2StUM/M+4L6z2GRb+yl9wZzdNyhZzdldg5ITBisr0KMHVCVJc8u3H5CkGmXmnH4Aa4CDwBiwqYfH+QRwHHi4ZWwhsAc4VG4XtNy3uWQ6SPNpnSfHrwEeKvf9Laf++jkP+GwZvx9Y1rLN+nKMQ8D6NjmXAl8DHgUOALf1Y1bgxcA+4Lsl53v7MWfL/HnAd4Av9nnOw+UYDwIP9GtW4CLgc8D3aX6vvrbfctJ8ivWDLR+/AN7Vbzl79TGrB5vmB+4HwCuBF9EsihU9Otbrgas5vdzfT/mFAmwC3leWV5Qs5wGXlYzzyn37yjdyAF8C3lTG/xT4+7K8Dvhsyw/mY+V2QVlecIaci4Cry/JLgf8qefoqa9lnoyy/sHxjv6bfcrbk/XPg05wq937NeRi4eNJY32UFdgDvLMsvoln2fZdzUtf8CHhFP+fsaufN5sGm+IK/FvhKy/pmYHMPj7eM08v9ILCoLC8CDk6Vg+azfl5b5ny/ZfxtwMdb55Tl+TRf8BCtc8p9HwfedhaZ7wH+qJ+zAi8Bvk3zVch9l5Pm6yz2Am/gVLn3Xc4y5zDPL/e+ygq8DHiccvbarzknZXsj8B/9nrObH3N9zX2qtylYPIvHH8rMowDl9tI2uRaX5cnjp22TmSeAnwO/dYZ9tRURy4CraJ4V913WiJgXEQ/SvNy1JzP7MifwYeDdwK9bxvoxJ0ACX42I/eVV3P2Y9ZXAj4FPRsR3ImJ7RFzQhzlbrQM+U5b7OWfXzHW5xxRjOespnm+6XGfKO5Ntpg8Q0QA+D7wrM39xpqkzOG5Xsmbms5l5Jc0z41UR8ep+yxkRbwGOZ+b+M2Q7bZMZHLOb//avy8yrab6j6saIeP0Z5s5V1vk0L3H+XWZeBfyS5uWNfsvZ3FHzhZRvBf7pTPNmeMyu/tx301yXe9u3KeixYxGxCKDcHm+Ta7wsTx4/bZuImA9cCPz0DPuaVkS8kGaxfyozv9DPWQEy82lglOaD4/2W83XAWyPiMLATeENE3NWHOQHIzCPl9jhwN813WO23rOPAePlLDZoPrF7dhzlPehPw7cw8Vtb7NWd3zeY1oCmug82n+UDDZZx6QPXyHh5vGadfc/8Apz+w8v6yfDmnP7DyGKceWPkWzQcOTz6w8uYyvpHTH1jZVZYX0rw+uaB8PA4sPEPGAP4R+PCk8b7KClwCXFSWzwe+Dryl33JOyjzCqWvufZcTuAB4acvyf9L8hdmPWb8OvKos/03J2Hc5yzY7gXf0689Sz/puNg82zRf+zTSfEfID4D09PM5ngKPAr2j+Vr2F5rWxvTSfqrS39YsPvKdkOkh5ZLyMDwMPl/s+yqmnRL2Y5p99YzQfWX9lyzZ/UsbHWr/Jpsn5BzT/fPsep57C9eZ+ywr8Hs2nFn6vHOOvynhf5ZyUeYRT5d53OWley/4up55e+p4+znol8ED59/9nmgXWjzlfQvMdZy9sGeu7nL348BWqklShub7mLknqActdkipkuUtShSx3SaqQ5S5JFbLcJalClrskVchyl6QK/T+r1mP2O+nUsgAAAABJRU5ErkJggg=="
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Applying a logarithm will allow the distribution to be centralized, allowing our model that will be trained to be more performant. We'll specifically use the `numpy.log1p()` function in case that there are values are 0. "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "source": [
    "y_train_log = numpy.log1p(y_train)\n",
    "y_test_log = numpy.log1p(y_test)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "source": [
    "y_train_log.hist();"
   ],
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ],
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"248.518125pt\" version=\"1.1\" viewBox=\"0 0 375.2875 248.518125\" width=\"375.2875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <metadata>\n  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2021-08-27T10:07:42.234684</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.3.4, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 248.518125 \nL 375.2875 248.518125 \nL 375.2875 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 33.2875 224.64 \nL 368.0875 224.64 \nL 368.0875 7.2 \nL 33.2875 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 48.505682 224.64 \nL 78.942045 224.64 \nL 78.942045 222.448617 \nL 48.505682 222.448617 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 78.942045 224.64 \nL 109.378409 224.64 \nL 109.378409 219.709388 \nL 78.942045 219.709388 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 109.378409 224.64 \nL 139.814773 224.64 \nL 139.814773 200.534785 \nL 109.378409 200.534785 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 139.814773 224.64 \nL 170.251136 224.64 \nL 170.251136 145.202358 \nL 139.814773 145.202358 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_7\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 170.251136 224.64 \nL 200.6875 224.64 \nL 200.6875 17.554286 \nL 170.251136 17.554286 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_8\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 200.6875 224.64 \nL 231.123864 224.64 \nL 231.123864 53.164263 \nL 200.6875 53.164263 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_9\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 231.123864 224.64 \nL 261.560227 224.64 \nL 261.560227 128.219138 \nL 231.123864 128.219138 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_10\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 261.560227 224.64 \nL 291.996591 224.64 \nL 291.996591 182.455873 \nL 261.560227 182.455873 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_11\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 291.996591 224.64 \nL 322.432955 224.64 \nL 322.432955 215.874467 \nL 291.996591 215.874467 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"patch_12\">\n    <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 322.432955 224.64 \nL 352.869318 224.64 \nL 352.869318 221.352925 \nL 322.432955 221.352925 \nz\n\" style=\"fill:#1f77b4;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 52.439102 224.64 \nL 52.439102 7.2 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m3db2f5d3c4\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.439102\" xlink:href=\"#m3db2f5d3c4\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 10.5 -->\n      <g transform=\"translate(41.30629 239.238438)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n        <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n        <path d=\"M 10.6875 12.40625 \nL 21 12.40625 \nL 21 0 \nL 10.6875 0 \nz\n\" id=\"DejaVuSans-46\"/>\n        <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 101.941946 224.64 \nL 101.941946 7.2 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"101.941946\" xlink:href=\"#m3db2f5d3c4\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 11.0 -->\n      <g transform=\"translate(90.809133 239.238438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 151.444789 224.64 \nL 151.444789 7.2 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"151.444789\" xlink:href=\"#m3db2f5d3c4\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 11.5 -->\n      <g transform=\"translate(140.311976 239.238438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 200.947632 224.64 \nL 200.947632 7.2 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"200.947632\" xlink:href=\"#m3db2f5d3c4\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 12.0 -->\n      <g transform=\"translate(189.814819 239.238438)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 250.450475 224.64 \nL 250.450475 7.2 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"250.450475\" xlink:href=\"#m3db2f5d3c4\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 12.5 -->\n      <g transform=\"translate(239.317662 239.238438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 299.953318 224.64 \nL 299.953318 7.2 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"299.953318\" xlink:href=\"#m3db2f5d3c4\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 13.0 -->\n      <g transform=\"translate(288.820506 239.238438)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_7\">\n     <g id=\"line2d_13\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 349.456161 224.64 \nL 349.456161 7.2 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"349.456161\" xlink:href=\"#m3db2f5d3c4\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 13.5 -->\n      <g transform=\"translate(338.323349 239.238438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-46\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_15\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 33.2875 224.64 \nL 368.0875 224.64 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"mbff3452ffd\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#mbff3452ffd\" y=\"224.64\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0 -->\n      <g transform=\"translate(19.925 228.439219)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_17\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 33.2875 197.24771 \nL 368.0875 197.24771 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#mbff3452ffd\" y=\"197.24771\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 50 -->\n      <g transform=\"translate(13.5625 201.046929)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_19\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 33.2875 169.85542 \nL 368.0875 169.85542 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#mbff3452ffd\" y=\"169.85542\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 100 -->\n      <g transform=\"translate(7.2 173.654638)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 33.2875 142.463129 \nL 368.0875 142.463129 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#mbff3452ffd\" y=\"142.463129\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 150 -->\n      <g transform=\"translate(7.2 146.262348)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_23\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 33.2875 115.070839 \nL 368.0875 115.070839 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_24\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#mbff3452ffd\" y=\"115.070839\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 200 -->\n      <g transform=\"translate(7.2 118.870058)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_25\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 33.2875 87.678549 \nL 368.0875 87.678549 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_26\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#mbff3452ffd\" y=\"87.678549\"/>\n      </g>\n     </g>\n     <g id=\"text_13\">\n      <!-- 250 -->\n      <g transform=\"translate(7.2 91.477768)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_27\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 33.2875 60.286259 \nL 368.0875 60.286259 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_28\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#mbff3452ffd\" y=\"60.286259\"/>\n      </g>\n     </g>\n     <g id=\"text_14\">\n      <!-- 300 -->\n      <g transform=\"translate(7.2 64.085477)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_29\">\n      <path clip-path=\"url(#p9f4440c5bd)\" d=\"M 33.2875 32.893968 \nL 368.0875 32.893968 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_30\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"33.2875\" xlink:href=\"#mbff3452ffd\" y=\"32.893968\"/>\n      </g>\n     </g>\n     <g id=\"text_15\">\n      <!-- 350 -->\n      <g transform=\"translate(7.2 36.693187)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-51\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"patch_13\">\n    <path d=\"M 33.2875 224.64 \nL 33.2875 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_14\">\n    <path d=\"M 368.0875 224.64 \nL 368.0875 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_15\">\n    <path d=\"M 33.2875 224.64 \nL 368.0875 224.64 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_16\">\n    <path d=\"M 33.2875 7.2 \nL 368.0875 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p9f4440c5bd\">\n   <rect height=\"217.44\" width=\"334.8\" x=\"33.2875\" y=\"7.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAUPElEQVR4nO3df4xlZX3H8fdXoEgYC0vBybJsuhjXRmAjyoSakJg7YmUrjQtpadYQsgSa9Q9INd0/XDSpGLIJtqJpAmrHrnFT1OkGJGz5UcWNE2IiQdYAw/KjbGWKu0t2o/LDsWTbWb/9Yw7xOtyZe2fuuXPnPr5fyeSe+5xznvt888x85syZc8+NzESSVJa39HsAkqT6Ge6SVCDDXZIKZLhLUoEMd0kq0In9HgDAmWeemevWraulr1//+teceuqptfTVTyXUUUINUEYdJdQAZdRRZw379u37eWae1Wrdigj3devW8dhjj9XS18TEBI1Go5a++qmEOkqoAcqoo4QaoIw66qwhIv57vnWelpGkAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAKtiHeoSu2s235/3177GxsH++3u+v3kkbskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklSgtuEeEW+NiEcj4omI2B8Rn6vab46IQxHxePX1kaZ9boqIAxHxXERc1ssCJElv1smNw44BH8zM6Yg4CfhhRDxYrftSZn6heeOIOA/YDJwPnA18PyLelZnH6xy4JGl+bY/cc9Z09fSk6isX2GUTMJ6ZxzLzBeAAcHHXI5UkdSwyF8rpaqOIE4B9wDuBOzLzUxFxM3At8BrwGLAtM1+OiNuBRzLzzmrfncCDmXnXnD63AlsBhoeHLxofH6+loOnpaYaGhmrpq59KqKPOGiYPvVpLP0tx7mknOBcrRAl11FnD6OjovswcabWuo/u5V6dULoyI04F7IuIC4CvALcwexd8C3AZcB0SrLlr0OQaMAYyMjGSj0ehkKG1NTExQV1/9VEIdddZwbZ/v5+5crAwl1LFcNSzqapnMfAWYADZm5pHMPJ6ZvwG+xm9PvRwE1jbtdg5wuPuhSpI61cnVMmdVR+xExCnAh4BnI2J102ZXAk9Vy3uAzRFxckScC6wHHq111JKkBXVyWmY1sKs67/4WYHdm3hcR/xoRFzJ7ymUK+DhAZu6PiN3A08AMcINXykjS8mob7pn5JPDeFu3XLLDPDmBHd0OTJC2V71CVpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQXq6GP2pN9nk4de7cvH/E3devmyv6bK4ZG7JBXIcJekAhnuklSgTj4g+60R8WhEPBER+yPic1X7GRHxUEQ8Xz2uatrnpog4EBHPRcRlvSxAkvRmnRy5HwM+mJnvAS4ENkbE+4HtwN7MXA/srZ4TEecBm4HzgY3Al6sP15YkLZO24Z6zpqunJ1VfCWwCdlXtu4ArquVNwHhmHsvMF4ADwMV1DlqStLDIzPYbzR557wPeCdyRmZ+KiFcy8/SmbV7OzFURcTvwSGbeWbXvBB7MzLvm9LkV2AowPDx80fj4eC0FTU9PMzQ0VEtf/VRCHXXWMHno1Vr6WYrhU+DI68v/uhvWnFZbXyV8P0EZddRZw+jo6L7MHGm1rqPr3DPzOHBhRJwO3BMRFyywebTqokWfY8AYwMjISDYajU6G0tbExAR19dVPJdRRZw39uM78Dds2zHDb5PK/JWTq6kZtfZXw/QRl1LFcNSzqapnMfAWYYPZc+pGIWA1QPR6tNjsIrG3a7RzgcLcDlSR1rpOrZc6qjtiJiFOADwHPAnuALdVmW4B7q+U9wOaIODkizgXWA4/WPG5J0gI6+VtzNbCrOu/+FmB3Zt4XET8CdkfE9cCLwFUAmbk/InYDTwMzwA3VaR1J0jJpG+6Z+STw3hbtvwAunWefHcCOrkcnSVoS36EqSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFahtuEfE2oj4QUQ8ExH7I+ITVfvNEXEoIh6vvj7StM9NEXEgIp6LiMt6WYAk6c3afkA2MANsy8yfRMTbgH0R8VC17kuZ+YXmjSPiPGAzcD5wNvD9iHhXZh6vc+CSpPm1PXLPzJcy8yfV8q+AZ4A1C+yyCRjPzGOZ+QJwALi4jsFKkjoTmdn5xhHrgIeBC4C/A64FXgMeY/bo/uWIuB14JDPvrPbZCTyYmXfN6WsrsBVgeHj4ovHx8a6LAZienmZoaKiWvvqphDrqrGHy0Ku19LMUw6fAkdeX/3U3rDmttr5K+H6CMuqos4bR0dF9mTnSal0np2UAiIgh4G7gk5n5WkR8BbgFyOrxNuA6IFrs/qbfIJk5BowBjIyMZKPR6HQoC5qYmKCuvvqphDrqrOHa7ffX0s9SbNsww22THf+o1Gbq6kZtfZXw/QRl1LFcNXR0tUxEnMRssH8zM78DkJlHMvN4Zv4G+Bq/PfVyEFjbtPs5wOH6hixJaqeTq2UC2Ak8k5lfbGpf3bTZlcBT1fIeYHNEnBwR5wLrgUfrG7IkqZ1O/ta8BLgGmIyIx6u2TwMfi4gLmT3lMgV8HCAz90fEbuBpZq+0ucErZSRpebUN98z8Ia3Poz+wwD47gB1djEuS1AXfoSpJBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCdfED22oj4QUQ8ExH7I+ITVfsZEfFQRDxfPa5q2uemiDgQEc9FxGW9LECS9GadHLnPANsy893A+4EbIuI8YDuwNzPXA3ur51TrNgPnAxuBL0fECb0YvCSptbbhnpkvZeZPquVfAc8Aa4BNwK5qs13AFdXyJmA8M49l5gvAAeDimsctSVpAZGbnG0esAx4GLgBezMzTm9a9nJmrIuJ24JHMvLNq3wk8mJl3zelrK7AVYHh4+KLx8fEuS5k1PT3N0NBQLX31Uwl11FnD5KFXa+lnKYZPgSOvL//rblhzWm19lfD9BGXUUWcNo6Oj+zJzpNW6EzvtJCKGgLuBT2bmaxEx76Yt2t70GyQzx4AxgJGRkWw0Gp0OZUETExPU1Vc/lVBHnTVcu/3+WvpZim0bZrhtsuMfldpMXd2ora8Svp+gjDqWq4aOrpaJiJOYDfZvZuZ3quYjEbG6Wr8aOFq1HwTWNu1+DnC4nuFKkjrRydUyAewEnsnMLzat2gNsqZa3APc2tW+OiJMj4lxgPfBofUOWJLXTyd+alwDXAJMR8XjV9mngVmB3RFwPvAhcBZCZ+yNiN/A0s1fa3JCZx+seuCRpfm3DPTN/SOvz6ACXzrPPDmBHF+OSJHXBd6hKUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklSg5b9hhqSOrKvxfjrbNsws6v48U7deXttrqz88cpekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUoE4+IPvrEXE0Ip5qars5Ig5FxOPV10ea1t0UEQci4rmIuKxXA5ckza+TI/dvABtbtH8pMy+svh4AiIjzgM3A+dU+X46IE+oarCSpM23DPTMfBn7ZYX+bgPHMPJaZLwAHgIu7GJ8kaQm6Oed+Y0Q8WZ22WVW1rQF+1rTNwapNkrSMIjPbbxSxDrgvMy+ong8DPwcSuAVYnZnXRcQdwI8y885qu53AA5l5d4s+twJbAYaHhy8aHx+vpaDp6WmGhoZq6aufSqijzhomD71aSz9LMXwKHHm9by9fi8XWsGHNab0bTBf8ufhdo6Oj+zJzpNW6Jd3PPTOPvLEcEV8D7queHgTWNm16DnB4nj7GgDGAkZGRbDQaSxnKm0xMTFBXX/1UQh111rCYe5HXbduGGW6bHOyPPlhsDVNXN3o3mC74c9G5JX3HRsTqzHypenol8MaVNHuAb0XEF4GzgfXAo12PUivGYj5AYrEfECGpPm3DPSK+DTSAMyPiIPBZoBERFzJ7WmYK+DhAZu6PiN3A08AMcENmHu/JyCVJ82ob7pn5sRbNOxfYfgewo5tBSZK64ztUJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAK1DfeI+HpEHI2Ip5razoiIhyLi+epxVdO6myLiQEQ8FxGX9WrgkqT5dXLk/g1g45y27cDezFwP7K2eExHnAZuB86t9vhwRJ9Q2WklSR9qGe2Y+DPxyTvMmYFe1vAu4oql9PDOPZeYLwAHg4nqGKknqVGRm+40i1gH3ZeYF1fNXMvP0pvUvZ+aqiLgdeCQz76zadwIPZuZdLfrcCmwFGB4evmh8fLyGcmB6epqhoaFa+uqnlVrH5KFXO952+BQ48noPB7NMSqhjsTVsWHNa7wbThZX6c7EYddYwOjq6LzNHWq07sZZX+K1o0dbyt0dmjgFjACMjI9loNGoZwMTEBHX11U8rtY5rt9/f8bbbNsxw22Td32LLr4Q6FlvD1NWN3g2mCyv152IxlquGpV4tcyQiVgNUj0er9oPA2qbtzgEOL314kqSlWGq47wG2VMtbgHub2jdHxMkRcS6wHni0uyFKkhar7d9pEfFtoAGcGREHgc8CtwK7I+J64EXgKoDM3B8Ru4GngRnghsw83qOxS5Lm0TbcM/Nj86y6dJ7tdwA7uhmUJKk7vkNVkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCDfYNMyT1xLpF3EOoTlO3Xt6X1y2RR+6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCtTV7QciYgr4FXAcmMnMkYg4A/g3YB0wBfx1Zr7c3TAlSYtRx5H7aGZemJkj1fPtwN7MXA/srZ5LkpZRL07LbAJ2Vcu7gCt68BqSpAVEZi5954gXgJeBBP45M8ci4pXMPL1pm5czc1WLfbcCWwGGh4cvGh8fX/I4mk1PTzM0NFRLX/20UuuYPPRqx9sOnwJHXu/hYJZJCXUMSg0b1py24PqV+nOxGHXWMDo6uq/prMnv6PaWv5dk5uGIeDvwUEQ82+mOmTkGjAGMjIxko9HociizJiYmqKuvflqpdVy7iFvBbtsww22Tg39X6RLqGJQapq5uLLh+pf5cLMZy1dDVaZnMPFw9HgXuAS4GjkTEaoDq8Wi3g5QkLc6Swz0iTo2It72xDHwYeArYA2ypNtsC3NvtICVJi9PN32nDwD0R8UY/38rM/4iIHwO7I+J64EXgqu6HqWb9+pQcSYNjyeGemT8F3tOi/RfApd0MSpLUHd+hKkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVKCV/9Eskn5vtLud9bYNM4v6NLBOTd16ee199ptH7pJUIMNdkgpkuEtSgTzn3oVeftxdr84tSvr94JG7JBWoZ0fuEbER+CfgBOBfMvPWXr2WJHVjOT90fu5f5b26Uqcn4R4RJwB3AH8GHAR+HBF7MvPpXrxe88R4OkOSenda5mLgQGb+NDP/FxgHNvXotSRJc0Rm1t9pxF8BGzPzb6rn1wB/mpk3Nm2zFdhaPf0T4LmaXv5M4Oc19dVPJdRRQg1QRh0l1ABl1FFnDX+cmWe1WtGrc+7Rou13fotk5hgwVvsLRzyWmSN197vcSqijhBqgjDpKqAHKqGO5aujVaZmDwNqm5+cAh3v0WpKkOXoV7j8G1kfEuRHxB8BmYE+PXkuSNEdPTstk5kxE3Ah8l9lLIb+emft78Vot1H6qp09KqKOEGqCMOkqoAcqoY1lq6Mk/VCVJ/eU7VCWpQIa7JBVoYMI9Ir4eEUcj4qmmtjMi4qGIeL56XDXPvlMRMRkRj0fEY8s36pZjaVXHVRGxPyJ+ExHzXiIVERsj4rmIOBAR25dnxC3H0U0NK30u/jEino2IJyPinog4fZ59V/JcdFrDSp+LW6oaHo+I70XE2fPsu5LnotMa6p+LzByIL+ADwPuAp5ra/gHYXi1vBz4/z75TwJn9rmGBOt7N7Bu5JoCRefY7Afgv4B3AHwBPAOcNUg0DMhcfBk6slj/f6ntqAOaibQ0DMhd/2LT8t8BXB3Au2tbQq7kYmCP3zHwY+OWc5k3Armp5F3DFco5pKVrVkZnPZGa7d+iumFs6dFHDijJPHd/LzJnq6SPMvkdjrpU+F53UsKLMU8drTU9PZc4bISsrfS46qaEnBibc5zGcmS8BVI9vn2e7BL4XEfuq2x4MojXAz5qeH6zaBs0gzcV1wIMt2gdpLuarAQZgLiJiR0T8DLga+PsWm6z4ueigBujBXAx6uHfqksx8H/DnwA0R8YF+D2gJ2t7SYUAMxFxExGeAGeCbrVa3aFtxc9GmBhiAucjMz2TmWmZruLHFJit+LjqoAXowF4Me7kciYjVA9Xi01UaZebh6PArcw+yfcoOmiFs6DMJcRMQW4C+Aq7M6ITrHip+LDmoYiLlo8i3gL1u0r/i5aDJfDT2Zi0EP9z3Almp5C3Dv3A0i4tSIeNsby8z+s+mpudsNgIG/pcMgzEXMfsjMp4CPZub/zLPZip6LTmoYkLlY3/T0o8CzLTZb6XPRtoaezUU//qu8xP9Efxt4Cfg/Zn9bXw/8EbAXeL56PKPa9mzggWr5Hcz+B/0JYD/wmRVYx5XV8jHgCPDduXVUzz8C/CezVwf0rY6l1jAgc3GA2XO4j1dfXx3AuWhbw4DMxd3MhtyTwL8DawZwLtrW0Ku58PYDklSgQT8tI0lqwXCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBfp/Bbmp3U0VvJgAAAAASUVORK5CYII="
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The distribution is now roughly normalized, and the model should be able to predict better. One thing we need to keep in mind, however, is that if we want predictions, we will have to inverse the logarithm on the outputs of the model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will train a **Lasso** linear regression model. Let's fit the model to the training data."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "source": [
    "# finding the cross-validation score with 5 (default) folds, showing how the model should generally perform, irrespective to how the data was split\n",
    "cross_val_score(LassoCV(max_iter=5000), X_train_ss, y_train_log).mean()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "0.8167504671863934"
      ]
     },
     "metadata": {},
     "execution_count": 36
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "source": [
    "lasso = LassoCV(max_iter=5000)\n",
    "lasso.fit(X_train_ss, y_train_log)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "LassoCV(alphas=None, copy_X=True, cv=None, eps=0.001, fit_intercept=True,\n",
       "        max_iter=5000, n_alphas=100, n_jobs=None, normalize=False,\n",
       "        positive=False, precompute='auto', random_state=None,\n",
       "        selection='cyclic', tol=0.0001, verbose=False)"
      ]
     },
     "metadata": {},
     "execution_count": 37
    }
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "source": [
    "lasso.score(X_train_ss, y_train_log)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "0.8430704847122994"
      ]
     },
     "metadata": {},
     "execution_count": 38
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "source": [
    "lasso.score(X_test_ss, y_test_log)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "0.8580636796558804"
      ]
     },
     "metadata": {},
     "execution_count": 39
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**III - Model Evaluation**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Before saving our trained model for further use, let's take a look at some performance metrics. We will evaluate the model on both the training and test sets; we want to see a stable performance between the two.  \n",
    "\n",
    "For repeatability, let's define a function which computes multiple metrics at once."
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "source": [
    "def compute_metrics(y, y_preds):\n",
    "    \"\"\"\n",
    "    A function to evaluate a regression model.\n",
    "    \n",
    "    param: y: true (ground truth) values\n",
    "    param: y_preds: predicted values (as predicted by model)\n",
    "    \n",
    "    return: multiple regression performance metrics\n",
    "    \"\"\"\n",
    "    \n",
    "    return {\n",
    "        'Mean Absolute Error' : round(mean_absolute_error(y, y_preds), 2),\n",
    "        'Root Mean Squared Error' : round(mean_squared_error(y, y_preds) ** 0.5, 2),\n",
    "        'R2 Score' : round(r2_score(y, y_preds), 3)\n",
    "    }"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's compute predictions on both training and test sets. Remember, we need to apply an inverse log to convert the outputs into the same units as our inputs."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "source": [
    "y_train_preds = numpy.round(numpy.expm1(lasso.predict(X_train_ss)), 2)\n",
    "y_test_preds = numpy.round(numpy.expm1(lasso.predict(X_test_ss)), 2)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "source": [
    "performance_df = pandas.DataFrame(\n",
    "    data=[{}],\n",
    "    columns=['Mean Absolute Error', 'Root Mean Squared Error', 'R2 Score'],\n",
    "    index=['Training Set', 'Test Set']\n",
    ")\n",
    "performance_df.loc['Training Set', :] = compute_metrics(y_train, y_train_preds)\n",
    "performance_df.loc['Test Set', :] = compute_metrics(y_test, y_test_preds)"
   ],
   "outputs": [],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's look at how our model performed:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "source": [
    "performance_df"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "              Mean Absolute Error  Root Mean Squared Error  R2 Score\n",
       "Training Set             20914.02                 36422.12     0.801\n",
       "Test Set                 16715.31                 23450.18     0.885"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mean Absolute Error</th>\n",
       "      <th>Root Mean Squared Error</th>\n",
       "      <th>R2 Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Training Set</th>\n",
       "      <td>20914.02</td>\n",
       "      <td>36422.12</td>\n",
       "      <td>0.801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Test Set</th>\n",
       "      <td>16715.31</td>\n",
       "      <td>23450.18</td>\n",
       "      <td>0.885</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "execution_count": 43
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "There is a slight difference in performance between the training set and the test set on some of the metrics. This could have come from the way that the data was split, or other issues. Further model improvements are needed to achieve more accurate inferences. For now, we will contend with this model and use it to produce new predictions."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**IV - Saving and Loading the Trained Model**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now that the model is **trained** and **evaluated**, we save it in a binary format. It will later be loaded and used to make new predictions."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "source": [
    "pickle.dump(lasso, open('lasso.pickle', 'wb'))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The model is reloaded on-demands as follows:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "source": [
    "lasso_loaded = pickle.load(open('lasso.pickle', 'rb'))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Predictions can be produced on-demand by calling the `predict()` function:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "source": [
    "new_preds = lasso_loaded.predict(X_test)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Before heading into the next section, let's append our predictions to our `_scored` DataFrames and save them. Once again, these data sets will be used mainly for monitoring purposes. Note that the features on these datasets will be the final list of features used that the model expects during prediction."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "source": [
    "all_cols = train_encoded_columns.copy()\n",
    "\n",
    "all_cols.append('SalePrice')\n",
    "\n",
    "cols_reordered = ['Id', 'prediction', 'SalePrice']\n",
    "cols_reordered.extend(train_encoded_columns)\n",
    "\n",
    "X_train['SalePrice'] = y_train\n",
    "X_test['SalePrice'] = y_test\n",
    "\n",
    "df_baseline_scored = X_train[all_cols]\n",
    "df_baseline_scored['prediction'] = y_train_preds\n",
    "df_baseline_scored['Id'] = df_baseline_ids\n",
    "df_baseline_scored = df_baseline_scored[cols_reordered]\n",
    "\n",
    "df_sample_scored = X_test[all_cols]\n",
    "df_sample_scored['prediction'] = y_test_preds\n",
    "df_sample_scored['Id'] = df_sample_ids\n",
    "df_sample_scored = df_sample_scored[cols_reordered]\n",
    "\n",
    "df_baseline_scored.to_json('df_baseline_scored.json', orient='records', lines=True)\n",
    "df_sample_scored.to_json('df_sample_scored.json', orient='records', lines=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "**V - Conforming Model Code to MOC Requirements**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Conformance is best demonstrated through example. Let's look at the code below:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "source": [
    "# modelop.slot.0: in-use\n",
    "# modelop.slot.1: in-use\n",
    "\n",
    "\n",
    "import pandas\n",
    "import pickle\n",
    "import numpy\n",
    "import logging\n",
    "from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error\n",
    "from sklearn.linear_model import LassoCV\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "logger = logging.getLogger(__name__)\n",
    "logging.basicConfig(level=\"INFO\")\n",
    "\n",
    "\n",
    "# modelop.init\n",
    "def begin():\n",
    "    \n",
    "    global lasso_model\n",
    "    global standard_scaler\n",
    "    global train_encoded_columns\n",
    "\n",
    "    # Load pickled Lasso linear regression model\n",
    "    lasso_model = pickle.load(open(\"lasso.pickle\", \"rb\"))\n",
    "    # Load pickled standard scaler\n",
    "    standard_scaler = pickle.load(open(\"standard_scaler.pickle\", \"rb\"))\n",
    "    # Load train_encoded_columns\n",
    "    train_encoded_columns = pickle.load(open(\"train_encoded_columns.pickle\", \"rb\"))\n",
    "\n",
    "    logger.info(\n",
    "        \"'lasso.pickle', 'standard_scaler.pickle', and 'train_encoded_columns.pickle' files loaded to respective variables\"\n",
    "    )\n",
    "\n",
    "\n",
    "# modelop.score\n",
    "def action(data):\n",
    "    \n",
    "    # Turning data into a dataframe\n",
    "    logger.info(\"Loading input record into a pandas.DataFrame\")\n",
    "    input_data = pandas.DataFrame([data])\n",
    "\n",
    "    # Set aside ground truth to later re-append to dataframe\n",
    "    ground_truth = input_data[\"SalePrice\"]\n",
    "\n",
    "    # Dictionaries to convert values in certain columns\n",
    "    generic = {\"Ex\": 4, \"Gd\": 3, \"TA\": 2, \"Fa\": 1, \"None\": 0}\n",
    "    fireplace_quality = {\"Ex\": 5, \"Gd\": 4, \"TA\": 3, \"Fa\": 2, \"Po\": 1, \"None\": 0}\n",
    "    garage_finish = {\"Fin\": 3, \"RFn\": 2, \"Unf\": 1, \"None\": 0}\n",
    "\n",
    "    # Imputations\n",
    "    logger.info(\"Conforming input dataset to be model-ready\")\n",
    "    input_data.loc[:, \"GarageYrBlt\"] = input_data.loc[:, \"GarageYrBlt\"].fillna(input_data[\"YearBuilt\"])\n",
    "    for col in [\"GarageFinish\", \"BsmtQual\", \"FireplaceQu\"]:\n",
    "        input_data.loc[:, col] = input_data.loc[:, col].fillna(\"None\")\n",
    "    # The rest of NaNs will be filled with 0s - end model only uses numerical features\n",
    "    for col in input_data.columns:\n",
    "        input_data[col] = input_data[col].fillna(0)\n",
    "\n",
    "    # Converting categorical values from certain features into numerical\n",
    "    for col in [\"BsmtQual\", \"KitchenQual\", \"ExterQual\"]:\n",
    "        input_data.loc[:, col] = input_data[col].map(generic)\n",
    "    input_data.loc[:, \"GarageFinish\"] = input_data[\"GarageFinish\"].map(garage_finish)\n",
    "    input_data.loc[:, \"FireplaceQu\"] = input_data[\"FireplaceQu\"].map(fireplace_quality)\n",
    "\n",
    "    # Feature engineering\n",
    "    f = lambda x: bool(1) if x > 0 else bool(0)\n",
    "    input_data[\"eHasPool\"] = input_data[\"PoolArea\"].apply(f)\n",
    "    input_data[\"eHasGarage\"] = input_data[\"GarageArea\"].apply(f)\n",
    "    input_data[\"eHasBsmt\"] = input_data[\"TotalBsmtSF\"].apply(f)\n",
    "    input_data[\"eHasFireplace\"] = input_data[\"Fireplaces\"].apply(f)\n",
    "    input_data[\"eHasRemodeling\"] = input_data[\"YearRemodAdd\"] - input_data[\"YearBuilt\"] > 0\n",
    "    input_data[\"eTotalSF\"] = input_data[\"TotalBsmtSF\"] + input_data[\"1stFlrSF\"] + input_data[\"2ndFlrSF\"]\n",
    "    input_data[\"eTotalBathrooms\"] = (\n",
    "        input_data[\"FullBath\"]\n",
    "        + (0.5 * input_data[\"HalfBath\"])\n",
    "        + input_data[\"BsmtFullBath\"]\n",
    "        + (0.5 * input_data[\"BsmtHalfBath\"])\n",
    "    )\n",
    "    input_data[\"eOverallQual_TotalSF\"] = input_data[\"OverallQual\"] * input_data[\"eTotalSF\"]\n",
    "\n",
    "    # Limiting features to just the ones the model needs\n",
    "    logger.info(\"Selecting columns that model is expecting\")\n",
    "    input_data = input_data[train_encoded_columns]\n",
    "\n",
    "    # Scale inputs\n",
    "    logger.info(\"Scaling data with pickled standard scaler\")\n",
    "    df_ss = standard_scaler.transform(input_data)\n",
    "\n",
    "    # generate predictions and rename columns\n",
    "    logger.info(\"Generating predictions with the model and appending onto DataFrame\")\n",
    "    input_data.loc[:, \"prediction\"] = numpy.round(numpy.expm1(lasso_model.predict(df_ss)), 2)\n",
    "    input_data.loc[:, \"SalePrice\"] = ground_truth\n",
    "\n",
    "    # MOC expects the action function to be a \"yield\" function\n",
    "    yield input_data.to_dict(orient=\"records\")\n",
    "\n",
    "\n",
    "# modelop.metrics\n",
    "def metrics(metrics_data):\n",
    "    \n",
    "    logger.info(\"metrics_data is of shape: %s\", metrics_data.shape)\n",
    "\n",
    "    logger.info(\"Grabbing relevant columns to calculate metrics\")\n",
    "    y = metrics_data[\"SalePrice\"]\n",
    "    y_preds = metrics_data[\"prediction\"]\n",
    "\n",
    "    logger.info(\"Computing MAE, RMSE, R2 scores\")\n",
    "    output_metrics = {\n",
    "        \"MAE\": mean_absolute_error(y, y_preds),\n",
    "        \"RMSE\": mean_squared_error(y, y_preds) ** 0.5,\n",
    "        \"R2\": r2_score(y, y_preds),\n",
    "    }\n",
    "    \n",
    "    logger.info(\"Metrics job complete!\")\n",
    "\n",
    "    # MOC expects the metrics function to be a \"yield\" function\n",
    "    yield output_metrics\n",
    "\n",
    "\n",
    "# modelop.train\n",
    "def train(training_data):\n",
    "    \n",
    "    logger.info(\"training_data is of shape: %s\", training_data.shape)\n",
    "    \n",
    "    # Set aside ground truth to later re-append to dataframe\n",
    "    y_train = training_data[\"SalePrice\"]\n",
    "\n",
    "    # Dictionaries to convert values in certain columns\n",
    "    generic = {\"Ex\": 4, \"Gd\": 3, \"TA\": 2, \"Fa\": 1, \"None\": 0}\n",
    "    fireplace_quality = {\"Ex\": 5, \"Gd\": 4, \"TA\": 3, \"Fa\": 2, \"Po\": 1, \"None\": 0}\n",
    "    garage_finish = {\"Fin\": 3, \"RFn\": 2, \"Unf\": 1, \"None\": 0}\n",
    "\n",
    "    # Imputations\n",
    "    logger.info(\"Imputing Nulls\")\n",
    "    training_data.loc[:, \"GarageYrBlt\"] = training_data.loc[:, \"GarageYrBlt\"].fillna(training_data[\"YearBuilt\"])\n",
    "    for col in [\"GarageFinish\", \"BsmtQual\", \"FireplaceQu\"]:\n",
    "        training_data.loc[:, col] = training_data.loc[:, col].fillna(\"None\")\n",
    "    # The rest of NaNs will be filled with 0s - end model only uses numerical features\n",
    "    for col in training_data.columns:\n",
    "        training_data[col] = training_data[col].fillna(0)\n",
    "\n",
    "    # Converting categorical values from certain features into numerical\n",
    "    logger.info(\"Converting categorical values to numerical values\")\n",
    "    for col in [\"BsmtQual\", \"KitchenQual\", \"ExterQual\"]:\n",
    "        training_data.loc[:, col] = training_data[col].map(generic)\n",
    "    training_data.loc[:, \"GarageFinish\"] = training_data[\"GarageFinish\"].map(garage_finish)\n",
    "    training_data.loc[:, \"FireplaceQu\"] = training_data[\"FireplaceQu\"].map(fireplace_quality)\n",
    "\n",
    "    # Feature engineering\n",
    "    logger.info(\"Creating new features with feature engineering\")\n",
    "    f = lambda x: 1 if x > 0 else 0\n",
    "    training_data[\"eHasPool\"] = training_data[\"PoolArea\"].apply(f)\n",
    "    training_data[\"eHasGarage\"] = training_data[\"GarageArea\"].apply(f)\n",
    "    training_data[\"eHasBsmt\"] = training_data[\"TotalBsmtSF\"].apply(f)\n",
    "    training_data[\"eHasFireplace\"] = training_data[\"Fireplaces\"].apply(f)\n",
    "    training_data[\"eHasRemodeling\"] = (training_data[\"YearRemodAdd\"] - training_data[\"YearBuilt\"] > 0).astype(int)\n",
    "    training_data[\"eTotalSF\"] = training_data[\"TotalBsmtSF\"] + training_data[\"1stFlrSF\"] + training_data[\"2ndFlrSF\"]\n",
    "    training_data[\"eTotalBathrooms\"] = (\n",
    "        training_data[\"FullBath\"]\n",
    "        + (0.5 * training_data[\"HalfBath\"])\n",
    "        + training_data[\"BsmtFullBath\"]\n",
    "        + (0.5 * training_data[\"BsmtHalfBath\"])\n",
    "    )\n",
    "    training_data[\"eOverallQual_TotalSF\"] = training_data[\"OverallQual\"] * training_data[\"eTotalSF\"]\n",
    "\n",
    "    # Final list of encoded columns\n",
    "    train_encoded_columns = [\n",
    "        \"eOverallQual_TotalSF\",\n",
    "        \"OverallQual\",\n",
    "        \"eTotalSF\",\n",
    "        \"GrLivArea\",\n",
    "        \"ExterQual\",\n",
    "        \"KitchenQual\",\n",
    "        \"GarageCars\",\n",
    "        \"eTotalBathrooms\",\n",
    "        \"BsmtQual\",\n",
    "        \"GarageArea\",\n",
    "        \"TotalBsmtSF\",\n",
    "        \"GarageFinish\",\n",
    "        \"YearBuilt\",\n",
    "        \"eHasGarage\",\n",
    "        \"TotRmsAbvGrd\",\n",
    "        \"eHasRemodeling\",\n",
    "        \"FireplaceQu\",\n",
    "        \"MasVnrArea\",\n",
    "        \"eHasFireplace\",\n",
    "        \"eHasBsmt\",\n",
    "    ]\n",
    "\n",
    "    # Saving the final list of encoded columns\n",
    "    logger.info(\"Pickling final list of columns for model to predict with\")\n",
    "    \n",
    "    # Pickle file should be written to outputDir/    \n",
    "    with open(\"outputDir/train_encoded_columns.pickle\", \"wb\") as columns_file:\n",
    "        pickle.dump(train_encoded_columns, columns_file)\n",
    "\n",
    "    # Choosing only the final list of encoded columns\n",
    "    X_train = training_data[train_encoded_columns]\n",
    "    \n",
    "    # Standard scale data and pickle scaler\n",
    "    standard_scaler = StandardScaler()\n",
    "    X_train_ss = standard_scaler.fit_transform(numpy.array(X_train))\n",
    "    logger.info(\"Pickling trained standard scaler object\")\n",
    "    \n",
    "    # Pickle file should be written to outputDir/    \n",
    "    with open(\"outputDir/standard_scaler.pickle\", \"wb\") as scaler_file:\n",
    "        pickle.dump(standard_scaler, scaler_file)\n",
    "    \n",
    "    # Apply log to distribution of y-values\n",
    "    y_train_log = numpy.log1p(y_train)\n",
    "\n",
    "    # Train and pickle model artifact\n",
    "    logger.info(\"Fitting LASSO model\")\n",
    "    lasso = LassoCV(max_iter=1000)\n",
    "    lasso.fit(X_train_ss, y_train_log)\n",
    "    logger.info(\"Pickling trained LASSO model\")\n",
    "\n",
    "    # Pickle file should be written to outputDir/   \n",
    "    with open(\"outputDir/lasso.pickle\", \"wb\") as lasso_file:\n",
    "        pickle.dump(lasso, lasso_file)\n",
    "\n",
    "    logger.info(\"Training job complete!\")\n",
    "    pass"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "There are five main sections that are standard to almost any model in MOC:\n",
    "1. Library imports\n",
    "2. `init` function\n",
    "3. `score` function\n",
    "4. `metrics` function\n",
    "5. `train` function\n",
    "\n",
    "**Library** imports are always at the top. We don't need to include all libraries that we used for training and model evaluation. We just need the libraries for processing and scoring.\n",
    "\n",
    "The **`init`** function runs once per deployment, and is used to load and persist into memory any variable that needs to be accessed at scoring time. For example, the init function is where we load the saved model binary, scalers, and any other dependencies. We make the variables global so they can be accessed from the scoring function. In our example, we also included the `train_encoded_columns` as this information will not change per prediction and only needs to be instantiated once.\n",
    "\n",
    "The **`score`** function is the function that runs anytime we make a scoring (prediction) request. This is where we put our prediction code. We have to remember to include any steps that were not captured by the pipeline, such as feature engineering or re-encoding.\n",
    "\n",
    "The **`metrics`** functions is where model evaluation is carried out. In our example, this is the place where we compute regression metrics.\n",
    "\n",
    "The **`train`** function is where model training is carried out. This function does not have to `return` or `yield`, and can instead simply `pass`. Any trained artifact should be written - for example as a pickle file) - to `outputDir/`. ModelOp Center will read the contents of this directory once the training job is complete, and make the contents available as job outputs."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let us test our source code to see if we missed anything. We will load input data and scored input data to test both the scoring and metrics functions:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "source": [
    "# this is for the action function (scoring)\n",
    "scoring_sample = pandas.read_json('df_baseline.json', orient='records', lines=True)\n",
    "\n",
    "# these are for the metrics function\n",
    "metrics_baseline_input = pandas.read_json('df_baseline_scored.json', orient='records', lines=True)\n",
    "metrics_sample_input = pandas.read_json('df_sample_scored.json', orient='records', lines=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's check that the **`init`** function can load the trained model binary:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "source": [
    "begin()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "INFO:__main__:'lasso.pickle', 'standard_scaler.pickle', and 'train_encoded_columns.pickle' files loaded to respective variables\n"
     ]
    }
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "No errors from the **`init`** function. Let's make a call to the **`score`** function on input data:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "source": [
    "scores = pandas.DataFrame(\n",
    "    next(\n",
    "        action(scoring_sample.iloc[0])\n",
    "    )\n",
    ")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "INFO:__main__:Loading input record into a pandas.DataFrame\n",
      "INFO:__main__:Conforming input dataset to be model-ready\n",
      "INFO:__main__:Selecting columns that model is expecting\n",
      "INFO:__main__:Scaling data with pickled standard scaler\n",
      "INFO:__main__:Generating predictions with the model and appending onto DataFrame\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "source": [
    "scores"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   eOverallQual_TotalSF  OverallQual  eTotalSF  GrLivArea  ExterQual  \\\n",
       "0                 18192            6      3032       1516          2   \n",
       "\n",
       "   KitchenQual  GarageCars  eTotalBathrooms  BsmtQual  GarageArea  ...  \\\n",
       "0            2           2              1.5         2         472  ...   \n",
       "\n",
       "   YearBuilt  eHasGarage  TotRmsAbvGrd  eHasRemodeling  FireplaceQu  \\\n",
       "0       1964        True             6           False            0   \n",
       "\n",
       "   MasVnrArea  eHasFireplace  eHasBsmt  prediction  SalePrice  \n",
       "0       180.0          False      True   156491.21     167000  \n",
       "\n",
       "[1 rows x 22 columns]"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eOverallQual_TotalSF</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>eTotalSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>ExterQual</th>\n",
       "      <th>KitchenQual</th>\n",
       "      <th>GarageCars</th>\n",
       "      <th>eTotalBathrooms</th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>...</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>eHasGarage</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>eHasRemodeling</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>eHasFireplace</th>\n",
       "      <th>eHasBsmt</th>\n",
       "      <th>prediction</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18192</td>\n",
       "      <td>6</td>\n",
       "      <td>3032</td>\n",
       "      <td>1516</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>472</td>\n",
       "      <td>...</td>\n",
       "      <td>1964</td>\n",
       "      <td>True</td>\n",
       "      <td>6</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>156491.21</td>\n",
       "      <td>167000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 22 columns</p>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "execution_count": 52
    }
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We scores! Finally, let's call the **`metrics`** function on scored data:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "source": [
    "metrics_compare = pandas.DataFrame(\n",
    "    [\n",
    "        next(\n",
    "            metrics(metrics_baseline_input)\n",
    "        ), \n",
    "        next(\n",
    "            metrics(metrics_sample_input)\n",
    "        )\n",
    "    ],\n",
    "    index=['baseline', 'sample']\n",
    ")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "INFO:__main__:metrics_data is of shape: (1168, 23)\n",
      "INFO:__main__:Grabbing relevant columns to calculate metrics\n",
      "INFO:__main__:Computing MAE, RMSE, R2 scores\n",
      "INFO:__main__:Metrics job complete!\n",
      "INFO:__main__:metrics_data is of shape: (292, 23)\n",
      "INFO:__main__:Grabbing relevant columns to calculate metrics\n",
      "INFO:__main__:Computing MAE, RMSE, R2 scores\n",
      "INFO:__main__:Metrics job complete!\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "source": [
    "metrics_compare"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                   MAE          RMSE        R2\n",
       "baseline  20914.018767  36422.119215  0.801362\n",
       "sample    16715.310240  23450.183792  0.885231"
      ],
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MAE</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>R2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>baseline</th>\n",
       "      <td>20914.018767</td>\n",
       "      <td>36422.119215</td>\n",
       "      <td>0.801362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sample</th>\n",
       "      <td>16715.310240</td>\n",
       "      <td>23450.183792</td>\n",
       "      <td>0.885231</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "execution_count": 54
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.6.9 64-bit ('modelop': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "interpreter": {
   "hash": "1bc0af681b6debbd76b10de7ba3014cd62e1e3ee55e8e9a892bbbe23fd8d618c"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}