05-NNinPyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification using Neural Networks\n",
    "\n",
    "We will now train our a neural network classifier using PyTorch.  We will use the titanic dataset for this.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the data using pandas as we learnt in the previous notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd  # type: ignore\n",
    "import numpy as np   # type: ignore\n",
    "# we are loading data from github. \n",
    "dataurl = 'https://github.com/rrr-uom-projects/MPiCRT-AI/raw/main/Data/titanic.csv' \n",
    "pax = pd.read_csv(dataurl, sep = ',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remember the data to make sense of it. Here is a short description of the series:\n",
    "\n",
    "- **PassengerId** Arbitrary nr between 1 and 841\n",
    "- **Survived** Weather Survived or not: 0 = No, 1 = Yes\n",
    "- **Pclass** Ticket class: 1 = 1st, 2 = 2nd, 3 = 3rd\n",
    "- **Name** Name of the Passenger\n",
    "- **Sex** Female/male\n",
    "- **Age** Age in years\n",
    "- **SibSp** No. of siblings / spouses aboard the Titanic\n",
    "- **Parch** No. of parents / children aboard the Titanic\n",
    "- **Ticket** Ticket number\n",
    "- **Fare** Passenger fare\n",
    "- **Cabin** Cabin number\n",
    "- **Embarked** Port of Embarkation:C = Cherbourg, Q = Queenstown, S = Southampton\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing and creatomg dummy variables\n",
    "\n",
    "During the last tutorials we processed the data to get it in numerical coding.  Let's bring the relevant code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imputing Age\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "medianAges = pax.groupby(['Sex','Pclass','Embarked'], observed=True)[['Age']].median()\n",
    "medianAges = medianAges.reset_index()\n",
    "\n",
    "def getMedianAgeForCategory(row):\n",
    "    # using the dataframe medianAges created above.\n",
    "    condition = (\n",
    "        (medianAges['Sex'] == row['Sex']) & \n",
    "        (medianAges['Pclass'] == row['Pclass']) & \n",
    "        (medianAges['Embarked'] == row['Embarked'])\n",
    "    ) \n",
    "    return medianAges[condition]['Age'].values[0]\n",
    "\n",
    "def imputeIfNeeded(row):\n",
    "    return getMedianAgeForCategory(row) if np.isnan(row['Age']) else row['Age']\n",
    "\n",
    "#let's make a copy of the values before imputing\n",
    "pax['Age'] = pax.apply(imputeIfNeeded, axis=1)\n",
    "pax.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# eliminate cabin and eliminate missing values --> 889 \n",
    "pax.dropna(subset=['Embarked'],inplace=True)\n",
    "pax.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Titles and tytle types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we need to cast the type of the Name series to str. \n",
    "pax['Name'] = pax['Name'].astype('string')\n",
    "surnamefirstnames = pax['Name'].str.split(',')  # this splits the string by the token given (,)\n",
    "pax['Surname'] = surnamefirstnames.str.get(0)   # here we get the first bit of the divided sentence\n",
    "afterComma = surnamefirstnames.str.get(1).str.split('.')# this splits the string by the token given (.)\n",
    "pax['Title'] = afterComma.str.get(0).str.strip()        # here we get the first bit of the divided sentence and eliminate empty spaces\n",
    "\n",
    "Title_Dictionary = {\n",
    "    \"Capt\": \"Officer\",\n",
    "    \"Col\": \"Officer\",\n",
    "    \"Major\": \"Officer\",\n",
    "    \"Jonkheer\": \"Royalty\",\n",
    "    \"Don\": \"Royalty\",\n",
    "    \"Sir\" : \"Royalty\",\n",
    "    \"Dr\": \"Officer\",\n",
    "    \"Rev\": \"Officer\",\n",
    "    \"the Countess\":\"Royalty\",\n",
    "    \"Mme\": \"Mrs\",\n",
    "    \"Mlle\": \"Miss\",\n",
    "    \"Ms\": \"Mrs\",\n",
    "    \"Mr\" : \"Mr\",\n",
    "    \"Mrs\" : \"Mrs\",\n",
    "    \"Miss\" : \"Miss\",\n",
    "    \"Master\" : \"Master\",\n",
    "    \"Lady\" : \"Royalty\"\n",
    "}\n",
    "pax['TitleType'] = pax['Title'].map(Title_Dictionary)\n",
    "pax['TitleType'] = pax['TitleType'].astype('category')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Family sizes and types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pax['FamilySize'] = pax['SibSp']+pax['Parch']+1 \n",
    "def getFamilyType(famsize):\n",
    "    return 'single' if famsize == 1 else ('smallFamily' if famsize < 5 else 'largeFamily')\n",
    "\n",
    "pax['FamilyType'] = pax['FamilySize'].apply(getFamilyType)\n",
    "pax['FamilyType'] = pax['FamilyType'].astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical variables to dummy variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pax['Sex'] = pax['Sex'].map({'male': 0, 'female': 1}).astype(int)\n",
    "# pax['Survived'] is already 0s and 1s. Not converted to category\n",
    "# pax['Pclass'] is already 1s, 2s, or 3s.  Not converted to category\n",
    "pax['Embarked'] = pax['Embarked'].astype(\"category\")\n",
    "pax = pd.get_dummies(pax, prefix='Embarked',columns=['Embarked'],dtype=int)\n",
    "pax = pd.get_dummies(pax, prefix='TitleType',columns=['TitleType'],dtype=int)\n",
    "pax = pd.get_dummies(pax, prefix='FamilyType',columns=['FamilyType'],dtype=int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eliminate variables \n",
    "\n",
    "Now we have extracted extra information from the data stored for each passanger. We can now clean up our dataframe in preparation to model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pax.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pax.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleanpax = pax.loc[:,['Survived', 'Pclass', 'Sex', 'Age', 'SibSp',\n",
    "       'Parch', 'Fare', 'FamilySize',\n",
    "       'Embarked_C', 'Embarked_Q', 'Embarked_S', 'TitleType_Master',\n",
    "       'TitleType_Miss', 'TitleType_Mr', 'TitleType_Mrs', 'TitleType_Officer',\n",
    "       'TitleType_Royalty', 'FamilyType_largeFamily', 'FamilyType_single',\n",
    "       'FamilyType_smallFamily']]\n",
    "\n",
    "\n",
    "cleanpax.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data splitting\n",
    "\n",
    "Before we do any training, let's divide the dataset in *training* and *validation*.  Ideally, we will have another dataset, *test*, to test for generalisability.  Kaggle kept a good portion of the data as test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = cleanpax.loc[:,'Survived'] # This is the target!\n",
    "X = cleanpax.loc[:, cleanpax.columns != 'Survived'] # This are the features/variables we wll use to predict\n",
    "\n",
    "# to divide the data in train/validation, we an use train_test_split from sklearn\n",
    "from sklearn import model_selection # type: ignore\n",
    "X_train, x_val, Y_train, y_val = model_selection.train_test_split(X, Y, test_size=0.2, random_state=1234) # train 80%, validation 20%\n",
    "print(X_train.shape,x_val.shape,Y_train.shape,y_val.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use X_train and Y_train to create our models, and use x_test and y_test to test for overfitting!  We will learn more about this later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch\n",
    "\n",
    "PyTorch is an open-source machine learning framework developed by Meta and widely used for deep learning applications. It is very popular as it enables to build and train neural networks 'easily'.\n",
    "\n",
    "Some key features of PyTorch:\n",
    "- Autograd (Automatic Differentiation) is built-in to automatically computate gradients. This is what allows neural network optimisation (backpropagation).\n",
    "- GPU Acceleration, supporting CUDA, which allows to train much faster than in normal CPUs.\n",
    "- Integration with NumPy, allowing easy data conversion.\n",
    "\n",
    "For learning PyTorch in depth, it is worth following their tutorials: https://pytorch.org/tutorials/beginner/basics/intro.html  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch  # type: ignore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important point in training is knowing whether we have GPU acceleration or not.  Let's find which device we are using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors\n",
    "*PyTorch* encodes inputs and outputs of a model, as well as the model’s parameters as *tensors*.  Tensor is similar to arrays/matrics, very similar to *ndarray*'s in *NumPy*. An important difference is that tensors include ways to store and use the data in GPUs and they are optimised for automatic differentation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.autograd import Variable  # type: ignore\n",
    "X_train_t = Variable(torch.Tensor(X_train.values), requires_grad=True)\n",
    "Y_train_t = Variable(torch.Tensor(Y_train.values), requires_grad=False).unsqueeze_(-1)\n",
    "x_val_t = Variable(torch.Tensor(x_val.values), requires_grad=False)\n",
    "y_val_t = Variable(torch.Tensor(y_val.values), requires_grad=False).unsqueeze_(-1)\n",
    "print('Dataframes:', X_train.shape,x_val.shape,Y_train.shape,y_val.shape)\n",
    "print('Tensors:', X_train_t.shape, Y_train_t.shape, x_val_t.shape, y_val_t.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Move tensor to GPU if available\n",
    "X_train_t = X_train_t.to(device)\n",
    "Y_train_t = Y_train_t.to(device)\n",
    "x_val_t = x_val_t.to(device)\n",
    "y_val_t = y_val_t.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Neural Network\n",
    "Let's define a neural network with 3 hidden layers where each layer has 7 neurons, with a ReLU activation function. I came across this configuration from the optimisation presented here: https://github.com/davidtvs/kaggle-titanic/blob/master/pytorch/surviving-the-titanic.ipynb\n",
    "\n",
    "To create a network, the best practice is to create a class with your configuration. For this example, we will inherite from the class [Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html) in PyTorch. We need to define the initialiser (what goes in the __init__() function) as well as the forward function, which is how you apply the neural network to a given 'x'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn  # type: ignore\n",
    "\n",
    "# Define Neural Network class\n",
    "class NeuralNetwork(nn.Module): # here we are inheriting from the Module class\n",
    "    def __init__(self):\n",
    "        super().__init__()          # here we call the 'parent' initialiser. It should be called first!!!\n",
    "        self.model = nn.Sequential(\n",
    "            nn.Linear(19, 7),       # Layer 1. From 19 variables in originally in our dataframe to 7 neurons in the input layer\n",
    "            nn.ReLU(),              # activation\n",
    "            nn.Linear(7, 7),        # Hidden layer\n",
    "            nn.ReLU(),              # activation\n",
    "            nn.Linear(7, 7),        # Hidden layer\n",
    "            nn.ReLU(),              # activation\n",
    "            nn.Linear(7, 1),        # Hidden layer\n",
    "            nn.Sigmoid()            # sigmoid activation to finish the network\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.model(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create the model. The model is an object of the class NeuralNetwork, and we need to assign the device we are working with (either cpu or cuda)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = NeuralNetwork().to(device)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the neural network \n",
    "\n",
    "Remember that training is optimising the values of the trainable parameters using some hyperparameters to define the model characteristics and guide the optimisation process.  Here we have already decided on the model characteristics (nr of layers, nr of hidden neurons, etc).  The next part is how to set up the training process.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function\n",
    "We also need a loss function.  We learnt about MSE (in registration), and cross-entropy.  For binary classification, binary cross-entropy (BCE) is the most appropriate loss function.  Here you can find the documentation: https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_fn = torch.nn.BCELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training 1 epoch\n",
    "\n",
    "We need to write what to do when the model is train for one epoch (that is, when the optimiser sees the complete dataset and updates the parameters).  In this case, we need to compute the error (loss function) and do backpropagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_1epoch(X, Y, model, loss_fn, optimiser ):\n",
    "    model.train()   # we tell the model we will be training it now\n",
    "\n",
    "    # Compute prediction error\n",
    "    pred = model(X)\n",
    "    loss = loss_fn(pred, Y)\n",
    "\n",
    "    # let's compute accuracy to check the training accuracy and compare it to what we got in the previous tutorial\n",
    "    accuracy = (pred.round() == Y).type(torch.float).sum().item() / Y.size(0)\n",
    "\n",
    "    # Backpropagation\n",
    "    loss.backward()\n",
    "    optimiser.step()\n",
    "    optimiser.zero_grad()\n",
    "    return loss.item(), accuracy # return the training loss and accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating without changing parameters\n",
    "We should also check the model's performance against the validation dataset to be sure it is learning and assess wether it is underfiting, overfitting, or doing just right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(x, y, model, loss_fn):\n",
    "    model.eval()    # we tell the model we will use it to evaluate or predict now.\n",
    "    with torch.no_grad():   # we do not need to keep track of the gradients here --> faster and lighter in memory\n",
    "        pred = model(x)\n",
    "        test_loss = loss_fn(pred, y).item()\n",
    "        accuracy = (pred.round() == y).type(torch.float).sum().item() / y.size(0)\n",
    "    return test_loss, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimiser\n",
    "To train a model, we need an optimizer. We learnt that *Adam* is the most popular optimiser.  Adam requires a hyperparameter to control the optimisation: learning rate.  It is always good to check for published values to guide your selection.  A very small value will make the training very slow but it is likely to converge. In contrast, a very large value will be quick, but it may miss the global minima.  We will use 0.01."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimiser = torch.optim.Adam(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training loop\n",
    "\n",
    "Finally, we need to make the trainable loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 200\n",
    "trainloss = []\n",
    "validationloss = []\n",
    "trainaccuracies = []\n",
    "valaccuracies = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for t in range(epochs):\n",
    "    tl, ta = train_1epoch(X_train_t,Y_train_t, model, loss_fn, optimiser)\n",
    "    vl, va = evaluate(x_val_t,y_val_t, model, loss_fn)\n",
    "    if (t+1)%10 == 0 :\n",
    "        print(f\"Epoch {t+1}\\tLosses:  Training {tl}, Validation {vl}.\" )\n",
    "    trainloss.append(tl)\n",
    "    validationloss.append(vl)\n",
    "    trainaccuracies.append(ta)\n",
    "    valaccuracies.append(va)\n",
    "print(\"Done!\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting learning curves\n",
    "Let's try and identify if we trained the best possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt  # type: ignore\n",
    "\n",
    "fig, axs = plt.subplots(1,2,figsize=(9, 4)) # plotting multiple panels: 1 row, 2 columns\n",
    "axs[0].plot(trainloss,'b-',label=\"Training Loss\")\n",
    "axs[0].plot(validationloss,'r-', label=\"Validation Loss\")\n",
    "axs[0].set_ylim(0, 1)\n",
    "axs[0].set_xlabel(\"Epochs\")\n",
    "axs[0].set_ylabel(\"Loss\")\n",
    "axs[0].legend()\n",
    "\n",
    "axs[1].plot(trainaccuracies,'b-',label=\"Training Accuracy\")\n",
    "axs[1].plot(valaccuracies,'r-', label=\"Validation Accuracy\")\n",
    "axs[1].set_ylim(0, 1)\n",
    "axs[1].set_xlabel(\"Epochs\")\n",
    "axs[1].set_ylabel(\"Loss\")\n",
    "axs[1].legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now you could test the model in the external test dataset to check generalisability. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training models is more of an art. I encourage you to look at other sources to learn more about it. A good resource is PyTorch's website. For example https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html or implementing grid-search with k-fold validation as in https://github.com/davidtvs/kaggle-titanic/blob/master/pytorch/surviving-the-titanic.ipynb\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}