Final_Modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Titanic Predictive Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will compare several different classification algorithms to determine which algorithm is best suited for the kaggle competition. We will use the following modeling packages\n",
    "* glmnet\n",
    "* nnet\n",
    "* gbm\n",
    "* glm\n",
    "* randomForest\n",
    "* K-nearest neighbors\n",
    "\n",
    "We will also be performing repeated cross validation on our models as well as parameter tuning using the `caret` package\n",
    "\n",
    "At the end we will also be experimenting with k-nearest neighbor imputation for missing values in the age column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Libraries and Data Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data is already cleaned of missing values and useless columns and is stored in `train_edited.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(caret)\n",
    "library(plyr)\n",
    "library(dplyr)\n",
    "library(nnet)\n",
    "library(glmnet)\n",
    "library(gbm)\n",
    "library(randomForest)\n",
    "library(class)\n",
    "library(doMC)\n",
    "registerDoMC(cores = 3)\n",
    "set.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t889 obs. of  11 variables:\n",
      " $ PassengerId: int  1 2 3 4 5 6 7 8 9 10 ...\n",
      " $ Survived   : Factor w/ 2 levels \"0\",\"1\": 1 2 2 2 1 1 1 1 2 2 ...\n",
      " $ Pclass     : int  3 1 3 1 3 3 1 3 3 2 ...\n",
      " $ Name       : Factor w/ 889 levels \"Abbing, Mr. Anthony\",..: 109 191 358 277 16 558 519 628 416 580 ...\n",
      " $ Sex        : Factor w/ 2 levels \"female\",\"male\": 2 1 1 1 2 2 2 2 1 1 ...\n",
      " $ Age        : num  22 38 26 35 35 ...\n",
      " $ SibSp      : int  1 1 0 1 0 0 0 3 0 1 ...\n",
      " $ Parch      : int  0 0 0 0 0 0 0 1 2 0 ...\n",
      " $ Ticket     : Factor w/ 680 levels \"110152\",\"110413\",..: 523 596 669 49 472 275 85 395 344 132 ...\n",
      " $ Fare       : num  7.25 71.28 7.92 53.1 8.05 ...\n",
      " $ Embarked   : Factor w/ 3 levels \"C\",\"Q\",\"S\": 3 1 3 3 3 2 3 3 3 1 ...\n"
     ]
    }
   ],
   "source": [
    "train <- read.csv('datasets/train_edited.csv', stringsAsFactors = TRUE)\n",
    "train$Survived <- as.factor(train$Survived)\n",
    "str(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we remove unecessary columns from our data that provide no current use to us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t889 obs. of  8 variables:\n",
      " $ Survived: Factor w/ 2 levels \"0\",\"1\": 1 2 2 2 1 1 1 1 2 2 ...\n",
      " $ Pclass  : int  3 1 3 1 3 3 1 3 3 2 ...\n",
      " $ Sex     : Factor w/ 2 levels \"female\",\"male\": 2 1 1 1 2 2 2 2 1 1 ...\n",
      " $ Age     : num  22 38 26 35 35 ...\n",
      " $ SibSp   : int  1 1 0 1 0 0 0 3 0 1 ...\n",
      " $ Parch   : int  0 0 0 0 0 0 0 1 2 0 ...\n",
      " $ Fare    : num  7.25 71.28 7.92 53.1 8.05 ...\n",
      " $ Embarked: Factor w/ 3 levels \"C\",\"Q\",\"S\": 3 1 3 3 3 2 3 3 3 1 ...\n"
     ]
    }
   ],
   "source": [
    "train <- train %>% select(-PassengerId, -Name, -Ticket)\n",
    "str(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Scaling and Splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t889 obs. of  8 variables:\n",
      " $ Survived: Factor w/ 2 levels \"0\",\"1\": 1 2 2 2 1 1 1 1 2 2 ...\n",
      " $ Pclass  : num  0.825 -1.571 0.825 -1.571 0.825 ...\n",
      " $ Sex     : Factor w/ 2 levels \"female\",\"male\": 2 1 1 1 2 2 2 2 1 1 ...\n",
      " $ Age     : num  -0.59 0.644 -0.282 0.412 0.412 ...\n",
      " $ SibSp   : num  0.431 0.431 -0.475 0.431 -0.475 ...\n",
      " $ Parch   : num  -0.474 -0.474 -0.474 -0.474 -0.474 ...\n",
      " $ Fare    : num  -0.5 0.789 -0.486 0.423 -0.484 ...\n",
      " $ Embarked: Factor w/ 3 levels \"C\",\"Q\",\"S\": 3 1 3 3 3 2 3 3 3 1 ...\n"
     ]
    }
   ],
   "source": [
    "preProcValues <- preProcess(train, method = c(\"center\", \"scale\"))\n",
    "train <- predict(preProcValues, train)\n",
    "str(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we skip over converting our factor columns to dummy variables although this is an option. Next we use createDataPartition to split our data into train and test. Here we use a 70/30 split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t623 obs. of  8 variables:\n",
      " $ Survived: Factor w/ 2 levels \"0\",\"1\": 1 2 2 1 1 1 1 2 2 2 ...\n",
      " $ Pclass  : num  0.825 -1.571 -1.571 0.825 0.825 ...\n",
      " $ Sex     : Factor w/ 2 levels \"female\",\"male\": 2 1 1 2 2 2 2 1 1 1 ...\n",
      " $ Age     : num  -0.59016 0.64361 0.41228 0.41228 0.00352 ...\n",
      " $ SibSp   : num  0.431 0.431 0.431 -0.475 -0.475 ...\n",
      " $ Parch   : num  -0.474 -0.474 -0.474 -0.474 -0.474 ...\n",
      " $ Fare    : num  -0.5 0.789 0.423 -0.484 -0.476 ...\n",
      " $ Embarked: Factor w/ 3 levels \"C\",\"Q\",\"S\": 3 1 3 3 2 3 3 3 1 3 ...\n"
     ]
    }
   ],
   "source": [
    "index <- createDataPartition(train$Survived, p = .70, list = FALSE)\n",
    "trainSet <- train[index, ]\n",
    "testSet <- train[-index, ]\n",
    "str(trainSet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Selection\n",
    "Here we use recursive feature elimination to select the top 5 most important predictors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Recursive feature selection\n",
       "\n",
       "Outer resampling method: Cross-Validated (10 fold, repeated 3 times) \n",
       "\n",
       "Resampling performance over subset size:\n",
       "\n",
       " Variables Accuracy  Kappa AccuracySD KappaSD Selected\n",
       "         4   0.8192 0.6049    0.03770 0.08757         \n",
       "         7   0.8342 0.6362    0.03048 0.06748        *\n",
       "\n",
       "The top 5 variables (out of 7):\n",
       "   Sex, Pclass, Age, Fare, Embarked\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "control <- rfeControl(functions = rfFuncs, method = \"repeatedcv\", repeats = 3, verbose = FALSE)\n",
    "titanic_pred_prof <- rfe(select(trainSet, -Survived), pull(trainSet, Survived), rfeControl = control)\n",
    "titanic_pred_prof"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These five predictors returned by caret will be used from now on in our data analysis. We will now update our `trainSet` and `testSet` data frames to only include these `Survived`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t623 obs. of  6 variables:\n",
      " $ Survived: Factor w/ 2 levels \"0\",\"1\": 1 2 2 1 1 1 1 2 2 2 ...\n",
      " $ Pclass  : num  0.825 -1.571 -1.571 0.825 0.825 ...\n",
      " $ Sex     : Factor w/ 2 levels \"female\",\"male\": 2 1 1 2 2 2 2 1 1 1 ...\n",
      " $ Age     : num  -0.59016 0.64361 0.41228 0.41228 0.00352 ...\n",
      " $ SibSp   : num  0.431 0.431 0.431 -0.475 -0.475 ...\n",
      " $ Fare    : num  -0.5 0.789 0.423 -0.484 -0.476 ...\n"
     ]
    }
   ],
   "source": [
    "trainSet <- trainSet %>% select(-Parch, -Embarked)\n",
    "testSet <- testSet %>% select(-Parch, -Embarked)\n",
    "str(trainSet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## glmnet Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we will look up the different parameters that we can modify for the glmnet model. Then we build a train control stating we will be using repeated cross validation to train our data. We specify the number of different parameter options to try for each parameter with tuneLength. Finally we train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>model</th><th scope=col>parameter</th><th scope=col>label</th><th scope=col>forReg</th><th scope=col>forClass</th><th scope=col>probModel</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>glmnet                  </td><td>alpha                   </td><td>Mixing Percentage       </td><td>TRUE                    </td><td>TRUE                    </td><td>TRUE                    </td></tr>\n",
       "\t<tr><td>glmnet                  </td><td>lambda                  </td><td>Regularization Parameter</td><td>TRUE                    </td><td>TRUE                    </td><td>TRUE                    </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " model & parameter & label & forReg & forClass & probModel\\\\\n",
       "\\hline\n",
       "\t glmnet                   & alpha                    & Mixing Percentage        & TRUE                     & TRUE                     & TRUE                    \\\\\n",
       "\t glmnet                   & lambda                   & Regularization Parameter & TRUE                     & TRUE                     & TRUE                    \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "model | parameter | label | forReg | forClass | probModel | \n",
       "|---|---|\n",
       "| glmnet                   | alpha                    | Mixing Percentage        | TRUE                     | TRUE                     | TRUE                     | \n",
       "| glmnet                   | lambda                   | Regularization Parameter | TRUE                     | TRUE                     | TRUE                     | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  model  parameter label                    forReg forClass probModel\n",
       "1 glmnet alpha     Mixing Percentage        TRUE   TRUE     TRUE     \n",
       "2 glmnet lambda    Regularization Parameter TRUE   TRUE     TRUE     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "modelLookup(model = 'glmnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>alpha</th><th scope=col>lambda</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>466</th><td>1         </td><td>0.01607879</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       "  & alpha & lambda\\\\\n",
       "\\hline\n",
       "\t466 & 1          & 0.01607879\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| <!--/--> | alpha | lambda | \n",
       "|---|\n",
       "| 466 | 1          | 0.01607879 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "    alpha lambda    \n",
       "466 1     0.01607879"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>TrainAccuracy</th><th scope=col>TrainKappa</th><th scope=col>method</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.8096413</td><td>0.5918576</td><td>glmnet   </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " TrainAccuracy & TrainKappa & method\\\\\n",
       "\\hline\n",
       "\t 0.8096413 & 0.5918576 & glmnet   \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "TrainAccuracy | TrainKappa | method | \n",
       "|---|\n",
       "| 0.8096413 | 0.5918576 | glmnet    | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  TrainAccuracy TrainKappa method\n",
       "1 0.8096413     0.5918576  glmnet"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Total time elapsed in minutes since training started 0.342666666666673\"\n"
     ]
    }
   ],
   "source": [
    "startTime <- proc.time()\n",
    "trainControl <- trainControl(method = 'repeatedcv', number = 5, repeats = 5)\n",
    "model_glmnet <- train(Survived ~., data = trainSet, method = 'glmnet', trControl = trainControl, tuneLength = 25)\n",
    "model_glmnet$bestTune\n",
    "getTrainPerf(model_glmnet)\n",
    "print(paste(\"Total time elapsed in minutes since training started\", (proc.time()[3] - startTime[3]) / 60))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we can see our glmnet parameters as well as our accuracy and kappa scores. Now we will use our model to predict on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Confusion Matrix and Statistics\n",
       "\n",
       "          Reference\n",
       "Prediction   0   1\n",
       "         0 145  43\n",
       "         1  19  59\n",
       "                                          \n",
       "               Accuracy : 0.7669          \n",
       "                 95% CI : (0.7114, 0.8164)\n",
       "    No Information Rate : 0.6165          \n",
       "    P-Value [Acc > NIR] : 1.294e-07       \n",
       "                                          \n",
       "                  Kappa : 0.4841          \n",
       " Mcnemar's Test P-Value : 0.003489        \n",
       "                                          \n",
       "            Sensitivity : 0.8841          \n",
       "            Specificity : 0.5784          \n",
       "         Pos Pred Value : 0.7713          \n",
       "         Neg Pred Value : 0.7564          \n",
       "             Prevalence : 0.6165          \n",
       "         Detection Rate : 0.5451          \n",
       "   Detection Prevalence : 0.7068          \n",
       "      Balanced Accuracy : 0.7313          \n",
       "                                          \n",
       "       'Positive' Class : 0               \n",
       "                                          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions_glmnet <- predict(model_glmnet, newdata = select(testSet, -Survived), type = 'raw')\n",
    "confusionMatrix(predictions_glmnet, testSet$Survived)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## nnet Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we will repeat the steps of glmnet modeling except we will now use the neural net package nnet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>model</th><th scope=col>parameter</th><th scope=col>label</th><th scope=col>forReg</th><th scope=col>forClass</th><th scope=col>probModel</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>nnet         </td><td>size         </td><td>#Hidden Units</td><td>TRUE         </td><td>TRUE         </td><td>TRUE         </td></tr>\n",
       "\t<tr><td>nnet         </td><td>decay        </td><td>Weight Decay </td><td>TRUE         </td><td>TRUE         </td><td>TRUE         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " model & parameter & label & forReg & forClass & probModel\\\\\n",
       "\\hline\n",
       "\t nnet            & size            & \\#Hidden Units & TRUE            & TRUE            & TRUE           \\\\\n",
       "\t nnet          & decay         & Weight Decay  & TRUE          & TRUE          & TRUE         \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "model | parameter | label | forReg | forClass | probModel | \n",
       "|---|---|\n",
       "| nnet          | size          | #Hidden Units | TRUE          | TRUE          | TRUE          | \n",
       "| nnet          | decay         | Weight Decay  | TRUE          | TRUE          | TRUE          | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  model parameter label         forReg forClass probModel\n",
       "1 nnet  size      #Hidden Units TRUE   TRUE     TRUE     \n",
       "2 nnet  decay     Weight Decay  TRUE   TRUE     TRUE     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "modelLookup(model = 'nnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# weights:  78\n",
      "initial  value 521.919047 \n",
      "iter  10 value 262.034850\n",
      "iter  20 value 251.836160\n",
      "iter  30 value 248.078092\n",
      "iter  40 value 245.276956\n",
      "iter  50 value 243.820814\n",
      "iter  60 value 243.503463\n",
      "iter  70 value 243.158848\n",
      "iter  80 value 242.941994\n",
      "iter  90 value 242.743149\n",
      "iter 100 value 242.676261\n",
      "final  value 242.676261 \n",
      "stopped after 100 iterations\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>size</th><th scope=col>decay</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>60</th><td>11 </td><td>0.1</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       "  & size & decay\\\\\n",
       "\\hline\n",
       "\t60 & 11  & 0.1\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| <!--/--> | size | decay | \n",
       "|---|\n",
       "| 60 | 11  | 0.1 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "   size decay\n",
       "60 11   0.1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>TrainAccuracy</th><th scope=col>TrainKappa</th><th scope=col>method</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.812849 </td><td>0.5939004</td><td>nnet     </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " TrainAccuracy & TrainKappa & method\\\\\n",
       "\\hline\n",
       "\t 0.812849  & 0.5939004 & nnet     \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "TrainAccuracy | TrainKappa | method | \n",
       "|---|\n",
       "| 0.812849  | 0.5939004 | nnet      | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  TrainAccuracy TrainKappa method\n",
       "1 0.812849      0.5939004  nnet  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Total time elapsed in minutes since training started 1.34325\"\n"
     ]
    }
   ],
   "source": [
    "startTime <- proc.time()\n",
    "trainControl <- trainControl(method = 'repeatedcv', number = 5, repeats = 5)\n",
    "model_nnet <- train(Survived ~., data = trainSet, method = 'nnet', trControl = trainControl, tuneLength = 10)\n",
    "model_nnet$bestTune\n",
    "getTrainPerf(model_nnet)\n",
    "print(paste(\"Total time elapsed in minutes since training started\", (proc.time()[3] - startTime[3]) / 60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Confusion Matrix and Statistics\n",
       "\n",
       "          Reference\n",
       "Prediction   0   1\n",
       "         0 147  33\n",
       "         1  17  69\n",
       "                                          \n",
       "               Accuracy : 0.812           \n",
       "                 95% CI : (0.7598, 0.8571)\n",
       "    No Information Rate : 0.6165          \n",
       "    P-Value [Acc > NIR] : 4.357e-12       \n",
       "                                          \n",
       "                  Kappa : 0.5903          \n",
       " Mcnemar's Test P-Value : 0.03389         \n",
       "                                          \n",
       "            Sensitivity : 0.8963          \n",
       "            Specificity : 0.6765          \n",
       "         Pos Pred Value : 0.8167          \n",
       "         Neg Pred Value : 0.8023          \n",
       "             Prevalence : 0.6165          \n",
       "         Detection Rate : 0.5526          \n",
       "   Detection Prevalence : 0.6767          \n",
       "      Balanced Accuracy : 0.7864          \n",
       "                                          \n",
       "       'Positive' Class : 0               \n",
       "                                          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions_nnet <- predict(model_nnet, newdata = select(testSet, -Survived), type = 'raw')\n",
    "confusionMatrix(predictions_nnet, testSet$Survived)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## gbm Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>model</th><th scope=col>parameter</th><th scope=col>label</th><th scope=col>forReg</th><th scope=col>forClass</th><th scope=col>probModel</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>gbm                    </td><td>n.trees                </td><td># Boosting Iterations  </td><td>TRUE                   </td><td>TRUE                   </td><td>TRUE                   </td></tr>\n",
       "\t<tr><td>gbm                    </td><td>interaction.depth      </td><td>Max Tree Depth         </td><td>TRUE                   </td><td>TRUE                   </td><td>TRUE                   </td></tr>\n",
       "\t<tr><td>gbm                    </td><td>shrinkage              </td><td>Shrinkage              </td><td>TRUE                   </td><td>TRUE                   </td><td>TRUE                   </td></tr>\n",
       "\t<tr><td>gbm                    </td><td>n.minobsinnode         </td><td>Min. Terminal Node Size</td><td>TRUE                   </td><td>TRUE                   </td><td>TRUE                   </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " model & parameter & label & forReg & forClass & probModel\\\\\n",
       "\\hline\n",
       "\t gbm                       & n.trees                   & \\# Boosting Iterations   & TRUE                      & TRUE                      & TRUE                     \\\\\n",
       "\t gbm                     & interaction.depth       & Max Tree Depth          & TRUE                    & TRUE                    & TRUE                   \\\\\n",
       "\t gbm                     & shrinkage               & Shrinkage               & TRUE                    & TRUE                    & TRUE                   \\\\\n",
       "\t gbm                     & n.minobsinnode          & Min. Terminal Node Size & TRUE                    & TRUE                    & TRUE                   \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "model | parameter | label | forReg | forClass | probModel | \n",
       "|---|---|---|---|\n",
       "| gbm                     | n.trees                 | # Boosting Iterations   | TRUE                    | TRUE                    | TRUE                    | \n",
       "| gbm                     | interaction.depth       | Max Tree Depth          | TRUE                    | TRUE                    | TRUE                    | \n",
       "| gbm                     | shrinkage               | Shrinkage               | TRUE                    | TRUE                    | TRUE                    | \n",
       "| gbm                     | n.minobsinnode          | Min. Terminal Node Size | TRUE                    | TRUE                    | TRUE                    | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  model parameter         label                   forReg forClass probModel\n",
       "1 gbm   n.trees           # Boosting Iterations   TRUE   TRUE     TRUE     \n",
       "2 gbm   interaction.depth Max Tree Depth          TRUE   TRUE     TRUE     \n",
       "3 gbm   shrinkage         Shrinkage               TRUE   TRUE     TRUE     \n",
       "4 gbm   n.minobsinnode    Min. Terminal Node Size TRUE   TRUE     TRUE     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "modelLookup(model = 'gbm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter   TrainDeviance   ValidDeviance   StepSize   Improve\n",
      "     1        1.2407             nan     0.1000    0.0392\n",
      "     2        1.1769             nan     0.1000    0.0331\n",
      "     3        1.1276             nan     0.1000    0.0238\n",
      "     4        1.0843             nan     0.1000    0.0200\n",
      "     5        1.0482             nan     0.1000    0.0175\n",
      "     6        1.0168             nan     0.1000    0.0129\n",
      "     7        0.9877             nan     0.1000    0.0104\n",
      "     8        0.9617             nan     0.1000    0.0114\n",
      "     9        0.9400             nan     0.1000    0.0097\n",
      "    10        0.9200             nan     0.1000    0.0084\n",
      "    20        0.8141             nan     0.1000   -0.0005\n",
      "    40        0.7137             nan     0.1000   -0.0002\n",
      "    60        0.6639             nan     0.1000   -0.0028\n",
      "    80        0.6287             nan     0.1000   -0.0025\n",
      "   100        0.5958             nan     0.1000   -0.0021\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>n.trees</th><th scope=col>interaction.depth</th><th scope=col>shrinkage</th><th scope=col>n.minobsinnode</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>42</th><td>100</td><td>5  </td><td>0.1</td><td>10 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       "  & n.trees & interaction.depth & shrinkage & n.minobsinnode\\\\\n",
       "\\hline\n",
       "\t42 & 100 & 5   & 0.1 & 10 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| <!--/--> | n.trees | interaction.depth | shrinkage | n.minobsinnode | \n",
       "|---|\n",
       "| 42 | 100 | 5   | 0.1 | 10  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "   n.trees interaction.depth shrinkage n.minobsinnode\n",
       "42 100     5                 0.1       10            "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>TrainAccuracy</th><th scope=col>TrainKappa</th><th scope=col>method</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.8244155</td><td>0.6195497</td><td>gbm      </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " TrainAccuracy & TrainKappa & method\\\\\n",
       "\\hline\n",
       "\t 0.8244155 & 0.6195497 & gbm      \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "TrainAccuracy | TrainKappa | method | \n",
       "|---|\n",
       "| 0.8244155 | 0.6195497 | gbm       | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  TrainAccuracy TrainKappa method\n",
       "1 0.8244155     0.6195497  gbm   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Total time elapsed in minutes since training started 0.449133333333324\"\n"
     ]
    }
   ],
   "source": [
    "startTime <- proc.time()\n",
    "trainControl <- trainControl(method = 'repeatedcv', number = 5, repeats = 5)\n",
    "model_gbm <- train(Survived ~., data = trainSet, method = 'gbm', trControl = trainControl, tuneLength = 10)\n",
    "model_gbm$bestTune\n",
    "getTrainPerf(model_gbm)\n",
    "print(paste(\"Total time elapsed in minutes since training started\", (proc.time()[3] - startTime[3]) / 60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Confusion Matrix and Statistics\n",
       "\n",
       "          Reference\n",
       "Prediction   0   1\n",
       "         0 151  30\n",
       "         1  13  72\n",
       "                                          \n",
       "               Accuracy : 0.8383          \n",
       "                 95% CI : (0.7885, 0.8805)\n",
       "    No Information Rate : 0.6165          \n",
       "    P-Value [Acc > NIR] : 2.231e-15       \n",
       "                                          \n",
       "                  Kappa : 0.647           \n",
       " Mcnemar's Test P-Value : 0.01469         \n",
       "                                          \n",
       "            Sensitivity : 0.9207          \n",
       "            Specificity : 0.7059          \n",
       "         Pos Pred Value : 0.8343          \n",
       "         Neg Pred Value : 0.8471          \n",
       "             Prevalence : 0.6165          \n",
       "         Detection Rate : 0.5677          \n",
       "   Detection Prevalence : 0.6805          \n",
       "      Balanced Accuracy : 0.8133          \n",
       "                                          \n",
       "       'Positive' Class : 0               \n",
       "                                          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions_gbm <- predict(model_gbm, newdata = select(testSet, -Survived), type = 'raw')\n",
    "confusionMatrix(predictions_gbm, testSet$Survived)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## glm modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>model</th><th scope=col>parameter</th><th scope=col>label</th><th scope=col>forReg</th><th scope=col>forClass</th><th scope=col>probModel</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>glm      </td><td>parameter</td><td>parameter</td><td>TRUE     </td><td>TRUE     </td><td>TRUE     </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " model & parameter & label & forReg & forClass & probModel\\\\\n",
       "\\hline\n",
       "\t glm       & parameter & parameter & TRUE      & TRUE      & TRUE     \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "model | parameter | label | forReg | forClass | probModel | \n",
       "|---|\n",
       "| glm       | parameter | parameter | TRUE      | TRUE      | TRUE      | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  model parameter label     forReg forClass probModel\n",
       "1 glm   parameter parameter TRUE   TRUE     TRUE     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "modelLookup(model = 'glm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>parameter</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>none</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " parameter\\\\\n",
       "\\hline\n",
       "\t none\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "parameter | \n",
       "|---|\n",
       "| none | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  parameter\n",
       "1 none     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>TrainAccuracy</th><th scope=col>TrainKappa</th><th scope=col>method</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.7980697</td><td>0.5700364</td><td>glm      </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " TrainAccuracy & TrainKappa & method\\\\\n",
       "\\hline\n",
       "\t 0.7980697 & 0.5700364 & glm      \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "TrainAccuracy | TrainKappa | method | \n",
       "|---|\n",
       "| 0.7980697 | 0.5700364 | glm       | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  TrainAccuracy TrainKappa method\n",
       "1 0.7980697     0.5700364  glm   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Total time elapsed in minutes since training started 0.0192000000000007\"\n"
     ]
    }
   ],
   "source": [
    "startTime <- proc.time()\n",
    "trainControl <- trainControl(method = 'repeatedcv', number = 5, repeats = 5)\n",
    "model_glm <- train(Survived ~., data = trainSet, method = 'glm', trControl = trainControl)\n",
    "model_glm$bestTune\n",
    "getTrainPerf(model_glm)\n",
    "print(paste(\"Total time elapsed in minutes since training started\", (proc.time()[3] - startTime[3]) / 60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Confusion Matrix and Statistics\n",
       "\n",
       "          Reference\n",
       "Prediction   0   1\n",
       "         0 142  39\n",
       "         1  22  63\n",
       "                                          \n",
       "               Accuracy : 0.7707          \n",
       "                 95% CI : (0.7154, 0.8198)\n",
       "    No Information Rate : 0.6165          \n",
       "    P-Value [Acc > NIR] : 6.178e-08       \n",
       "                                          \n",
       "                  Kappa : 0.4992          \n",
       " Mcnemar's Test P-Value : 0.0405          \n",
       "                                          \n",
       "            Sensitivity : 0.8659          \n",
       "            Specificity : 0.6176          \n",
       "         Pos Pred Value : 0.7845          \n",
       "         Neg Pred Value : 0.7412          \n",
       "             Prevalence : 0.6165          \n",
       "         Detection Rate : 0.5338          \n",
       "   Detection Prevalence : 0.6805          \n",
       "      Balanced Accuracy : 0.7418          \n",
       "                                          \n",
       "       'Positive' Class : 0               \n",
       "                                          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions_glm <- predict(model_glm, newdata = select(testSet, -Survived), type = 'raw')\n",
    "confusionMatrix(predictions_glm, testSet$Survived)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## randomForest Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>model</th><th scope=col>parameter</th><th scope=col>label</th><th scope=col>forReg</th><th scope=col>forClass</th><th scope=col>probModel</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>rf                           </td><td>mtry                         </td><td>#Randomly Selected Predictors</td><td>TRUE                         </td><td>TRUE                         </td><td>TRUE                         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " model & parameter & label & forReg & forClass & probModel\\\\\n",
       "\\hline\n",
       "\t rf                              & mtry                            & \\#Randomly Selected Predictors & TRUE                            & TRUE                            & TRUE                           \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "model | parameter | label | forReg | forClass | probModel | \n",
       "|---|\n",
       "| rf                            | mtry                          | #Randomly Selected Predictors | TRUE                          | TRUE                          | TRUE                          | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  model parameter label                         forReg forClass probModel\n",
       "1 rf    mtry      #Randomly Selected Predictors TRUE   TRUE     TRUE     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "modelLookup(model = 'rf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "note: only 4 unique complexity parameters in default grid. Truncating the grid to 4 .\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>mtry</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>2</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " mtry\\\\\n",
       "\\hline\n",
       "\t 2\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "mtry | \n",
       "|---|\n",
       "| 2 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  mtry\n",
       "1 2   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>TrainAccuracy</th><th scope=col>TrainKappa</th><th scope=col>method</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.8211974</td><td>0.6124393</td><td>rf       </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " TrainAccuracy & TrainKappa & method\\\\\n",
       "\\hline\n",
       "\t 0.8211974 & 0.6124393 & rf       \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "TrainAccuracy | TrainKappa | method | \n",
       "|---|\n",
       "| 0.8211974 | 0.6124393 | rf        | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  TrainAccuracy TrainKappa method\n",
       "1 0.8211974     0.6124393  rf    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Total time elapsed in minutes since training started 0.232233333333321\"\n"
     ]
    }
   ],
   "source": [
    "startTime <- proc.time()\n",
    "trainControl <- trainControl(method = 'repeatedcv', number = 5, repeats = 5)\n",
    "model_rf <- train(Survived ~., data = trainSet, method = 'rf', trControl = trainControl, tuneLength = 10)\n",
    "model_rf$bestTune\n",
    "getTrainPerf(model_rf)\n",
    "print(paste(\"Total time elapsed in minutes since training started\", (proc.time()[3] - startTime[3]) / 60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Confusion Matrix and Statistics\n",
       "\n",
       "          Reference\n",
       "Prediction   0   1\n",
       "         0 151  32\n",
       "         1  13  70\n",
       "                                          \n",
       "               Accuracy : 0.8308          \n",
       "                 95% CI : (0.7803, 0.8738)\n",
       "    No Information Rate : 0.6165          \n",
       "    P-Value [Acc > NIR] : 2.211e-14       \n",
       "                                          \n",
       "                  Kappa : 0.6292          \n",
       " Mcnemar's Test P-Value : 0.00729         \n",
       "                                          \n",
       "            Sensitivity : 0.9207          \n",
       "            Specificity : 0.6863          \n",
       "         Pos Pred Value : 0.8251          \n",
       "         Neg Pred Value : 0.8434          \n",
       "             Prevalence : 0.6165          \n",
       "         Detection Rate : 0.5677          \n",
       "   Detection Prevalence : 0.6880          \n",
       "      Balanced Accuracy : 0.8035          \n",
       "                                          \n",
       "       'Positive' Class : 0               \n",
       "                                          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions_rf <- predict(model_rf, newdata = select(testSet, -Survived), type = 'raw')\n",
    "confusionMatrix(predictions_rf, testSet$Survived)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## knn Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>model</th><th scope=col>parameter</th><th scope=col>label</th><th scope=col>forReg</th><th scope=col>forClass</th><th scope=col>probModel</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>knn       </td><td>k         </td><td>#Neighbors</td><td>TRUE      </td><td>TRUE      </td><td>TRUE      </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " model & parameter & label & forReg & forClass & probModel\\\\\n",
       "\\hline\n",
       "\t knn          & k            & \\#Neighbors & TRUE         & TRUE         & TRUE        \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "model | parameter | label | forReg | forClass | probModel | \n",
       "|---|\n",
       "| knn        | k          | #Neighbors | TRUE       | TRUE       | TRUE       | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  model parameter label      forReg forClass probModel\n",
       "1 knn   k         #Neighbors TRUE   TRUE     TRUE     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "modelLookup(model = 'knn')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>k</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>15</th><td>33</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       "  & k\\\\\n",
       "\\hline\n",
       "\t15 & 33\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| <!--/--> | k | \n",
       "|---|\n",
       "| 15 | 33 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "   k \n",
       "15 33"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>TrainAccuracy</th><th scope=col>TrainKappa</th><th scope=col>method</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.8051535</td><td>0.5835262</td><td>knn      </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " TrainAccuracy & TrainKappa & method\\\\\n",
       "\\hline\n",
       "\t 0.8051535 & 0.5835262 & knn      \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "TrainAccuracy | TrainKappa | method | \n",
       "|---|\n",
       "| 0.8051535 | 0.5835262 | knn       | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  TrainAccuracy TrainKappa method\n",
       "1 0.8051535     0.5835262  knn   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Total time elapsed in minutes since training started 0.058766666666664\"\n"
     ]
    }
   ],
   "source": [
    "startTime <- proc.time()\n",
    "trainControl <- trainControl(method = 'repeatedcv', number = 5, repeats = 5)\n",
    "model_knn <- train(Survived ~., data = trainSet, method = 'knn', trControl = trainControl, tuneLength = 20)\n",
    "model_knn$bestTune\n",
    "getTrainPerf(model_knn)\n",
    "print(paste(\"Total time elapsed in minutes since training started\", (proc.time()[3] - startTime[3]) / 60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Confusion Matrix and Statistics\n",
       "\n",
       "          Reference\n",
       "Prediction   0   1\n",
       "         0 147  32\n",
       "         1  17  70\n",
       "                                          \n",
       "               Accuracy : 0.8158          \n",
       "                 95% CI : (0.7639, 0.8605)\n",
       "    No Information Rate : 0.6165          \n",
       "    P-Value [Acc > NIR] : 1.592e-12       \n",
       "                                          \n",
       "                  Kappa : 0.5993          \n",
       " Mcnemar's Test P-Value : 0.0455          \n",
       "                                          \n",
       "            Sensitivity : 0.8963          \n",
       "            Specificity : 0.6863          \n",
       "         Pos Pred Value : 0.8212          \n",
       "         Neg Pred Value : 0.8046          \n",
       "             Prevalence : 0.6165          \n",
       "         Detection Rate : 0.5526          \n",
       "   Detection Prevalence : 0.6729          \n",
       "      Balanced Accuracy : 0.7913          \n",
       "                                          \n",
       "       'Positive' Class : 0               \n",
       "                                          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions_knn <- predict(model_knn, newdata = select(testSet, -Survived), type = 'raw')\n",
    "confusionMatrix(predictions_knn, testSet$Survived)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further things to look into\n",
    "* Using all features\n",
    "* Results that are not reproducible - parameters change every time train is run - seeding problem because we are using doMC for parallel sampling\n",
    "* knn - number of neighbors changes wildly - 11 to 33 - but accuracy still ~80%\n",
    "* knn imputation for age variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "Overall, the models that performed > 80% were gbm and rf with 83%. nnet, knn both had accuracies of 81% when used to predict test data values. From here it is obvious that more work must be done with modeling tuning as well as data refinement (possibly look for new features to make) to increase accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}