Fainal Project The best classifier.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36057288-cd97-4358-ba70-e2ac3c367204",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "Classification with Python\n",
    "We load a dataset using Pandas library, and apply the following algorithms, and find the best one for this specific dataset by accuracy evaluation methods.\n",
    "\n",
    "Lets first load required libraries:\n",
    "\n",
    "import itertools\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.ticker import NullFormatter\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.ticker as ticker\n",
    "from sklearn import preprocessing\n",
    "%matplotlib inline\n",
    "About dataset\n",
    "This dataset is about past loans. The Loan_train.csv data set includes details of 346 customers whose loan are already paid off or defaulted. It includes following fields:\n",
    "\n",
    "Field\tDescription\n",
    "Loan_status\tWhether a loan is paid off on in collection\n",
    "Principal\tBasic principal loan amount at the\n",
    "Terms\tOrigination terms which can be weekly (7 days), biweekly, and monthly payoff schedule\n",
    "Effective_date\tWhen the loan got originated and took effects\n",
    "Due_date\tSince it’s one-time payoff schedule, each loan has one single due date\n",
    "Age\tAge of applicant\n",
    "Education\tEducation of applicant\n",
    "Gender\tThe gender of applicant\n",
    "Lets download the dataset\n",
    "\n",
    "!wget -O loan_train.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_train.csv\n",
    "--2019-01-04 13:17:13--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_train.csv\n",
    "Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 67.228.254.193\n",
    "Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|67.228.254.193|:443... connected.\n",
    "HTTP request sent, awaiting response... 200 OK\n",
    "Length: 23101 (23K) [text/csv]\n",
    "Saving to: ‘loan_train.csv’\n",
    "\n",
    "100%[======================================>] 23,101      --.-K/s   in 0.002s  \n",
    "\n",
    "2019-01-04 13:17:13 (14.0 MB/s) - ‘loan_train.csv’ saved [23101/23101]\n",
    "\n",
    "Load Data From CSV File\n",
    "df = pd.read_csv('loan_train.csv')\n",
    "df.head()\n",
    "Unnamed: 0\tUnnamed: 0.1\tloan_status\tPrincipal\tterms\teffective_date\tdue_date\tage\teducation\tGender\n",
    "0\t0\t0\tPAIDOFF\t1000\t30\t9/8/2016\t10/7/2016\t45\tHigh School or Below\tmale\n",
    "1\t2\t2\tPAIDOFF\t1000\t30\t9/8/2016\t10/7/2016\t33\tBechalor\tfemale\n",
    "2\t3\t3\tPAIDOFF\t1000\t15\t9/8/2016\t9/22/2016\t27\tcollege\tmale\n",
    "3\t4\t4\tPAIDOFF\t1000\t30\t9/9/2016\t10/8/2016\t28\tcollege\tfemale\n",
    "4\t6\t6\tPAIDOFF\t1000\t30\t9/9/2016\t10/8/2016\t29\tcollege\tmale\n",
    "df.shape\n",
    "(346, 10)\n",
    "Convert to date time object\n",
    "df['due_date'] = pd.to_datetime(df['due_date'])\n",
    "df['effective_date'] = pd.to_datetime(df['effective_date'])\n",
    "df.head()\n",
    "Unnamed: 0\tUnnamed: 0.1\tloan_status\tPrincipal\tterms\teffective_date\tdue_date\tage\teducation\tGender\n",
    "0\t0\t0\tPAIDOFF\t1000\t30\t2016-09-08\t2016-10-07\t45\tHigh School or Below\tmale\n",
    "1\t2\t2\tPAIDOFF\t1000\t30\t2016-09-08\t2016-10-07\t33\tBechalor\tfemale\n",
    "2\t3\t3\tPAIDOFF\t1000\t15\t2016-09-08\t2016-09-22\t27\tcollege\tmale\n",
    "3\t4\t4\tPAIDOFF\t1000\t30\t2016-09-09\t2016-10-08\t28\tcollege\tfemale\n",
    "4\t6\t6\tPAIDOFF\t1000\t30\t2016-09-09\t2016-10-08\t29\tcollege\tmale\n",
    "Data visualization and pre-processing\n",
    "Let’s see how many of each class is in our data set\n",
    "\n",
    "df['loan_status'].value_counts()\n",
    "PAIDOFF       260\n",
    "COLLECTION     86\n",
    "Name: loan_status, dtype: int64\n",
    "260 people have paid off the loan on time while 86 have gone into collection\n",
    "\n",
    "Lets plot some columns to underestand data better:\n",
    "\n",
    "# notice: installing seaborn might takes a few minutes\n",
    "!conda install -c anaconda seaborn -y\n",
    "Solving environment: done\n",
    "\n",
    "## Package Plan ##\n",
    "\n",
    "  environment location: /Users/Saeed/anaconda/envs/python3.6\n",
    "\n",
    "  added / updated specs: \n",
    "    - seaborn\n",
    "\n",
    "\n",
    "The following packages will be downloaded:\n",
    "\n",
    "    package                    |            build\n",
    "    ---------------------------|-----------------\n",
    "    openssl-1.0.2o             |       h26aff7b_0         3.4 MB  anaconda\n",
    "    ca-certificates-2018.03.07 |                0         124 KB  anaconda\n",
    "    ------------------------------------------------------------\n",
    "                                           Total:         3.5 MB\n",
    "\n",
    "The following packages will be UPDATED:\n",
    "\n",
    "    ca-certificates: 2018.03.07-0      --> 2018.03.07-0      anaconda\n",
    "    openssl:         1.0.2o-h26aff7b_0 --> 1.0.2o-h26aff7b_0 anaconda\n",
    "\n",
    "\n",
    "Downloading and Extracting Packages\n",
    "openssl-1.0.2o       |  3.4 MB | ####################################### | 100% \n",
    "ca-certificates-2018 |  124 KB | ####################################### | 100% \n",
    "Preparing transaction: done\n",
    "Verifying transaction: done\n",
    "Executing transaction: done\n",
    "import seaborn as sns\n",
    "\n",
    "bins = np.linspace(df.Principal.min(), df.Principal.max(), 10)\n",
    "g = sns.FacetGrid(df, col=\"Gender\", hue=\"loan_status\", palette=\"Set1\", col_wrap=2)\n",
    "g.map(plt.hist, 'Principal', bins=bins, ec=\"k\")\n",
    "\n",
    "g.axes[-1].legend()\n",
    "plt.show()\n",
    "\n",
    "bins = np.linspace(df.age.min(), df.age.max(), 10)\n",
    "g = sns.FacetGrid(df, col=\"Gender\", hue=\"loan_status\", palette=\"Set1\", col_wrap=2)\n",
    "g.map(plt.hist, 'age', bins=bins, ec=\"k\")\n",
    "\n",
    "g.axes[-1].legend()\n",
    "plt.show()\n",
    "\n",
    "Pre-processing: Feature selection/extraction\n",
    "Lets look at the day of the week people get the loan\n",
    "df['dayofweek'] = df['effective_date'].dt.dayofweek\n",
    "bins = np.linspace(df.dayofweek.min(), df.dayofweek.max(), 10)\n",
    "g = sns.FacetGrid(df, col=\"Gender\", hue=\"loan_status\", palette=\"Set1\", col_wrap=2)\n",
    "g.map(plt.hist, 'dayofweek', bins=bins, ec=\"k\")\n",
    "g.axes[-1].legend()\n",
    "plt.show()\n",
    "\n",
    "We see that people who get the loan at the end of the week dont pay it off, so lets use Feature binarization to set a threshold values less then day 4\n",
    "\n",
    "df['weekend'] = df['dayofweek'].apply(lambda x: 1 if (x>3)  else 0)\n",
    "df.head()\n",
    "Unnamed: 0\tUnnamed: 0.1\tloan_status\tPrincipal\tterms\teffective_date\tdue_date\tage\teducation\tGender\tdayofweek\tweekend\n",
    "0\t0\t0\tPAIDOFF\t1000\t30\t2016-09-08\t2016-10-07\t45\tHigh School or Below\tmale\t3\t0\n",
    "1\t2\t2\tPAIDOFF\t1000\t30\t2016-09-08\t2016-10-07\t33\tBechalor\tfemale\t3\t0\n",
    "2\t3\t3\tPAIDOFF\t1000\t15\t2016-09-08\t2016-09-22\t27\tcollege\tmale\t3\t0\n",
    "3\t4\t4\tPAIDOFF\t1000\t30\t2016-09-09\t2016-10-08\t28\tcollege\tfemale\t4\t1\n",
    "4\t6\t6\tPAIDOFF\t1000\t30\t2016-09-09\t2016-10-08\t29\tcollege\tmale\t4\t1\n",
    "Convert Categorical features to numerical values\n",
    "Lets look at gender:\n",
    "\n",
    "df.groupby(['Gender'])['loan_status'].value_counts(normalize=True)\n",
    "Gender  loan_status\n",
    "female  PAIDOFF        0.865385\n",
    "        COLLECTION     0.134615\n",
    "male    PAIDOFF        0.731293\n",
    "        COLLECTION     0.268707\n",
    "Name: loan_status, dtype: float64\n",
    "86 % of female pay there loans while only 73 % of males pay there loan\n",
    "\n",
    "Lets convert male to 0 and female to 1:\n",
    "\n",
    "df['Gender'].replace(to_replace=['male','female'], value=[0,1],inplace=True)\n",
    "df.head()\n",
    "Unnamed: 0\tUnnamed: 0.1\tloan_status\tPrincipal\tterms\teffective_date\tdue_date\tage\teducation\tGender\tdayofweek\tweekend\n",
    "0\t0\t0\tPAIDOFF\t1000\t30\t2016-09-08\t2016-10-07\t45\tHigh School or Below\t0\t3\t0\n",
    "1\t2\t2\tPAIDOFF\t1000\t30\t2016-09-08\t2016-10-07\t33\tBechalor\t1\t3\t0\n",
    "2\t3\t3\tPAIDOFF\t1000\t15\t2016-09-08\t2016-09-22\t27\tcollege\t0\t3\t0\n",
    "3\t4\t4\tPAIDOFF\t1000\t30\t2016-09-09\t2016-10-08\t28\tcollege\t1\t4\t1\n",
    "4\t6\t6\tPAIDOFF\t1000\t30\t2016-09-09\t2016-10-08\t29\tcollege\t0\t4\t1\n",
    "One Hot Encoding\n",
    "How about education?\n",
    "df.groupby(['education'])['loan_status'].value_counts(normalize=True)\n",
    "education             loan_status\n",
    "Bechalor              PAIDOFF        0.750000\n",
    "                      COLLECTION     0.250000\n",
    "High School or Below  PAIDOFF        0.741722\n",
    "                      COLLECTION     0.258278\n",
    "Master or Above       COLLECTION     0.500000\n",
    "                      PAIDOFF        0.500000\n",
    "college               PAIDOFF        0.765101\n",
    "                      COLLECTION     0.234899\n",
    "Name: loan_status, dtype: float64\n",
    "Feature befor One Hot Encoding\n",
    "df[['Principal','terms','age','Gender','education']].head()\n",
    "Principal\tterms\tage\tGender\teducation\n",
    "0\t1000\t30\t45\t0\tHigh School or Below\n",
    "1\t1000\t30\t33\t1\tBechalor\n",
    "2\t1000\t15\t27\t0\tcollege\n",
    "3\t1000\t30\t28\t1\tcollege\n",
    "4\t1000\t30\t29\t0\tcollege\n",
    "Use one hot encoding technique to conver categorical varables to binary variables and append them to the feature Data Frame\n",
    "Feature = df[['Principal','terms','age','Gender','weekend']]\n",
    "Feature = pd.concat([Feature,pd.get_dummies(df['education'])], axis=1)\n",
    "Feature.drop(['Master or Above'], axis = 1,inplace=True)\n",
    "Feature.head()\n",
    "Principal\tterms\tage\tGender\tweekend\tBechalor\tHigh School or Below\tcollege\n",
    "0\t1000\t30\t45\t0\t0\t0\t1\t0\n",
    "1\t1000\t30\t33\t1\t0\t1\t0\t0\n",
    "2\t1000\t15\t27\t0\t0\t0\t0\t1\n",
    "3\t1000\t30\t28\t1\t1\t0\t0\t1\n",
    "4\t1000\t30\t29\t0\t1\t0\t0\t1\n",
    "Feature selection\n",
    "Lets defind feature sets, X:\n",
    "\n",
    "X = Feature\n",
    "X[0:5]\n",
    "Principal\tterms\tage\tGender\tweekend\tBechalor\tHigh School or Below\tcollege\n",
    "0\t1000\t30\t45\t0\t0\t0\t1\t0\n",
    "1\t1000\t30\t33\t1\t0\t1\t0\t0\n",
    "2\t1000\t15\t27\t0\t0\t0\t0\t1\n",
    "3\t1000\t30\t28\t1\t1\t0\t0\t1\n",
    "4\t1000\t30\t29\t0\t1\t0\t0\t1\n",
    "What are our lables?\n",
    "\n",
    "y = df['loan_status'].values\n",
    "y[0:5]\n",
    "array(['PAIDOFF', 'PAIDOFF', 'PAIDOFF', 'PAIDOFF', 'PAIDOFF'], dtype=object)\n",
    "Normalize Data\n",
    "Data Standardization give data zero mean and unit variance (technically should be done after train test split )\n",
    "\n",
    "X= preprocessing.StandardScaler().fit(X).transform(X)\n",
    "X[0:5]\n",
    "array([[ 0.51578458,  0.92071769,  2.33152555, -0.42056004, -1.20577805,\n",
    "        -0.38170062,  1.13639374, -0.86968108],\n",
    "       [ 0.51578458,  0.92071769,  0.34170148,  2.37778177, -1.20577805,\n",
    "         2.61985426, -0.87997669, -0.86968108],\n",
    "       [ 0.51578458, -0.95911111, -0.65321055, -0.42056004, -1.20577805,\n",
    "        -0.38170062, -0.87997669,  1.14984679],\n",
    "       [ 0.51578458,  0.92071769, -0.48739188,  2.37778177,  0.82934003,\n",
    "        -0.38170062, -0.87997669,  1.14984679],\n",
    "       [ 0.51578458,  0.92071769, -0.3215732 , -0.42056004,  0.82934003,\n",
    "        -0.38170062, -0.87997669,  1.14984679]])\n",
    "Classification\n",
    "Now, it is your turn, use the training set to build an accurate model. Then use the test set to report the accuracy of the model You should use the following algorithm:\n",
    "\n",
    "K Nearest Neighbor(KNN)\n",
    "Decision Tree\n",
    "Support Vector Machine\n",
    "Logistic Regression\n",
    "Notice:\n",
    "\n",
    "You can go above and change the pre-processing, feature selection, feature-extraction, and so on, to make a better model.\n",
    "You should use either scikit-learn, Scipy or Numpy libraries for developing the classification algorithms.\n",
    "You should include the code of the algorithm in the following cells.\n",
    "K Nearest Neighbor(KNN)\n",
    "Notice: You should find the best k to build the model with the best accuracy.\n",
    "warning: You should not use the loan_test.csv for finding the best k, however, you can split your train_loan.csv into train and test to find the best k.\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)\n",
    "print ('Train set:', X_train.shape,  y_train.shape)\n",
    "print ('Test set:', X_test.shape,  y_test.shape)\n",
    "Train set: (276, 8) (276,)\n",
    "Test set: (70, 8) (70,)\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "k = 6\n",
    "\n",
    "neighK6 = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)\n",
    "neighK6\n",
    " \n",
    "yhat = neighK6.predict(X_test)\n",
    "yhat[0:5]\n",
    "\n",
    "from sklearn import metrics\n",
    "print(\"Train set Accuracy: \", metrics.accuracy_score(y_train, neighK6.predict(X_train)))\n",
    "print(\"Test set Accuracy: \", metrics.accuracy_score(y_test, yhat))\n",
    "Train set Accuracy:  0.800724637681\n",
    "Test set Accuracy:  0.714285714286\n",
    "Ks = 10\n",
    "mean_acc = np.zeros((Ks-1))\n",
    "std_acc = np.zeros((Ks-1))\n",
    "\n",
    "ConfustionMx = [];\n",
    "for n in range(1,Ks):\n",
    "    \n",
    "    \n",
    "    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)\n",
    "    yhat=neigh.predict(X_test)\n",
    "    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)\n",
    "\n",
    "    \n",
    "    std_acc[n-1]=np.std(yhat==y_test)/np.sqrt(yhat.shape[0])\n",
    "\n",
    "plt.plot(range(1,Ks),mean_acc,'g')\n",
    "plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)\n",
    "plt.legend(('Accuracy ', '+/- 3xstd'))\n",
    "plt.ylabel('Accuracy ')\n",
    "plt.xlabel('Number of Nabors (K)')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "print( \"Best accuracy:\", mean_acc.max(), \"k=\", mean_acc.argmax()+1)\n",
    "\n",
    "Best accuracy: 0.785714285714 k= 7\n",
    "Decision Tree\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_trainset, X_testset, y_trainset, y_testset = train_test_split(X, y, test_size=0.3, random_state=3)\n",
    "#Modelling\n",
    "Tree = DecisionTreeClassifier(criterion=\"entropy\", max_depth = 6)\n",
    "Tree\n",
    "DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=6,\n",
    "            max_features=None, max_leaf_nodes=None,\n",
    "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
    "            min_samples_leaf=1, min_samples_split=2,\n",
    "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
    "            splitter='best')\n",
    "Tree.fit(X_trainset,y_trainset)\n",
    "DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=6,\n",
    "            max_features=None, max_leaf_nodes=None,\n",
    "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
    "            min_samples_leaf=1, min_samples_split=2,\n",
    "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
    "            splitter='best')\n",
    "predTree = Tree.predict(X_testset)\n",
    "print (predTree [0:5])\n",
    "print (y_testset [0:5])\n",
    "\n",
    "from sklearn import metrics\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(\"Accuracy: \", metrics.accuracy_score(y_testset, predTree))\n",
    "\n",
    "\n",
    "!conda install -c conda-forge pydotplus -y\n",
    "!conda install -c conda-forge python-graphviz -y\n",
    "from sklearn.externals.six import StringIO\n",
    "import pydotplus\n",
    "import matplotlib.image as mpimg\n",
    "from sklearn import tree\n",
    "%matplotlib inline \n",
    "dot_data = StringIO()\n",
    "filename = \"loan.png\"\n",
    "featureNames = df.columns[0:8]\n",
    "targetNames = df['loan_status'].unique().tolist()\n",
    "out=tree.export_graphviz(Tree,feature_names=featureNames, out_file=dot_data, class_names= np.unique(y_trainset), filled=True,  special_characters=True,rotate=False)  \n",
    "graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \n",
    "graph.write_png(filename)\n",
    "img = mpimg.imread(filename)\n",
    "plt.figure(figsize=(100, 200))\n",
    "plt.imshow(img,interpolation='nearest')\n",
    "['PAIDOFF' 'PAIDOFF' 'PAIDOFF' 'PAIDOFF' 'PAIDOFF']\n",
    "['PAIDOFF' 'PAIDOFF' 'COLLECTION' 'COLLECTION' 'PAIDOFF']\n",
    "Accuracy:  0.701923076923\n",
    "Fetching package metadata .............\n",
    "Solving package specifications: .\n",
    "\n",
    "Package plan for installation in environment /opt/conda/envs/DSX-Python35:\n",
    "\n",
    "The following NEW packages will be INSTALLED:\n",
    "\n",
    "    pydotplus: 2.0.2-py_2 conda-forge\n",
    "\n",
    "pydotplus-2.0. 100% |################################| Time: 0:00:00  18.80 MB/s\n",
    "Fetching package metadata .............\n",
    "Solving package specifications: .\n",
    "\n",
    "Package plan for installation in environment /opt/conda/envs/DSX-Python35:\n",
    "\n",
    "The following NEW packages will be INSTALLED:\n",
    "\n",
    "    python-graphviz: 0.8.4-py35_2 conda-forge\n",
    "\n",
    "python-graphvi 100% |################################| Time: 0:00:00  18.89 MB/s\n",
    "<matplotlib.image.AxesImage at 0x2af2b9463320>\n",
    "\n",
    "Support Vector Machine\n",
    "df.dtypes\n",
    "df = df[pd.to_numeric(df['education'], errors='coerce').notnull()]\n",
    "df['education'] = df['education'].astype('int')\n",
    "df.dtypes\n",
    "\n",
    "from sklearn import svm\n",
    "clf = svm.SVC(kernel='rbf')\n",
    "clf.fit(X_train, y_train)\n",
    "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
    "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
    "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
    "  tol=0.001, verbose=False)\n",
    "yhat = clf.predict(X_test)\n",
    "yhat [0:5]\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "import itertools\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "cnf_matrix = confusion_matrix(y_test, yhat, labels=['PAIDOFF','COLLECTION'])\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "print (classification_report(y_test, yhat))\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cnf_matrix, classes=['PAIDOFF','COLLECTION'],normalize= False,  title='Confusion matrix')\n",
    "\n",
    "from sklearn.metrics import f1_score\n",
    "f1_score(y_test, yhat, average='weighted')\n",
    "\n",
    "from sklearn.metrics import jaccard_similarity_score\n",
    "jaccard_similarity_score(y_test, yhat)\n",
    "             precision    recall  f1-score   support\n",
    "\n",
    " COLLECTION       0.36      0.27      0.31        15\n",
    "    PAIDOFF       0.81      0.87      0.84        55\n",
    "\n",
    "avg / total       0.72      0.74      0.73        70\n",
    "\n",
    "0.74285714285714288\n",
    "\n",
    "Logistic Regression\n",
    "df = df[['loan_status', 'Principal', 'terms', 'effective_date', 'due_date', 'age', 'education', 'Gender']]\n",
    "df['loan_status'] = df['loan_status'].astype('int')\n",
    "\n",
    "from sklearn import preprocessing\n",
    "X = preprocessing.StandardScaler().fit(X).transform(X)\n",
    "X[0:5]\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)\n",
    "print ('Train set:', X_train.shape,  y_train.shape)\n",
    "print ('Test set:', X_test.shape,  y_test.shape)\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import confusion_matrix\n",
    "LogR = LogisticRegression(C=0.01, solver='liblinear').fit(X_train,y_train)\n",
    "LogR\n",
    "Train set: (276, 8) (276,)\n",
    "Test set: (70, 8) (70,)\n",
    "LogisticRegression(C=0.01, class_weight=None, dual=False, fit_intercept=True,\n",
    "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
    "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
    "          verbose=0, warm_start=False)\n",
    "yhat = LogR.predict(X_test)\n",
    "yhat\n",
    "yhat_prob = LogR.predict_proba(X_test)\n",
    "yhat_prob\n",
    "array([[ 0.5 ,  0.5 ],\n",
    "       [ 0.45,  0.55],\n",
    "       [ 0.31,  0.69],\n",
    "       [ 0.34,  0.66],\n",
    "       [ 0.32,  0.68],\n",
    "       [ 0.32,  0.68],\n",
    "       [ 0.49,  0.51],\n",
    "       [ 0.48,  0.52],\n",
    "       [ 0.34,  0.66],\n",
    "       [ 0.49,  0.51],\n",
    "       [ 0.34,  0.66],\n",
    "       [ 0.5 ,  0.5 ],\n",
    "       [ 0.25,  0.75],\n",
    "       [ 0.34,  0.66],\n",
    "       [ 0.44,  0.56],\n",
    "       [ 0.26,  0.74],\n",
    "       [ 0.52,  0.48],\n",
    "       [ 0.3 ,  0.7 ],\n",
    "       [ 0.5 ,  0.5 ],\n",
    "       [ 0.32,  0.68],\n",
    "       [ 0.44,  0.56],\n",
    "       [ 0.49,  0.51],\n",
    "       [ 0.51,  0.49],\n",
    "       [ 0.47,  0.53],\n",
    "       [ 0.41,  0.59],\n",
    "       [ 0.51,  0.49],\n",
    "       [ 0.51,  0.49],\n",
    "       [ 0.37,  0.63],\n",
    "       [ 0.5 ,  0.5 ],\n",
    "       [ 0.25,  0.75],\n",
    "       [ 0.47,  0.53],\n",
    "       [ 0.46,  0.54],\n",
    "       [ 0.46,  0.54],\n",
    "       [ 0.48,  0.52],\n",
    "       [ 0.39,  0.61],\n",
    "       [ 0.46,  0.54],\n",
    "       [ 0.5 ,  0.5 ],\n",
    "       [ 0.29,  0.71],\n",
    "       [ 0.46,  0.54],\n",
    "       [ 0.45,  0.55],\n",
    "       [ 0.51,  0.49],\n",
    "       [ 0.32,  0.68],\n",
    "       [ 0.45,  0.55],\n",
    "       [ 0.51,  0.49],\n",
    "       [ 0.31,  0.69],\n",
    "       [ 0.5 ,  0.5 ],\n",
    "       [ 0.47,  0.53],\n",
    "       [ 0.5 ,  0.5 ],\n",
    "       [ 0.46,  0.54],\n",
    "       [ 0.46,  0.54],\n",
    "       [ 0.28,  0.72],\n",
    "       [ 0.47,  0.53],\n",
    "       [ 0.31,  0.69],\n",
    "       [ 0.49,  0.51],\n",
    "       [ 0.28,  0.72],\n",
    "       [ 0.25,  0.75],\n",
    "       [ 0.32,  0.68],\n",
    "       [ 0.43,  0.57],\n",
    "       [ 0.47,  0.53],\n",
    "       [ 0.34,  0.66],\n",
    "       [ 0.42,  0.58],\n",
    "       [ 0.33,  0.67],\n",
    "       [ 0.46,  0.54],\n",
    "       [ 0.53,  0.47],\n",
    "       [ 0.32,  0.68],\n",
    "       [ 0.49,  0.51],\n",
    "       [ 0.33,  0.67],\n",
    "       [ 0.42,  0.58],\n",
    "       [ 0.45,  0.55],\n",
    "       [ 0.32,  0.68]])\n",
    "from sklearn.metrics import jaccard_similarity_score\n",
    "jaccard_similarity_score(y_test, yhat)\n",
    "from sklearn.metrics import log_loss\n",
    "log_loss(y_test, yhat_prob)\n",
    "0.57722876094796538\n",
    "Model Evaluation using Test set\n",
    "from sklearn.metrics import jaccard_similarity_score\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import log_loss\n",
    "First, download and load the test set:\n",
    "\n",
    "!wget -O loan_test.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_test.csv\n",
    "--2019-01-04 13:28:15--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/loan_test.csv\n",
    "Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 67.228.254.193\n",
    "Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|67.228.254.193|:443... connected.\n",
    "HTTP request sent, awaiting response... 200 OK\n",
    "Length: 3642 (3.6K) [text/csv]\n",
    "Saving to: ‘loan_test.csv’\n",
    "\n",
    "100%[======================================>] 3,642       --.-K/s   in 0s      \n",
    "\n",
    "2019-01-04 13:28:15 (491 MB/s) - ‘loan_test.csv’ saved [3642/3642]\n",
    "\n",
    "Load Test set for evaluation\n",
    "test_df = pd.read_csv('loan_test.csv')\n",
    "test_df.head()\n",
    "Unnamed: 0\tUnnamed: 0.1\tloan_status\tPrincipal\tterms\teffective_date\tdue_date\tage\teducation\tGender\n",
    "0\t1\t1\tPAIDOFF\t1000\t30\t9/8/2016\t10/7/2016\t50\tBechalor\tfemale\n",
    "1\t5\t5\tPAIDOFF\t300\t7\t9/9/2016\t9/15/2016\t35\tMaster or Above\tmale\n",
    "2\t21\t21\tPAIDOFF\t1000\t30\t9/10/2016\t10/9/2016\t43\tHigh School or Below\tfemale\n",
    "3\t24\t24\tPAIDOFF\t1000\t30\t9/10/2016\t10/9/2016\t26\tcollege\tmale\n",
    "4\t35\t35\tPAIDOFF\t800\t15\t9/11/2016\t9/25/2016\t29\tBechalor\tmale\n",
    "X= preprocessing.StandardScaler().fit(X).transform(X)\n",
    "X[0:5]\n",
    "Y = test_df['loan_status'].values\n",
    "Y[0:5]\n",
    "array(['PAIDOFF', 'PAIDOFF', 'PAIDOFF', 'PAIDOFF', 'PAIDOFF'], dtype=object)\n",
    "#test the KNN algorithm already trained with K=6\n",
    "yhatKNN=neigh.predict(X)\n",
    "KNNJaccard = jaccard_similarity_score(y, yhatKNN)\n",
    "KNNF1 = f1_score(y, yhatKNN, average='weighted')\n",
    "print(\"Avg F1-score: %.2f\" % KNNF1 )\n",
    "print(\"KNN Jaccard Score: %.2f\" % KNNJaccard)\n",
    "\n",
    "\n",
    "yhatDEC = Tree.predict(X)\n",
    "DTJaccard = jaccard_similarity_score(y, yhatDEC)\n",
    "DTF1 = f1_score(y, yhatDEC, average='weighted')\n",
    "print(\"Avg F1-score: %.2f\" % DTF1 )\n",
    "print(\"Decision Tree Jaccard Score: %.2f\" % DTJaccard)\n",
    "\n",
    "yhatSVM=clf.predict(X)\n",
    "SVMJaccard = jaccard_similarity_score(y, yhatSVM)\n",
    "SVMF1 = f1_score(y, yhatSVM, average='weighted')\n",
    "print(\"Avg F1-score: %.2f\" % SVMF1)\n",
    "print(\"SVM Jaccard score: %.2f\" % SVMJaccard)\n",
    "\n",
    "yhatLOG = LogR.predict(X)\n",
    "yhatLOGproba = LogR.predict_proba(X)\n",
    "LogRJaccard = jaccard_similarity_score(y, yhatLOG)\n",
    "LogRF1 = f1_score(y, yhatLOG, average='weighted')\n",
    "Logloss = log_loss(y, yhatLOGproba)\n",
    "print(\"LogLoss: : %.2f\" % Logloss)\n",
    "print(\"Avg F1-score: %.4f\" % LogRF1)\n",
    "print(\"LOG Jaccard score: %.4f\" % LogRJaccard)\n",
    "Avg F1-score: 0.78\n",
    "KNN Jaccard Score: 0.79\n",
    "Avg F1-score: 0.78\n",
    "Decision Tree Jaccard Score: 0.79\n",
    "Avg F1-score: 0.76\n",
    "SVM Jaccard score: 0.77\n",
    "LogLoss: : 0.56\n",
    "Avg F1-score: 0.7199\n",
    "LOG Jaccard score: 0.7428\n",
    " \n",
    "Report\n",
    "You should be able to report the accuracy of the built model using different evaluation metrics:\n",
    "\n",
    "Algorithm\tJaccard\tF1-score\tLogLoss\n",
    "KNN\t0.79\t0.78\tNA\n",
    "Decision Tree\t0.79\t0.78\tNA\n",
    "SVM\t0.77\t0.76\tNA\n",
    "LogisticRegression\t0.7428\t0.7199\t0.56\n",
    "Want to learn more?\n",
    "IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler\n",
    "\n",
    "Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio\n",
    "\n",
    "Thanks for completing this lesson!\n",
    "Author: Saeed Aghabozorgi\n",
    "Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.\n",
    "\n",
    "Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}