Cross_Validation_and_Model_Optimization.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Cross_Validation_and_Model_Optimization.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPwuOlCT86dzZ3fQoh5GFDq",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/numerai/example-scripts/blob/master/Cross_Validation_and_Model_Optimization.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xCr2GyNS54zR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#import your packages and modules\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import sklearn"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CzRoYGfs575H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_datalink = 'https://numerai-public-datasets.s3-us-west-2.amazonaws.com/latest_numerai_training_data.csv.xz'\n",
        "\n",
        "df = pd.read_csv(train_datalink, nrows=50000) #download the training data and keep only the first 50,000 rows"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VVkcvVw06INs",
        "colab_type": "text"
      },
      "source": [
        "We have an additional data step to consider before we define X and y.\n",
        "\n",
        "Numerai's training data consists of eras, which are time-ordered groupings of data.\n",
        "\n",
        "How many eras does our subsample contain?\n",
        "\n",
        "We can use pandas to identify the unique observations in a column.\n",
        "\n",
        "https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nunique.html\n",
        "\n",
        "We've done similar things in the past, but this may refresh your memory.\n",
        "\n",
        "df.era.nunique()"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TzRSeo9E6EGa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#identify how many eras are contained in the subsample\n",
        "df.era.nunique()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oTPoMbUv6e2b",
        "colab_type": "text"
      },
      "source": [
        "We need to change the 'era' column to contain only numbers so that we can easily split the data into a train and test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NCNkT2kC6hnT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df['era'] = df.loc[:, 'era'].str[3:].astype('int32')\n",
        "\n",
        "#what this line does:\n",
        "\n",
        "#1 - over-write the existing 'era' column: df['era'] =\n",
        "#2 - select only the column named 'era' and all of the observations in that column: df.loc[:, 'era']\n",
        "#3 - for each observation in the selected column, remove the first 3 characters: .str[3:]\n",
        "#4 - convert the data type to an integer32 which uses less memory: .astype('int32')\n",
        "\n",
        "#this is called method chaining"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GmWGa0gO6q0-",
        "colab_type": "text"
      },
      "source": [
        "This is a small subset of the 120 total training eras available.\n",
        "\n",
        "We need to split the data into training and testing sets.\n",
        "\n",
        "We need to manually partition the data into train and test sets. Let's set aside 8 eras for testing.\n",
        "\n",
        "`df_train = df[df.era < 10].copy()`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Md4xCxIT6xSr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_train = df[df.era < 10].copy() #nothing is printed. if you want to see the results of this operation, then add a new line and print the first 5 rows of df_train."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nijF_bbP7gA5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_train.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kU8K0m8o7mbM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_train.era.unique() #this shows you the actual eras that were selected in the code to define df_train"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-irD6ec97p0j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#now define your test set.\n",
        "#HINT: above, you defined df_train. define df_test, and convert the operation above to something that would select all eras other than what was already selected.\n",
        "#Verify you got the right answer by following the code directly above this cell but change df_train to df_test\n",
        "\n",
        "df_test = df[df.era > 9].copy() #this code tells python to select all eras that are greater than 9, whereas the train set was defined as all eras less than 10. This is a simple logic test."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FaSK3HlI78BC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Now we have two dataframes, and need to define X, y and era twice. Let's do that now.\n",
        "\n",
        "X_train = df_train.iloc[:, 3:313].values\n",
        "\n",
        "X_test = df_test.iloc[:, 3:313].values\n",
        "\n",
        "y_train = df_train.target.values\n",
        "\n",
        "y_test = df_test.target.values\n",
        "\n",
        "era_train = df_train.era.values\n",
        "\n",
        "era_test = df_test.era.values"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E8WbXKiq8EAA",
        "colab_type": "text"
      },
      "source": [
        "I mentioned that cross validation helps to evaluate models and to mitigate the effects of overfitting.\n",
        "\n",
        "We need to further partition our data into cross validation folds.\n",
        "\n",
        "Thankfully, Scikit-Learn has functions to do that for us. Let's take a look at the Cross Validation functions in scikit-learn.\n",
        "\n",
        "https://scikit-learn.org/stable/modules/cross_validation.html\n",
        "\n",
        "We know already that our data has groups, so we should prefer cross validators that account for groups.\n",
        "\n",
        "https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-for-grouped-data\n",
        "\n",
        "Within this section, there are a couple of strategies.\n",
        "\n",
        "We can visualize the methodology:\n",
        "\n",
        "![alt text](https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_005.png)\n",
        " \n",
        "Please note that there are four different cross validation iterators for grouped data.\n",
        "\n",
        "I want you to read about each one and think about your future production algorithm. Which one do you want to rely on to generate the best possible inference from the training data?\n",
        "\n",
        "I will demonstrate how to use a cross-validation iterator using the first option. This may not be appropriate for your model!\n",
        "\n",
        "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html#sklearn.model_selection.GroupKFold\n",
        "\n",
        "This time, we'll need to choose parameters and not rely on the default settings. Visit the link above and read the Parameters section.\n",
        "\n",
        " \n",
        "You'll notice that the default number of splits is 5. We happen to have 9 eras, which splits evenly into 3 groups.\n",
        "\n",
        "The warning in the parameter section tells you the default setting changed in version 0.22 from 3 to 5. This is why it is important to define the settings for production algorithms which may require stability over several years.\n",
        "\n",
        "While we could rely on the default settings for now, let's instead use n_splits=3\n",
        "\n",
        "![alt text](https://i.postimg.cc/WpqxTCv7/image.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jDTeHLX7H9t1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#import the model_selection module\n",
        "\n",
        "from sklearn import model_selection\n",
        "\n",
        "#define your cross-validation iterator\n",
        "\n",
        "CV = model_selection.GroupKFold(n_splits = 3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dELDjlSDICCz",
        "colab_type": "text"
      },
      "source": [
        "Now that we've defined our splitter, we have to use it to generate our cross-validation partitions.\n",
        "\n",
        "I demonstrated last lecture that you can access the functions of a model (or iterator) by viewing the Methods section.\n",
        "\n",
        "![alt text](https://i.postimg.cc/mZc3qGsT/Capture.jpg)\n",
        " \n",
        "We can access the methods using '.' on CV:\n",
        "\n",
        "![alt text](https://i.postimg.cc/BbZrSQc7/method-parameters.jpg)\n",
        " \n",
        "What happens behind the scenes is that the data is sliced into further train and test sets, and the output from using .split() is an array which models in the next step can use automatically to train and test on the different partitions. This isn't easy to visualize, but the example code in the link gives a pretty good explanation of the output. You are given train and test sets which identify the rows that belong to each grouping."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5GMlkXq0Lhev",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#We can now define a new variable which stores these row identifiers for algorithms to use.\n",
        "\n",
        "grp = list(CV.split(X = X_train, y = y_train,  groups = era_train))\n",
        "\n",
        "#what this line does is:\n",
        "\n",
        "#1 - define a new variable: grp = \n",
        "#2 - use the python 'list' function: list(\n",
        "#3 - within the list function, we use the .split() method on our group k-fold iterator, which we defined as CV: CV.split(\n",
        "#4 - within the .split() method, we defined the required parameters and closed the .split parenthesis: X = X_train, y = y_train, groups = era_train)\n",
        "#5 - we enclosed the list function: )\n",
        "\n",
        "#this is called nested function."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lOaY-d3gLvLI",
        "colab_type": "text"
      },
      "source": [
        "You may have noticed that many models include parameters which can be changed from the default value. Doing so can increase the model's performance, and also lead to overfitting. The benefit of cross validation, and models with built-in cross validation optimization, is that you don't have to do the work of determining what value is the most performant.\n",
        "\n",
        "Let's use Ridge Regression as an example.\n",
        "\n",
        "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html#sklearn.linear_model.RidgeCV\n",
        "\n",
        "![alt text](https://i.postimg.cc/sxRp7v0K/ridge-CV.jpg)\n",
        "\n",
        "The model may improve by changing the alpha parameter, but we don't know which value to choose in advance. So we should just try a whole bunch and let the math wizards who wrote this code figure it out!\n",
        "\n",
        "We can define the search space ourselves by creating a new variable and listing the values to evaluate explicitly like so:\n",
        "\n",
        "`alphas = (0.1, 0.5, 1.0, 10.0)`\n",
        "\n",
        "Since we are altering the default settings of RidgeCV then we have to modify those terms within the parenthesis:\n",
        "\n",
        "`RidgeCV(alphas = alphas, cv = grp)`\n",
        "\n",
        "above, we deviate from the default parameters for alphas and cv, which we've defined as variables ourselves above. Your python environment has stored those variables in memory, and RidgeCV will \"call\" those variables as part of the fit function automatically. Let's put this to work and determine which alpha gives the best in-sample performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5va96B1_Myx0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn import linear_model\n",
        "alphas = (0.1, 0.5, 1.0, 10.0)\n",
        "REG1 = linear_model.RidgeCV(alphas = alphas, cv = grp)\n",
        "REG1.fit(X_train, y_train)\n",
        "#you can just ignore the output that is generated below once the code finishes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O4tJ_p2kM28k",
        "colab_type": "text"
      },
      "source": [
        "We can access stored values:\n",
        "\n",
        "![alt text](https://i.postimg.cc/LX4SXTmV/access.jpg)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nv6ZrkzFM-Qa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "REG1.alpha_ #This line prints the alpha value that gave the model the best fit"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-V74IC-8M_RK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "REG1.score(X_train, y_train) #scoring the model gives you the R-sq of the model using the best alpha parameter"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h32jQbbmNCFz",
        "colab_type": "text"
      },
      "source": [
        "This may be insufficent for your analysis. What if you wanted to see the performance of each term?\n",
        "\n",
        "We can use a model selection tool called Grid Search CV.\n",
        "\n",
        "Let's turn this up to 11. \n",
        "\n",
        "![alt text](https://i.postimg.cc/SxjDyz0z/to-11.jpg)\n",
        "\n",
        "https://scikit-learn.org/stable/modules/grid_search.html\n",
        "\n",
        "Grid search allows us to \"span the space\" and determine which parameter is best and also to give you the performance in a table. This is a very powerful tool!\n",
        "\n",
        "Let's keep using Ridge, but now we will use the non-CV version."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zl7MNPAZNWM-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "REG2 = linear_model.Ridge()\n",
        "\n",
        "#we also have to create a python dictionary for grid search to use:\n",
        "\n",
        "params1 = {'alpha': [0.1, 0.3, 0.5, 0.8, 1.0, 2.0, 5.0, 10.0]}\n",
        "\n",
        "#what this line does is:\n",
        "\n",
        "#a dictionary has a dimension n,j (think columns and rows, although that isn't quite right)\n",
        "\n",
        "#1 - define a new variable: params = \n",
        "#2 - create a python 'dictionary': {\n",
        "#3 - define the name of the nth item as: 'alpha':\n",
        "#4 - the jth item is a list: [\n",
        "#5 - inside the jth item list is: 0.1, 0.3, 0.5, 0.8, 1.0, 2.0, 5.0, 10.0]\n",
        "#6 - close the dictionary: }\n",
        "\n",
        "#we created a dictionary that contains a list: https://docs.python.org/3/tutorial/datastructures.html#dictionaries"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IvfgXRt5Ndlm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "GS1 = model_selection.GridSearchCV(estimator = REG2, param_grid = params1, cv = grp, return_train_score = True)\n",
        "GS1.fit(X_train, y_train)\n",
        "#you can ignore the output that is generated below after the code finishes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f1rBxEPyNfZb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "scores1 = pd.DataFrame(GS1.cv_results_); scores1\n",
        "\n",
        "#if we type GS.cv_results_ then we are given a dictionary. We can use Pandas to convert a dictionary to a dataframe using method chaining.\n",
        "\n",
        "#the code above defines a new variable called scores1, and we use the DataFrames() function from Pandas to interpret the GS1.cv_results_ method's output as a dataframe.\n",
        "#we then used the shortcut method: ; scores1 to print the table below."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I-XE99BnNiX7",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://i.postimg.cc/PfQjD0CQ/winning-jpg.jpg)\n",
        "\n",
        "WE HAVE DATA!!!\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ndgsm_zOfPY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#which model is the best?\n",
        "\n",
        "GS1.best_estimator_\n",
        "\n",
        "#the output below contains the parameters used in the best model. You could copy this and define it as a new variable for production if you wanted to.\n",
        "#alpha=10.0, copy_X=True, fit_intercept=True, max_iter=None,\n",
        "#      normalize=False, random_state=None, solver='auto', tol=0.001"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QNM5sCttOg_R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "GS1.best_score_\n",
        "\n",
        "#this is the score given by the model above. Our score is negative! We've actually overfit, and the model is worse than just guessing randomly.\n",
        "#However, this statement only holds for eras 1-9; perhaps the model would perform better with more data.\n",
        "#I leave that to you to evaluate with a computer that has more resources available."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E5Q88dvXOsT-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#this code will take almost 10 minutes to run. Please be patient!\n",
        "\n",
        "#####################################################################################################################################################################################################################################\n",
        "\n",
        "#What if a model has more than one parameter which can be optimized? Grid Search CV can handle that scenario as well. Here's how we handle more than one parameter optimization.\n",
        "#Let's find the best combination of l1_ratio and tol in the ElasticNet model:\n",
        "\n",
        "#https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html\n",
        "\n",
        "params2 = {'l1_ratio': [0.0, 0.25, 0.33, 0.5, 0.66, 0.75, 1], 'tol': [0.01, 0.001, 0.0001, 0.00001]}\n",
        "\n",
        "#what will happen in the background is that grid search will use l1_ratio of 0.0 and evaluate performance for each value of 'tol', and so on for every value of l1_ratio provided. This takes a long time with just 9 eras; imagine how long it would take to do this with all 120 eras! While trying a lot of parameters is a good idea, you've got to balance efficiency with your desire to find the best settings for the model. This is an art, not a science.\n",
        "\n",
        "#You'll see some warning messages about convergence. You can ignore them for now.\n",
        "\n",
        "REG3 = linear_model.ElasticNet(max_iter = 3000) #note I changed a default parameter! I am using a maximum of 3,000 training iterations to find the best fit for each run of the model.\n",
        "\n",
        "GS2 = model_selection.GridSearchCV(estimator = REG3, param_grid = params2, cv = grp, return_train_score = True)\n",
        "\n",
        "GS2.fit(X_train, y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uCJxOYMCOw9y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "scores2 = pd.DataFrame(GS2.cv_results_); scores2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yQzgElS-OyKW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#determine which model was the best\n",
        "GS2.best_estimator_"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BeWkunHHOz94",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#determine the score of the best model\n",
        "GS2.best_score_"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lKmEQNQPO28l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Now that we've determined which model is the best, we can evaluate the performance of the model on the out-of-sample test set.\n",
        "\n",
        "#First, we have to re-train the model on the training data.\n",
        "\n",
        "REG4 = GS2.best_estimator_ #we can easily define the best model without writing very much, since GridSearchCV allows us to access the best model directly\n",
        "REG4.fit(X_train, y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fUn6ZWtoO5I0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Let's double check the in-sample performance\n",
        "REG4.score(X_train, y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gFz3b3OIO76W",
        "colab_type": "text"
      },
      "source": [
        "Numerai scores your predictions based on correlation with the target variable. Until this point, we've only considered the performance of the model based on the R-sq. Numerai provides the scoring code.\n",
        "\n",
        "We can define a python function (https://docs.python.org/3/tutorial/controlflow.html#defining-functions) to calculate the correlation score:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e53tCUldO6gV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def correlation_score(y_true, y_pred):\n",
        "    return np.corrcoef(y_true, y_pred)[0,1]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mzm86ROqPCpA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Let's generate some predictions. Since we've fit the model to the training data, we can generate predictions based on the test set.\n",
        "#We want the highest possible correlation score with the understanding that a very high score may be a sign that the model has overfit.\n",
        "\n",
        "preds1 = REG4.predict(X_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TMJBmYgFPEw2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Now let's calculate the out-of-sample correlation using Numerai's scoring function and print the score:\n",
        "\n",
        "OOS_score1 = correlation_score(y_test, preds1); OOS_score1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kNWoJAPPPHA3",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://i.postimg.cc/7LR1Yx8P/thumbs-up.jpg)\n",
        "\n",
        "5.7% correlation is pretty good! There are some caveats to remember. You trained the model on 9 out of 120 eras, and evaluated the performance based on 8 of the 111 eras remaining. We know that there is a time-series component to the data, so perhaps eras 1-9 are similar to eras 10-17. We still don't know if the model is any good, but we did take a scientific approach to the evaluation.\n",
        "\n",
        "Because we held out data that the model has never seen, we can infer that if live stock market data is similar to eras 10-17, then we should expect our models to generate a 5.7% correlation, on average. This is highly unlikely in practice!\n",
        "\n",
        "We have another problem. We trained the model on eras 1-9, so the assumption is that each era is nearly identical in composition. We know this to be false as well. The more likely case is that some of the eras are more similar than others. What if we used grid search to identify the best parameters to use on eras 1-3, 4-6, and 7-9? What if we fit those best models to 1-3, 4-6, and 7-9 and then generated predictions on the test set for each of them and averaged the predictions across the three models? That would be a much better approach, wouldn't it?\n",
        "\n",
        "Here's the thing: Even if you did average across groups like that, there's no guarantee that eras 1-3 are sufficiently similar to eachother but sufficiently different from 4-6 and 7-9 to improve the forecast. If 1-3, 4-6, and 7-9 are actually similar, then averaging the 3 prediction sets would yield an answer that was no different from any one of the 3 prediction sets.\n",
        "\n",
        "And another layer of complexity exists that we haven't even addressed yet. Some of the features are highly correlated. If your model does not address this multicollinearity then the model is likely to overfit those features, causing bad inference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kMBNpbv1Ph4b",
        "colab_type": "text"
      },
      "source": [
        "ASSIGNMENT: Use the scikit-learn website to identify three models (in this case, a model is a sequence of tasks that eventually generate predictions) which work with our dataset (regression algorithms) that:\n",
        "\n",
        "-either automatically handle multicollinearity or offer some way to mitigate multicollinearity\n",
        "\n",
        "-mitigate problems with training on the entire dataset, such as averaging across a number of models (hint: ensemble methods)\n",
        "\n",
        "-or a combination of a feature selection tool followed by an algorithm which will work with our dataset\n",
        "\n",
        "You must also identify which parameters can be optimized.\n",
        "\n",
        "![alt text](https://i.postimg.cc/V6D8bHgN/supervised-learning.jpg)\n",
        " \n",
        "Below is an example of a correct answer. Note that just because there are three models within the answer, that doesn't mean that you are finished. You still must develop 2 additional models. Perhaps you find an algorithm that does everything you want and doesn't require a long answer such as below. That's fine, but you should justify that decision in the description. This is an opportunity for you to think about your production model and to show me some of your ideas, and also that you understand the concepts we've discussed. If you find one you like but are struggling to understand how to set it up properly, then please join Numerai's Rocket Chat in the #newusers channel and I can help you."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NnNi-6LpQFUC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#MODEL 1: Voting Regressor\n",
        "\n",
        "#LINK: https://scikit-learn.org/stable/modules/ensemble.html#voting-regressor\n",
        "\n",
        "#Description of methodology and how the model helps to give better performance:\n",
        "\n",
        "#Voting Regressor requires that several models be defined which will each be fit on the training data.\n",
        "#The voting regressor then averages the predictions made by each model, which balances out the individual weaknesses of the models.\n",
        "#This will help to reduce overfitting, but only if the models are sufficienty different.\n",
        "\n",
        "#Import the relevant modules, define the model(s), identify parameters to be optimized:\n",
        "\n",
        "from sklearn import ensemble\n",
        "from sklearn import kernel_ridge\n",
        "from sklearn import neural_network\n",
        "VR1 = kernel_ridge.KernelRidge() #kernel ridge allows for optimization of the alpha parameter.\n",
        "VR2 = neural_network.MLPRegressor() #MLP regressor allows for optimization of many parameters: the activation function, the solver, alpha, batch_size, learning_rate_init, max_iter, tol, and many other parameters. So many, that if I choose to use this model in production, then I will ask the Professor for help before I try to use this in practice!\n",
        "VR3 = linear_model.ElasticNet() #ElasticNet allows for optimization of these parameters: alpha, l1_ratio, tol\n",
        "\n",
        "#the example code given for Voting Regressor tells me that I have to provide a list of estimators within the parenthesis, so I follow the example like this:\n",
        "\n",
        "META = ensemble.VotingRegressor(estimators=[('VR1', VR1), ('VR2', VR2), ('VR3', VR3)])\n",
        "\n",
        "#Write the correct code to fit your model:\n",
        "\n",
        "META.fit(X_train, y_train)\n",
        "\n",
        "#Write the code to generate and store predictions from your fitted model:\n",
        "\n",
        "#you can name this whatever you want, here are some examples:\n",
        "\n",
        "#PREDS1\n",
        "#y_pred\n",
        "#predictions\n",
        "\n",
        "VR_pred = META.predict(X_test)\n",
        "\n",
        "#Write the code to score the out-of-sample predictions using the correlation function and print the correlation score:\n",
        "\n",
        "#you can name this whatever you want\n",
        "\n",
        "OOS_VR_score = correlation_score(y_test, VR_pred); OOS_VR_score"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "obJ16xMWQb_E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#MODEL 1: Elastic Net\n",
        "\n",
        "#LINK: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html#sklearn.linear_model.ElasticNet\n",
        "\n",
        "#Description of methodology and how the model helps to give better performance:\n",
        "\n",
        "#Elastic-net is useful when there are multiple features which are correlated with one another. Lasso is likely to pick one of these at random, while elastic-net is likely to pick both.\n",
        "\n",
        "#Import the relevant modules, define the models, identify parameters to be optimized:\n",
        "\n",
        "M1 = linear_model.ElasticNet() # parameters that can be optimized are: alpha, l1_ratio, tol\n",
        "\n",
        "#Write the correct code to fit your model:\n",
        "\n",
        "M1.fit(X_train, y_train)\n",
        "\n",
        "#Write the code to generate and store predictions from your fitted model:\n",
        "\n",
        "OOS_M1 = M1.predict(X_test)\n",
        "\n",
        "#Write the code to score the out-of-sample predictions using the correlation function and print the correlation score:\n",
        "\n",
        "OOS_M1_score = correlation_score(y_test, OOS_M1); OOS_M1_score"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xxhq7lcMQc8o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#MODEL 2: \n",
        "\n",
        "#LINK:\n",
        "\n",
        "#Description of methodology and how the model helps to give better performance:\n",
        "\n",
        "#Import the relevant modules, define the models, identify parameters to be optimized:\n",
        "#parameters that can be optimized are: base_estimator, n_estimators, learning_rate, loss, random_state\n",
        "\n",
        "#Write the correct code to fit your model:\n",
        "\n",
        "#Write the code to generate and store predictions from your fitted model:\n",
        "\n",
        "#Write the code to score the out-of-sample predictions using the correlation function and print the correlation score:\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "evpW52SqQe3Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#MODEL 3: \n",
        "\n",
        "#LINK:\n",
        "\n",
        "#Description of methodology and how the model helps to give better performance:\n",
        "\n",
        "#Import the relevant modules, define the models, identify parameters to be optimized:\n",
        "\n",
        "#Write the correct code to fit your model:\n",
        "\n",
        "#Write the code to generate and store predictions from your fitted model:\n",
        "\n",
        "#Write the code to score the out-of-sample predictions using the correlation function and print the correlation score:"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}