SLF_Project_LearnerNotebook_LowCode_2.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CoderSales/machine-learning-classification/blob/main/SLF_Project_LearnerNotebook_LowCode_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RA_GXJwuhmVz"
      },
      "source": [
        "### Data Description\n",
        "\n",
        "The data contains the different attributes of used/refurbished phones and tablets. The data was collected in the year 2021. The detailed data dictionary is given below.\n",
        "\n",
        "\n",
        "- brand_name: Name of manufacturing brand\n",
        "- os: OS on which the device runs\n",
        "- screen_size: Size of the screen in cm\n",
        "- 4g: Whether 4G is available or not\n",
        "- 5g: Whether 5G is available or not\n",
        "- main_camera_mp: Resolution of the rear camera in megapixels\n",
        "- selfie_camera_mp: Resolution of the front camera in megapixels\n",
        "- int_memory: Amount of internal memory (ROM) in GB\n",
        "- ram: Amount of RAM in GB\n",
        "- battery: Energy capacity of the device battery in mAh\n",
        "- weight: Weight of the device in grams\n",
        "- release_year: Year when the device model was released\n",
        "- days_used: Number of days the used/refurbished device has been used\n",
        "- normalized_new_price: Normalized price of a new device of the same model in euros\n",
        "- normalized_used_price: Normalized price of the used/refurbished device in euros"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zeF8YaNKDPyK"
      },
      "outputs": [],
      "source": [
        "# read/manipulate:\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "# see:\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "# split: train/test:\n",
        "from sklearn.model_selection import train_test_split\n",
        "# model:\n",
        "from sklearn.linear_model import LinearRegression\n",
        "# model performance:\n",
        "from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\n",
        "# model w statsmodels:\n",
        "import statsmodels.api as sm\n",
        "# VIF (variance inflation factor):\n",
        "from statsmodels.stats.outliers_influence import variance_inflation_factor"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D4id2YLjhmV3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 73
        },
        "outputId": "65cc2c6f-403a-433f-fd4c-b7257810af62"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-abfd7a91-7c93-4bbc-854f-dd22fe8cf220\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-abfd7a91-7c93-4bbc-854f-dd22fe8cf220\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving used_device_data.csv to used_device_data.csv\n"
          ]
        }
      ],
      "source": [
        "# loading data\n",
        "# data = pd.read_csv('_______') ## Complete the code to read the data\n",
        "\n",
        "from google.colab import files\n",
        "import io\n",
        "\n",
        "try:\n",
        "    uploaded\n",
        "except NameError:\n",
        "    uploaded = files.upload()\n",
        "\n",
        "data = pd.read_csv(io.BytesIO(uploaded['used_device_data.csv']))\n",
        "#cData = pd.read_csv(\"auto-mpg.csv\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aNvXgPEhWaLv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "outputId": "1d63dc9e-d20d-4cb8-94b6-4c3f77ce57b7"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  brand_name       os  screen_size   4g   5g  main_camera_mp  \\\n",
              "0      Honor  Android        14.50  yes   no            13.0   \n",
              "1      Honor  Android        17.30  yes  yes            13.0   \n",
              "2      Honor  Android        16.69  yes  yes            13.0   \n",
              "\n",
              "   selfie_camera_mp  int_memory  ram  battery  weight  release_year  \\\n",
              "0               5.0        64.0  3.0   3020.0   146.0          2020   \n",
              "1              16.0       128.0  8.0   4300.0   213.0          2020   \n",
              "2               8.0       128.0  8.0   4200.0   213.0          2020   \n",
              "\n",
              "   days_used  normalized_used_price  normalized_new_price  \n",
              "0        127               4.307572              4.715100  \n",
              "1        325               5.162097              5.519018  \n",
              "2        162               5.111084              5.884631  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ca797411-df80-46d9-a72c-019f37390fe8\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>brand_name</th>\n",
              "      <th>os</th>\n",
              "      <th>screen_size</th>\n",
              "      <th>4g</th>\n",
              "      <th>5g</th>\n",
              "      <th>main_camera_mp</th>\n",
              "      <th>selfie_camera_mp</th>\n",
              "      <th>int_memory</th>\n",
              "      <th>ram</th>\n",
              "      <th>battery</th>\n",
              "      <th>weight</th>\n",
              "      <th>release_year</th>\n",
              "      <th>days_used</th>\n",
              "      <th>normalized_used_price</th>\n",
              "      <th>normalized_new_price</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Honor</td>\n",
              "      <td>Android</td>\n",
              "      <td>14.50</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "      <td>13.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>64.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3020.0</td>\n",
              "      <td>146.0</td>\n",
              "      <td>2020</td>\n",
              "      <td>127</td>\n",
              "      <td>4.307572</td>\n",
              "      <td>4.715100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Honor</td>\n",
              "      <td>Android</td>\n",
              "      <td>17.30</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>13.0</td>\n",
              "      <td>16.0</td>\n",
              "      <td>128.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>4300.0</td>\n",
              "      <td>213.0</td>\n",
              "      <td>2020</td>\n",
              "      <td>325</td>\n",
              "      <td>5.162097</td>\n",
              "      <td>5.519018</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Honor</td>\n",
              "      <td>Android</td>\n",
              "      <td>16.69</td>\n",
              "      <td>yes</td>\n",
              "      <td>yes</td>\n",
              "      <td>13.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>128.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>4200.0</td>\n",
              "      <td>213.0</td>\n",
              "      <td>2020</td>\n",
              "      <td>162</td>\n",
              "      <td>5.111084</td>\n",
              "      <td>5.884631</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ca797411-df80-46d9-a72c-019f37390fe8')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ca797411-df80-46d9-a72c-019f37390fe8 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ca797411-df80-46d9-a72c-019f37390fe8');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "data.head(3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VsQmSVQDWdQB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa0b66f4-5b3f-4175-d0d6-54f767887bac"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3454, 15)"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "data.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5TcqcxbK-qT3"
      },
      "source": [
        "### data types of columns:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4sMqr8mnWf2v",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b3b670e6-4777-4d90-c600-6769f985a6f8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 3454 entries, 0 to 3453\n",
            "Data columns (total 15 columns):\n",
            " #   Column                 Non-Null Count  Dtype  \n",
            "---  ------                 --------------  -----  \n",
            " 0   brand_name             3454 non-null   object \n",
            " 1   os                     3454 non-null   object \n",
            " 2   screen_size            3454 non-null   float64\n",
            " 3   4g                     3454 non-null   object \n",
            " 4   5g                     3454 non-null   object \n",
            " 5   main_camera_mp         3275 non-null   float64\n",
            " 6   selfie_camera_mp       3452 non-null   float64\n",
            " 7   int_memory             3450 non-null   float64\n",
            " 8   ram                    3450 non-null   float64\n",
            " 9   battery                3448 non-null   float64\n",
            " 10  weight                 3447 non-null   float64\n",
            " 11  release_year           3454 non-null   int64  \n",
            " 12  days_used              3454 non-null   int64  \n",
            " 13  normalized_used_price  3454 non-null   float64\n",
            " 14  normalized_new_price   3454 non-null   float64\n",
            "dtypes: float64(9), int64(2), object(4)\n",
            "memory usage: 404.9+ KB\n"
          ]
        }
      ],
      "source": [
        "data.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yXcbwEJcWWWI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 394
        },
        "outputId": "1fa359c6-fee4-41b7-95ca-67b395d730f0"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                        count         mean          std          min  \\\n",
              "screen_size            3454.0    13.713115     3.805280     5.080000   \n",
              "main_camera_mp         3275.0     9.460208     4.815461     0.080000   \n",
              "selfie_camera_mp       3452.0     6.554229     6.970372     0.000000   \n",
              "int_memory             3450.0    54.573099    84.972371     0.010000   \n",
              "ram                    3450.0     4.036122     1.365105     0.020000   \n",
              "battery                3448.0  3133.402697  1299.682844   500.000000   \n",
              "weight                 3447.0   182.751871    88.413228    69.000000   \n",
              "release_year           3454.0  2015.965258     2.298455  2013.000000   \n",
              "days_used              3454.0   674.869716   248.580166    91.000000   \n",
              "normalized_used_price  3454.0     4.364712     0.588914     1.536867   \n",
              "normalized_new_price   3454.0     5.233107     0.683637     2.901422   \n",
              "\n",
              "                               25%          50%          75%          max  \n",
              "screen_size              12.700000    12.830000    15.340000    30.710000  \n",
              "main_camera_mp            5.000000     8.000000    13.000000    48.000000  \n",
              "selfie_camera_mp          2.000000     5.000000     8.000000    32.000000  \n",
              "int_memory               16.000000    32.000000    64.000000  1024.000000  \n",
              "ram                       4.000000     4.000000     4.000000    12.000000  \n",
              "battery                2100.000000  3000.000000  4000.000000  9720.000000  \n",
              "weight                  142.000000   160.000000   185.000000   855.000000  \n",
              "release_year           2014.000000  2015.500000  2018.000000  2020.000000  \n",
              "days_used               533.500000   690.500000   868.750000  1094.000000  \n",
              "normalized_used_price     4.033931     4.405133     4.755700     6.619433  \n",
              "normalized_new_price      4.790342     5.245892     5.673718     7.847841  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4618fc11-f000-48e0-91d1-ec43cc3657c6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "      <th>mean</th>\n",
              "      <th>std</th>\n",
              "      <th>min</th>\n",
              "      <th>25%</th>\n",
              "      <th>50%</th>\n",
              "      <th>75%</th>\n",
              "      <th>max</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>screen_size</th>\n",
              "      <td>3454.0</td>\n",
              "      <td>13.713115</td>\n",
              "      <td>3.805280</td>\n",
              "      <td>5.080000</td>\n",
              "      <td>12.700000</td>\n",
              "      <td>12.830000</td>\n",
              "      <td>15.340000</td>\n",
              "      <td>30.710000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>main_camera_mp</th>\n",
              "      <td>3275.0</td>\n",
              "      <td>9.460208</td>\n",
              "      <td>4.815461</td>\n",
              "      <td>0.080000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>13.000000</td>\n",
              "      <td>48.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>selfie_camera_mp</th>\n",
              "      <td>3452.0</td>\n",
              "      <td>6.554229</td>\n",
              "      <td>6.970372</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>8.000000</td>\n",
              "      <td>32.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>int_memory</th>\n",
              "      <td>3450.0</td>\n",
              "      <td>54.573099</td>\n",
              "      <td>84.972371</td>\n",
              "      <td>0.010000</td>\n",
              "      <td>16.000000</td>\n",
              "      <td>32.000000</td>\n",
              "      <td>64.000000</td>\n",
              "      <td>1024.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ram</th>\n",
              "      <td>3450.0</td>\n",
              "      <td>4.036122</td>\n",
              "      <td>1.365105</td>\n",
              "      <td>0.020000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>12.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>battery</th>\n",
              "      <td>3448.0</td>\n",
              "      <td>3133.402697</td>\n",
              "      <td>1299.682844</td>\n",
              "      <td>500.000000</td>\n",
              "      <td>2100.000000</td>\n",
              "      <td>3000.000000</td>\n",
              "      <td>4000.000000</td>\n",
              "      <td>9720.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>weight</th>\n",
              "      <td>3447.0</td>\n",
              "      <td>182.751871</td>\n",
              "      <td>88.413228</td>\n",
              "      <td>69.000000</td>\n",
              "      <td>142.000000</td>\n",
              "      <td>160.000000</td>\n",
              "      <td>185.000000</td>\n",
              "      <td>855.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>release_year</th>\n",
              "      <td>3454.0</td>\n",
              "      <td>2015.965258</td>\n",
              "      <td>2.298455</td>\n",
              "      <td>2013.000000</td>\n",
              "      <td>2014.000000</td>\n",
              "      <td>2015.500000</td>\n",
              "      <td>2018.000000</td>\n",
              "      <td>2020.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>days_used</th>\n",
              "      <td>3454.0</td>\n",
              "      <td>674.869716</td>\n",
              "      <td>248.580166</td>\n",
              "      <td>91.000000</td>\n",
              "      <td>533.500000</td>\n",
              "      <td>690.500000</td>\n",
              "      <td>868.750000</td>\n",
              "      <td>1094.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>normalized_used_price</th>\n",
              "      <td>3454.0</td>\n",
              "      <td>4.364712</td>\n",
              "      <td>0.588914</td>\n",
              "      <td>1.536867</td>\n",
              "      <td>4.033931</td>\n",
              "      <td>4.405133</td>\n",
              "      <td>4.755700</td>\n",
              "      <td>6.619433</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>normalized_new_price</th>\n",
              "      <td>3454.0</td>\n",
              "      <td>5.233107</td>\n",
              "      <td>0.683637</td>\n",
              "      <td>2.901422</td>\n",
              "      <td>4.790342</td>\n",
              "      <td>5.245892</td>\n",
              "      <td>5.673718</td>\n",
              "      <td>7.847841</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4618fc11-f000-48e0-91d1-ec43cc3657c6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-4618fc11-f000-48e0-91d1-ec43cc3657c6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-4618fc11-f000-48e0-91d1-ec43cc3657c6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "data.describe().T # statistical summary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xNr4bWoM-qT5"
      },
      "source": [
        "### Check for duplicate values"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "list(data.columns.values)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-59W5ssf2z73",
        "outputId": "d368e7ac-de83-4ce0-f9f5-ece209eb17af"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['brand_name',\n",
              " 'os',\n",
              " 'screen_size',\n",
              " '4g',\n",
              " '5g',\n",
              " 'main_camera_mp',\n",
              " 'selfie_camera_mp',\n",
              " 'int_memory',\n",
              " 'ram',\n",
              " 'battery',\n",
              " 'weight',\n",
              " 'release_year',\n",
              " 'days_used',\n",
              " 'normalized_used_price',\n",
              " 'normalized_new_price']"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "headings_list=list(data.columns.values)"
      ],
      "metadata": {
        "id": "8fTQTzGd3Wkx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print('brand_name')\n",
        "data['brand_name'].head(2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yP12lVNF6ERY",
        "outputId": "7601afb9-bba4-41bb-c06b-f5f2fd5340ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "brand_name\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    Honor\n",
              "1    Honor\n",
              "Name: brand_name, dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for head_n,heading in enumerate(headings_list):\n",
        "  # print(heading)\n",
        "  print(data[heading].head(2), '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "azoB3Vsc53dk",
        "outputId": "fe74651d-9dc5-41c3-b383-06ae88660401"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0    Honor\n",
            "1    Honor\n",
            "Name: brand_name, dtype: object \n",
            "\n",
            "0    Android\n",
            "1    Android\n",
            "Name: os, dtype: object \n",
            "\n",
            "0    14.5\n",
            "1    17.3\n",
            "Name: screen_size, dtype: float64 \n",
            "\n",
            "0    yes\n",
            "1    yes\n",
            "Name: 4g, dtype: object \n",
            "\n",
            "0     no\n",
            "1    yes\n",
            "Name: 5g, dtype: object \n",
            "\n",
            "0    13.0\n",
            "1    13.0\n",
            "Name: main_camera_mp, dtype: float64 \n",
            "\n",
            "0     5.0\n",
            "1    16.0\n",
            "Name: selfie_camera_mp, dtype: float64 \n",
            "\n",
            "0     64.0\n",
            "1    128.0\n",
            "Name: int_memory, dtype: float64 \n",
            "\n",
            "0    3.0\n",
            "1    8.0\n",
            "Name: ram, dtype: float64 \n",
            "\n",
            "0    3020.0\n",
            "1    4300.0\n",
            "Name: battery, dtype: float64 \n",
            "\n",
            "0    146.0\n",
            "1    213.0\n",
            "Name: weight, dtype: float64 \n",
            "\n",
            "0    2020\n",
            "1    2020\n",
            "Name: release_year, dtype: int64 \n",
            "\n",
            "0    127\n",
            "1    325\n",
            "Name: days_used, dtype: int64 \n",
            "\n",
            "0    4.307572\n",
            "1    5.162097\n",
            "Name: normalized_used_price, dtype: float64 \n",
            "\n",
            "0    4.715100\n",
            "1    5.519018\n",
            "Name: normalized_new_price, dtype: float64 \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# duplicates_list=data.duplicated(subset=None, keep='first')\n",
        "# print(duplicates_list)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PeAjz2NcCpTT",
        "outputId": "4e853d19-ba5e-408e-ebc1-abae47d4229d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0       False\n",
            "1       False\n",
            "2       False\n",
            "3       False\n",
            "4       False\n",
            "        ...  \n",
            "3449    False\n",
            "3450    False\n",
            "3451    False\n",
            "3452    False\n",
            "3453    False\n",
            "Length: 3454, dtype: bool\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# countT=0\n",
        "# countF=0\n",
        "# for n,boolean_value in enumerate(duplicates_list):\n",
        "#   if (boolean_value==True):\n",
        "#     countT+=1\n",
        "#   elif (boolean_value==False):\n",
        "#     countF+=1"
      ],
      "metadata": {
        "id": "FpS4khfwnF4V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# print(countT)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qJ6KYUiQlEuF",
        "outputId": "f5029abe-4839-4358-ef27-ba2d5b0baea8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# print(countF)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jjVGGX0rlILf",
        "outputId": "618466a8-be18-42ac-a8a5-2eb36baf7493"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3454\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# len(duplicates_list)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d_-SRjz1qLam",
        "outputId": "20a515a5-13a4-4e15-98b2-8585dcefddbf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3454"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# if len(duplicates_list)==countF:\n",
        "#   print(\"all values are False\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7yhjsEOup3OJ",
        "outputId": "af236f96-4605-4e04-f931-e5b94f9c052e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "all values are False\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "All values are False means that there are no duplicates"
      ],
      "metadata": {
        "id": "YdQtqo3JqUVs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "duplicate = data[data.duplicated()]\n",
        "print(duplicate)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W9SG4hsk9SXM",
        "outputId": "10ba9c80-a82a-465c-8442-7dc375947cf9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Empty DataFrame\n",
            "Columns: [brand_name, os, screen_size, 4g, 5g, main_camera_mp, selfie_camera_mp, int_memory, ram, battery, weight, release_year, days_used, normalized_used_price, normalized_new_price]\n",
            "Index: []\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ch_TjRfF-qT5"
      },
      "source": [
        "### Checking for missing values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c10B9HMNWq9K"
      },
      "outputs": [],
      "source": [
        "data.'_______' ## Complete the code to check duplicate entries in the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z4HfMUSiWxBT"
      },
      "outputs": [],
      "source": [
        "# creating a copy of the data so that original data remains unchanged\n",
        "df = data.copy()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DhPuzWO7hmV8"
      },
      "source": [
        "## Exploratory Data Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uxiNWC9JhmV8"
      },
      "source": [
        "### Univariate Analysis"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "opT6eO8qhmV8"
      },
      "outputs": [],
      "source": [
        "# function to plot a boxplot and a histogram along the same scale.\n",
        "\n",
        "\n",
        "def histogram_boxplot(data, feature, figsize=(15, 10), kde=False, bins=None):\n",
        "    \"\"\"\n",
        "    Boxplot and histogram combined\n",
        "\n",
        "    data: dataframe\n",
        "    feature: dataframe column\n",
        "    figsize: size of figure (default (15,10))\n",
        "    kde: whether to show the density curve (default False)\n",
        "    bins: number of bins for histogram (default None)\n",
        "    \"\"\"\n",
        "    f2, (ax_box2, ax_hist2) = plt.subplots(\n",
        "        nrows=2,  # Number of rows of the subplot grid= 2\n",
        "        sharex=True,  # x-axis will be shared among all subplots\n",
        "        gridspec_kw={\"height_ratios\": (0.25, 0.75)},\n",
        "        figsize=figsize,\n",
        "    )  # creating the 2 subplots\n",
        "    sns.boxplot(\n",
        "        data=data, x=feature, ax=ax_box2, showmeans=True, color=\"violet\"\n",
        "    )  # boxplot will be created and a triangle will indicate the mean value of the column\n",
        "    sns.histplot(\n",
        "        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins\n",
        "    ) if bins else sns.histplot(\n",
        "        data=data, x=feature, kde=kde, ax=ax_hist2\n",
        "    )  # For histogram\n",
        "    ax_hist2.axvline(\n",
        "        data[feature].mean(), color=\"green\", linestyle=\"--\"\n",
        "    )  # Add mean to the histogram\n",
        "    ax_hist2.axvline(\n",
        "        data[feature].median(), color=\"black\", linestyle=\"-\"\n",
        "    )  # Add median to the histogram"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1I7psbWYhmWD"
      },
      "outputs": [],
      "source": [
        "# function to create labeled barplots\n",
        "\n",
        "\n",
        "def labeled_barplot(data, feature, perc=False, n=None):\n",
        "    \"\"\"\n",
        "    Barplot with percentage at the top\n",
        "\n",
        "    data: dataframe\n",
        "    feature: dataframe column\n",
        "    perc: whether to display percentages instead of count (default is False)\n",
        "    n: displays the top n category levels (default is None, i.e., display all levels)\n",
        "    \"\"\"\n",
        "\n",
        "    total = len(data[feature])  # length of the column\n",
        "    count = data[feature].nunique()\n",
        "    if n is None:\n",
        "        plt.figure(figsize=(count + 2, 6))\n",
        "    else:\n",
        "        plt.figure(figsize=(n + 2, 6))\n",
        "\n",
        "    plt.xticks(rotation=90, fontsize=15)\n",
        "    ax = sns.countplot(\n",
        "        data=data,\n",
        "        x=feature,\n",
        "        palette=\"Paired\",\n",
        "        order=data[feature].value_counts().index[:n],\n",
        "    )\n",
        "\n",
        "    for p in ax.patches:\n",
        "        if perc == True:\n",
        "            label = \"{:.1f}%\".format(\n",
        "                100 * p.get_height() / total\n",
        "            )  # percentage of each class of the category\n",
        "        else:\n",
        "            label = p.get_height()  # count of each level of the category\n",
        "\n",
        "        x = p.get_x() + p.get_width() / 2  # width of the plot\n",
        "        y = p.get_height()  # height of the plot\n",
        "\n",
        "        ax.annotate(\n",
        "            label,\n",
        "            (x, y),\n",
        "            ha=\"center\",\n",
        "            va=\"center\",\n",
        "            size=12,\n",
        "            xytext=(0, 5),\n",
        "            textcoords=\"offset points\",\n",
        "        )  # annotate the percentage\n",
        "\n",
        "    plt.show()  # show the plot"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "woJI2SC5hmV9"
      },
      "source": [
        "**`normalized_used_price`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KOz9j7M_hmV9"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot(df, \"normalized_used_price\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YYIpGoRWhmV9"
      },
      "source": [
        "**`normalized_new_price`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K_AVTS9-hmV_"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'normalized_new_price'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVuQ8smPhmV_"
      },
      "source": [
        "**`screen_size`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kDcR4t5uhmV_"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'screen_size'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zeDwMOz6hmV_"
      },
      "source": [
        "**`main_camera_mp`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qZHkTniOhmWA"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'main_camera_mp'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MZav4AxrhmWA"
      },
      "source": [
        "**`selfie_camera_mp`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZxX6OlLehmWA"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'selfie_camera_mp'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oFXxdInchmWA"
      },
      "source": [
        "**`int_memory`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FhrZ64l_hmWA"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'int_memory'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eXF46221hmWA"
      },
      "source": [
        "**`ram`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2TfjASGThmWA"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'ram'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fTZI2sO1hmWB"
      },
      "source": [
        "**`weight`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PDxaAIDZhmWB"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'weight'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VXgPSxGbhmWB"
      },
      "source": [
        "**`battery`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9Wo--kEQhmWB"
      },
      "outputs": [],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'battery'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9WPI4QMjhmWB"
      },
      "source": [
        "**`days_used`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v3JU7JxehmWB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 165
        },
        "outputId": "f8afaa08-3507-47d5-8d58-76c0c3e76aa5"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-2-08b228970eed>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mhistogram_boxplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'_______'\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m## Complete the code to create histogram_boxplot for 'days_used'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'histogram_boxplot' is not defined"
          ]
        }
      ],
      "source": [
        "histogram_boxplot('_______')  ## Complete the code to create histogram_boxplot for 'days_used'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "utY4IudWhmWD"
      },
      "source": [
        "**`brand_name`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5VvbAPpYhmWD"
      },
      "outputs": [],
      "source": [
        "labeled_barplot(df, \"brand_name\", perc=True, n=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NIqtv7SHhmWD"
      },
      "source": [
        "**`os`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pG7BdJ30hmWD"
      },
      "outputs": [],
      "source": [
        "labeled_barplot('_______') ## Complete the code to create labeled_barplot for 'os'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5rGNYyuahmWD"
      },
      "source": [
        "**`4g`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wtGMz_aEhmWD"
      },
      "outputs": [],
      "source": [
        "labeled_barplot('_______') ## Complete the code to create labeled_barplot for '4g'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7k69m1GPhmWD"
      },
      "source": [
        "**`5g`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A5s9SA49hmWE"
      },
      "outputs": [],
      "source": [
        "labeled_barplot('_______') ## Complete the code to create labeled_barplot for '5g'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bTnwoEZ6hmWE"
      },
      "source": [
        "**`release_year`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lgV3q1tUhmWE"
      },
      "outputs": [],
      "source": [
        "labeled_barplot('_______') ## Complete the code to create labeled_barplot for 'release_year'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ga_huJrnhmWE"
      },
      "source": [
        "### Bivariate Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jc8kJMd6VxHC"
      },
      "source": [
        "**Correlation Check**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BpJxawjAhmWE"
      },
      "outputs": [],
      "source": [
        "cols_list = df.select_dtypes(include=np.number).columns.tolist()\n",
        "# dropping release_year as it is a temporal variable\n",
        "cols_list.remove(\"release_year\")\n",
        "\n",
        "plt.figure(figsize=(15, 7))\n",
        "sns.heatmap(\n",
        "    df[cols_list].corr(), annot=True, vmin=-1, vmax=1, fmt=\".2f\", cmap=\"Spectral\"\n",
        ")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QLjArqLFhmWF"
      },
      "source": [
        "**The amount of RAM is important for the smooth functioning of a device. Let's see how the amount of RAM varies across brands.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q77kiPCxhmWF"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(15, 5))\n",
        "sns.boxplot(data=df, x=\"brand_name\", y=\"ram\")\n",
        "plt.xticks(rotation=90)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeDTwk21hmWF"
      },
      "source": [
        "**People who travel frequently require devices with large batteries to run through the day. But large battery often increases weight, making it feel uncomfortable in the hands. Let's create a new dataframe of only those devices which offer a large battery and analyze.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cbcd1vbZhmWF"
      },
      "outputs": [],
      "source": [
        "df_large_battery = df[df.battery > 4500]\n",
        "df_large_battery.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kcrj-ImghmWF"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(15, 5))\n",
        "sns.boxplot('______') ## Complete the code to create a boxplot for 'brand_name' and 'weight'\n",
        "plt.xticks(rotation=90)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E9o5QoShhmWF"
      },
      "source": [
        "**People who buy phones and tablets primarily for entertainment purposes prefer a large screen as they offer a better viewing experience. Let's create a new dataframe of only those devices which are suitable for such people and analyze.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i1DT9SDBhmWF"
      },
      "outputs": [],
      "source": [
        "df_large_screen = df[df.screen_size > 6 * 2.54]\n",
        "df_large_screen.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CkSkJ0GIhmWG"
      },
      "outputs": [],
      "source": [
        "labeled_barplot('_______') ## Complete the code to create labeled_barplot for 'brand_name' in large screen dataframe"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UZ4IWWn7hmWG"
      },
      "source": [
        "**Everyone likes a good camera to capture their favorite moments with loved ones. Some customers specifically look for good front cameras to click cool selfies. Let's create a new dataframe of only those devices which are suitable for this customer segment and analyze.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lAond0uQhmWG"
      },
      "outputs": [],
      "source": [
        "df_selfie_camera = df[df.selfie_camera_mp > 8]\n",
        "df_selfie_camera.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OdP27nZBhmWG"
      },
      "outputs": [],
      "source": [
        "labeled_barplot('_______') ## Complete the code to create labeled_barplot for 'brand_name' in high selfie camera dataframe"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BmFNBDmxhmWH"
      },
      "source": [
        "**Let's do a similar analysis for rear cameras.**\n",
        "\n",
        "- Rear cameras generally have a better resolution than front cameras, so we set the threshold higher for them at 16MP."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-xoY4afZhmWH"
      },
      "outputs": [],
      "source": [
        "df_main_camera = df[df.main_camera_mp > 16]\n",
        "df_main_camera.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gegyv89HhmWH"
      },
      "outputs": [],
      "source": [
        "labeled_barplot('_______') ## Complete the code to create labeled_barplot for 'brand_name' in high main camera dataframe"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SDx7dJtEhmWH"
      },
      "source": [
        "**Let's see how the price of used devices varies across the years.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HrdhOpHthmWH"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(12, 5))\n",
        "sns.lineplot('_____') ## Complete the code to create a lineplot for release year and used price\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vvLSkuZ2hmWI"
      },
      "source": [
        "**Let's check how the prices vary for used phones and tablets offering 4G and 5G networks.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x5ni9yVPhmWI"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(10, 4))\n",
        "\n",
        "plt.subplot(121)\n",
        "sns.boxplot(data=df, x=\"4g\", y=\"normalized_used_price\")\n",
        "\n",
        "plt.subplot(122)\n",
        "sns.boxplot(data=df, x=\"5g\", y=\"normalized_used_price\")\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sbDJ_Vl8hmWI"
      },
      "source": [
        "## Data Preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Jnj657ShmWI"
      },
      "source": [
        "### Missing Value Imputation\n",
        "\n",
        "- We will impute the missing values in the data by the column medians grouped by `release_year` and `brand_name`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dzp0-E7rKffM"
      },
      "outputs": [],
      "source": [
        "# let's create a copy of the data\n",
        "df1 = df.copy()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0QfWi8PYhmWI"
      },
      "outputs": [],
      "source": [
        "# checking for missing values\n",
        "df1.'_______' ## Complete the code to check missing values in all the columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KQEug5qNhmWJ"
      },
      "outputs": [],
      "source": [
        "cols_impute = [\n",
        "    \"main_camera_mp\",\n",
        "    \"selfie_camera_mp\",\n",
        "    \"int_memory\",\n",
        "    \"ram\",\n",
        "    \"battery\",\n",
        "    \"weight\",\n",
        "]\n",
        "\n",
        "for col in cols_impute:\n",
        "    df1[col] = df1[col].fillna(\n",
        "        value=df1.groupby(['_____'])[col].transform(\"median\")\n",
        "    )   ## Complete the code to impute missing values in cols_impute with median by grouping the data on release year and brand name \n",
        "\n",
        "# checking for missing values\n",
        "df1.'_______' ## Complete the code to check missing values after imputing the above columns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XF_SHRjPhmWJ"
      },
      "source": [
        "- We will impute the remaining missing values in the data by the column medians grouped by `brand_name`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IzHBI4C6hmWJ"
      },
      "outputs": [],
      "source": [
        "cols_impute = [\n",
        "    \"main_camera_mp\",\n",
        "    \"selfie_camera_mp\",\n",
        "    \"battery\",\n",
        "    \"weight\",\n",
        "]\n",
        "\n",
        "for col in cols_impute:\n",
        "    df1[col] = df1[col].fillna(\n",
        "        value=df1.groupby(['_____'])[col].transform(\"median\")\n",
        "    ) ## Complete the code to impute the missing values in cols_impute with median by grouping the data on brand name\n",
        "\n",
        "# checking for missing values\n",
        "df1.'_______' ## Complete the code to check missing values after imputing the above columns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B6JWbAyQhmWJ"
      },
      "source": [
        "- We will fill the remaining missing values in the `main_camera_mp` column by the column median."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uQJgQmzohmWJ"
      },
      "outputs": [],
      "source": [
        "df1[\"main_camera_mp\"] = df1[\"main_camera_mp\"].fillna(df1[\"main_camera_mp\"].'_______') ## Complete the code to impute the data with median\n",
        "\n",
        "# checking for missing values\n",
        "df1.'_______' ## Complete the code to check missing values after imputing the above columns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OPB6dxxmWq1s"
      },
      "source": [
        "### Feature Engineering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zYOa-FxlhmWG"
      },
      "source": [
        "\n",
        "\n",
        "- Let's create a new column `years_since_release` from the `release_year` column.\n",
        "- We will consider the year of data collection, 2021, as the baseline.\n",
        "- We will drop the `release_year` column."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ihTY3XRkhmWG"
      },
      "outputs": [],
      "source": [
        "df1[\"years_since_release\"] = 2021 - df1[\"release_year\"]\n",
        "df1.drop(\"release_year\", axis=1, inplace=True)\n",
        "df1[\"years_since_release\"].describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4jUrmT40Wx7M"
      },
      "source": [
        "### Outlier Check\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLoHqqf1hmWJ"
      },
      "source": [
        "\n",
        "- Let's check for outliers in the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2RSTq3PWhmWJ"
      },
      "outputs": [],
      "source": [
        "# outlier detection using boxplot\n",
        "num_cols = df1.select_dtypes(include=np.number).columns.tolist()\n",
        "\n",
        "plt.figure(figsize=(15, 15))\n",
        "\n",
        "for i, variable in enumerate(num_cols):\n",
        "    plt.subplot(4, 3, i + 1)\n",
        "    sns.boxplot(data=df1, x=variable)\n",
        "    plt.tight_layout(pad=2)\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QChrTYFaKvTk"
      },
      "source": [
        "### Data Preparation for modeling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wwu8iCRQhmWK"
      },
      "source": [
        "- We want to predict the normalized price of used devices\n",
        "- Before we proceed to build a model, we'll have to encode categorical features\n",
        "- We'll split the data into train and test to be able to evaluate the model that we build on the train data\n",
        "- We will build a Linear Regression model using the train data and then check it's performance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "372WHRhJGpj4"
      },
      "outputs": [],
      "source": [
        "## Complete the code to define the dependent and independent variables\n",
        "X = '_______'\n",
        "y = '_______'\n",
        "\n",
        "print(X.head())\n",
        "print()\n",
        "print(y.head())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Py6xQDcfLQLk"
      },
      "outputs": [],
      "source": [
        "# let's add the intercept to data\n",
        "X = sm.add_constant(X)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xWw6sT0yhmWK"
      },
      "outputs": [],
      "source": [
        "# creating dummy variables\n",
        "X = pd.'_______'(\n",
        "    X,\n",
        "    columns=X.select_dtypes(include=[\"object\", \"category\"]).columns.tolist(),\n",
        "    drop_first=True,\n",
        ")  ## Complete the code to create dummies for independent features\n",
        "\n",
        "X.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CBXfTtAPGoaF"
      },
      "outputs": [],
      "source": [
        "# splitting the data in 70:30 ratio for train to test data\n",
        "\n",
        "x_train, x_test, y_train, y_test = '_______' ## Complete the code to split the data into train and test in specified ratio"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xUGp866qhmWK"
      },
      "outputs": [],
      "source": [
        "print(\"Number of rows in train data =\", x_train.shape[0])\n",
        "print(\"Number of rows in test data =\", x_test.shape[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1JsqBJEohmWK"
      },
      "source": [
        "## Model Building - Linear Regression\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-uylJTNMqH8b",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "olsmodel1 = sm.'_______' ## Complete the code to fit OLS model\n",
        "print(olsmodel1.summary())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dgIbf7ALlYm"
      },
      "source": [
        "### Model Performance Check"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5MyYq4vaKffS"
      },
      "source": [
        "**Let's check the performance of the model using different metrics.**\n",
        "\n",
        "* We will be using metric functions defined in sklearn for RMSE, MAE, and $R^2$.\n",
        "* We will define a function to calculate MAPE and adjusted $R^2$.    \n",
        "* We will create a function which will print out all the above metrics in one go."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "deaapIIvlJIL"
      },
      "outputs": [],
      "source": [
        "# function to compute adjusted R-squared\n",
        "def adj_r2_score(predictors, targets, predictions):\n",
        "    r2 = r2_score(targets, predictions)\n",
        "    n = predictors.shape[0]\n",
        "    k = predictors.shape[1]\n",
        "    return 1 - ((1 - r2) * (n - 1) / (n - k - 1))\n",
        "\n",
        "\n",
        "# function to compute MAPE\n",
        "def mape_score(targets, predictions):\n",
        "    return np.mean(np.abs(targets - predictions) / targets) * 100\n",
        "\n",
        "\n",
        "# function to compute different metrics to check performance of a regression model\n",
        "def model_performance_regression(model, predictors, target):\n",
        "    \"\"\"\n",
        "    Function to compute different metrics to check regression model performance\n",
        "\n",
        "    model: regressor\n",
        "    predictors: independent variables\n",
        "    target: dependent variable\n",
        "    \"\"\"\n",
        "\n",
        "    # predicting using the independent variables\n",
        "    pred = model.predict(predictors)\n",
        "\n",
        "    r2 = r2_score(target, pred)  # to compute R-squared\n",
        "    adjr2 = adj_r2_score(predictors, target, pred)  # to compute adjusted R-squared\n",
        "    rmse = np.sqrt(mean_squared_error(target, pred))  # to compute RMSE\n",
        "    mae = mean_absolute_error(target, pred)  # to compute MAE\n",
        "    mape = mape_score(target, pred)  # to compute MAPE\n",
        "\n",
        "    # creating a dataframe of metrics\n",
        "    df_perf = pd.DataFrame(\n",
        "        {\n",
        "            \"RMSE\": rmse,\n",
        "            \"MAE\": mae,\n",
        "            \"R-squared\": r2,\n",
        "            \"Adj. R-squared\": adjr2,\n",
        "            \"MAPE\": mape,\n",
        "        },\n",
        "        index=[0],\n",
        "    )\n",
        "\n",
        "    return df_perf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vVBBaPtf-BZd"
      },
      "outputs": [],
      "source": [
        "# checking model performance on train set (seen 70% data)\n",
        "print(\"Training Performance\\n\")\n",
        "olsmodel1_train_perf = model_performance_regression(olsmodel1, x_train, y_train)\n",
        "olsmodel1_train_perf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Sv51vNBhlJIL"
      },
      "outputs": [],
      "source": [
        "# checking model performance on test set (seen 30% data)\n",
        "print(\"Test Performance\\n\")\n",
        "olsmodel1_test_perf = model_performance_regression('_______') ## Complete the code to check the performance on test data\n",
        "olsmodel1_test_perf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a9GxSQf-qH8e"
      },
      "source": [
        "## Checking Linear Regression Assumptions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Wr6XkwoqH8f"
      },
      "source": [
        "We will be checking the following Linear Regression assumptions:\n",
        "\n",
        "1. **No Multicollinearity**\n",
        "\n",
        "2. **Linearity of variables**\n",
        "\n",
        "3. **Independence of error terms**\n",
        "\n",
        "4. **Normality of error terms**\n",
        "\n",
        "5. **No Heteroscedasticity**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SB4GanGWSOBT"
      },
      "source": [
        "### TEST FOR MULTICOLLINEARITY\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mROHQdzZqH8f"
      },
      "source": [
        "\n",
        "- We will test for multicollinearity using VIF.\n",
        "\n",
        "- **General Rule of thumb**:\n",
        "    - If VIF is 1 then there is no correlation between the $k$th predictor and the remaining predictor variables.\n",
        "    - If VIF exceeds 5 or is close to exceeding 5, we say there is moderate multicollinearity.\n",
        "    - If VIF is 10 or exceeding 10, it shows signs of high multicollinearity."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hm6Y_e-mTvDl"
      },
      "source": [
        "Let's define a function to check VIF."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZV-69rVWqH8g"
      },
      "outputs": [],
      "source": [
        "#Let's define a function to check VIF.\n",
        "\n",
        "def checking_vif(predictors):\n",
        "    vif = pd.DataFrame()\n",
        "    vif[\"feature\"] = predictors.columns\n",
        "\n",
        "    # calculating VIF for each feature\n",
        "    vif[\"VIF\"] = [\n",
        "        variance_inflation_factor(predictors.values, i)\n",
        "        for i in range(len(predictors.columns))\n",
        "    ]\n",
        "    return vif"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xPdvX4xZhmWL"
      },
      "outputs": [],
      "source": [
        "checking_vif('_______')  ## Complete the code to check VIF on train data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ulUc-ceISeyu"
      },
      "source": [
        "### Removing Multicollinearity (if needed)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XSiS4SRaqH8q"
      },
      "source": [
        "\n",
        "To remove multicollinearity\n",
        "\n",
        "1. Drop every column one by one that has a VIF score greater than 5.\n",
        "2. Look at the adjusted R-squared and RMSE of all these models.\n",
        "3. Drop the variable that makes the least change in adjusted R-squared.\n",
        "4. Check the VIF scores again.\n",
        "5. Continue till you get all VIF scores under 5.\n",
        "\n",
        "Let's define a function that will help us do this."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "93U9udWjKffT"
      },
      "outputs": [],
      "source": [
        "def treating_multicollinearity(predictors, target, high_vif_columns):\n",
        "    \"\"\"\n",
        "    Checking the effect of dropping the columns showing high multicollinearity\n",
        "    on model performance (adj. R-squared and RMSE)\n",
        "\n",
        "    predictors: independent variables\n",
        "    target: dependent variable\n",
        "    high_vif_columns: columns having high VIF\n",
        "    \"\"\"\n",
        "    # empty lists to store adj. R-squared and RMSE values\n",
        "    adj_r2 = []\n",
        "    rmse = []\n",
        "\n",
        "    # build ols models by dropping one of the high VIF columns at a time\n",
        "    # store the adjusted R-squared and RMSE in the lists defined previously\n",
        "    for cols in high_vif_columns:\n",
        "        # defining the new train set\n",
        "        train = predictors.loc[:, ~predictors.columns.str.startswith(cols)]\n",
        "\n",
        "        # create the model\n",
        "        olsmodel = sm.OLS(target, train).fit()\n",
        "\n",
        "        # adding adj. R-squared and RMSE to the lists\n",
        "        adj_r2.append(olsmodel.rsquared_adj)\n",
        "        rmse.append(np.sqrt(olsmodel.mse_resid))\n",
        "\n",
        "    # creating a dataframe for the results\n",
        "    temp = pd.DataFrame(\n",
        "        {\n",
        "            \"col\": high_vif_columns,\n",
        "            \"Adj. R-squared after_dropping col\": adj_r2,\n",
        "            \"RMSE after dropping col\": rmse,\n",
        "        }\n",
        "    ).sort_values(by=\"Adj. R-squared after_dropping col\", ascending=False)\n",
        "    temp.reset_index(drop=True, inplace=True)\n",
        "\n",
        "    return temp"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r08PiCLiKffT"
      },
      "outputs": [],
      "source": [
        "col_list = ['____'] ## Complete the code to specify the columns with high VIF\n",
        "\n",
        "res = treating_multicollinearity('_____', y_train, col_list) ## Complete the code to check the effect on model performance after dropping specified columns from train data\n",
        "res"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JDJlsR2dKffU"
      },
      "outputs": [],
      "source": [
        "col_to_drop = '_____' ## Complete the code to specify the column to drop\n",
        "x_train2 = '_____'.loc[:, ~'_____'.columns.str.startswith(col_to_drop)] ## Complete the code to specify the train data from which to drop the column specified\n",
        "x_test2 = '_____'.loc[:, ~'_____'.columns.str.startswith(col_to_drop)] ## Complete the code to specify the test data from which to drop the column specified\n",
        "\n",
        "# Check VIF now\n",
        "vif = checking_vif(x_train2)\n",
        "print(\"VIF after dropping \", col_to_drop)\n",
        "vif"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j2hb9VujhmWL"
      },
      "source": [
        "### Dropping high p-value variables (if needed)\n",
        "\n",
        "- We will drop the predictor variables having a p-value greater than 0.05 as they do not significantly impact the target variable.\n",
        "- But sometimes p-values change after dropping a variable. So, we'll not drop all variables at once.\n",
        "- Instead, we will do the following:\n",
        "    - Build a model, check the p-values of the variables, and drop the column with the highest p-value.\n",
        "    - Create a new model without the dropped feature, check the p-values of the variables, and drop the column with the highest p-value.\n",
        "    - Repeat the above two steps till there are no columns with p-value > 0.05.\n",
        "\n",
        "The above process can also be done manually by picking one variable at a time that has a high p-value, dropping it, and building a model again. But that might be a little tedious and using a loop will be more efficient."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kTFISTQlSy6K"
      },
      "outputs": [],
      "source": [
        "# initial list of columns\n",
        "predictors = '_____'.copy()  ## Complete the code to check for p-values on the right dataset\n",
        "cols = predictors.columns.tolist()\n",
        "\n",
        "# setting an initial max p-value\n",
        "max_p_value = 1\n",
        "\n",
        "while len(cols) > 0:\n",
        "    # defining the train set\n",
        "    x_train_aux = predictors[cols]\n",
        "\n",
        "    # fitting the model\n",
        "    model = sm.OLS(y_train, x_train_aux).fit()\n",
        "\n",
        "    # getting the p-values and the maximum p-value\n",
        "    p_values = model.pvalues\n",
        "    max_p_value = max(p_values)\n",
        "\n",
        "    # name of the variable with maximum p-value\n",
        "    feature_with_p_max = p_values.idxmax()\n",
        "\n",
        "    if max_p_value > 0.05:\n",
        "        cols.remove(feature_with_p_max)\n",
        "    else:\n",
        "        break\n",
        "\n",
        "selected_features = cols\n",
        "print(selected_features)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p0CfofLchmWL"
      },
      "outputs": [],
      "source": [
        "x_train3 = '____'[selected_features]  ## Complete the code to specify the train data from which to select the specified columns\n",
        "x_test3 = '____'[selected_features]  ## Complete the code to specify the test data from which to select the specified columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qaaUG3jDhmWM"
      },
      "outputs": [],
      "source": [
        "olsmodel2 = sm.'_______' ## Complete the code fit OLS() on updated dataset (no multicollinearity and no insignificant predictors)\n",
        "print(olsmodel2.summary())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pgmwbtRXhmWM"
      },
      "outputs": [],
      "source": [
        "# checking model performance on train set (seen 70% data)\n",
        "print(\"Training Performance\\n\")\n",
        "olsmodel2_train_perf = model_performance_regression('_______') ## Complete the code to check performance on train data\n",
        "olsmodel2_train_perf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-xMPZ9pRhmWM"
      },
      "outputs": [],
      "source": [
        "# checking model performance on test set (seen 30% data)\n",
        "print(\"Test Performance\\n\")\n",
        "olsmodel2_test_perf = model_performance_regression('_______') ## Complete the code to check performance on test data\n",
        "olsmodel2_test_perf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ctk29PsRqH90"
      },
      "source": [
        "**Now we'll check the rest of the assumptions on *olsmod2*.**\n",
        "\n",
        "2. **Linearity of variables**\n",
        "\n",
        "3. **Independence of error terms**\n",
        "\n",
        "4. **Normality of error terms**\n",
        "\n",
        "5. **No Heteroscedasticity**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykvaeUZ-TkjD"
      },
      "source": [
        "### TEST FOR LINEARITY AND INDEPENDENCE\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MnOY6E5uqH-C"
      },
      "source": [
        "\n",
        "- We will test for linearity and independence by making a plot of fitted values vs residuals and checking for patterns.\n",
        "- If there is no pattern, then we say the model is linear and residuals are independent.\n",
        "- Otherwise, the model is showing signs of non-linearity and residuals are not independent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aNkYG_qihmWM"
      },
      "outputs": [],
      "source": [
        "# let us create a dataframe with actual, fitted and residual values\n",
        "df_pred = pd.DataFrame()\n",
        "\n",
        "df_pred[\"Actual Values\"] = y_train  # actual values\n",
        "df_pred[\"Fitted Values\"] = olsmodel2.fittedvalues  # predicted values\n",
        "df_pred[\"Residuals\"] = olsmodel2.resid  # residuals\n",
        "\n",
        "df_pred.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NJ4ivHQiqH-G"
      },
      "outputs": [],
      "source": [
        "# let's plot the fitted values vs residuals\n",
        "\n",
        "sns.residplot(\n",
        "    data=df_pred, x=\"Fitted Values\", y=\"Residuals\", color=\"purple\", lowess=True\n",
        ")\n",
        "plt.xlabel(\"Fitted Values\")\n",
        "plt.ylabel(\"Residuals\")\n",
        "plt.title(\"Fitted vs Residual plot\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VOxf_qsQqH-N"
      },
      "source": [
        "### TEST FOR NORMALITY\n",
        "\n",
        "- We will test for normality by checking the distribution of residuals, by checking the Q-Q plot of residuals, and by using the Shapiro-Wilk test.\n",
        "- If the residuals follow a normal distribution, they will make a straight line plot, otherwise not.\n",
        "- If the p-value of the Shapiro-Wilk test is greater than 0.05, we can say the residuals are normally distributed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MUdQ6XJLqH-N"
      },
      "outputs": [],
      "source": [
        "sns.histplot(data=df_pred, '_______') ## Complete the code to plot the distribution of residuals\n",
        "plt.title(\"Normality of residuals\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Wj8y7-1nqH-Q"
      },
      "outputs": [],
      "source": [
        "import pylab\n",
        "import scipy.stats as stats\n",
        "\n",
        "stats.probplot('_____', dist=\"norm\", plot=pylab) ## Complete the code check Q-Q plot\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b1LIATpBqH-U"
      },
      "outputs": [],
      "source": [
        "stats.shapiro('_______') ## Complete the code to apply the Shapiro-Wilks test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FTnuLaLBXvdR"
      },
      "source": [
        "### TEST FOR HOMOSCEDASTICITY\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gZ4GKY_qH-Z"
      },
      "source": [
        "\n",
        "- We will test for homoscedasticity by using the goldfeldquandt test.\n",
        "- If we get a p-value greater than 0.05, we can say that the residuals are homoscedastic. Otherwise, they are heteroscedastic."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "naX-iXItqH-b"
      },
      "outputs": [],
      "source": [
        "import statsmodels.stats.api as sms\n",
        "from statsmodels.compat import lzip\n",
        "\n",
        "name = [\"F statistic\", \"p-value\"]\n",
        "test = sms.het_goldfeldquandt(df_pred[\"Residuals\"], '_____') ## Complete the code with the right train data to apply the Goldfeldquandt test\n",
        "lzip(name, test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aIjiQ1NghmWN"
      },
      "source": [
        "## Final Model Summary"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x_Sqvs4TMKtn"
      },
      "outputs": [],
      "source": [
        "olsmodel_final = sm.'_______' ## Complete the code to fit the final model\n",
        "print(olsmodel_final.summary())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4KbUJPfhhmWN"
      },
      "outputs": [],
      "source": [
        "# checking model performance on train set (seen 70% data)\n",
        "print(\"Training Performance\\n\")\n",
        "olsmodel_final_train_perf = model_performance_regression('_______') ## Complete the code to check the performance on train data\n",
        "olsmodel_final_train_perf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T-WL13KBhmWN"
      },
      "outputs": [],
      "source": [
        "# checking model performance on test set (seen 30% data)\n",
        "print(\"Test Performance\\n\")\n",
        "olsmodel_final_test_perf = model_performance_regression('_______') ## Complete the code to check performance on test data\n",
        "olsmodel_final_test_perf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2BkZh6eHluZK"
      },
      "source": [
        "##  Actionable Insights and Recommendations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pj1rya_NXY-y"
      },
      "source": [
        "- \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W9V-w49_ULvd"
      },
      "source": [
        "---------"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}