Location_Extraction_Naive_approach.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Extracting State, District, Village, geocodes using traditional methods."
      ],
      "metadata": {
        "id": "LxvymghxqzB2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "There aren't many reliable resources that provides geographical data of india. In this notebook, there are simple scraping functions that can be used to get data related to states, districts, villages, and their geo coordinates.\n",
        "\n",
        "This notebook can be used to access location data or be extended for extraction of villages and their codes. It is a starter notebook, created to understand how far we can go with simple approaches without using ML. So it can be considered as a baseline while evaluating other complex models."
      ],
      "metadata": {
        "id": "wPedbJ2hs--L"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "TODO:\n",
        "\n",
        "*   To get started, a naive function for extracting district is provided in this notebook, which has an accuracy of 80%. It can be further improved using fuzzy matching. Whether you're working on naive approaches or ML, fuzzy matching can be helpful especially during evaluation with our data. Since there can be errors at different levels of the manual coding of SATP database and other data sources.\n",
        "\n",
        "*   New functions are needed for extracting villages and their geocodes from the input summaries. It would be better to try ML approaches, since these approaches doesn't care about context, which is necessary for some inputs.\n",
        "\n"
      ],
      "metadata": {
        "id": "hK7glHb-vXS2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install unidecode"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dpulx37ODLc5",
        "outputId": "60c52f46-1b90-4267-e441-aeff376d227d"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: unidecode in /usr/local/lib/python3.10/dist-packages (1.3.8)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Import Libraries"
      ],
      "metadata": {
        "id": "goD-R75rzEFg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "import random\n",
        "import requests\n",
        "import unidecode\n",
        "import pandas as pd\n",
        "from tqdm.auto import tqdm\n",
        "from bs4 import BeautifulSoup"
      ],
      "metadata": {
        "id": "77a6TManDFC_"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Get Data"
      ],
      "metadata": {
        "id": "LMYNn9BBzGqG"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 314
        },
        "id": "3vhalsQ_zHN9",
        "outputId": "c9d116cc-3bf3-4367-dcdc-ee0487e55908"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   incident_number           state   district                    block  \\\n",
              "0      101010701.0  Andhra Pradesh  Hyderabad  Gachibowli (Rangareddy)   \n",
              "1      101010901.0  Andhra Pradesh  Nizamabad                      NaN   \n",
              "2      101030601.0  Andhra Pradesh    Khammam                      NaN   \n",
              "\n",
              "   village_name other_areas     constituency  longitude   latitude  year  ...  \\\n",
              "0           NaN   Cyberabad  Serilingampally  17.432500  78.371806  2007  ...   \n",
              "1     Kamareddy         NaN        Kamareddy  18.320889  78.337139  2009  ...   \n",
              "2  Bhadrachalam         NaN     Bhadrachalam  17.668056  80.896861  2006  ...   \n",
              "\n",
              "  commander_arrests cadre_arrests sympathizer_arrests unknown_arrests  \\\n",
              "0                 0           0.0                   1             0.0   \n",
              "1                 0           0.0                   0             0.0   \n",
              "2                 1           0.0                   0             0.0   \n",
              "\n",
              "  total_surrenders  commander_surrenders  cadre_surrenders  \\\n",
              "0              0.0                     0               0.0   \n",
              "1              1.0                     0               1.0   \n",
              "2              0.0                     0               0.0   \n",
              "\n",
              "   sympathizer_surrenders  unknown_surrenders  \\\n",
              "0                       0                   0   \n",
              "1                       0                   0   \n",
              "2                       0                   0   \n",
              "\n",
              "                                    incident_summary  \n",
              "0  An alleged arms supplier to the Communist Part...  \n",
              "1  A Kamareddy dalam (squad) member belonging to ...  \n",
              "2  Senior CPI-Maoist 'Polit Bureau' and 'central ...  \n",
              "\n",
              "[3 rows x 70 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5cbff9fa-a5db-4f9d-ad7b-9f8452b0de52\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>incident_number</th>\n",
              "      <th>state</th>\n",
              "      <th>district</th>\n",
              "      <th>block</th>\n",
              "      <th>village_name</th>\n",
              "      <th>other_areas</th>\n",
              "      <th>constituency</th>\n",
              "      <th>longitude</th>\n",
              "      <th>latitude</th>\n",
              "      <th>year</th>\n",
              "      <th>...</th>\n",
              "      <th>commander_arrests</th>\n",
              "      <th>cadre_arrests</th>\n",
              "      <th>sympathizer_arrests</th>\n",
              "      <th>unknown_arrests</th>\n",
              "      <th>total_surrenders</th>\n",
              "      <th>commander_surrenders</th>\n",
              "      <th>cadre_surrenders</th>\n",
              "      <th>sympathizer_surrenders</th>\n",
              "      <th>unknown_surrenders</th>\n",
              "      <th>incident_summary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>101010701.0</td>\n",
              "      <td>Andhra Pradesh</td>\n",
              "      <td>Hyderabad</td>\n",
              "      <td>Gachibowli (Rangareddy)</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Cyberabad</td>\n",
              "      <td>Serilingampally</td>\n",
              "      <td>17.432500</td>\n",
              "      <td>78.371806</td>\n",
              "      <td>2007</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>An alleged arms supplier to the Communist Part...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>101010901.0</td>\n",
              "      <td>Andhra Pradesh</td>\n",
              "      <td>Nizamabad</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Kamareddy</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Kamareddy</td>\n",
              "      <td>18.320889</td>\n",
              "      <td>78.337139</td>\n",
              "      <td>2009</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>A Kamareddy dalam (squad) member belonging to ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>101030601.0</td>\n",
              "      <td>Andhra Pradesh</td>\n",
              "      <td>Khammam</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Bhadrachalam</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Bhadrachalam</td>\n",
              "      <td>17.668056</td>\n",
              "      <td>80.896861</td>\n",
              "      <td>2006</td>\n",
              "      <td>...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>Senior CPI-Maoist 'Polit Bureau' and 'central ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3 rows × 70 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5cbff9fa-a5db-4f9d-ad7b-9f8452b0de52')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-5cbff9fa-a5db-4f9d-ad7b-9f8452b0de52 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-5cbff9fa-a5db-4f9d-ad7b-9f8452b0de52');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-9ff4b928-d9f2-4c1b-8070-06ddc4ebb40a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-9ff4b928-d9f2-4c1b-8070-06ddc4ebb40a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-9ff4b928-d9f2-4c1b-8070-06ddc4ebb40a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "data"
            }
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "data = pd.read_csv(\"https://raw.githubusercontent.com/eteitelbaum/code-satp/refs/heads/main/data/satp_clean.csv\")\n",
        "data.head(3)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# convert the names into lowercase for easier comparisions.\n",
        "data[\"state\"] = data[\"state\"].apply(lambda x: x.lower() if isinstance(x, str) else x)\n",
        "data[\"district\"] = data[\"district\"].apply(lambda x: x.lower() if isinstance(x, str) else x)\n",
        "data[\"village_name\"] = data[\"village_name\"].apply(lambda x: x.lower() if isinstance(x, str) else x)\n",
        "data[[\"state\", \"district\", \"village_name\"]].head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "nmkHlipb9IY0",
        "outputId": "81ae7ed5-5e4a-433e-9a89-b92da6dfaef2"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            state        district  village_name\n",
              "0  andhra pradesh       hyderabad           NaN\n",
              "1  andhra pradesh       nizamabad     kamareddy\n",
              "2  andhra pradesh         khammam  bhadrachalam\n",
              "3  andhra pradesh  vishakhapatnam           NaN\n",
              "4  andhra pradesh   visakhapatnam  teegalabanda"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8cfdef45-6516-4041-ba0f-ddfde036a627\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>state</th>\n",
              "      <th>district</th>\n",
              "      <th>village_name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>andhra pradesh</td>\n",
              "      <td>hyderabad</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>andhra pradesh</td>\n",
              "      <td>nizamabad</td>\n",
              "      <td>kamareddy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>andhra pradesh</td>\n",
              "      <td>khammam</td>\n",
              "      <td>bhadrachalam</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>andhra pradesh</td>\n",
              "      <td>vishakhapatnam</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>andhra pradesh</td>\n",
              "      <td>visakhapatnam</td>\n",
              "      <td>teegalabanda</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8cfdef45-6516-4041-ba0f-ddfde036a627')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8cfdef45-6516-4041-ba0f-ddfde036a627 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8cfdef45-6516-4041-ba0f-ddfde036a627');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-cef53625-3e36-4daa-95e0-60cfbbf611b6\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-cef53625-3e36-4daa-95e0-60cfbbf611b6')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-cef53625-3e36-4daa-95e0-60cfbbf611b6 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"data[[\\\"state\\\", \\\"district\\\", \\\"village_name\\\"]]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"state\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"andhra pradesh\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"district\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"nizamabad\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"village_name\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"kamareddy\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Function to scrape States and Districts in India from Wikipedia."
      ],
      "metadata": {
        "id": "az-swp3JzJSu"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qh7KP4xlGt6N",
        "outputId": "f7a608c8-dec0-4ed6-a8f6-37fe8f612509"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "No of states found: 36\n",
            "No of districts found: 793\n"
          ]
        }
      ],
      "source": [
        "def get_states_and_districts():\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "      state and district mapping, a list of states, and a list of districts.\n",
        "\n",
        "      Args:\n",
        "        None\n",
        "      Returns:\n",
        "        state_districts: state and district mapping\n",
        "        states: list of states\n",
        "        districts: list of districts\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "    url = \"https://en.wikipedia.org/wiki/List_of_districts_in_India\"\n",
        "    response = requests.get(url)\n",
        "\n",
        "    if response.status_code != 200:\n",
        "        return \"Error fetching data from Wikipedia\"\n",
        "\n",
        "    soup = BeautifulSoup(response.content, 'html.parser')\n",
        "\n",
        "    # Dictionary to hold state names as keys and their districts as values\n",
        "    states_districts = {}\n",
        "\n",
        "    # Find all divs with the class 'mw-heading mw-heading3' (which contain the state names)\n",
        "    state_names = soup.find_all('div', {'class': 'mw-heading mw-heading3'})\n",
        "    state_names = [state_name.text[:-10].strip().lower() for state_name in state_names[3:-1]]\n",
        "\n",
        "    states = soup.find_all('table', {'class': \"wikitable sortable\"})\n",
        "    districts_list = []\n",
        "    for state in states:\n",
        "      districts_list.append([])\n",
        "      districts = state.find_all('tr')[1:]\n",
        "\n",
        "      for district in districts:\n",
        "        districts_list[-1].append(district.find_all('td')[2].text.strip().lower())\n",
        "\n",
        "    for i, state_name in enumerate(state_names):\n",
        "        states_districts[state_name] = districts_list[i]\n",
        "\n",
        "    all_districts = [d for district in districts_list for d in district]\n",
        "\n",
        "    print(f\"No of states found: {len(state_names)}\")\n",
        "    print(f\"No of districts found: {len(all_districts)}\")\n",
        "\n",
        "    return states_districts, state_names, all_districts\n",
        "\n",
        "\n",
        "# Fetch the states and districts\n",
        "state_district_data, states, districts = get_states_and_districts()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "next(iter(state_district_data)), state_district_data[next(iter(state_district_data))][:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h692eC4DjYMw",
        "outputId": "cadbab4f-8680-4feb-d759-fe0b89a4443f"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('andhra pradesh',\n",
              " ['alluri sitharama raju',\n",
              "  'anakapalli',\n",
              "  'ananthapuramu',\n",
              "  'annamayya',\n",
              "  'bapatla'])"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "states[:5], districts[:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Aju8QspLqYQK",
        "outputId": "f991aba7-d874-408e-851a-c5aff759e594"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(['andhra pradesh', 'arunachal pradesh', 'assam', 'bihar', 'chhattisgarh'],\n",
              " ['alluri sitharama raju',\n",
              "  'anakapalli',\n",
              "  'ananthapuramu',\n",
              "  'annamayya',\n",
              "  'bapatla'])"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Function to scrape villages and their geocodes from GeoNames website"
      ],
      "metadata": {
        "id": "H1XI91XEzWV7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# https://www.geonames.org/advanced-search.html?"
      ],
      "metadata": {
        "id": "TxFk_U0K9JbQ"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "all_villages = {}\n",
        "def get_villages(sd_data):\n",
        "\n",
        "    \"\"\"\n",
        "    Returns a dictionary with hierarchical structure of states, districts, and villages with their geocodes.\n",
        "\n",
        "    Args:\n",
        "      sd_data: state and district mapping\n",
        "    Returns:\n",
        "      all_villages: a dictionary of structure {state:{district{village: {latitude:, longitude:}}}}\n",
        "    \"\"\"\n",
        "\n",
        "    for state, districts in sd_data.items():\n",
        "        all_villages[state] = {}\n",
        "        for district in tqdm(districts):\n",
        "            all_villages[state][district] = {}\n",
        "            base_url = \"https://www.geonames.org/search.html\"\n",
        "            start_row = 0\n",
        "\n",
        "            while True:\n",
        "                # Step 1: Prepare the request URL\n",
        "                params = {\n",
        "                    'q': district,\n",
        "                    'country': 'IN',\n",
        "                    'continentCode': 'AS',\n",
        "                    'startRow': start_row\n",
        "                }\n",
        "\n",
        "                # Step 2: Send the HTTP request to fetch the page content\n",
        "                response = requests.get(base_url, params=params)\n",
        "\n",
        "                if response.status_code != 200:\n",
        "                    print(f\"Failed to retrieve page for : {district}\")\n",
        "                    break\n",
        "\n",
        "                # Step 3: Parse the page content\n",
        "                soup = BeautifulSoup(response.content, 'html.parser')\n",
        "\n",
        "                # Step 4: Find the table with village names and coordinates\n",
        "                table = soup.find('table', class_='restable')\n",
        "                if not table:\n",
        "                    break  # Exit when there are no more tables (i.e., no more results)\n",
        "\n",
        "                # Step 5: Loop through each row and extract village name, latitude, and longitude\n",
        "                rows = table.find_all('tr')[2:-1]  # Skip the header row\n",
        "                for row in rows:\n",
        "                    try:\n",
        "                      cols = row.find_all('td')\n",
        "\n",
        "                      # Extract village name from the first column's href attribute\n",
        "                      village_name = cols[1].find('a')\n",
        "\n",
        "                      latitude = cols[4].get_text(strip=True)\n",
        "                      longitude = cols[5].get_text(strip=True)\n",
        "\n",
        "                      if village_name is not None and village_name not in states and village_name not in districts:\n",
        "                          village_name = unidecode.unidecode(village_name.text.strip().lower())\n",
        "                          all_villages[state][district][village_name] = {'Latitude': latitude, 'Longitude': longitude}\n",
        "                          village_count += 1\n",
        "                    except Exception as e:\n",
        "                      continue\n",
        "\n",
        "                # Step 6: Move to the next page (next 50 results)\n",
        "                start_row += 50\n",
        "        with open('/content/drive/MyDrive/SATP/location_data.json', 'w') as json_file:\n",
        "              json.dump(all_villages, json_file)\n",
        "    return all_villages\n",
        "\n",
        "#location_data = get_villages(state_district_data) # Takes around 30 min to get all the data.\n",
        "#No need to run, since this is already saved and can be accessed from the data folder."
      ],
      "metadata": {
        "collapsed": true,
        "id": "vOVyxm-_f82N"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://raw.githubusercontent.com/eteitelbaum/code-satp/refs/heads/main/data/location_data.json"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KchkRjhF-WJz",
        "outputId": "9044fa2e-0283-424f-8ce6-9b099d3ef919"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-10-05 04:39:50--  https://raw.githubusercontent.com/eteitelbaum/code-satp/refs/heads/main/data/location_data.json\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 51732669 (49M) [text/plain]\n",
            "Saving to: ‘location_data.json’\n",
            "\n",
            "location_data.json  100%[===================>]  49.34M   194MB/s    in 0.3s    \n",
            "\n",
            "2024-10-05 04:39:50 (194 MB/s) - ‘location_data.json’ saved [51732669/51732669]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "with open('location_data.json', 'r') as file:\n",
        "    location_data = json.load(file)"
      ],
      "metadata": {
        "id": "utLAfVC8-ZRO"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "villages = []\n",
        "for state in location_data.keys():\n",
        "  for district in location_data[state].keys():\n",
        "    for village in location_data[state][district].keys():\n",
        "      villages.append(village)\n",
        "\n",
        "print(len(villages), villages[:5])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zJ9OgeIYx6WF",
        "outputId": "2a343ff4-dd31-4b83-836c-99318795db66"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "539142 ['alluri sitharama raju', 'rampachodavaram', 'paderu', 'koyyuru', 'hukumpeta']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(districts), districts[:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h2O-D_2gz7tq",
        "outputId": "42e136f9-ab3e-4b98-c673-36c5bf9a958c"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(793,\n",
              " ['alluri sitharama raju',\n",
              "  'anakapalli',\n",
              "  'ananthapuramu',\n",
              "  'annamayya',\n",
              "  'bapatla'])"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dqET3ewA-AeR",
        "outputId": "795048f8-d840-40d8-c36e-0a22576a73f4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "953 ['alluri', 'sitharama', 'raju', 'anakapalli', 'ananthapuramu']\n"
          ]
        }
      ],
      "source": [
        "district_words = [dws for d in districts for dws in d.split()]\n",
        "print(len(district_words), district_words[:5])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U1BMxu_N-0zg",
        "outputId": "63f29abd-1a1a-4089-d896-714a30a8c2e6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "938\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['north garo hills', 'gramin', 'kodagu', 'purba medinipur', 'katni']"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "district_words = set(district_words + [d for d in districts])\n",
        "print(len(district_words))\n",
        "list(district_words)[:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Naive approach to extract district from the input summary."
      ],
      "metadata": {
        "id": "5pBxi_xV08Jm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_district(x, district_words):\n",
        "\n",
        "  \"\"\"\n",
        "\n",
        "    Given a news summary, returns the district found in the input.\n",
        "\n",
        "    Args:\n",
        "        x: news summary,\n",
        "        district_words: a set of all words in districts and districts themselves.\n",
        "\n",
        "    Returns:\n",
        "        district name found in the input.\n",
        "\n",
        "  \"\"\"\n",
        "\n",
        "  x = x.lower()\n",
        "  if \" district\" in x: # check if the summary has district keyword in it.\n",
        "    pre_words = x.split(' district')[0].strip().split() # split at that index\n",
        "    res = ''\n",
        "\n",
        "    for i in range(len(pre_words)-1, -1, -1): # loop back from the district keyword\n",
        "      if (pre_words[i] in district_words) or (pre_words[i].replace('-', ' ') in district_words): # check if the words preceding are present in the scraped data.\n",
        "        res = pre_words[i] + ' ' + res\n",
        "      else:\n",
        "        break # break if there is a keyword that's not present in the district_words set.\n",
        "\n",
        "    res = res.strip().replace('-', ' ')\n",
        "    if res in districts:\n",
        "      return res\n",
        "\n",
        "  # additional process for explicit matching when the above method fail or district keyword is not present.\n",
        "  for d in districts: #loop through all the districts and check if there is match.\n",
        "    if d in x:\n",
        "      return d\n",
        "\n",
        "  return \"\""
      ],
      "metadata": {
        "id": "fLjDeXbJ3y2p"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QpsjDPEx7UO8",
        "outputId": "102e5499-3c50-4f40-d99c-c2a4a3ab1fe1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ITEM: 8208\n",
            "ACTUAl: malkangiri\n",
            "Extracted: malkangiri\n",
            "Summary: CPI-Maoist cadres abducted five villagers, including one former Maoist, who was about to surrender before the Police from Potteru village under Kalimela block of Motu Police limits in Malkangiri District.\n"
          ]
        }
      ],
      "source": [
        "# run the function on random samples\n",
        "item = random.randint(1, 9000)\n",
        "print(\"ITEM:\", item)\n",
        "print(\"ACTUAl:\", data[\"district\"][item])\n",
        "print(\"Extracted:\", get_district(data[\"incident_summary\"][item], district_words))\n",
        "print(\"Summary:\", data[\"incident_summary\"][item])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "xjUVg6gNHI0z",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "c2ccadf8-2089-40a0-f1f6-a958b4519c8f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    hyderabad\n",
              "1    nizamabad\n",
              "2      khammam\n",
              "3        patna\n",
              "4          una\n",
              "Name: extracted_district, dtype: object"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>extracted_district</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>hyderabad</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>nizamabad</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>khammam</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>patna</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>una</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div><br><label><b>dtype:</b> object</label>"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ],
      "source": [
        "data[\"extracted_district\"] = data[\"incident_summary\"].apply(lambda x: get_district(x, district_words))\n",
        "data[\"extracted_district\"][:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Evaluation"
      ],
      "metadata": {
        "id": "9Zy3vGYI572u"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LsKyxj4JIGRy",
        "outputId": "9e992369-c570-4112-b22a-fe88a32a0df3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "District Extraction Score: 0.7934684003628667\n"
          ]
        }
      ],
      "source": [
        "print(\"District Extraction Score:\", sum(data[\"district\"] == data[\"extracted_district\"]) / data.shape[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Ideas: add fuzzy matching, preprocessing functions."
      ],
      "metadata": {
        "id": "cjfvkbxV9uGR"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}