4-regex.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "4-regex.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yIeCBLNOE-ya",
        "colab_type": "text"
      },
      "source": [
        "# Regex"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c91ZsNlWE-yc",
        "colab_type": "text"
      },
      "source": [
        "In this lesson, we'll learn about a useful tool in the NLP toolkit: regex.\n",
        "\n",
        "Let's consider two motivating examples:\n",
        "\n",
        "#### 1. The phone number problem\n",
        "\n",
        "Suppose we are given some data that includes phone numbers:\n",
        "\n",
        "123-456-7890\n",
        "\n",
        "123 456 7890\n",
        "\n",
        "101 Howard\n",
        "\n",
        "Some of the phone numbers have different formats (hyphens, no hyphens).  Also, there are some errors in the data-- 101 Howard isn't a phon number!  How can we find all the phone numbers?\n",
        "\n",
        "#### 2. Creating our own tokens\n",
        "\n",
        "In the previous lessons, we used sklearn or fastai to tokenize our text.  What if we want to do it ourselves?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_qLC0aKE-yd",
        "colab_type": "text"
      },
      "source": [
        "## The phone number problem"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "367fgYXOE-yd",
        "colab_type": "text"
      },
      "source": [
        "Suppose we are given some data that includes phone numbers:\n",
        "\n",
        "123-456-7890\n",
        "\n",
        "123 456 7890\n",
        "\n",
        "(123)456-7890\n",
        "\n",
        "101 Howard\n",
        "\n",
        "Some of the phone numbers have different formats (hyphens, no hyphens, parentheses).  Also, there are some errors in the data-- 101 Howard isn't a phone number!  How can we find all the phone numbers?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7QlQBfMWE-ye",
        "colab_type": "text"
      },
      "source": [
        "We will attempt this without regex, but will see that this quickly leads to lot of if/else branching statements and isn't a veyr promising approach:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PFzOiXKgE-yf",
        "colab_type": "text"
      },
      "source": [
        "### Attempt 1 (without regex)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M_3zzvlsE-yf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "phone1 = \"123-456-7890\"\n",
        "\n",
        "phone2 = \"123 456 7890\"\n",
        "\n",
        "not_phone1 = \"101 Howard\""
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kFt2eq-ZE-yk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9c49b6dd-7da9-432b-8230-36546fdd47a4"
      },
      "source": [
        "import string\n",
        "string.digits"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'0123456789'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WjgJJC6JE-yo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def check_phone(inp):\n",
        "    valid_chars = string.digits + ' -()'\n",
        "    for char in inp:\n",
        "        if char not in valid_chars:\n",
        "            return False\n",
        "    return True"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T5b2R1E5E-yq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "assert(check_phone(phone1))\n",
        "assert(check_phone(phone2))\n",
        "assert(not check_phone(not_phone1))"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "15QFgFtFE-yt",
        "colab_type": "text"
      },
      "source": [
        "### Attempt 2  (without regex)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ciYwIF9EE-yu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "not_phone2 = \"1234\""
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G5aWJ9Q4E-yw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 164
        },
        "outputId": "792941b6-f471-4301-9219-3f4e2aa19432"
      },
      "source": [
        "assert(not check_phone(not_phone2))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AssertionError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-7-325c150ba276>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32massert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mnot\u001b[0m \u001b[0mcheck_phone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnot_phone2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p3_My_49E-yz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def check_phone(inp):\n",
        "    nums = string.digits\n",
        "    valid_chars = nums + ' -()'\n",
        "    num_counter = 0\n",
        "    for char in inp:\n",
        "        if char not in valid_chars:\n",
        "            return False\n",
        "        if char in nums:\n",
        "            num_counter += 1\n",
        "    if num_counter==10:\n",
        "        return True\n",
        "    else:\n",
        "        return False"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uq942Gc5E-y3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "assert(check_phone(phone1))\n",
        "assert(check_phone(phone2))\n",
        "assert(not check_phone(not_phone1))\n",
        "assert(not check_phone(not_phone2))"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HL0-RUb5E-y5",
        "colab_type": "text"
      },
      "source": [
        "### Attempt 3  (without regex)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-m4qu4QHE-y5",
        "colab_type": "text"
      },
      "source": [
        "But we also need to extract the digits!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l4Pwa1_IE-y6",
        "colab_type": "text"
      },
      "source": [
        "Also, what about:\n",
        "\n",
        "34!NA5098gn#213ee2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LRyESZ4xE-y7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 198
        },
        "outputId": "d64eb8e6-c5a2-4723-89ea-884d5555e385"
      },
      "source": [
        "not_phone3 = \"34 50 98 21 32\"\n",
        "\n",
        "assert(not check_phone(not_phone3))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AssertionError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-10-63ad4ac6a61d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mnot_phone3\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"34 50 98 21 32\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32massert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mnot\u001b[0m \u001b[0mcheck_phone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnot_phone3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vrNN09V-E-y-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 198
        },
        "outputId": "d1fb0b21-6ca5-4efa-9b7e-9cb8b15f291f"
      },
      "source": [
        "not_phone4 = \"(34)(50)()()982132\"\n",
        "\n",
        "assert(not check_phone(not_phone3))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AssertionError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-11-081f5987a5f8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mnot_phone4\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"(34)(50)()()982132\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32massert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mnot\u001b[0m \u001b[0mcheck_phone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnot_phone3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KFFO5IZLE-zB",
        "colab_type": "text"
      },
      "source": [
        "This is getting increasingly unwieldy.  We need a different approach."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P1vOAE2xE-zC",
        "colab_type": "text"
      },
      "source": [
        "## Introducing regex"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0XD-_pEoE-zC",
        "colab_type": "text"
      },
      "source": [
        "Useful regex resources:\n",
        "\n",
        "- https://regexr.com/\n",
        "- http://callumacrae.github.io/regex-tuesday/\n",
        "- https://regexone.com/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWgjmIvME-zC",
        "colab_type": "text"
      },
      "source": [
        "**Best practice: Be as specific as possible.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Z3qZ9iJE-zD",
        "colab_type": "text"
      },
      "source": [
        "Parts of the following section were adapted from Brian Spiering, who taught the MSDS [NLP elective last summer](https://github.com/brianspiering/nlp-course)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "bjJArmhlE-zF",
        "colab_type": "text"
      },
      "source": [
        "### What is regex?\n",
        "\n",
        "Regular expressions is a pattern matching language. \n",
        "\n",
        "Instead of writing `0 1 2 3 4 5 6 7 8 9`, you can write `[0-9]` or `\\d`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "3ZFxo7ZBE-zG",
        "colab_type": "text"
      },
      "source": [
        "It is Domain Specific Language (DSL). Powerful (but limited language). \n",
        "\n",
        "**What other DSLs do you already know?**\n",
        "- SQL  \n",
        "- Markdown\n",
        "- TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DIsje6qxE-zG",
        "colab_type": "text"
      },
      "source": [
        "### Matching Phone Numbers (The \"Hello, world!\" of Regex)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NRSKHvCkE-zH",
        "colab_type": "text"
      },
      "source": [
        "`[0-9][0-9][0-9]-[0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]` matches US telephone number."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9oHHv1JyE-zH",
        "colab_type": "text"
      },
      "source": [
        "Refactored: `\\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d`\n",
        "\n",
        "A **metacharacter** is one or more special characters that have a unique meaning and are NOT used as literals in the search expression. For example \"\\d\" means any digit.\n",
        "\n",
        "**Metacharacters are the special sauce of regex.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IcaQyd-tE-zI",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "Quantifiers\n",
        "-----\n",
        "\n",
        "Allow you to specify how many times the preceding expression should match. \n",
        "\n",
        "`{}` is an extact qualifer\n",
        "\n",
        "Refactored: `\\d{3}-\\d{3}-\\d{4}`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DEfHhYjgE-zI",
        "colab_type": "text"
      },
      "source": [
        "Unexact quantifiers\n",
        "-----\n",
        "\n",
        "1. `?` question mark - zero or one \n",
        "2. `*` star - zero or more\n",
        "3. `+` plus sign - one or more | "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WUr7_AMNE-zI",
        "colab_type": "text"
      },
      "source": [
        "### Regex can look really weird, since it's so concise"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xlC-JE9dE-zJ",
        "colab_type": "text"
      },
      "source": [
        "The best (only?) way to learn it is through practice.  Otherwise, you feel like you're just reading lists of rules.\n",
        "\n",
        "Let's take 15 minutes to begin working through the lessons on [regexone](https://regexone.com/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NEBOql50E-zJ",
        "colab_type": "text"
      },
      "source": [
        "**Reminder: Be as specific as possible!**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CvSrYIutE-zK",
        "colab_type": "text"
      },
      "source": [
        "### Pros & Cons of Regex"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vJjNUVhcE-zK",
        "colab_type": "text"
      },
      "source": [
        "**What are the advantages of regex?**\n",
        "\n",
        "1. Concise and powerful pattern matching DSL\n",
        "2. Supported by many computer languages, including SQL"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eIaZGZIqE-zK",
        "colab_type": "text"
      },
      "source": [
        "**What are the disadvantages of regex?**\n",
        "\n",
        "1. Brittle \n",
        "2. Hard to write, can get complex to be correct\n",
        "3. Hard to read"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dgm_RSZjE-zL",
        "colab_type": "text"
      },
      "source": [
        "## Revisiting tokenization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJaolzyiE-zM",
        "colab_type": "text"
      },
      "source": [
        "In the previous lessons, we used a tokenizer.  Now, let's learn how we could do this ourselves, and get a better understanding of tokenization."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_lejrkVYE-zM",
        "colab_type": "text"
      },
      "source": [
        "What if we needed to create our own tokens?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bfjg7xK4E-zN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N73kz4rYE-zQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "re_punc = re.compile(\"([\\\"\\''().,;:/_?!—\\-])\") # add spaces around punctuation\n",
        "re_apos = re.compile(r\"n ' t \")    # n't\n",
        "re_bpos = re.compile(r\" ' s \")     # 's\n",
        "re_mult_space = re.compile(r\"  *\") # replace multiple spaces with just one\n",
        "\n",
        "def simple_toks(sent):\n",
        "    sent = re_punc.sub(r\" \\1 \", sent)\n",
        "    sent = re_apos.sub(r\" n't \", sent)\n",
        "    sent = re_bpos.sub(r\" 's \", sent)\n",
        "    sent = re_mult_space.sub(' ', sent)\n",
        "    return sent.lower().split()"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JAc1cGVHE-zV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "text = \"I don't know who Kara's new friend is-- is it 'Mr. Toad'?\""
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GsA_dPRlE-zY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f06c3cf0-13be-4847-bbb5-215faf70e378"
      },
      "source": [
        "' '.join(simple_toks(text))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"i do n't know who kara 's new friend is - - is it ' mr . toad ' ?\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xcuUl9FQE-za",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "40f2786e-91ea-4d0d-8796-de650a807bfd"
      },
      "source": [
        "text2 = re_punc.sub(r\" \\1 \", text); text2"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"I don ' t know who Kara ' s new friend is -  -  is it  ' Mr .  Toad '  ? \""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LUmm3nLhE-zc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9c1b9770-dafd-482e-9bc3-8fc6e05ac8a7"
      },
      "source": [
        "text3 = re_apos.sub(r\" n't \", text2); text3"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"I do n't know who Kara ' s new friend is -  -  is it  ' Mr .  Toad '  ? \""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V4hDjpHmE-ze",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e91f7131-7a87-4c91-fbac-b5b083f53f21"
      },
      "source": [
        "text4 = re_bpos.sub(r\" 's \", text3); text4"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"I do n't know who Kara 's new friend is -  -  is it  ' Mr .  Toad '  ? \""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q4Sx4WunE-zg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "23b5d835-b05c-44c0-faee-fe56d628d930"
      },
      "source": [
        "re_mult_space.sub(' ', text4)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"I do n't know who Kara 's new friend is - - is it ' Mr . Toad ' ? \""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bdYvj0txE-zj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sentences = ['All this happened, more or less.',\n",
        "             'The war parts, anyway, are pretty much true.',\n",
        "             \"One guy I knew really was shot for taking a teapot that wasn't his.\",\n",
        "             'Another guy I knew really did threaten to have his personal enemies killed by hired gunmen after the war.',\n",
        "             'And so on.',\n",
        "             \"I've changed all their names.\"]"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PbnHmn3RE-zq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tokens = list(map(simple_toks, sentences))"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WEOt6-UFE-zs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 867
        },
        "outputId": "bd4f38c6-1b54-40cb-a154-756c70e76e66"
      },
      "source": [
        "tokens"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['all', 'this', 'happened', ',', 'more', 'or', 'less', '.'],\n",
              " ['the',\n",
              "  'war',\n",
              "  'parts',\n",
              "  ',',\n",
              "  'anyway',\n",
              "  ',',\n",
              "  'are',\n",
              "  'pretty',\n",
              "  'much',\n",
              "  'true',\n",
              "  '.'],\n",
              " ['one',\n",
              "  'guy',\n",
              "  'i',\n",
              "  'knew',\n",
              "  'really',\n",
              "  'was',\n",
              "  'shot',\n",
              "  'for',\n",
              "  'taking',\n",
              "  'a',\n",
              "  'teapot',\n",
              "  'that',\n",
              "  'was',\n",
              "  \"n't\",\n",
              "  'his',\n",
              "  '.'],\n",
              " ['another',\n",
              "  'guy',\n",
              "  'i',\n",
              "  'knew',\n",
              "  'really',\n",
              "  'did',\n",
              "  'threaten',\n",
              "  'to',\n",
              "  'have',\n",
              "  'his',\n",
              "  'personal',\n",
              "  'enemies',\n",
              "  'killed',\n",
              "  'by',\n",
              "  'hired',\n",
              "  'gunmen',\n",
              "  'after',\n",
              "  'the',\n",
              "  'war',\n",
              "  '.'],\n",
              " ['and', 'so', 'on', '.'],\n",
              " ['i', \"'\", 've', 'changed', 'all', 'their', 'names', '.']]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UHRojdK8E-zv",
        "colab_type": "text"
      },
      "source": [
        "Once we have our tokens, we need to convert them to integer ids.  We will also need to know our vocabulary, and have a way to convert between words and ids."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VVvmMeY0E-zv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import collections"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "__I1aRHTE-zy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "PAD = 0; SOS = 1\n",
        "\n",
        "def toks2ids(sentences):\n",
        "    voc_cnt = collections.Counter(t for sent in sentences for t in sent)\n",
        "    vocab = sorted(voc_cnt, key=voc_cnt.get, reverse=True)\n",
        "    vocab.insert(PAD, \"<PAD>\")\n",
        "    vocab.insert(SOS, \"<SOS>\")\n",
        "    w2id = {w:i for i,w in enumerate(vocab)}\n",
        "    ids = [[w2id[t] for t in sent] for sent in sentences]\n",
        "    return ids, vocab, w2id, voc_cnt"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tb7oyRnpE-z0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ids, vocab, w2id, voc_cnt = toks2ids(tokens)"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oo17gW1cE-z3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "c9bfa24f-6608-448b-8fe5-482fff27c9e6"
      },
      "source": [
        "ids"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[5, 13, 14, 3, 15, 16, 17, 2],\n",
              " [6, 7, 18, 3, 19, 3, 20, 21, 22, 23, 2],\n",
              " [24, 8, 4, 9, 10, 11, 25, 26, 27, 28, 29, 30, 11, 31, 12, 2],\n",
              " [32, 8, 4, 9, 10, 33, 34, 35, 36, 12, 37, 38, 39, 40, 41, 42, 43, 6, 7, 2],\n",
              " [44, 45, 46, 2],\n",
              " [4, 47, 48, 49, 5, 50, 51, 2]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aJSbkVaYE-z6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 901
        },
        "outputId": "39ec0b83-0175-406c-a981-6f9d840e689f"
      },
      "source": [
        "vocab"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<PAD>',\n",
              " '<SOS>',\n",
              " '.',\n",
              " ',',\n",
              " 'i',\n",
              " 'all',\n",
              " 'the',\n",
              " 'war',\n",
              " 'guy',\n",
              " 'knew',\n",
              " 'really',\n",
              " 'was',\n",
              " 'his',\n",
              " 'this',\n",
              " 'happened',\n",
              " 'more',\n",
              " 'or',\n",
              " 'less',\n",
              " 'parts',\n",
              " 'anyway',\n",
              " 'are',\n",
              " 'pretty',\n",
              " 'much',\n",
              " 'true',\n",
              " 'one',\n",
              " 'shot',\n",
              " 'for',\n",
              " 'taking',\n",
              " 'a',\n",
              " 'teapot',\n",
              " 'that',\n",
              " \"n't\",\n",
              " 'another',\n",
              " 'did',\n",
              " 'threaten',\n",
              " 'to',\n",
              " 'have',\n",
              " 'personal',\n",
              " 'enemies',\n",
              " 'killed',\n",
              " 'by',\n",
              " 'hired',\n",
              " 'gunmen',\n",
              " 'after',\n",
              " 'and',\n",
              " 'so',\n",
              " 'on',\n",
              " \"'\",\n",
              " 've',\n",
              " 'changed',\n",
              " 'their',\n",
              " 'names']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dwivly95E-z8",
        "colab_type": "text"
      },
      "source": [
        "Q: what could be another name of the `vocab` variable above?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "03QLDYMaE-z9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 901
        },
        "outputId": "7f7a6e6b-0d85-48a5-8f96-d449098a219e"
      },
      "source": [
        "w2id"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{\"'\": 47,\n",
              " ',': 3,\n",
              " '.': 2,\n",
              " '<PAD>': 0,\n",
              " '<SOS>': 1,\n",
              " 'a': 28,\n",
              " 'after': 43,\n",
              " 'all': 5,\n",
              " 'and': 44,\n",
              " 'another': 32,\n",
              " 'anyway': 19,\n",
              " 'are': 20,\n",
              " 'by': 40,\n",
              " 'changed': 49,\n",
              " 'did': 33,\n",
              " 'enemies': 38,\n",
              " 'for': 26,\n",
              " 'gunmen': 42,\n",
              " 'guy': 8,\n",
              " 'happened': 14,\n",
              " 'have': 36,\n",
              " 'hired': 41,\n",
              " 'his': 12,\n",
              " 'i': 4,\n",
              " 'killed': 39,\n",
              " 'knew': 9,\n",
              " 'less': 17,\n",
              " 'more': 15,\n",
              " 'much': 22,\n",
              " \"n't\": 31,\n",
              " 'names': 51,\n",
              " 'on': 46,\n",
              " 'one': 24,\n",
              " 'or': 16,\n",
              " 'parts': 18,\n",
              " 'personal': 37,\n",
              " 'pretty': 21,\n",
              " 'really': 10,\n",
              " 'shot': 25,\n",
              " 'so': 45,\n",
              " 'taking': 27,\n",
              " 'teapot': 29,\n",
              " 'that': 30,\n",
              " 'the': 6,\n",
              " 'their': 50,\n",
              " 'this': 13,\n",
              " 'threaten': 34,\n",
              " 'to': 35,\n",
              " 'true': 23,\n",
              " 've': 48,\n",
              " 'war': 7,\n",
              " 'was': 11}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uu9GcluCE-z_",
        "colab_type": "text"
      },
      "source": [
        "What are the uses of RegEx?\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hz9fDomqE-z_",
        "colab_type": "text"
      },
      "source": [
        "1. Find / Search\n",
        "1. Find & Replace\n",
        "2. Cleaning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2F6w31eE-z_",
        "colab_type": "text"
      },
      "source": [
        "Don't forgot about Python's `str` methods\n",
        "-----\n",
        "\n",
        "`str.<tab>`\n",
        "    \n",
        "str.find()"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HgBdtSZLE-z_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "str.find?"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ABPc3Ex3E-0B",
        "colab_type": "text"
      },
      "source": [
        "Regex vs. String methods\n",
        "-----\n",
        "\n",
        "1. String methods are easier to understand.\n",
        "1. String methods express the intent more clearly. \n",
        "\n",
        "-----\n",
        "\n",
        "1. Regex handle much broader use cases.\n",
        "1. Regex can be language independent.\n",
        "1. Regex can be faster at scale."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O7RIf9TnE-0B",
        "colab_type": "text"
      },
      "source": [
        "## What about unicode?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iFwp_kgGE-0C",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "952cb3f9-11e7-4411-bf4a-3bb5ba3a32bf"
      },
      "source": [
        "message = \"😒🎦 🤢🍕\"\n",
        "\n",
        "re_frown = re.compile(r\"😒|🤢\")\n",
        "re_frown.sub(r\"😊\", message)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'😊🎦 😊🍕'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6V6oLE_CE-0G",
        "colab_type": "text"
      },
      "source": [
        "## Regex Errors:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6F9lUC3TE-0G",
        "colab_type": "text"
      },
      "source": [
        "__False positives__ (Type I): Matching strings that we should __not__ have\n",
        "matched\n",
        "\n",
        "__False negatives__ (Type II): __Not__ matching strings that we should have matched"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vi7PRuD4E-0H",
        "colab_type": "text"
      },
      "source": [
        "Reducing the error rate for a task often involves two antagonistic efforts:\n",
        "\n",
        "1. Minimizing false positives\n",
        "2. Minimizing false negatives\n",
        "\n",
        "**Important to have tests for both!**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ZOGuw_sE-0H",
        "colab_type": "text"
      },
      "source": [
        "In a perfect world, you would be able to minimize both but in reality you often have to trade one for the other."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "at-TYZ-HE-0H",
        "colab_type": "text"
      },
      "source": [
        "Useful Tools:\n",
        "----\n",
        "- [Regex cheatsheet](http://www.cheatography.com/davechild/cheat-sheets/regular-expressions/)\n",
        "- [regexr.com](http://regexr.com/) Realtime regex engine\n",
        "- [pyregex.com](https://pythex.org/) Realtime Python regex engine"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dY5LyUQrE-0I",
        "colab_type": "text"
      },
      "source": [
        "Summary\n",
        "----\n",
        "\n",
        "1. We use regex as a metalanguage to find string patterns in blocks of text\n",
        "1. `r\"\"` are your IRL friends for Python regex\n",
        "1. We are just doing binary classification so use the same performance metrics\n",
        "1. You'll make a lot of mistakes in regex 😩. \n",
        "    - False Positive: Thinking you are right but you are wrong\n",
        "    - False Negative: Missing something"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9IU475GiE-0I",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"https://github.com/rantanplan77/course-nlp/blob/master/images/face_tat.png?raw=1\" width=\"700\"/></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bt2EcsDOE-0I",
        "colab_type": "text"
      },
      "source": [
        "<br>\n",
        "<br>\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ACjbRABDE-0I",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"https://imgs.xkcd.com/comics/perl_problems.png\" width=\"700\"/></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jMWcGwZSE-0J",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"https://imgs.xkcd.com/comics/regex_golf.png\" width=\"700\"/></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMifzhhiE-0J",
        "colab_type": "text"
      },
      "source": [
        "Regex Terms\n",
        "----\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0NIODCjE-0J",
        "colab_type": "text"
      },
      "source": [
        "- __target string__:\tThis term describes the string that we will be searching, that is, the string in which we want to find our match or search pattern.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3V07bLoJE-0K",
        "colab_type": "text"
      },
      "source": [
        "- __search expression__: The pattern we use to find what we want. Most commonly called the regular expression. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "atCwYRS_E-0K",
        "colab_type": "text"
      },
      "source": [
        "- __literal__:\tA literal is any character we use in a search or matching expression, for example, to find 'ind' in 'windows' the 'ind' is a literal string - each character plays a part in the search, it is literally the string we want to find."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d9uG_PTdE-0K",
        "colab_type": "text"
      },
      "source": [
        "- __metacharacter__: A metacharacter is one or more special characters that have a unique meaning and are NOT used as literals in the search expression. For example \".\" means any character.\n",
        "\n",
        "Metacharacters are the special sauce of regex."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DVDdI7bhE-0L",
        "colab_type": "text"
      },
      "source": [
        "- __escape sequence__:\tAn escape sequence is a way of indicating that we want to use a metacharacters as a literal. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e7KEkYfUE-0L",
        "colab_type": "text"
      },
      "source": [
        "In a regular expression an escape sequence involves placing the metacharacter \\ (backslash) in front of the metacharacter that we want to use as a literal. \n",
        "\n",
        "`'\\.'` means find literal period character (not match any character)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DV6ujCtXE-0L",
        "colab_type": "text"
      },
      "source": [
        "Regex Workflow\n",
        "---\n",
        "1. Create pattern in Plain English\n",
        "2. Map to regex language\n",
        "3. Make sure results are correct:\n",
        "    - All Positives: Captures all examples of pattern\n",
        "    - No Negatives: Everything captured is from the pattern\n",
        "4. Don't over-engineer your regex. \n",
        "    - Your goal is to Get Stuff Done, not write the best regex in the world\n",
        "    - Filtering before and after are okay."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3CSvDjVVF199",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 30,
      "outputs": []
    }
  ]
}