Vital Extraction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **Cloudphysician's The Vital Extraction Challenge** "
      ],
      "metadata": {
        "id": "RkALnbLNHeI7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Problem Statement:**\n",
        "Patient monitoring is crucial in healthcare, as it allows healthcare professionals to closely track a patient's vital signs and detect any potential issues before they become serious. In particular, monitoring a patient's vitals, such as heart rate, blood pressure, and oxygen levels, can provide valuable information about a patient's overall health and well-being. The core problem statement is to extract *Heart Rate, SpO2, RR, Systolic Blood Pressure, Diabolic Blood Pressure, and MAP* from images of ECG Machines.\n"
      ],
      "metadata": {
        "id": "AfF5vs4rHkfo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Proposed Pipeline:**\n",
        "The proposed pipeline consists of a total of four stages. The whole pipeline is made such that it can work in two modes: **'Fast'** and **'Accurate**'. \n",
        "**By default, the pipeline is set to 'Accurate' mode**. \n",
        "\n",
        "The 'Fast' mode ensures that the whole pipeline runs under 2 seconds on CPU, while the 'Accurate’ mode ensures maximum accuracy at each stage, thus ensuring the best overall performance. \n",
        "\n",
        "\n",
        " The four stages of the pipeline are as follows:\n"
      ],
      "metadata": {
        "id": "B-ZMOPZ7HnTN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[![url](https://i.postimg.cc/k58kF1Bw/Untitled-Diagram-drawio-1.png)"
      ],
      "metadata": {
        "id": "ofDIXWxIHqNu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "1. Screen Extraction\n",
        "2. Number Detection\n",
        "3. OCR\n",
        "4. Number Classification\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "CFTEigZNHtGj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Downloading Libraries"
      ],
      "metadata": {
        "id": "CknZtzg6wr6q"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "oziZrSNyHxT3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "o_qcfB4FUb7K"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install yolov5\n",
        "!pip install paddlepaddle -qq\n",
        "!pip install paddleocr -qq\n",
        "!pip install ensemble-boxes -qq\n",
        "!pip install timm\n",
        "!pip install torchvision\n",
        "!pip install pytorch_lightning\n",
        "!pip install imutils\n",
        "\n",
        "## Kindly Restart your Runtime after this cell, to execute them on your system"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Importing Libraries"
      ],
      "metadata": {
        "id": "36hxbgA6w4L6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install segmentation-models-pytorch -qq\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "from PIL import Image\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import cv2\n",
        "\n",
        "import torch\n",
        "from torch import nn, optim\n",
        "import torchvision.transforms as T\n",
        "from torchvision.utils import make_grid\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "\n",
        "import timm\n",
        "import segmentation_models_pytorch as smp\n",
        "import imutils\n",
        "from skimage.transform import ProjectiveTransform\n",
        "\n",
        "import os\n",
        "from tqdm import tqdm\n",
        "from PIL import Image\n",
        "import albumentations as A\n",
        "from sklearn.model_selection import train_test_split\n",
        "import gc\n",
        "import glob\n",
        "\n",
        "import random\n",
        "import yolov5\n",
        "import paddleocr\n",
        "from paddleocr import PaddleOCR,draw_ocr\n",
        "from ensemble_boxes import *\n",
        "import re\n",
        "import torch.nn.functional as F\n",
        "import copy\n",
        "%matplotlib inline"
      ],
      "metadata": {
        "id": "sHLD-7-Mw53i"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
      ],
      "metadata": {
        "id": "Zl7iCnpoxVMT"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vT1oFNh90rlv",
        "outputId": "b17320df-4651-4a12-ac98-fef0a0b71e3e"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Change root to your location\n",
        "root = '/content/drive/MyDrive/InterIIT11MP/'"
      ],
      "metadata": {
        "id": "99SslfSR1BNW"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Screen Extraction"
      ],
      "metadata": {
        "id": "hwLy0LxxxRfk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "The first stage of the pipeline is the extraction of monitor screens from the given images. We have approached the problem in two ways, which are as follows:\n",
        "-  #### **Corner Regression**: \n",
        "\n",
        "    The Corner Regression approach uses a CNN encoder to regress on the values of the bounding boxes of the desired screen, thereby producing an output vector of length eight (x1,y1,x2,y2,x3,y3,x4,y4 - coordinates of the bounding box). The followings are the details of the training parameters and augmentations for the CNN:\n",
        "\n",
        "   *  Model: **Fine-Tuned Resnet 34**\n",
        " * Augmentations: \n",
        "\n",
        "    We used a few custom augmentations in our pipeline because the standard libraries (e.g., Albumentations) do not provide augmentations on the random 4-sided polygon and are specifically tuned to rectangles. These custom augmentations improved the performance of our model and made it more robust.\n",
        "\n",
        "    **Custom Horizontal Flip**: Our custom augmentation performs a horizontal flip on the image and accordingly changes the bounding box coordinates, even in non-rectangular cases for the newly flipped image.\n",
        "\n",
        "    **Custom Random Crop**: Our custom implementation performs random crops to ensure that the full area of the screen is always visible in the augmented image. This random crop is implemented to mimic the scenarios where the model generally performs poorly (i.e., the original model performed poorly in images where the monitor was too close to the edge of the image). This augmentation significantly improved the model's performance on the edge cases, making it more robust to the actual scenarios.\n",
        "\n",
        "    **RandomBrightnessContrast**: RandomBrightnessContrast from the Albumentations library for more robust training on various input image scenarios. \n",
        "\n",
        "**Loss: Mean Squared Error**\n",
        "\n",
        "**Learning Rate: 0.001**\n",
        "\n",
        "**Epochs: 30**\n",
        "\n",
        "**Mean IoU: 0.873** \n",
        "\n",
        "* #### **UNet++ Semantic Segmentation**:\n",
        "\n",
        "UNet++ takes the original image as input and mask as ground truth. We generated the ground truth mask from the given bounding box coordinates using the cv2.fillPoly function. The details of the training, preprocessing, and post-processing are as follows:\n",
        "\n",
        "* Model: **UNET++ (from segmentation_models_pytorch)**\n",
        "* Encoder: **resnext101_32x8d**\n",
        "* Encoder weights: imagenet\n",
        "* Augmentations:\n",
        "    Except for the augmentations used in \"Corner Regression,\" we used one more augmentation for training purposes: ShiftScaleRotate: translate, scale, and rotate the input. \n",
        "* Preprocessing:\n",
        "    We use the same preprocessing as done on the original UNET++ pretraining. This normalizes the image to a distribution accustomed to the pre-trained weights.\n",
        "\n",
        "**Epochs: 15**\n",
        "\n",
        "**Learning Rate: 0.001**\n",
        "\n",
        "**Mean IoU: 0.896**\n",
        "\n",
        "**Loss: 0.75*Dice Loss + 0.25*BCE Loss**\n",
        "\n",
        "\n",
        "We made a custom weighted loss for the training of UNET++. Dice Loss increases the intersection with the ground truth image, while BCE Loss ensures confidence in predictions. This weighted average of losses resulted in better overall learning of the model.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "qs-N-nOMH9sP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "* #### **Area Union Fusion Ensembling**:\n",
        "    We used the predicted boxes from Corner Regression (I) and UNET++ (II) to ensemble for final prediction. We used Area Union Prediction method, i.e, the largest polygon that covers both the predicted polygons, thus ensuring maximum coverage.\n",
        "\n",
        "    ***Mean IoU Score: 0.91***\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "RpJgPi4mIGNy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## Defining Corner Regression Model\n",
        "\n",
        "class CNN(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.model = timm.create_model('resnet34', pretrained=True, num_classes=768)\n",
        "        self.ll = nn.Linear(768,8)\n",
        "    \n",
        "    def forward(self, img):\n",
        "        x = self.model(img)\n",
        "        x = self.ll(x)\n",
        "\n",
        "        return x\n",
        "\n",
        "model_reg = CNN()\n",
        "\n",
        "\n",
        "## Define UNET++ Model\n",
        "\n",
        "ENCODER = 'resnext101_32x8d'\n",
        "ENCODER_WEIGHTS = 'imagenet'\n",
        "\n",
        "model_unet = smp.UnetPlusPlus(\n",
        "    encoder_name=ENCODER, \n",
        "    encoder_weights=ENCODER_WEIGHTS, \n",
        "    classes=1,\n",
        "    in_channels=3,\n",
        "    activation=None,\n",
        ")\n",
        "\n",
        "preprocessing_unet = smp.encoders.get_preprocessing_fn(ENCODER, ENCODER_WEIGHTS)"
      ],
      "metadata": {
        "id": "gPn-9FihxS9x"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Loading weights\n",
        "\n",
        "model_reg = model_reg.to(device)\n",
        "model_unet = model_unet.to(device)\n",
        "\n",
        "model_reg.load_state_dict(torch.load(root + '/weights/corner_reg.pt', map_location=device))\n",
        "model_unet.load_state_dict(torch.load(root + '/weights/unet++_weights.pt', map_location=device))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4sN33pXpyks3",
        "outputId": "4d7bb115-b889-4532-bfd8-edeeda425d69"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<All keys matched successfully>"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## Helper Functions\n",
        "\n",
        "def get_preprocessing(preprocessing_fn):\n",
        "    \n",
        "    \"\"\"Construct preprocessing transform\n",
        "    \n",
        "    Args:\n",
        "        preprocessing_fn (callbale): data normalization function \n",
        "            (can be specific for each pretrained neural network)\n",
        "    Return:\n",
        "        transform: albumentations.Compose\n",
        "    \n",
        "    \"\"\"\n",
        "    \n",
        "    _transform = [\n",
        "        A.Lambda(image=preprocessing_fn),\n",
        "        A.Lambda(image=to_tensor, mask=to_tensor),\n",
        "    ]\n",
        "    return A.Compose(_transform)\n",
        "\n",
        "\n",
        "def to_tensor(x, **kwargs):\n",
        "      \n",
        "    return x.transpose(2, 0, 1).astype('float32')\n",
        "\n",
        "\n",
        "def get_contour_from_mask(img):\n",
        "\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        img (np.array): mask for which bounding box has to be formed\n",
        "    Returns:\n",
        "        cnr (np.array): corners of the bounding box\n",
        "    \"\"\"\n",
        "\n",
        "    assert img.ndim == 2\n",
        "    h, w = img.shape[:2]\n",
        "    img = (img>0.9*img.max()) * 255\n",
        "    img = np.ascontiguousarray(img, dtype=np.uint8)\n",
        "\n",
        "    im_floodfill = img.copy()\n",
        "    mask = np.zeros((h+2, w+2), np.uint8)\n",
        "    cv2.floodFill(im_floodfill, mask, (0,0), 255)\n",
        "    im_floodfill_inv = cv2.bitwise_not(im_floodfill)\n",
        "    im_out = img | im_floodfill_inv\n",
        "    \n",
        "    im_open = cv2.morphologyEx(np.uint8(im_out), cv2.MORPH_OPEN, np.ones((5, 5)),iterations= 5)\n",
        "    image_sharp = cv2.morphologyEx(im_open, cv2.MORPH_CLOSE, np.ones((5, 5)),iterations= 5)\n",
        "\n",
        "    cnts = cv2.findContours(image_sharp, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n",
        "    cnts = imutils.grab_contours(cnts)\n",
        "    c = max(cnts, key=cv2.contourArea)\n",
        "    peri = cv2.arcLength(c, True)\n",
        "    cnt = cv2.approxPolyDP(c, 0.02 * peri, True)\n",
        "    \n",
        "    cnr = np.zeros((4,2))\n",
        "    cnr[0] = cnt[(cnt[:,:,0] + cnt[:,:,1]).argmin()][0]\n",
        "    cnr[1] = cnt[(cnt[:,:,0] - cnt[:,:,1]).argmax()][0]\n",
        "    cnr[2] = cnt[(cnt[:,:,0] + cnt[:,:,1]).argmax()][0]\n",
        "    cnr[3] = cnt[(cnt[:,:,1] - cnt[:,:,0]).argmax()][0]\n",
        "    \n",
        "    return cnr\n",
        "\n",
        "def corner_regression(img_path, model, size = 224):\n",
        "\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        img_path (str): path to the image\n",
        "        model (torch.nn.Module): model for corner regression\n",
        "        size (int): size of the image to be fed to the model\n",
        "    Returns:\n",
        "        corner_preds (np.array): corners of the bounding box of mask\n",
        "    \"\"\"\n",
        "    \n",
        "    model.eval()\n",
        "    img_orig = cv2.imread(img_path)\n",
        "    img_orig = cv2.cvtColor(img_orig, cv2.COLOR_BGR2RGB)\n",
        "    \n",
        "    img = A.Compose([A.Resize(size, size)])(image = img_orig)[\"image\"]\n",
        "    \n",
        "    img = (np.transpose(img, (2, 0, 1))) / 255.0\n",
        "    img = torch.tensor(img[np.newaxis,:,:,:])\n",
        "    \n",
        "    with torch.no_grad():\n",
        "        corner_preds = model(img.to(device, dtype=torch.float32))\n",
        "        \n",
        "    corner_preds = corner_preds.detach().cpu().numpy()\n",
        "    corner_preds = np.float32(corner_preds.reshape((4,2)))\n",
        "\n",
        "    for pt in corner_preds:\n",
        "        pt[0] = pt[0]/size * 1280.0\n",
        "        pt[1] = pt[1]/size * 720.0\n",
        "\n",
        "    width, height = 1280, 720\n",
        "    target_corners = np.array([(0, 0), (width, 0), (width, height), (0, height)])\n",
        "    \n",
        "    H, _ = cv2.findHomography(corner_preds, target_corners, params=None)\n",
        "    \n",
        "    transformed_image = cv2.warpPerspective(\n",
        "        img_orig, H, (img_orig.shape[1], img_orig.shape[0]))\n",
        "    \n",
        "    return corner_preds\n",
        "\n",
        "def unet_prediction(img_path, model, preprocessing_fn, size = 320):\n",
        "\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        img_path (str): path to the image\n",
        "        model (torch.nn.Module): model for segmentation\n",
        "        preprocessing_fn (callbale): data normalization function\n",
        "        size (int): size of the image to be fed to the model\n",
        "    Returns:\n",
        "        corners (np.array): corners of the bounding box of mask\n",
        "    \"\"\"\n",
        "    \n",
        "    model.eval()\n",
        "    im2 = np.array(Image.open(img_path))\n",
        "    h, w, n = im2.shape\n",
        "    true = im2.copy()\n",
        "    resize_img = A.Compose([\n",
        "            A.Resize(size, size)\n",
        "        ])\n",
        "\n",
        "    preprocessor = get_preprocessing(preprocessing_fn)\n",
        "    \n",
        "    im2 = resize_img(image = im2)['image']\n",
        "    im = im2.copy()\n",
        "    im2 = preprocessor(image = im2)['image']\n",
        "    \n",
        "    img = torch.tensor(im2)\n",
        "    \n",
        "    with torch.no_grad():\n",
        "        out = model(img.unsqueeze(0).to(device, dtype = torch.float32))\n",
        "        \n",
        "    img = img.numpy()\n",
        "    out = out.sigmoid().detach().cpu().numpy()\n",
        "    temp = np.transpose(out.squeeze(0), [1,2,0]).copy()\n",
        "    \n",
        "    corners = get_contour_from_mask(np.uint8(temp*255).squeeze())\n",
        "    \n",
        "    for pt in corners:\n",
        "        pt[0] = pt[0]/size * w*1.0\n",
        "        pt[1] = pt[1]/size * h*1.0\n",
        "        \n",
        "    return corners\n",
        "\n",
        "def min_max_corner_fusion(corner_unet, corner_reg):\n",
        "\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        corner_unet (np.array): corners of the bounding box from segmentation\n",
        "        corner_reg (np.array): corners of the bounding box from regression\n",
        "    Returns:\n",
        "        corner_min_max (np.array): corners of the bounding box from Min Max Corner Fusion Algorithm\n",
        "    \"\"\"\n",
        "    \n",
        "    corner_min_max = np.zeros((4,2))\n",
        "        \n",
        "    corner_min_max[0][0] = min(corner_unet[0][0], corner_reg[0][0])\n",
        "    corner_min_max[0][1] = min(corner_unet[0][1], corner_reg[0][1])\n",
        "    corner_min_max[1][0] = max(corner_unet[1][0], corner_reg[1][0])\n",
        "    corner_min_max[1][1] = min(corner_unet[1][1], corner_reg[1][1])\n",
        "    corner_min_max[2][0] = max(corner_unet[2][0], corner_reg[2][0])\n",
        "    corner_min_max[2][1] = max(corner_unet[2][1], corner_reg[2][1])\n",
        "    corner_min_max[3][0] = min(corner_unet[3][0], corner_reg[3][0])\n",
        "    corner_min_max[3][1] = max(corner_unet[3][1], corner_reg[3][1])\n",
        "    \n",
        "    return corner_min_max\n",
        "\n",
        "def screen_extraction(img_path, model_unet, model_reg, preprocessing_unet, mode = 'accurate'):\n",
        "\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        img_path (str): path to the image\n",
        "        model_unet (torch.nn.Module): unet model\n",
        "        model_reg (torch.nn.Module): corner regression model\n",
        "        preprocessing_unet (callbale): data normalization function of unet\n",
        "        mode (str): mode of the algorithm\n",
        "    Returns:\n",
        "        transformed_image (np.array): transformed image\n",
        "    \"\"\"\n",
        "    \n",
        "    img = cv2.imread(img_path)\n",
        "    \n",
        "    if mode == 'accurate':\n",
        "        \n",
        "        corner_unet = unet_prediction(img_path, model_unet, preprocessing_unet)\n",
        "        corner_reg = corner_regression(img_path, model_reg)\n",
        "        corner_preds = min_max_corner_fusion(corner_unet, corner_reg)\n",
        "        \n",
        "    else:\n",
        "        corner_preds = corner_regression(img_path, model_reg)\n",
        "    \n",
        "    for pt in corner_preds:\n",
        "        pt[0] = pt[0]/img.shape[1] * 1280.0\n",
        "        pt[1] = pt[1]/img.shape[0] * 720.0\n",
        "        \n",
        "        \n",
        "    width, height = 1280, 720\n",
        "    target_corners = np.array([(0, 0), (width, 0), (width, height), (0, height)])\n",
        "\n",
        "    # Get matrix H that maps source_corners to target_corners\n",
        "    H, _ = cv2.findHomography(corner_preds, target_corners, params=None)\n",
        "\n",
        "    # Apply matrix H to source image.\n",
        "    transformed_image = cv2.warpPerspective(\n",
        "        img, H, (img.shape[1], img.shape[0]))\n",
        "    \n",
        "#     plt.imshow(transformed_image)\n",
        "    \n",
        "    return transformed_image"
      ],
      "metadata": {
        "id": "fCO6FmWA0HLw"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Number Detection + OCR"
      ],
      "metadata": {
        "id": "Iu5njhI-105W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "After extracting the screens, our next stage in the pipeline is to find all the numbers on the screen and return the appropriate bounding boxes across them. This task is achieved with the help of two separate models: \n",
        "  \n",
        "  - **YOLOv5** \n",
        "  - **PaddleOCR**\n",
        "      \n",
        "      * YOLOv5: \n",
        "              \n",
        "          We trained YOLOv5 model for the detection of numbers on a screen. We assigned a “number” class to all given bounding box and then fine-tuned YOLO for the same. The details of training and hyperparameters are as follows:\n",
        "\n",
        "          * Model: YOLOv5\n",
        "          * Size: M\n",
        "          * Epochs: 25\n",
        "          * Learning Rate: 0.01 \n",
        "          * Mean Inference time: 650 ms\n",
        "          * Average Precision: 0.77\n",
        "\n",
        "      * PaddleOCR:\n",
        "              \n",
        "         PaddleOCR is an optical character detection and recognition model implemented using PaddlePaddle (PArallel Distributed Deep LEarning) framework. In our pipeline, we have used Pre-trained PaddleOCR to detect numbers from the extracted screen. The details of the hyperparameters are as follows:\n",
        "          * Recognition Algorithm: CRNN\n",
        "          * Detection Algorithm: DB\n",
        "          * Mean Inference Time: 2.5 s\n",
        "\n",
        "    * Weighted Box Fusion of PaddleOCR and YOLO:\n",
        "              \n",
        "         Fine-tuned YOLOv5 is showing promising results on layouts it was trained on but a relatively low accuracy on unseen layouts. In any case, it was not giving any noise in the prediction. Pre-trained PaddleOCR captures boxes of all numbers on the screens, but being a text recognition model, it also predicts few noise in the screens, which is not required. Hence, we ensemble the predictions of both the models, using Weighted Box Fusion algorithm, taking the good points of both algorithms, thereby resulting in much more robust predictions.\n",
        "      * Weighted Box Fusion algorithm utilizes confidence scores of all proposed bounding boxes to construct the averaged boxes. \n",
        "\n",
        "       * Mean Inference Time - 3.3 s\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "dnqX6USFIM8H"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAr4AAAE4CAYAAAC9hvAtAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEnQAABJ0Ad5mH3gAADXeSURBVHhe7d0JmFzVYeb9t/bqfVG3Wrsa7UIIERAyCEkYsJCNjQHbmHggjJ3YinEcxk6YSZwMzpcvn7dknDzGjmcm8YaxWYzBYFazGCQjIQFCIIT2fWmpW93qtbau7TvnVskIaOGW1K2uqvv/PRyqq+p0te6tW+e+99S553qyhgAAAIAS583fAgAAACWN4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABX8GSN/M8YIfYNSKWziiWySmWyzn3gGI8pAZ9HFSGPvF57DwAAnAqC7wizaz+RzOpIT1p72pLqM+GXdwTH8/mkqjKPZowJqrrc64RgD/kXAICTRvAdYTb0rt0R17NvxLX9UMqEXt4OvFso4NE5EwNaNq9M08cGFQ6SfAEAOFkE3xHWG8vokXVRPfJK1PzMW4GB+bxSU41P1y4o14JpITVU+/LPAACAwSL4jrDuaEb3vxjRQy9Hfz/EwW9CjtcUvs52N7s9pDO5Yof2VoY9um5BhRbNCmlcvT9fCwAADBbBd4TZ4Hvf6ogeNsHXsqG3qdanhiqvyvg629XiyawOdabV2p1xDoIqQx6nx3fx7LDGE3wBADhpBN8RdnzwteGmLODR5XPLtGBaUOPq+DrbzY70ZPTkazGt2BQn+AIAMAQIviNsoOD74QvKdenZYU1qINy4WWtXSg+sjTrhl+ALAMDp4wIWBci+KXZMpz2hieLewjhvAACGltm1AgAAAKWP4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsARiabVSKZViSRUjKdkbkLACgxBF8AMDr7+rVyyxHds2qvtrb0KJ5M5Z8BAJQKgi8AGEePHNELjz6ue7/zPW177TXFo7H8MzmxVEYvtfTpoS1Htak16vQKAwCKC8EXAIz+WJ9a976pXW+sUHd7i9KpZP6ZnJgJwmtWr9b9v7hPGze+of7+/vwzAIBiQfAFAEfalIgpXaYkTHl7j24yHtWudS9o3aP3qGX7BqWTSWUZCAwARYXgCwCDkTVBONYj9baZXGwDMqEXAIoNwRcAAACuQPAFAACAKxB8AQAA4AoEXwB4h0xWSpuSyry92Mft0F57a+8PVMfOcsY5bwBQmAi+AHAcewW3rli/WrvjOtQV/X1p644pmkiZcJtVbyypw/bxztjb6thypDeuaD8XvwCAQuTJMh/PiOqOZnTf6ogefjkqj0cqC3h09QXlunROWJMb/flacKPDXSn9ck1UT6yPOdtGZcijaxeUa/HssMbXs20MtTfffFN33HGHHnjoUc1ecrUmzT5fofKq/LNSrK9bG55/WAe3bdDMi67QjPPfr0C4LP/sWyaOrtWyBbO08Jyz8o8AAAoFwXeEEXxxIgTfM8sG3+985zu6975fyFtWrXBFjby+t9ZzOp1U5GibEtEeldc0qKy6Xl6v/dLMvDnHOXvWDP35n/6Jrr/2I/lHAACFguA7wgi+OBGC75nV2rJPq575tTa8vFL9iXj+0ZM3ZsJZWrT0Gp1/8eX5RwAAhYLgO8IIvjgRgu+Zle3vU6pzr9J9h6WMvYrbqfGEKuWvbZavelz+EQBAoSD4jjCCL06E4Fu4bKOZaznt/zy5wQ7mf84tAKBgMasDAJwEO5VZf1rqSaTVEU2pM5ZSn3kgaacxy9cBABQmgi8AHMcG2454Vm8eSeiFfb1aczCiQ9G04ibsxky63dnWp1+u2qG/++Fj+ux3fqG/+N6D+tY9K/XrF/doX1fC1CP+AkChYqjDCGOoA06EoQ5nnp2j90BPSivXbdFrW7br8NEOhYJBzZ02Q++/YLqTil98ZYt+/dhvtWfLK0pH+iSfX2V1jWqePkdLP3SZPnrJHE0eXeO8ZwCAwkKPLwAY9sIVPdGEnlq9Ub+470E9c9+d2vTYz7T5ybv12L1368nfvqhHn12jpx9/TIfW/Frn+ffrmqaIPlB7VGO6X9fW5+7TL+7+hTZt261oIpl/VQBAISH4AoART6a1u6VdP/7f/6mdz92rcxOb9bmpKS2fEVfT3if1+D0/010//rEOvvG0Pvm+sL79rb/QP/3k6/rW//mf+uqtV2vRhLi2PPUzvfTKOh1s78m/KgCgkBB8AcDo6enTGxveVPfetVo4d4xuuvXPdNO3vq2P3/6v+uJ//6xG6ZBS+1/W5LFVuvLmv1TDvOtUNnaRqqZdqZmX3airbrhJE5rKtGfrGzqwd0/+VQEAhYQxviOMMb44Ecb4nlk7du/TT+55UPf+5Pv67A0f1sev/4SmzZmnbNajntbd+vL/+Ee99voGLV50kf77bbdpXPMs+fwB85tZ9fV0auP6Nfr72/9fhesm6dP/9b/q+o99OPfCRc6e7JdMZRRJmOWMZ5Xi5D28g99n2qew1ylBP4PbUdgIviOM4IsTIfieWZu2bNW//5//1BOP/Ep/+9f/Tddce52axk10JuzNpBL6iy/+pV5//TVd9v5L9aW/+muNGjVaXp/Z4xuxWFTbt23R337lK4r3S5/59Kf1Jzd9ynmu2HWZNmrzgX5ta0ma9orgi3cwbZPXlIYqn+Y1BzV7fMAEYcIvChfBd4QRfHEiBN8z680339R3v/tdPfXUU7r99tt19dVXq6GhIf+s9IUvfEGvb3hdl19+ub70pS+prrZOXm9utFg8HteOHTv0N3/7N4pGovrMZz6jm2++2XmumGUy0q62pH71UkSv7OxXNGEOAthjYAA15R5ddX65Pnx+meoqcgeEQCFijC8AYED9qYw6etPaeySleH8u9NqDMNvDR6HYbeGYnmhWhzpTOtJz6pf7Bs4Egi8AYEAm9yqRzCrW/9ZV6cIBqbbCo7pKL8XlpSrscQKwZbePZMpuL7n7QKFiqMMIY6gDToShDmfWsaEOT/zmKX3xb2/XFVddrbpRbw11uP1LX9CWja/r4ksv15/+xZdUfdxQh0Q8rj07d+ifv/o3Ssei+rM/LY2hDn3xjNbtSujO5yNq700rbYLwtDF+zZ0UUGM1X2e7mb1s9/72lFZvTShuDo5sklg4M6SPzi8320cwXwsoPATfEUbwxYkQfM8sG3zvuOMOPfTrx3ThZZ/UtDkXqbyiKv+s9Jv7/11th3Zr8vR5unDJtQqXV5r3JdfdlUr262j7IT3/6I/VUBPQFz6/vGSD7/lTgvrA3LBmjyfcuFkkkdHre/t136qIM9uHHQZD8EUxIPiOMIIvToTge2Zt3bJJP/i/39evf/UreYKTFSpvktdrpyvL6T36hpL9vQqGG1RefZZ5zr4HueCbzWaUTkXV17VZ06eN1+c+f4uuv+FG57liNlDwvXBqUB8xbdT8qaF8LbhRTyyjl7Yn9KPn+tRrfib4olgQfEfYiAZfe8q2LRgZ9mvy/FflAyH4nlkt+3bq2Ud+rpdWPqH+eDT/6Mmb0DxDH7jmZl18+dX5R4rXsAZf2p+RZafiy39jcSoIvihWBN8RNqLBt6tLOnpUSnI2whkXMqGhulqqr88/8G4E3zMrGe9V5MguxbtalM2c+pnp/nC1KhunqKx+Qv6R4jVswTdt1m97u/kDfVIqlX8QZ4RtTPym/Rg9WqqoOOXwS/BFsSL4jrAzHnxtD0tHh/TII9KLL0otLfS6jIRjO57zz5c++clcAM5fDOEYgu+ZlTQBrC8aNeEupbBZvUHzdpzK6Vsej08ef9iU4h8KMOTB1wbe/fulZ56Rfve73IE3wffMsw3K2LHSFVfkSlNT/onBI/iiWBF8R9gZD762d3fnTulrX8vtfOyOh03gzLNDHGyP77x50r/8izRzpnnzy/JP5hB8z6wjXb16ZfM+HexIqK6xXmNG12lMbbkaqgOy8/HbEGzeBlcZ8uDb3y+tXSt9/evSyy+bP9DHgfdIqazMHXQvX547AD9JBF8UK+bxdRsbcu3Oxvb02p5fuyOyPS62J+bYmDvK8Ba7vhOJ3EHH1q2598Lex4jq7erSa6tX676f/UJ33v2Qfv7gU3rwqTV6as0uvbj9iLa2RtUaSaovmVHS7OXtjp5DxpNk25kjR6R166TOztx2bz8PA31OKMNT7Pq2HSB2/du2x74fgIvQ4zvCzniPrw26dqdz222S2ck7f9R+7V5TIwUC73myFYaA3fH09EixWO7nUaOkO+6Qli6VGhvzlXLo8T2zDu7Ypifu/LHuevgxtcUTSnuD8obHK9i0SGMumKZ5s6fonKmNmja2UmMqQqoKBxUO+hX0e+U3nxvnXMX8a5WKIe/xjUSkhx6Sbropd9+uNPtNR3m5FKSXcFjZXb1t/23Hhz3gsPeXLZNuuUW65pp8pcGjxxfFiuA7wkY8+NqTrOxYr499TJoyJbcDwvCwQdeeUGh3/Bs25AIwwbdgpBO9ih3Zoc7dr+rgzte0Z+s+bd7Rrg2tnert6VRn1ygdLZ8h75SzNf2cs3TejNmaPnWMpkxo0MRRFRpV5ZP5r6SGRAx78LWhd/586dJLc+0Pho9t+7dvl554Indre30JvnAhgu8IK4jg29wsfeUr0oIFUl1dviKGnN3RbNkifec70gsvmDe/m+BbSLJpk35jysR6FIscVaSzQ93tbTrS3qLOtv06vKdd+9o6dci8b5GeiI52TVDv6DGqHjde48c1a/SEGZo0c7zmNo/SjIawGt4+ZLsoDXvwtQfaNnzZ+wsX5h7D8LBDG2y7873vSZs3E3zhWgTfEVYQwdf2tNiTTRYvzgUxDA+77tevl77xDen55wm+Bc0kvEy/ssmoUrFexaOd6uvoUGdHq3OFto7Dh7RvT0x7Ir062hdVV5dHh/tq5B1TqwXnztXHr7xYiy6YkX+t4nVGgu/VV+dOsLr88txjGB52+rinn5a++U2CL1yNAZ0A8C52wG5YnlC9ArWTVTVunkaf8341X/IJzVr2OZ3zsS9q/iev08WXXKCZE6sUSu5R+xuPaPfj39fWZ+5R+54t+dcBABQSgi+Glf06wX6nQMkXuz48HnN7rOQfO/b8ccV54jjH7jpP5esUdcktTkGy/zb7b0yb/yXSWUX6MzoSSWtHR1Iv7k3okQ0x/fiFjO55uVxP7m3UluwYlY2r0SVnleuyOUE1N9G0AkAhonXGsLAjaFImMCRTWfWnMpRjJe1Rvyegfn8oV3ymZLzmuYHWU9Y5H+546UzprNOkKel0xtlWCon9yrbfbLt22rKW3qRWH4rop6+06St3v6ovffVu/eNf/IP+4/YvauX/vV2pZ+7Qwu4n9KUL4vruX12rf/zXr+nmL/4PzTp3Qf7VAACFhDG+I6wUx/jaLaovkdGm/f1q6Uwr3s8m5kinpEOHpZUrpd27c1MK2bPaP3yVNHVa7vKhx+mNm3V4IKnth1LOLAEBsznMHh9wtovqsuI/Zi0PeTS21qcpYwKqr/CN+Ex6cXNA0daX1La2XvP2tGrH/oPmdp+ObN2lUPcOydutWk9GE8u9ap4c1PgpM9Q0boLqx4xXdcNYVVQ3KlhepUC4Ut5AmTzeQP6VixdjfEsIY3wBB8F3hJVi8E2ms9rWktRTr8e0qy3l9FCykRn2oxaP5y4cEo3a7tvcZYrHNOVC7zsuWZwye5LeWFaReG7tee0JbmGPCYxe+d9etSiFzbY+cZRfF88I6fwpIee+/QyMlEMdXfrty5v0+Auvq3fbDqX69snn6VG1J6Tm6pTGTqrXhHHjnTJmYqNGjZ+iUN0YecP2ctP2oKX0vkAj+JYQgi/gIPiOsFIMvrH+rJ55I6bH1sW0vyOVfxR4O5txR9f4tGR2SNdcWK6a8pHt9d22Y5f+86f36s57H1ZtX7fmTAroj86fqDnnXaizmptN2G1WXeM4hatGyROsNAtgg98IJvUzgOBbQgi+gIMxvhhytgHsMDvJRCp3TGWjgc9saUG/RyG3F29WoUy/Qsm4KTGFUubWkzbPmWOQd9S168uut+MV+3o8tkz2IM9uHfbbAXvwl7BXrR3hY3BfOqGKSItqI7uliZVKnX+l+s//nHpnX6/eicvUUz9fPWVnKeKtV382rEz+5EQAQPGgx3eElWKPbySR1f0v9mnFpoTauk2oM8s0qsqj5saA/G4+1LJnqnV2SZs2SW1tuR6XUFA6/4LccAf7Xhwnnsxqf0dahzrTzn277iaM8qmx2qeyYHH2NNoTHg+a5Wntzjhjv+srvZo/JagbLqlUQ5UdwjFyy9XXflDb1z6h1U/8Slt2b9OB/RkdjDRpf+M0jZ8yXjOmnKvpc2ZrxuyJmjG2Vs0NPtX6pOIfyXti9PiWEHp8AQfBd4S5IfhWhT2aMS6gZeeVqabcqxHMNiPL7mi2bZfu+qn06qsmVfRJNTXSX/2VSRMXSrW1+Yo5R/syeu7NuF7clnC2DTsG9tKzQ5o3OeiE32IU7c/omQ1xbdiXVFckU1DBN52MKdbVos6Du9XRtl8dhw7p0IFD2tlifj58SIfba9QRMp+PUaPVWDtJdTPP0VnTJ2lGc6OmNVVqTKVf5T6fgmYZnF7t/OsWM4JvCSH4Ag6C7whzQ/C1YffcyQF96pIKZ0yn356l5UZ23b/+uvTP/yytXGHe/B6pvl76t3+TrjA7/Ya3X7mt1ay7B1+K6jev5a7cVhHyODuVRbNCGlc3DNvGGWCD1N2r+rR2e786egsr+DqDL+xli1MJpfoj6o92K9rVqs7WfSYE79WBvZ3af6RdrR1d6m6TWoK18leVqapyjGoaJqumeZwmjTYheOJozZxQq7G1xX/NYoJvCSH4Ag43f/GMM8jGGZ8JvDbYBPwuLt6sAtmUAumkKf0KZMytJ6PAAOvFztxw/DFCbh2qqNehXSZPwfaFmn+XxxxQBCrkrxit8sbpapi+SNMXfUoXXX+rPv755Vr+53+sWz71Ad24bLo+ODWh6ekNir35jFY/8rB+dOc9+v4P7tYvH3lO23e15F8TAFBICL4A8B7sFfYy2Qqly6cqMPkKjV78ac27+csmCH9Zn/j8F/Xh/7JUF54XVFXL77T98f/Qqw//XAe3vpn/bQBAISH4AsAJZEyJ23MS+6V93Sm92hLR41u69Z+ruvSlXx/Rn9/Zrm/+5JBWPbtXFckjunRyuT423a9zTn/EEABgGBB8ASDPnvAQMf/b25fVKy1xPbrhiH74my367l3P6Wv/fq/+4Zv/oW9/9d/05P/3NcV+8y3Nan9EV00+pM99dIpu//Kn9Pf/9GV9/C8/p7P+6LzcCwIACgrBFwCMzmhSv9vRrp88+aZ+cNeT+uEPfq6f/vAHuv/OH2jF3T9W64p7VL3/GZ0d3K5FU9O6duFU3XjVAv3xDVfqo5/6hK645k+04MqPqfn8RapsHJt/VQBAISH4ngQ7/4Wdh9ROuj9kJZV1zoY9Xtr8oSH/O8eK+XvJjEdJj19JXzBXvAEls+axtN5d/xSL+e9t7DIO2zIVSEmbhWSOlOJ1tKtbK154Sff+7G795ud36o1H71LPq79Sfedqzak4pA/NCurPlk7RrTcv0mdvvV7X/ekX9YFPfFYXLLtBkxZ8UJUT3ydfzRR5/HZaulKe3RcAihfBdxCOBV47ZYud0qe1a+jKkZ60ogk7kjDH/mSnA2vvG7j+aRfz91rjfrWWjVZr7SRTJqq1Yoxa+0Nq7c0O/DsnWdq6U4rEc0HQWSZzay9W4Ky77nfXL5XSGUkr1p9xpnxC8fH2HFH4jSc1fvdPtaB6k244P6SvfGyO/v0rH9K3v/NlfeGb/0sf/OK/6NyP/LXGXXCdqsf9kYI1k+QJ1ZlfDttXyL0QAKBgMY/vIPQnTSA0ge13W+I61JVSIpl/YgjYHtgDHWm15K/OZaerGlNnr87lVUVoGHakGfN37NXDXt8gdXSYfbX5GxUV0txzpMYGKXgKc3O+gz1IsMvU3psx6yqrgE/OXL4TG3wKBz3y2klpS5B9z2aODWjmuICaageYZ9fO47t+vfSNb0jPPy91d+fmTb7jDmnpUvMCb5/H97DZ1n65Jqon1ufm8a0MeXTtgnItnh3W+PrinMe3N5bWXSsjWrM9UXDz+EaOHlbLxlVKRPertnGSqhonKFzTKH+4Xl5fSB6fWecee+GQ0tx+B8I8viWEeXwBB8F3EDojGa03jb+9mEBXdGh79Ozqt0G63wRgywacoJ3z1M7hOhwXenC6r1O5HZBt+OwfPBZ+A4Hcz6fp2DLZAGwbQ/sn7OV27aWLnUUq0eAbNqtvWlPAucDEpXMGuHgBwbegg28mmVB/5KjZfhMKBCvkC5WbnGt24B73Dlsg+JYQgi/gGIYuxdJjv6Y/2Jl2en07+zLqMeF3qEpvLPv70GvZXGp7Sfvi2QHrn3Yxf68n6VNPsFo9FaPUU16vnnCtetIB9cRNYzbQ75xkObZM+ZEOzjLZ8cPOMtm/P8DvlEJp685oz5GUDhzN9d6juHgDIYVrm1RWN0n+ikZ5/OZg0MWhFwBKEcF3EGzPZZ8JbDbAWbb3rSzoUVON77SL/Xq8PPRWL5d9bXtp2lFV3gHrn34xrxtOqinaqqbOfWrq2q+mvkNqCsTVVKUB6p98yQ3T8DjDNizbyxsO5JZptP37A/xOsRbbS2mX9VgndswcJNnxzShWdqOlWQSAUsVQh0HY357So+tienZjzAk2QZ9HE0b5dOH00/86x/Ymbz6Q1LZDKee+vaTr9DF+TW70q6ZiGHbA6bR06JD09DPSwQOSHbdYWytddpk0aZJUPsBX9CcpaRZl04F+s97Szol6doiDDbxzJwZUEfYOxWiKgmF7t3e1JrXzcMrp1a4z79mS2WEtX2qOIt6JoQ4FPdQB78ZQhxLCUAfAQfAdhHcGX9vD977pIX3+yup8jVNnvyL/1UsRPfZqLtzYntEPnlemRbPCmtgwDOHGhq/XX5P+/n9KL62VQmbnNblZ+upXTau1UKqvz1c8dXaWiodejmrVloQza0V1mVfnmND7yYXlJgD7Syrc7DPbxiOvRJ0gZ4eoEHzfG8G3uBB8SwjBF3Dwnd4psief2QB8usUOczh+Z29/sj3KdijFQPWHpPgyqkhFVRHvzZVkxDyWVoVpqwasf5KlPOR11s+xr//tre3JLgvmhnUM9DvFWsrMgcqxIR0AAKCwscsGAACAKxB8AQAA4AoEXwAoMPbUi1Qqo3h/SunMEE4cDgAuR/AFgAITjSe1fX+7nn9ll1o7+pzLfgMATh/BFwAKTCyR1I59HXrg6Y16/HdbtX1fu1JDeclIAHApgi8AFJh0OquevoS27Tmip1/crhWv7Naels78swCAU0XwBYACZAc3JJJpbdzZqmfW7NBKE35bjvQoaR5j9nUAODUEXwAoWFlFY0mt39KiXz37pp5ctU1tRyNKptL55wEAJ4PgCwAFLpFMafv+Dt3/1Bt6/IUtau3odWZ+AACcHIIvABQ4m3HtCW+7D3bq4ec26bcv7dTeQ13M9gAAJ4ngCwBFwIbcaLxfW/e264kXtunZtTu0c38Hsz0AwEkg+AJAkbA9v/FEyhnza8OvnefXnvCWJvwCwKAQfAGgyNgLXLyx47Aee2GLVry8S119ccIvAAwCwRcAilAsntKWXe362WOv6dHnN+tAa7dzmWMAwIkRfAGgCNlZHWLxpHOS20PPbXIudLGn5SizPQDAe/CYRpJW8g/Y357So+tienZjTLH+rCpCHi2aFdKtV9Xka5y67mhG962O6OGXo/J4pLKAR1dfUK5L54Q1udGfrzWE+vuldeuk226TVq+WQiFpyhTp61+XFi+WRo3KVzx1kURW97/YpxWbEmrrTqum3Kt5kwO6cXGlmmp9CvjMgpaIPW0pPbA2olVbE0oks6qr8GrJ7LCWL63K1ziOXffr10vf+Ib0/PPmze/Ore877pCWLpUaG/MVcw53pfTLNVE9sT7mbBuVZru7dkG5FpvXH18/DNvGGdAbS+uulRGt2Z5QR29G9ZVezZ8S1A2XVKqhyiv/ANtGe2dEuw4edWY0cEtz1d2X0OZdbVqxbpe6euLveQKbx2wc5WG/5k4fo6sWzdTl75uqiU218npP/3PWF89o3a6E7nw+ovbetOw/48KpQX3EtFHzp5q242RFItJDD0k33ZS7X14uXX21tHy5dPnluccwPNrbpaeflr75TWnzZimZlJYtk265RbrmmnylweuJZfSS+Rz/6Lk+87nOyE4wsnBmSB+dX665k4L5WkDhIfgOAsH35BB8Cb4ncrLB1zZPG3e0OnPXPrlqu7Jm7+qGBsuO143Ek4rG+pXoT5nA+YeXujwc0Pmzx+lDJvy+/8IpGjOqSgG/z9l2ThXBt4QQfAEHQx0AFDQ7f+3h9oh27OvQzgNHtcsFxQ5fsD3d9iS2wYRey9Zdt7lFv3z6DT21eruOmN9PpbnCGwAcj+ALACXCTnW2fW+HHnxmozPm14ZfAMBbCL4AUCLs0JBo/gpvDzz7pp54Yau272tXktkeAMBB8AWAEmKv8GbHB2/ZfUS/Wb3dubzxzgMdzPYAAAbBFwBKjA258URSG7Yd1lMv7tCKV3ar5Ugv8/wCcD2CLwCUqEisX2+Y8PvY77bo2bU71N4dUTLFCW8A3IvgCwAlLN6f0rY9R/SzR9frwWff1L7DXUoRfgG4FMEXAEqYHfZgL29sp0h7/HdbnaEPu1s6888CgLsQfAGgxGVs+HWmOmt3pjn77dqd2mZ+tsMeOOUNgJsQfAHABZypzuJJbdx+WL9ZvU3PrNmhvS2dDHsA4CoEXwBwEdvz++bONj220l4GeptzAhxTnQFwC4IvALiMvcLbzv0dTvDtjSacoRAA4AYEXwBwmUDAq4ljanXZgqmqKAvK4/HknwGA0kbwBQAX8fk8mjKhXldcNFVLL5rmBF8vwReASxB8AcAFbLQN+L1qHlen98+fog9eMkOzzhqtoN+fqwAALkDwBYASZ0Ov34TexrpKXb1ktq69bI7mThujYMAnOnsBuAnBFwBKnN8E3DENVfrUB8/VVYtnOkMdfD6afwDuQ8sHACXM5/Vo8phaffLKubpy4XRNMj+Hgr78swDgLgRfAChR4ZDfGcdrx/N+aFGup7e8LMAsDgBci+ALACXI7/Nqqgm6Sy+epo8smaUZkxtMECb0AnA3gi8AlJBjszc01JVr2ULb0ztDM5obCLwAYBB8AaCEBAI+jWus1meuucDp6Z02oUF+P2N6AcAi+AJAibBjeu2QhuuvnKsPvG+axjfVOEGYvl4AyCH4Aiho1ZVhTZlQp3NnjHFNmTt9jGY2N57UmFw7pteG3mULp+vKi6dr8rg6lZkgzAgHAHiLJ2vkf8YJ7G9P6dF1MT27MaZYf1YVIY8WzQrp1qtq8jVOXXc0o/tWR/Twy1FnB1UW8OjqC8p16ZywJjcOwxWV+vuldeuk226TVq+WQiFpyhTp61+XFi+WRo3KVzx1kURW97/YpxWbEmrrTqum3Kt5kwO6cXGlmmp9CvhKZ0+8py2lB9ZGtGprQolkVnUVXi2ZHdbypVX5Gsex6379eukb35Cef968+d259X3HHdLSpVJjY75izuGulH65Jqon1secbaPSbHfXLijXYvP64+uL82pbvbG07loZ0ZrtCXX0ZlRf6dX8KUHdcEmlGqq8Jry9e9s42hPTzv0d2rSrLf9I6YsnUjrQ2q1HV25WZ09c6XQm/8y72W0jFPCboFvrzNzw4cWzNG1ivXz+0+/p7YtntG5XQnc+H1F7b1r2n3Hh1KA+Ytqo+VNN23GyIhHpoYekm27K3S8vl66+Wlq+XLr88txjGB7t7dLTT0vf/Ka0ebOUTErLlkm33CJdc02+0uD1xDJ6yXyOf/Rcn/lcZ5QxSWLhzJA+Or9ccycF87WAwkPwHQSC78kh+BJ8T+RUgq8btXb06ek1O/S9e1aZnyNKnSD42rVlhzJMHlOjT9h5ei+erqkTR5ntZWjWI8G3hBB8AQdDHQCgSNnLEI+ur9Qff+g8Lb1ouiaOqR2y0AsApYge30E4kz2+4YBH5zUHNH1swOk9HHKplLR3n3TffdLOHfZaprmexo9/TJoxQ6qoyFc8dYlUVi/v7Ne2lqR6Y1mVBT2aOMqni2aEVF3mVSldKbW9N9cjtvNwSsm0WX1m25g+1q9Lzw7naxwnbdf9fumxx6StW6RYzPxCpXTjf5HOPluqensvcXfMvLZZj2/sS9Lj6zKD6fEtC5l2YvIofXjxTF25cIbGN1Y7J7cNZfClx7eE0OMLOAi+g3Cmgq9lZx2yYaDKBMRhGRKQNXuuPrPz2bPHpJBeyWtSaNjswCZNlqpN8LJB+DRlzCZlQ02PWTYbBm3QteusodoOc7BjEksn3NjhDR0mEPTFs07Db98/+9411QwwfZRd93bHf+Cg2Wt0m4MQs3ICJsCedZZUU2u/s85XzEmmc+uxK5Ih+LrMHwq+5eGAZk8ZrcsXTNVVi2Zo0ti6YZm9geBbQgi+gIPgOwjDHXwfWJMLvqm379uA3/OaRFNd5tF176vQJTPDGltXnPOyEnwH572Cr5294ZxpTVp2yXRnyrLpkxrkG6avUQi+JYTgCzhK6Evn4mT3V5VlXtVW2B7eXI/hsBdvVv5MSv50f65kkvJ7zGPm3zJg/VMoNqgdH2Fsj+VQvn7BFLNMdlmPZ+8PWNcWu+6zdt2bde6sf7vuM++5bux2EfR7VFPuc8Jv6PQ75VGE7GcoaDaGxvoKZ3jDVYtmatqkUcMWegGgFNHjOwjD2eNrv87ecjDpHDlvO9Tv9KgM6xtiD8ttr8u7hjpMkqqrTdI6/a/QM2YZnKEOsayzfHa/XG7WWWN1bvhGCY10cIY6HDXL2ntsqINZ1ioTTu3sFe9ybN0fPCD19Aww1GHgdW+DtB37bXtRLpoR1rg6nwnExbkS6fEdnIF6fO2Y3ubxtfrUh87TZRdO0fjR1WaTGWA7G0L0+JYQenwBB8F3EIYz+Nq1b3cuHX1pHTXFZqFhlUzlTmr74Y9yjZ8dVzpmjHTzzdI5c951gtWpiJsw+MLmhDbuTzpDOez6ah7t12Vzwk7Pdil1ULV2Z7R6S1ybD9qT27KqDHs0Z2JAHzyvLF/jOPbEwp27pPvvlzZskKLR3Pr+/J9L551nwu/A25M9UAj57YGDzwRFn4ImBBdrPCT4Ds47g69zcYrmBl132Rxd/r4pGtdY45zINtwIviWE4As4CL6DMJzB17JvQMa0GikTnGzjMazsXLKvvS793d9Ja9fm5vFtbpb+n38wrdZCqb4+X/HURRNZPfhSVKu2JHSkJ+3M5DB3UkA3LKxQY03uBLdSsfdIWg+/EtVaswOwvb822F9iGv/PXFaZr3GcfrOj2WDW/be/La38Xa7X167v//Uv0mWXSQ0N+YpvZ6Og1+tRwITfYo+FBN/B+X3wvXe1evsSzhXZPnDRNOcCFeObqhX0+8wB0fCvK4JvCSH4Ag4GhxUAu/vymWATCnhVFjwDxZdRWTqusv5IrqRiKvOaxwKegeufZAmbYsPtsf2yvbW9vCHn9YfmbxRKsUMQ7PCGY+wi+817OVBdZ9l9WbPuEypLRnPr3t560u+57u36tGN8hz/moJDYz0046Nessxp15cLp+uAlM5zLENurtDFXLwCcGoIvABQYG2ydMb3janX1pbO0zATfZhN67QEyAODUEXwBoMBUV4Z04Tnj9ZefukQfvGSmJjTVMnsDAAwBWlIAKDB22jJ7KWI7zGFUbfmwz94AAG5B8AWAAuP12DH/flWUBZ3Qy5BeABgaBF8AAAC4AsEXAAAArkDwBQAAgCsQfAEAAOAKBF8AAAC4AsEXAAAArkDwBQAAgCsQfAEAAOAKBF8AAAC4AsEXOZmMlE4PXclmzYvakmfvZwaoV+zFLpOzrMfJvse6fGddwO2ctmGI2x/Ku4tdx7YALufJGvmfcQL721N6dF1Mz26MKdafVUXIo0WzQrr1qpp8jSLS3y+tWyfddpu0erUUCkkTJki33CKde65Uc/rLFEl5dP+eaq1orVRb3K+aYFrz6uK68awuNZWlFCihw609vQE9sLdaq9orlEh7VGeWdUlTRMtndOZrHCeZNL+wR/rxj6WXX5Z6eqRRo6Q77pCWLpUaG/MVS1dvLK27Vka0ZntCHb0Z1Vd6NX9KUDdcUqmGKq/8Pq7NW0j64hmt25XQnc9H1N6bVtrkpgunBvWRC8o1f6ppO05WJCI99JB00025+2Vl0pIl0rXXSuefn3sMw8O2N6+8Iv30p9KOHbn2aNmyXNt/zTX5SoPXE8voJfM5/tFzfeZznVHGJImFM0P66PxyzZ0UzNcCCg/BdxBKOvj6/bmwO2tW7tbeP00RX5nuH7dMKxrep7bQKNUkezWve4tuPPCwmhIdCmTT+ZrFb0/ZeD0wdqlWNVyghDekuv5uLel4Wcv3/iJfYwCbNkkHD0qxGMGX4FvQhj34Bk1AGj9emjxZqq7OPYbh090t7dwpHTqU6wUm+MKFCL6DUNLB1/J6pUDAbA0mdNhymiKhKt2/8PNacc5H1VY7QTWRDs3bvVo3rvhXNXUfVCCdzNcsfntGz9QDF31Oq86+SolAmep627Rk06Na/tQ/5WsMwPa02J2O/egRfAm+BWzYg69tb3y+XLHtEIaXHeqQSuXaH4vgCxeipXEbu6MpL5fq6nK3lm0MEwkpHs/1Qp5usa9jG9ffH1OZW9vQ2r8RG6K/USglbpbpWIh1FtXc2mUfqO6xYp+37DCT0aNzvV32K1+g1NmAa79Zmjo119tr2c+D0zYM8FmhDG2x6/lY6D32Xth9AeAi9PgOQkn1+NpGr6VFuvNOaeVKqbV1yE94iAQqdP/Zf6wVzZeprWKMauJdmte6XjduuFNNkVYFMiXU41t7lh6Ydb1WTVqihD+suliHluxdoeWvfj9f4wRsD7vt7bXjqm+9VRo3LvdYiaPHt7gMeY+v/bZj82bprrty3zx1dr51IIgzx/au2/bnqquk667LHYicJHp8UawIvoNQUsHXvt3RaG6cqd0BHT78Vg/AEIlkA7pf52qFZ4raVKUaxTQv26IbtV5N6lPAM7RBeyTtydbpAc3VKk+zEvKrTlEt0W4t19p8jROwvb12aMPMmdI55+R6fIdgmEmhI/gWlyEPvrb9OTbOdMMGqaMjF4ZxZtneXnuwPWeONGOGVFGRf2LwCL4oVgTfQSip4Hs8O97XfvU1xJtAJJHV/euSWrEjo7berGpMpps33qsbFwTUVO1RoITCzZ72jB5Yn9KqXWklUlJduUdLpnm1fPEfaPiPjau2AdhFCL7FZciD7zG2zXnXkCicMfYgOxzOncx8igfcBF8UK4LvIJRs8D321g9H8F0T1YrNcbV1Z1RT7tW8SQHduLhcTTV+E3zzFUvAniMpPbA2plXbEkoks6qr8GrJ7JCWX1GZr3ECx3Y2LujlPR7Bt7gMW/C1hqn9wSAMQftD8EWx4uQ2N7ONni2293Goi9OeHteo2h899rlh+nsjVewyHbeYOYNYxmPrHnCr4Wx/KO9daH/gYuYTAAAAAJQ+gi8AAABcgeALAAAAVyD4AgAAwBUIvgAAAHAFgi8AAABcgeALAAAAVyD4AgAAwBUIvgAAAHAFgi8AAABcgeALAAAAVyD4AgAAwBUIvgAAAHAFgi8AAABcgeALAAAAVyD4AgAAwBU8WSP/M05gf3tKj66L6dmNMcX6s6oIebRoVki3XlWTr4HjRRJZ3f9in1ZsSqitO62acq/mTQ7oxsWVaqr1KeDz5GsWvz1tKT2wNqJVWxNKJLOqq/Bqyeywli+tytfA8Xpjad21MqI12xPq6M2ovtKr+VOCuuGSSjVUeeUvoW2jFPTFM1q3K6E7n4+ovTetdEY6rzmoD5wbNrehfC24UV8so1d3J3TvqqiznWRMklg4M6SPzi/X3EnBfC2g8BB8B4Hge3IIvgTfEyH4FpeBgm/zaL/zeT5rdCBfC24U689oV2tKq7YkFDNtn00SBF8UA4LvIBB8Tw7Bl+B7IgTf4jJQ8PV7pYDfY255r9zMRF2l0lIilQu9FsEXxYAxvgCAAdkDkXDAlvwDRsqEX9sB0GtCMcW9pS+eVTzf02t5zHGQ3VbKghwQobARfAEAAwr4pFFVPs0YF9CoSq8qwx6Vm2BDofy+hDzOdjG+3qdJDX6znZiNBihgDHUYBIY6nByGOjDU4UQY6lB8uqMZbTrQr1d3JdQTyzrDHYDj+bxygu8FU0I6e0LQ6f0FChXBdxAIvieH4EvwPRGCb/HJZLLOtm0D8PHjOYFjbNCtLvOqIuxV0M9nGIWN4DsIBN+TQ/Al+J4Iwbc42b2Ena4qd0oT8Hb2U+sx6dcGYD7BKHSM8QUAvCcbaOzX2T5vbjYHCuX4YrcL8x+hF0WB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFTxZI/8zTmB/e0qProvp2Y0xxfqzqgh5tGhWSLdeVZOvgeNFElnd/2KfVmxKqK07reoyr86ZFND1F5Wrodonv690Lmy5z24br0T10s5+JZJZ1VZ4dfH0kG5+f2W+Bn7PtDSRREYPrI3opR396ujNqL7Sq/lTgrrhkko1VHlLatsAABQegu8gEHxPzjuDr11fzaP9WjQzpCoTgr32ou4l4khP2oS4hLa1pJRM57aNWeMDumJuWb4G3pJ1Dg5e3pnQpv0pdUUJvgCAM4vgOwgE35PzzuDr90rlYY/TG1pKoddKmiBnA1zULLP9JPnMstrto77al6+B49nmJmE+Q52RXAh2gu9UE3wXEnwBAMOP4DsIBN+TY4Pvgy/16bmNCbV2pZ3HbN715EspsZ+ejCnHf4rssnoZPT8wu67MzbF1NsoE3wXTgvqkCb6jTPD1ldiBEQCgsLB7xpCzPbyTRvlVV+FRIN/xaYNOOiOlTA4upWKX6Z2HjnZZB6pLMcWsr2PrzG4b1eVeja33q9wcTHpL7agIAFBw6PEdBHp8T07aJL+Woym9uC2h3W0pReJZp5evFNntobU7ra5Ixgm8NszVVXg1wQR/vLfKsEcTG/ya1xzUjLEBs+4IvgCA4UXwHQSC78mxG1QqldWR3rRz5n40nguFpciOYbYBf0v+5DYb5s4eH9CV8zi57Q+pLPM6wxvsgUI44C25YTAAgMJD8B0Egi9OZE9bypmea9XWhHOylg1xS2aHtXxpVb4GAAAoFIzxBQAAgCsQfAEAAOAKBF8AAAC4AsEXAAAArkDwBQAAgCsQfAEAAOAKBF8AAAC4AvP4DsI75/EN+j2aMtqvS88O52vArTr6Mnp9T0K7j6ScS/Iyjy8AAIWL4DsI+ztSeswE32feyAVfe4GpgAm/5SEuNeV26XRW8WTWCb32g1RfmQu+n/sAwRcAgEJD8B2Eg0fT+s1rUT2+Phd8gRMZXe3VpXPC+vT7Cb4AABQaxvgOQjggNdX65Gdt4Q+oq/RpjNlWAABA4aHHdxD6U1m192a0anNcB46m6fXFgOz43ulj/Tp7YkDj6vz5RwEAQKEg+A5SOiO1dafV2ZdxgjDwTpVhj+qrvKou98rvZfw3AACFhuALAAAAV2DUKgAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAWCLwAAAFyB4AsAAABXIPgCAADAFQi+AAAAcAHp/wd+Z5Os5p5WqgAAAABJRU5ErkJggg==)"
      ],
      "metadata": {
        "id": "uXUZ5nw8IRrg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "After getting all the bounding boxes around the numbers in the image, we use Optical Character Recognition to get the numbers written in the images. We implement this task using the ParSeq (Permuted AutoRegressive SEQuence) model. We are using the pre-trained model of ParSeq, and the details are as follows:\n",
        "\n",
        "  * **Model: ParSeq**\n",
        "  * **Accuracy: 0.95**\n",
        "  * **Mean Inference Time: 1 s**\n"
      ],
      "metadata": {
        "id": "jwJ5dozIIVls"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## Loading models for detection\n",
        "\n",
        "yolo_model_det = yolov5.load(root + '/weights/final_yolo_weights.pt')\n",
        "paddle_ocr_det_acc = PaddleOCR(cpu_threads=1, rec_batch_num=2, rec_algorithm='CRNN', rec_image_inverse=False)\n",
        "model_ocr = torch.hub.load('baudm/parseq', 'parseq', pretrained=True).eval().to(device)\n",
        "yolo_fast = yolov5.load(root + 'weights/yolo_on_6_fast.pt')\n",
        "paddle_fast = PaddleOCR(use_angle_cls=False, lang='en', ocr_version = 'PP-OCR', structure_version = 'PP-Structure', \n",
        "                rec_algorithm = 'CRNN', max_text_length = 200, use_space_char = False, lan = 'en', det = False,\n",
        "                cpu_threads = 12, cls = False,use_gpu=False )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xlRZLZmW13pg",
        "outputId": "da9d8f42-efc4-4c1d-a25e-255e6f254be9"
      },
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[2023/02/07 18:56:15] ppocr DEBUG: Namespace(alpha=1.0, benchmark=False, beta=1.0, cls_batch_num=6, cls_image_shape='3, 48, 192', cls_model_dir='/root/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer', cls_thresh=0.9, cpu_threads=1, crop_res_save_dir='./output', det=True, det_algorithm='DB', det_box_type='quad', det_db_box_thresh=0.6, det_db_score_mode='fast', det_db_thresh=0.3, det_db_unclip_ratio=1.5, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_east_score_thresh=0.8, det_limit_side_len=960, det_limit_type='max', det_model_dir='/root/.paddleocr/whl/det/ch/ch_PP-OCRv3_det_infer', det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_scale=1, det_pse_thresh=0, det_sast_nms_thresh=0.2, det_sast_score_thresh=0.5, draw_img_save_dir='./inference_results', drop_score=0.5, e2e_algorithm='PGNet', e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_limit_side_len=768, e2e_limit_type='max', e2e_model_dir=None, e2e_pgnet_mode='fast', e2e_pgnet_score_thresh=0.5, e2e_pgnet_valid_set='totaltext', enable_mkldnn=False, fourier_degree=5, gpu_mem=500, help='==SUPPRESS==', image_dir=None, image_orientation=False, ir_optim=True, kie_algorithm='LayoutXLM', label_list=['0', '180'], lang='ch', layout=True, layout_dict_path=None, layout_model_dir=None, layout_nms_threshold=0.5, layout_score_threshold=0.5, max_batch_size=10, max_text_length=25, merge_no_span_structure=True, min_subgraph_size=15, mode='structure', ocr=True, ocr_order_method=None, ocr_version='PP-OCRv3', output='./output', page_num=0, precision='fp32', process_id=0, re_model_dir=None, rec=True, rec_algorithm='CRNN', rec_batch_num=2, rec_char_dict_path='/usr/local/lib/python3.8/dist-packages/paddleocr/ppocr/utils/ppocr_keys_v1.txt', rec_image_inverse=False, rec_image_shape='3, 48, 320', rec_model_dir='/root/.paddleocr/whl/rec/ch/ch_PP-OCRv3_rec_infer', recovery=False, save_crop_res=False, save_log_path='./log_output/', scales=[8, 16, 32], ser_dict_path='../train_data/XFUND/class_list_xfun.txt', ser_model_dir=None, show_log=True, sr_batch_num=1, sr_image_shape='3, 32, 128', sr_model_dir=None, structure_version='PP-StructureV2', table=True, table_algorithm='TableAttn', table_char_dict_path=None, table_max_len=488, table_model_dir=None, total_process_num=1, type='ocr', use_angle_cls=False, use_dilation=False, use_gpu=False, use_mp=False, use_npu=False, use_onnx=False, use_pdf2docx_api=False, use_pdserving=False, use_space_char=True, use_tensorrt=False, use_visual_backbone=True, use_xpu=False, vis_font_path='./doc/fonts/simfang.ttf', warmup=False)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Using cache found in /root/.cache/torch/hub/baudm_parseq_main\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "download https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/en_ppocr_mobile_v2.0_det_infer.tar to /root/.paddleocr/whl/det/en/en_ppocr_mobile_v2.0_det_infer/en_ppocr_mobile_v2.0_det_infer.tar\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 3.16M/3.16M [00:14<00:00, 219kiB/s] \n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "download https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/en_number_mobile_v2.0_rec_infer.tar to /root/.paddleocr/whl/rec/en/en_number_mobile_v2.0_rec_infer/en_number_mobile_v2.0_rec_infer.tar\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 2.70M/2.70M [00:13<00:00, 195kiB/s] "
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[2023/02/07 18:56:50] ppocr DEBUG: Namespace(alpha=1.0, benchmark=False, beta=1.0, cls=False, cls_batch_num=6, cls_image_shape='3, 48, 192', cls_model_dir='/root/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer', cls_thresh=0.9, cpu_threads=12, crop_res_save_dir='./output', det=False, det_algorithm='DB', det_box_type='quad', det_db_box_thresh=0.6, det_db_score_mode='fast', det_db_thresh=0.3, det_db_unclip_ratio=1.5, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_east_score_thresh=0.8, det_limit_side_len=960, det_limit_type='max', det_model_dir='/root/.paddleocr/whl/det/en/en_ppocr_mobile_v2.0_det_infer', det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_scale=1, det_pse_thresh=0, det_sast_nms_thresh=0.2, det_sast_score_thresh=0.5, draw_img_save_dir='./inference_results', drop_score=0.5, e2e_algorithm='PGNet', e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_limit_side_len=768, e2e_limit_type='max', e2e_model_dir=None, e2e_pgnet_mode='fast', e2e_pgnet_score_thresh=0.5, e2e_pgnet_valid_set='totaltext', enable_mkldnn=False, fourier_degree=5, gpu_mem=500, help='==SUPPRESS==', image_dir=None, image_orientation=False, ir_optim=True, kie_algorithm='LayoutXLM', label_list=['0', '180'], lan='en', lang='en', layout=True, layout_dict_path=None, layout_model_dir=None, layout_nms_threshold=0.5, layout_score_threshold=0.5, max_batch_size=10, max_text_length=200, merge_no_span_structure=True, min_subgraph_size=15, mode='structure', ocr=True, ocr_order_method=None, ocr_version='PP-OCR', output='./output', page_num=0, precision='fp32', process_id=0, re_model_dir=None, rec=True, rec_algorithm='CRNN', rec_batch_num=6, rec_char_dict_path='/usr/local/lib/python3.8/dist-packages/paddleocr/ppocr/utils/en_dict.txt', rec_image_inverse=True, rec_image_shape='3, 32, 320', rec_model_dir='/root/.paddleocr/whl/rec/en/en_number_mobile_v2.0_rec_infer', recovery=False, save_crop_res=False, save_log_path='./log_output/', scales=[8, 16, 32], ser_dict_path='../train_data/XFUND/class_list_xfun.txt', ser_model_dir=None, show_log=True, sr_batch_num=1, sr_image_shape='3, 32, 128', sr_model_dir=None, structure_version='PP-Structure', table=True, table_algorithm='TableAttn', table_char_dict_path=None, table_max_len=488, table_model_dir=None, total_process_num=1, type='ocr', use_angle_cls=False, use_dilation=False, use_gpu=False, use_mp=False, use_npu=False, use_onnx=False, use_pdf2docx_api=False, use_pdserving=False, use_space_char=False, use_tensorrt=False, use_visual_backbone=True, use_xpu=False, vis_font_path='./doc/fonts/simfang.ttf', warmup=False)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "yolo_model_det.eval();\n",
        "model_ocr.eval();\n",
        "yolo_fast.eval();"
      ],
      "metadata": {
        "id": "WBNM8FCSGdba"
      },
      "execution_count": 62,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Helper Functions\n",
        "def return_fast_output(yolo_model, img):\n",
        "\n",
        "    image = img.copy()\n",
        "\n",
        "    results_yolo = yolo_model(img)\n",
        "\n",
        "    try:\n",
        "        boxes = results_yolo.pred[0][:, :4].tolist()\n",
        "        scores = results_yolo.pred[0][:, 4].tolist()\n",
        "        labels = results_yolo.pred[0][:, 5].tolist()\n",
        "    except:\n",
        "        boxes = []\n",
        "        scores_yolo = []\n",
        "        labels_yolo = []\n",
        "    \n",
        "    dic = {}\n",
        "    for each in labels:\n",
        "        if each not in dic.keys():\n",
        "            dic[each] = (0,[])\n",
        "    \n",
        "    for i in range(len(labels)):\n",
        "        score , box = dic[labels[i]]\n",
        "        if score < scores[i]:\n",
        "            dic[labels[i]] = (scores[i], boxes[i])\n",
        "    \n",
        "    # print(dic)\n",
        "    return dic\n",
        "\n",
        "def recognize_fast(image,dic,rec):\n",
        "\n",
        "    vitals = {}\n",
        "    labels = {0.0: 'DBP' , 1.0:'HR' , 2.0:'MAP', 3.0:'RR' ,4.0:'SBP' ,5.0:'SPO2' }\n",
        "    for each in dic.keys():\n",
        "        score, box = dic[each]\n",
        "        xmin = int(box[0])\n",
        "        xmax = int(box[2])\n",
        "        ymin = int(box[1])\n",
        "        ymax = int(box[3])\n",
        "        img = image[ymin:ymax,xmin:xmax]\n",
        "        text = rec.ocr(img,cls = False,det = False)[0][0][0]\n",
        "        text = text.replace('(','').replace(')','').replace('/','').replace('-','').replace('*','')\n",
        "        if text.isdigit():\n",
        "            vitals[labels[each]] = text\n",
        "\n",
        "    return vitals\n",
        "\n",
        "def return_output(yolo_model, paddle_ocr, img):\n",
        "  image = img.copy()\n",
        "  results_yolo = yolo_model(img)\n",
        "  try:\n",
        "    boxes = results_yolo.pred[0][:, :4].tolist()\n",
        "    scores_yolo = results_yolo.pred[0][:, 4].tolist()\n",
        "    labels_yolo = results_yolo.pred[0][:, 5].tolist()\n",
        "  except:\n",
        "    boxes = []\n",
        "    scores_yolo = []\n",
        "    labels_yolo = []\n",
        "  boxes_yolo = []\n",
        "  for box in boxes:\n",
        "    boxes_yolo.append([box[0]/1280, box[1]/720, box[2]/1280, box[3]/720])\n",
        "  # results_paddle = paddle_ocr.ocr(img, cls=False, rec = True)\n",
        "  # final_boxes_paddle = []\n",
        "  # final_confidence_paddle = []\n",
        "  # final_labels_paddle = [] \n",
        "  # for i in range(len(results_paddle[0])):\n",
        "  #   xmin = results_paddle[0][i][0][0][0]/1280.\n",
        "  #   ymin = results_paddle[0][i][0][0][1]/720.\n",
        "  #   xmax = results_paddle[0][i][0][2][0]/1280.\n",
        "  #   ymax = results_paddle[0][i][0][2][1]/720.\n",
        "  #   temp = [xmin, ymin, xmax, ymax]\n",
        "\n",
        "  #   # area\n",
        "  #   if (xmax*1280 - xmin*1280)*(ymax*720 - ymin*720) < 1000:\n",
        "  #     continue\n",
        "    \n",
        "  #   if (xmax*1280 - xmin*1280) < 120:\n",
        "  #     continue\n",
        "  #   if (xmax*1280 - xmin*1280) > 300:\n",
        "  #     continue\n",
        "  #   final_boxes_paddle.append(temp)\n",
        "  #   final_confidence_paddle.append(results_paddle[0][i][1][1])\n",
        "  # for i in range(len(final_confidence_paddle)):\n",
        "  #   final_labels_paddle.append(0)\n",
        "  result_box = boxes_yolo\n",
        "  result_conf = scores_yolo\n",
        "  result_label = labels_yolo\n",
        "  return result_box, result_conf, result_label, img\n",
        "  \n",
        "################################################################################\n",
        "\n",
        "def wbf_ensemble(boxes_list, scores_list, labels_list, image):\n",
        "  weights = [2, 1]\n",
        "  iou_thr = 0.6\n",
        "  skip_box_thr = 0.01\n",
        "  sigma = 0.1\n",
        "  boxes, scores, labels = weighted_boxes_fusion(boxes_list, scores_list, labels_list, weights=weights, iou_thr=iou_thr, skip_box_thr=skip_box_thr)\n",
        "  return boxes, scores\n",
        "\n",
        "def recognize(image,boxes,scores):\n",
        "    imgs = []\n",
        "    for box in boxes:\n",
        "        xmin = int(box[0]*1280)\n",
        "        ymin = int(box[1]*720)\n",
        "        xmax = int(box[2]*1280)\n",
        "        ymax = int(box[3]*720)\n",
        "        img = image[ymin:ymax,xmin:xmax]\n",
        "        imgs.append(img)\n",
        "    \n",
        "    procs = [preproc_image(img) for img in imgs]\n",
        "    preds = model_ocr(torch.cat(procs, dim=0))\n",
        "    labels = inference_pred(preds)\n",
        "    \n",
        "    return labels,image,boxes,scores\n",
        "\n",
        "def preproc_image(img):\n",
        "    img = Image.fromarray(img).convert('RGB')\n",
        "    transform = T.Compose([\n",
        "            T.Resize((32, 128)),\n",
        "            T.ToTensor(),\n",
        "            T.Normalize(0.5, 0.5)\n",
        "        ])\n",
        "    img = transform(img)\n",
        "    return img.unsqueeze(0)\n",
        "\n",
        "def inference_pred(pred):\n",
        "    pred = pred.softmax(-1)\n",
        "    label, _ = model_ocr.tokenizer.decode(pred)\n",
        "    return label"
      ],
      "metadata": {
        "id": "6rSob5U03g2D"
      },
      "execution_count": 113,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def check_string(string):\n",
        "  if(('(' in string or ')' in string) and '/' in string):\n",
        "      return False\n",
        "  elif (string.count('(') > 1 or string.count('/') > 1 or string.count(')') > 1):\n",
        "      return False\n",
        "  string = string.replace('(', '').replace('/', '').replace(')', '')\n",
        "  pattern = r'^[\\d]+$'\n",
        "  return re.match(pattern, string) != None\n",
        "\n",
        "#################\n",
        "\n",
        "\n",
        "def image_dict(text , boxes , scores, image):\n",
        "  c = 0\n",
        "  text_l = []\n",
        "  boxes_l = []\n",
        "  scores_l = []\n",
        "\n",
        "  for i, t in enumerate(text):\n",
        "    if check_string(t):\n",
        "      text_l.append(t)\n",
        "      boxes_l.append(boxes[i])\n",
        "      scores_l.append(scores[i])\n",
        "\n",
        "  boxes_l, scores_l = np.array(boxes_l), np.array(scores_l)\n",
        "  nums = np.array([float(txt.replace('(', '').replace('/', '').replace(')', '')) for txt in text_l])\n",
        "  try:\n",
        "    ind = np.argsort(scores_l)[-6:]\n",
        "    scores_l = scores_l[ind]\n",
        "    text_l = [text_l[x] for x in ind]\n",
        "    boxes_l = boxes_l[ind]\n",
        "    nums = nums[ind]\n",
        "  except:\n",
        "    pass\n",
        "  boxes_dic = []\n",
        "  for i,num in enumerate(text_l):\n",
        "    bbxi = boxes_l[i]\n",
        "    nm = nums[i]\n",
        "    text_data = np.array([0.0, 0.0, 0.0])\n",
        "    if '/' in num:\n",
        "      text_data[0] = 1.0\n",
        "    if '(' in num:\n",
        "      text_data[1] = 1.0\n",
        "    if ')' in num:\n",
        "      text_data[2] = 1.0\n",
        "    boxes_dic.append({'bbox': bbxi, 'num': nm, 'text_data': text_data})\n",
        "    \n",
        "  return {'image': image, 'val_vec': nums.tolist(), 'boxes': boxes_dic}"
      ],
      "metadata": {
        "id": "eK3rLVAc6qGQ"
      },
      "execution_count": 88,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def number_detection(img, mode = 'accurate'):\n",
        "\n",
        "  if mode == 'accurate':\n",
        "    \n",
        "    boxes, scores, result_label, img = return_output(yolo_model_det, paddle_ocr_det_acc, img)\n",
        "    # result_box, result_conf, result_label, img = return_output(yolo_model_det, paddle_ocr_det_acc, img)\n",
        "\n",
        "    # boxes, scores, img = wbf_ensemble(result_box, result_conf, result_label, img)\n",
        "\n",
        "    text , img, boxes , scores = recognize(img, boxes,scores)\n",
        "\n",
        "    number_dict = image_dict(text , boxes , scores, img)\n",
        "\n",
        "  else:\n",
        "\n",
        "    temp = return_fast_output(yolo_fast, img)\n",
        "\n",
        "    number_dict = recognize_fast(img, temp, paddle_fast)\n",
        "\n",
        "  return number_dict"
      ],
      "metadata": {
        "id": "-fTXJ6NQ32aB"
      },
      "execution_count": 117,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "An-1KX5j7Ufh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Number Classification"
      ],
      "metadata": {
        "id": "Yiuk34Bd7gE0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This task consists of classifying all the numbers detected by OCR into their respective classes. To implement this task, we have made **CRABBNet (Custom Recognition Assisted Bounding Box classification Network)**.  \n",
        "  CRABBNet takes 3 inputs, a 4-channel Image, all numbers on the screen, and the target number which needs to be classified. \n",
        "\n",
        "  The working of CRABBNet is explained below:\n",
        "  \n",
        "  * The screen-extracted image (of 3 RGB channels) on which the prediction has to be made is concatenated with a 1-channel mask of the bounding box of the target number to produce a 4-channel input for the network.\n",
        "\n",
        "  * This 4-channel input is passed to a ResNeXt - 50 model, which produces a 2048-length feature vector of the image. \n",
        "A 14-length vector is created, which contains information about all the numbers on the screen.\n",
        "\n",
        "  * The 10 numbers in the vector denote the 10 numbers which are present on the screen. If less numbers are present, the remaining values are filled with 0\n",
        "The next 3 numbers are binary values, indicating presence of ‘/’, ‘(‘ and ‘)’  in our target number which is to be classified. This is because, many a times in prediction of DBP and SBP, ‘/’ is seen in the image, while in prediction of MAP, ‘(‘ and ‘)’ are found.\n",
        "\n",
        "* The last number of the vector is the target number itself.\n",
        "\n",
        "* This 14-length vector is then passed to a linear layer, which converts this into 6-length vector. \n",
        "* This 6-length vector, is concatenated with the previous 2048-length feature vector, thereby resulting in 2054-length vector.\n",
        "* This 2054-length vector is again concatenated with the target number, to give more weight to the target number, informing model to focus on predicting the target number.\n",
        "* Hence, after this step, we would result in 2055-length vector.\n",
        "This 2055-length vector is then passed again to a linear layer, with softmax activation, which gives probability of 6 classes among which the number would be present.\n",
        "\n",
        "    * **Training Methodology**:\n",
        "        * **Model : Resnext50_32x4d**\n",
        "        * **Epochs : 15**\n",
        "        * **Learning_rate : 0.001**\n",
        "        * **Loss : CrossEntropy** \n",
        "        * **Accuracy :    77% (Out-of-fold Layout Validation) and 98% (Single Layout training)**\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "Y_TnvdPvIbvz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Custom Logit-Decoder for Multi-label Matching**:\n",
        "\n",
        "Using the above model we generate 6-class predictions for each bounding box on the screen. To combine predictions from multiple bounding boxes in multiple classes while ensuring single prediction for each vital, we use a customized probability decoder.\n",
        "\n",
        "\n",
        "We use variance measure across each class prediction to get the most confident class and decide the box prediction for that class. This step is repeated on each class in the order of decreasing variance while simultaneously already selected bounding boxes to ensure a one-to-one mapping of bounding boxes and classes.\n"
      ],
      "metadata": {
        "id": "e7lrNh5MIgW8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CRABBNET(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.model = timm.create_model('resnext50_32x4d', pretrained=True, num_classes=0, in_chans=4)\n",
        "        self.ll = nn.Linear(2048+7,6)\n",
        "        self.fl = nn.Linear(14,6)\n",
        "    \n",
        "    def forward(self, img, text_data, num, val_vec):\n",
        "        x = self.model(img)\n",
        "        x = torch.cat([x, num.view(-1, 1), self.fl(torch.cat([num.view(-1, 1), text_data.view(-1, 3), val_vec.view(-1, 10)], dim=1))], dim=1)\n",
        "        x = self.ll(x)\n",
        "        x = x.view(-1, 6)\n",
        "#         x = F.softmax(x, dim = 1)\n",
        "        \n",
        "        return x"
      ],
      "metadata": {
        "id": "8_Z7e3Um7iPH"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_crabb = CRABBNET()\n",
        "model_crabb = model_crabb.to(device)\n",
        "model_crabb.load_state_dict(torch.load(root + '/weights/crabbnet.pt', map_location = device))\n",
        "model_crabb.eval();"
      ],
      "metadata": {
        "id": "nFLVM4tC7wOa"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class InferDataset(Dataset):\n",
        "    def __init__(self, img_dict):\n",
        "        super().__init__()\n",
        "        \n",
        "        self.img_dict = img_dict\n",
        "                          \n",
        "    def __getitem__(self, idx):\n",
        "        img = self.img_dict['image']\n",
        "        val_vec = np.array(self.img_dict['val_vec'])\n",
        "        val_vec.resize(10,)\n",
        "        np.random.shuffle(val_vec)\n",
        "        \n",
        "        boxes = self.img_dict['boxes'][idx]  \n",
        "        bbox = boxes['bbox']\n",
        "        \n",
        "        xmin = int(bbox[0]*1280)\n",
        "        ymin = int(bbox[1]*720)\n",
        "        xmax = int(bbox[2]*1280)\n",
        "        ymax = int(bbox[3]*720)\n",
        "\n",
        "        mask = np.zeros((img.shape[0], img.shape[1], 1))\n",
        "        mask = cv2.rectangle((mask), (xmin, ymin), (xmax, ymax), 255, -1)\n",
        "\n",
        "        # plt.imshow(mask[:,:,0])\n",
        "        # plt.show()\n",
        "        \n",
        "        num = boxes['num']\n",
        "        text_data = boxes['text_data']\n",
        "\n",
        "        img = cv2.resize(img, (224, 224))\n",
        "        mask = cv2.resize(mask, (224,224))[:,:,None]\n",
        "                \n",
        "        arr4 = np.concatenate([img, mask], axis = 2)\n",
        "                   \n",
        "        arr4 = (np.transpose(arr4, (2, 0, 1))) / 255.0\n",
        "        arr4 = torch.tensor(arr4)\n",
        "        val_vec = torch.tensor(val_vec)\n",
        "                \n",
        "        return arr4, torch.tensor(num), torch.tensor(text_data), val_vec\n",
        "    \n",
        "    def __len__(self):\n",
        "        return len(self.img_dict['boxes'])\n",
        "\n",
        "########################\n",
        "\n",
        "def pred_organizing(pred_mat, nums):\n",
        "    num_boxes = nums.shape[0]\n",
        "    box_store = set(range(num_boxes))\n",
        "    mat_variance = torch.var(pred_mat, dim=0)\n",
        "    argmax_dim0 = torch.argmax(pred_mat, dim=0)\n",
        "    sorted_, indices = torch.sort(mat_variance, descending=True)\n",
        "    \n",
        "    res_dict = {'HR':None, 'RR':None, 'SPO2':None, 'SBP':None, 'DBP':None, 'MAP':None}\n",
        "    label_cols = ['HR', 'RR', 'SPO2', 'SBP', 'DBP', 'MAP']\n",
        "    \n",
        "    for ind in indices:\n",
        "        \n",
        "        argmax_dim0 = torch.argmax(pred_mat, dim=0)\n",
        "        box_ind = argmax_dim0[ind].item()\n",
        "\n",
        "        if box_ind not in box_store:\n",
        "            present = list(box_store)\n",
        "            if len(present) == 0:\n",
        "                continue\n",
        "            \n",
        "        res_dict[label_cols[ind]] = nums[box_ind]\n",
        "        pred_mat[box_ind, :] = -100000\n",
        "    \n",
        "        if torch.unique(pred_mat).shape[0] == 1:\n",
        "            break\n",
        "    \n",
        "    return res_dict\n",
        "\n",
        "############################\n",
        "\n",
        "def final_inference(img_dict):\n",
        "    test_img = InferDataset(img_dict)\n",
        "\n",
        "    test_loader = DataLoader(\n",
        "            dataset=test_img,\n",
        "            batch_size = len(img_dict['boxes']),\n",
        "            num_workers = 0,\n",
        "    )\n",
        "\n",
        "\n",
        "    imgs, nums, text_data, val_vec = next(iter(test_loader))\n",
        "\n",
        "    imgs = imgs.float()\n",
        "    nums = nums.float()\n",
        "    text_data = text_data.float()\n",
        "    val_vec = val_vec.float()\n",
        "\n",
        "    with torch.no_grad():\n",
        "        label_preds = model_crabb(imgs, text_data, nums, val_vec)\n",
        "\n",
        "    nums = nums.numpy()\n",
        "\n",
        "    yerr_dict = pred_organizing(label_preds, nums)\n",
        "    yerr_dict = {k:v for k,v in yerr_dict.items() if v is not None}\n",
        "\n",
        "    return yerr_dict\n",
        "\n",
        "#############################"
      ],
      "metadata": {
        "id": "a9-4PZun722n"
      },
      "execution_count": 118,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "2MYi-Xjk9Xf4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Final Inference"
      ],
      "metadata": {
        "id": "E1gLNFZZ9btR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def vital_extraction(img_path, mode = 'accurate'):\n",
        "\n",
        "  if mode == 'accurate':\n",
        "\n",
        "    img = cv2.imread(img_path)\n",
        "    transformed_image = screen_extraction(img_path, model_unet, model_reg, preprocessing_unet, mode)\n",
        "\n",
        "    detection_dict = number_detection(transformed_image, mode)\n",
        "\n",
        "    output_dict = final_inference(detection_dict)\n",
        "\n",
        "    return output_dict\n",
        "\n",
        "  else:\n",
        "    \n",
        "    img = cv2.imread(img_path)\n",
        "\n",
        "    transformed_image = screen_extraction(img_path, model_unet, model_reg, preprocessing_unet, mode)\n",
        "\n",
        "    detection_dict = number_detection(transformed_image, mode)\n",
        "\n",
        "    return detection_dict"
      ],
      "metadata": {
        "id": "dY_ctVF89eY8"
      },
      "execution_count": 119,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "XAiTOyJiSGS4"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}