PIFuHD_Demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hazael00/AndroidStudio/blob/master/PIFuHD_Demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eclLG4xlJRIE"
      },
      "source": [
        "# PIFuHD Demo: https://shunsukesaito.github.io/PIFuHD/\n",
        "\n",
        "![](https://shunsukesaito.github.io/PIFuHD/resources/images/pifuhd.gif)\n",
        "\n",
        "Made by [![Follow](https://img.shields.io/twitter/follow/psyth91?style=social)](https://twitter.com/psyth91)\n",
        "\n",
        "To see how the model works, visit the project repository.\n",
        "\n",
        "[![GitHub stars](https://img.shields.io/github/stars/facebookresearch/pifuhd?style=social)](https://github.com/facebookresearch/pifuhd)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmFdsTvLKtBO"
      },
      "source": [
        "## Note\n",
        "Make sure that your runtime type is 'Python 3 with GPU acceleration'. To do so, go to Edit > Notebook settings > Hardware Accelerator > Select \"GPU\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1TfPAtL4CyZw"
      },
      "source": [
        "## More Info\n",
        "- Paper: https://arxiv.org/pdf/2004.00452.pdf\n",
        "- Repo: https://github.com/facebookresearch/pifuhd\n",
        "- Project Page: https://shunsukesaito.github.io/PIFuHD/\n",
        "- 1-minute/5-minute Presentation (see below)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5DDpqpf2BABR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 931
        },
        "outputId": "2b2388c7-1dfc-4828-fab7-98b2b46c2818"
      },
      "source": [
        "import IPython\n",
        "IPython.display.HTML('<h2>1-Minute Presentation</h2><iframe width=\"720\" height=\"405\" src=\"https://www.youtube.com/embed/-1XYTmm8HhE\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe><br><h2>5-Minute Presentation</h2><iframe width=\"720\" height=\"405\" src=\"https://www.youtube.com/embed/uEDqCxvF5yc\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>')"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<h2>1-Minute Presentation</h2><iframe width=\"720\" height=\"405\" src=\"https://www.youtube.com/embed/-1XYTmm8HhE\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe><br><h2>5-Minute Presentation</h2><iframe width=\"720\" height=\"405\" src=\"https://www.youtube.com/embed/uEDqCxvF5yc\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8vZaAyhUJ9QC"
      },
      "source": [
        "## Requirements\n",
        "- Python 3\n",
        "- PyTorch tested on 1.4.0\n",
        "- json\n",
        "- PIL\n",
        "- skimage\n",
        "- tqdm\n",
        "- numpy\n",
        "- cv2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sfPDep8LlP_I"
      },
      "source": [
        "## Help! I'm new to Google Colab\n",
        "\n",
        "You can check out the following youtube video on how to upload your own picture and run PIFuHD. **Note that with new update, you can upload your own picture more easily with GUI down below.**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zaMP1EitljaA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "bee1da79-d1c0-4238-8472-cada6001c600"
      },
      "source": [
        "import IPython\n",
        "IPython.display.HTML('<iframe width=\"720\" height=\"405\" src=\"https://www.youtube.com/embed/LWDGR5v3-3o\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>')"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/IPython/core/display.py:701: UserWarning: Consider using IPython.display.IFrame instead\n",
            "  warnings.warn(\"Consider using IPython.display.IFrame instead\")\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<iframe width=\"720\" height=\"405\" src=\"https://www.youtube.com/embed/LWDGR5v3-3o\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WYhlsDkg1Hwb"
      },
      "source": [
        "## Clone PIFuHD repository"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BmpEwdOd1G1z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "75269f08-32a9-4613-d2d8-57bb0110034a"
      },
      "source": [
        "!git clone https://github.com/facebookresearch/pifuhd"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'pifuhd'...\n",
            "remote: Enumerating objects: 216, done.\u001b[K\n",
            "remote: Counting objects: 100% (3/3), done.\u001b[K\n",
            "remote: Compressing objects: 100% (3/3), done.\u001b[K\n",
            "remote: Total 216 (delta 0), reused 0 (delta 0), pack-reused 213\u001b[K\n",
            "Receiving objects: 100% (216/216), 406.47 KiB | 3.66 MiB/s, done.\n",
            "Resolving deltas: 100% (104/104), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QvQm-A8ESKb2"
      },
      "source": [
        "## Configure input data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xvle9T10fB6g",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6717fc36-b575-4823-ba11-ad2253e3db52"
      },
      "source": [
        "cd /content/pifuhd/sample_images"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/pifuhd/sample_images\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9SI7Ye1JfIim"
      },
      "source": [
        "**If you want to upload your own picture, run the next cell**. Otherwise, go to the next next cell. Currently PNG, JPEG files are supported."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jaV_7Yi8fM-B",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 38
        },
        "outputId": "109bcf32-6f19-4ef9-9d19-ae17a59762ff"
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "filename = list(files.upload().keys())[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-7061e13d-aa8e-446f-8429-f26a1b321c7e\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-7061e13d-aa8e-446f-8429-f26a1b321c7e\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AEzmmB01SOZp"
      },
      "source": [
        "import os\n",
        "\n",
        "try:\n",
        "  image_path = '/content/pifuhd/sample_images/%s' % filename\n",
        "except:\n",
        "  image_path = '/content/pifuhd/sample_images/test.png' # example image\n",
        "image_dir = os.path.dirname(image_path)\n",
        "file_name = os.path.splitext(os.path.basename(image_path))[0]\n",
        "\n",
        "# output pathes\n",
        "obj_path = '/content/pifuhd/results/pifuhd_final/recon/result_%s_256.obj' % file_name\n",
        "out_img_path = '/content/pifuhd/results/pifuhd_final/recon/result_%s_256.png' % file_name\n",
        "video_path = '/content/pifuhd/results/pifuhd_final/recon/result_%s_256.mp4' % file_name\n",
        "video_display_path = '/content/pifuhd/results/pifuhd_final/result_%s_256_display.mp4' % file_name"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "896EC7iQfXkj"
      },
      "source": [
        "cd /content"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JbVmda9J5TDL"
      },
      "source": [
        "## Preprocess (for cropping image)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UtMjWGNU5STe"
      },
      "source": [
        "!git clone https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F-vYklhI5dab"
      },
      "source": [
        "cd /content/lightweight-human-pose-estimation.pytorch/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dRod9SOu77I6"
      },
      "source": [
        "!wget https://download.01.org/opencv/openvino_training_extensions/models/human_pose_estimation/checkpoint_iter_370000.pth"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PdRcDXe38lHB"
      },
      "source": [
        "import torch\n",
        "import cv2\n",
        "import numpy as np\n",
        "from models.with_mobilenet import PoseEstimationWithMobileNet\n",
        "from modules.keypoints import extract_keypoints, group_keypoints\n",
        "from modules.load_state import load_state\n",
        "from modules.pose import Pose, track_poses\n",
        "import demo\n",
        "\n",
        "def get_rect(net, images, height_size):\n",
        "    net = net.eval()\n",
        "\n",
        "    stride = 8\n",
        "    upsample_ratio = 4\n",
        "    num_keypoints = Pose.num_kpts\n",
        "    previous_poses = []\n",
        "    delay = 33\n",
        "    for image in images:\n",
        "        rect_path = image.replace('.%s' % (image.split('.')[-1]), '_rect.txt')\n",
        "        img = cv2.imread(image, cv2.IMREAD_COLOR)\n",
        "        orig_img = img.copy()\n",
        "        orig_img = img.copy()\n",
        "        heatmaps, pafs, scale, pad = demo.infer_fast(net, img, height_size, stride, upsample_ratio, cpu=False)\n",
        "\n",
        "        total_keypoints_num = 0\n",
        "        all_keypoints_by_type = []\n",
        "        for kpt_idx in range(num_keypoints):  # 19th for bg\n",
        "            total_keypoints_num += extract_keypoints(heatmaps[:, :, kpt_idx], all_keypoints_by_type, total_keypoints_num)\n",
        "\n",
        "        pose_entries, all_keypoints = group_keypoints(all_keypoints_by_type, pafs)\n",
        "        for kpt_id in range(all_keypoints.shape[0]):\n",
        "            all_keypoints[kpt_id, 0] = (all_keypoints[kpt_id, 0] * stride / upsample_ratio - pad[1]) / scale\n",
        "            all_keypoints[kpt_id, 1] = (all_keypoints[kpt_id, 1] * stride / upsample_ratio - pad[0]) / scale\n",
        "        current_poses = []\n",
        "\n",
        "        rects = []\n",
        "        for n in range(len(pose_entries)):\n",
        "            if len(pose_entries[n]) == 0:\n",
        "                continue\n",
        "            pose_keypoints = np.ones((num_keypoints, 2), dtype=np.int32) * -1\n",
        "            valid_keypoints = []\n",
        "            for kpt_id in range(num_keypoints):\n",
        "                if pose_entries[n][kpt_id] != -1.0:  # keypoint was found\n",
        "                    pose_keypoints[kpt_id, 0] = int(all_keypoints[int(pose_entries[n][kpt_id]), 0])\n",
        "                    pose_keypoints[kpt_id, 1] = int(all_keypoints[int(pose_entries[n][kpt_id]), 1])\n",
        "                    valid_keypoints.append([pose_keypoints[kpt_id, 0], pose_keypoints[kpt_id, 1]])\n",
        "            valid_keypoints = np.array(valid_keypoints)\n",
        "            \n",
        "            if pose_entries[n][10] != -1.0 or pose_entries[n][13] != -1.0:\n",
        "              pmin = valid_keypoints.min(0)\n",
        "              pmax = valid_keypoints.max(0)\n",
        "\n",
        "              center = (0.5 * (pmax[:2] + pmin[:2])).astype(np.int)\n",
        "              radius = int(0.65 * max(pmax[0]-pmin[0], pmax[1]-pmin[1]))\n",
        "            elif pose_entries[n][10] == -1.0 and pose_entries[n][13] == -1.0 and pose_entries[n][8] != -1.0 and pose_entries[n][11] != -1.0:\n",
        "              # if leg is missing, use pelvis to get cropping\n",
        "              center = (0.5 * (pose_keypoints[8] + pose_keypoints[11])).astype(np.int)\n",
        "              radius = int(1.45*np.sqrt(((center[None,:] - valid_keypoints)**2).sum(1)).max(0))\n",
        "              center[1] += int(0.05*radius)\n",
        "            else:\n",
        "              center = np.array([img.shape[1]//2,img.shape[0]//2])\n",
        "              radius = max(img.shape[1]//2,img.shape[0]//2)\n",
        "\n",
        "            x1 = center[0] - radius\n",
        "            y1 = center[1] - radius\n",
        "\n",
        "            rects.append([x1, y1, 2*radius, 2*radius])\n",
        "\n",
        "        np.savetxt(rect_path, np.array(rects), fmt='%d')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M6cGZD6f6IaY"
      },
      "source": [
        "net = PoseEstimationWithMobileNet()\n",
        "checkpoint = torch.load('checkpoint_iter_370000.pth', map_location='cpu')\n",
        "load_state(net, checkpoint)\n",
        "\n",
        "get_rect(net.cuda(), [image_path], 512)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y0rgMInwTt0s"
      },
      "source": [
        "## Download the Pretrained Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UrIcZweSNRFI"
      },
      "source": [
        "cd /content/pifuhd/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k3jjm6HuQRk8"
      },
      "source": [
        "!sh ./scripts/download_trained_model.sh"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6heKcA-0QEBw"
      },
      "source": [
        "## Run PIFuHD!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5995t2PnQTmG"
      },
      "source": [
        "# Warning: all images with the corresponding rectangle files under -i will be processed. \n",
        "!python -m apps.simple_test -r 256 --use_rect -i $image_dir\n",
        "\n",
        "# seems that 256 is the maximum resolution that can fit into Google Colab. \n",
        "# If you want to reconstruct a higher-resolution mesh, please try with your own machine. "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EUZ8Nt5rNFXZ"
      },
      "source": [
        "## Render the result"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5xp5s5uiOiDv"
      },
      "source": [
        "!pip install pytorch3d\n",
        "\n",
        "# If you get an error in the next cell, you can instead try the following command (don't forget to comment out the one above!).\n",
        "# Note that this error is caused by inconsistent cuda version for the pytorch3d package and pytorch in Colab environment.\n",
        "# Thus, this issue may persist unless pytorch3d in the pip package is updated with the cuda version consistent with pytorch in Colab.\n",
        "# Also please be aware that the following command is much slower as it builds pytorch3d from scratch.\n",
        "\n",
        "# !pip install 'git+https://github.com/facebookresearch/pytorch3d.git@stable'\n",
        "\n",
        "# You can try another solution below as well. This is also slow and requires you to restart the runtime.\n",
        "\n",
        "# !pip install 'torch==1.6.0+cu101' -f https://download.pytorch.org/whl/torch_stable.html\n",
        "# !pip install 'torchvision==0.7.0+cu101' -f https://download.pytorch.org/whl/torch_stable.html\n",
        "# !pip install 'pytorch3d==0.2.5'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "afwL_-ROCmDf"
      },
      "source": [
        "from lib.colab_util import generate_video_from_obj, set_renderer, video\n",
        "\n",
        "renderer = set_renderer()\n",
        "generate_video_from_obj(obj_path, out_img_path, video_path, renderer)\n",
        "\n",
        "# we cannot play a mp4 video generated by cv2\n",
        "!ffmpeg -i $video_path -vcodec libx264 $video_display_path -y -loglevel quiet\n",
        "video(video_display_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eUEXAvcvkVYV"
      },
      "source": [
        "## Tips for Inputs: My results are broken!\n",
        "\n",
        "(Kudos to those who share results on twitter with [#pifuhd](https://twitter.com/search?q=%23pifuhd&src=recent_search_click&f=live) tag!!!!)\n",
        "\n",
        "Due to the limited variation in the training data, your results might be broken sometimes. Here I share some useful tips to get resonable results. \n",
        "\n",
        "*   Use high-res image. The model is trained with 1024x1024 images. Use at least 512x512 with fine-details. Low-res images and JPEG artifacts may result in unsatisfactory results. \n",
        "*   Use an image with a single person. If the image contain multiple people, reconstruction quality is likely degraded.\n",
        "*   Front facing with standing works best (or with fashion pose)\n",
        "*   The entire body is covered within the image. (Note: now missing legs is partially supported)\n",
        "*   Make sure the input image is well lit. Exteremy dark or bright image and strong shadow often create artifacts.\n",
        "*   I recommend nearly parallel camera angle to the ground. High camera height may result in distorted legs or high heels. \n",
        "*   If the background is cluttered, use less complex background or try removing it using https://www.remove.bg/ before processing.\n",
        "*   It's trained with human only. Anime characters may not work well (To my surprise, indeed many people tried it!!).\n",
        "*   Search on twitter with [#pifuhd](https://twitter.com/search?q=%23pifuhd&src=recent_search_click&f=live) tag to get a better sense of what succeeds and what fails. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u6U0K5CNAO_u"
      },
      "source": [
        "## Share your result! \n",
        "Please share your results with[ #pifuhd](https://twitter.com/search?q=%23pifuhd&src=recent_search_click&f=live) tag on Twitter. Sharing your good/bad results helps and encourages the authors to further push towards producition-quality human digitization at home.\n",
        "**As the tweet buttom below doesn't add the result video automatically, please download the result video above and manually add it to the tweet.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1CBxbdrM9F-9"
      },
      "source": [
        "import IPython\n",
        "IPython.display.HTML('<a href=\"https://twitter.com/intent/tweet?button_hashtag=pifuhd&ref_src=twsrc%5Etfw\" class=\"twitter-hashtag-button\" data-size=\"large\" data-text=\"Google Colab Link: \" data-url=\"https://bit.ly/37sfogZ\" data-show-count=\"false\">Tweet #pifuhd</a><script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>  (Don\\'t forget to add your result to the tweet!)')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2d-1pR8UR7PR"
      },
      "source": [
        "## Cool Applications\n",
        "Special thanks to those who play with PIFuHD and came up with many creative applications!! If you made any cool applications, please tweet your demo with [#pifuhd](https://twitter.com/search?q=%23pifuhd&src=recent_search_click&f=live). I'm constantly checking results there.\n",
        "If you need complete texture on the mesh, please try my previous work [PIFu](https://github.com/shunsukesaito/PIFu) as well! It supports 3D reconstruction + texturing from a single image although the geometry quality may not be as good as PIFuHD."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68JDAYJFSFMV"
      },
      "source": [
        "IPython.display.HTML('<h2>Rigging (Mixamo) + Photoreal Rendering (Blender)</h2><blockquote class=\"twitter-tweet\"><p lang=\"pt\" dir=\"ltr\">vcs ainda tem a PACHORRA de me dizer que eu não sei dançar<a href=\"https://twitter.com/hashtag/b3d?src=hash&amp;ref_src=twsrc%5Etfw\">#b3d</a> <a href=\"https://twitter.com/hashtag/pifuhd?src=hash&amp;ref_src=twsrc%5Etfw\">#pifuhd</a> <a href=\"https://t.co/kHCnLh6zxH\">pic.twitter.com/kHCnLh6zxH</a></p>&mdash; lukas arendero (@lukazvd) <a href=\"https://twitter.com/lukazvd/status/1274810484798128131?ref_src=twsrc%5Etfw\">June 21, 2020</a></blockquote> <script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script><h2>FaceApp + Rigging (Mixamo)</h2><blockquote class=\"twitter-tweet\"><p lang=\"ja\" dir=\"ltr\">カツラかぶってる自分に見える <a href=\"https://twitter.com/hashtag/pifuhd?src=hash&amp;ref_src=twsrc%5Etfw\">#pifuhd</a> <a href=\"https://t.co/V8o7VduTiG\">pic.twitter.com/V8o7VduTiG</a></p>&mdash; Shuhei Tsuchida (@shuhei2306) <a href=\"https://twitter.com/shuhei2306/status/1274507242910314498?ref_src=twsrc%5Etfw\">June 21, 2020</a></blockquote> <script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script><h2>Rigging (Mixamo) + AR (Adobe Aero)</AR><blockquote class=\"twitter-tweet\"><p lang=\"ja\" dir=\"ltr\">写真→PIFuHD→Mixamo→AdobeAeroでサウンド付きARを作成。Zip化してLINEでARコンテンツを共有。<br>写真が1枚あれば簡単にARの3Dアニメーションが作れる時代…凄い。<a href=\"https://twitter.com/hashtag/PIFuHD?src=hash&amp;ref_src=twsrc%5Etfw\">#PIFuHD</a> <a href=\"https://twitter.com/hashtag/AdobeAero?src=hash&amp;ref_src=twsrc%5Etfw\">#AdobeAero</a> <a href=\"https://twitter.com/hashtag/Mixamo?src=hash&amp;ref_src=twsrc%5Etfw\">#Mixamo</a> <a href=\"https://t.co/CbiMi4gZ0K\">pic.twitter.com/CbiMi4gZ0K</a></p>&mdash; モジョン (@mojon1) <a href=\"https://twitter.com/mojon1/status/1273217947872317441?ref_src=twsrc%5Etfw\">June 17, 2020</a></blockquote> <script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script><h2>3D Printing</h2><blockquote class=\"twitter-tweet\"><p lang=\"ja\" dir=\"ltr\"><a href=\"https://twitter.com/hashtag/pifuhd?src=hash&amp;ref_src=twsrc%5Etfw\">#pifuhd</a> 楽しい〜<br>小さい自分プリントした <a href=\"https://t.co/4qyWuij0Hs\">pic.twitter.com/4qyWuij0Hs</a></p>&mdash; isb (@vxzxzxzxv) <a href=\"https://twitter.com/vxzxzxzxv/status/1273136266406694913?ref_src=twsrc%5Etfw\">June 17, 2020</a></blockquote> <script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lX5CTTW_KWhQ"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    }
  ]
}