notebooks/compiler_caffe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the xfDNN Compiler w/ a Caffe Model\n",
    "\n",
    "## Introduction\n",
    "In this notebook, you will learn what steps are required to prepare and compile a model. Before being able to deploy networks/models to Xilinx FPGAs you will need to compile them.  \n",
    "\n",
    "This step is more than just converting the framework graph representation to one Xilinx can execute. The xfDNN Compiler is a high performance optimizer for Machine Learning inference. Some of the techniques it performs are fusing and merging layers, optimizing memory usage and pre-scheduling complete network deployment. These techniques increase inference rates and lower inference latency. \n",
    "\n",
    "Using the xfDNN Compiler is an offline process, which only needs to be performed once per network/model. As you will see, the process is simple and quick. \n",
    "\n",
    "First, we will look at a ready made example demonstrating an Inception v1 network trained on Imagenet. Lets look at each step: \n",
    "\n",
    "### 1. Import the required packages  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os,sys\n",
    "from __future__ import print_function\n",
    " \n",
    "\n",
    "# Bring in Xilinx ML-Suite Compiler\n",
    "from xfdnn.tools.compile.bin.xfdnn_compiler_caffe import CaffeFrontend as xfdnnCompiler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Define a new xfdnnCompiler instance and pass arguments  \n",
    "To simplify handling of arguments, a config dictionary is used. Take a look at the dictionary below. \n",
    "\n",
    "The arguments that need to be passed are: \n",
    "- `prototxt` - Caffe representation of the network\n",
    "- `caffemodel` - Pre-trained Model for the network \n",
    "- `outmodel` - Filename to save the prototxt of the optimized network\n",
    "- `fpgacommands` - Filename to save micro-instruction produced by the compiler needed to deploy\n",
    "- `memory` - Parameter to set the on-chip memory for the target xDNN overlay. This example will target an overlay with 5 MB of cache. \n",
    "- `dsp` - Parameter to set the size of the target xDNN overlay. This example uses an overlay of size 32x56 DSPs.  \n",
    "\n",
    "The xfDNN Compiler interfaces with Caffe to read a network graph, and generates a sequence of instructions for the xfDNN Deploy APIs to execute on the FPGA.  \n",
    "\n",
    "During this process the xfDNN Compiler performs computational graph traversal, node merging and optimization, memory allocation and optimization and, finally, micro-instruction generation.\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a config dictionary to pass parameters to the compiler\n",
    "config = {}\n",
    "\n",
    "# Compiler Arguments\n",
    "config[\"prototxt\"] = \"../models/caffe/bvlc_googlenet_without_lrn/fp32/bvlc_googlenet_without_lrn_deploy.prototxt\" \n",
    "config[\"caffemodel\"] = \"../models/caffe/bvlc_googlenet_without_lrn/fp32/bvlc_googlenet_without_lrn.caffemodel\"\n",
    "config[\"outmodel\"] = \"work/optimized_model\" # String for naming intermediate prototxt, caffemodel\n",
    "config[\"fpgacommands\"] = \"work/bvlc_googlenet_without_lrn.cmds\" # Compiler will generate FPGA instructions\n",
    "config[\"memory\"] = 5 # Available on-chip SRAM\n",
    "config[\"dsp\"] = 56 # Width of Systolic Array\n",
    "\n",
    "compiler = xfdnnCompiler(\n",
    "    verbose=False,\n",
    "    networkfile=config[\"prototxt\"],       # Prototxt filename: input file\n",
    "    weights=config[\"caffemodel\"],         # Floating Point Weights: input file\n",
    "    anew=config[\"outmodel\"],              # Filename for optimized prototxt/caffemodel\n",
    "    generatefile=config[\"fpgacommands\"],  # Script filename: output file\n",
    "    memory=config[\"memory\"],              # Available on chip SRAM within xclbin\n",
    "    dsp=config[\"dsp\"]                     # Rows in DSP systolic array within xclbin \n",
    ")\n",
    "\n",
    "# Invoke compiler\n",
    "try:\n",
    "    compiler.compile()\n",
    "\n",
    "    # The compiler extracts the floating point weights from the .caffemodel. \n",
    "    # This weights dir will be stored in the work dir with the appendex '_data'. \n",
    "    # The compiler will name it after the caffemodel, and append _data\n",
    "    config[\"datadir\"] = \"work/\" + config[\"caffemodel\"].split(\"/\")[-1]+\"_data\"\n",
    "        \n",
    "    if os.path.exists(config[\"datadir\"]) and os.path.exists(config[\"fpgacommands\"]+\".json\"):\n",
    "        print(\"Compiler successfully generated JSON and the data directory: %s\" % config[\"datadir\"])\n",
    "    else:\n",
    "        print(\"Compiler failed to generate the JSON or data directory: %s\" % config[\"datadir\"])\n",
    "        raise\n",
    "        \n",
    "    print(\"**********\\nCompilation Successful!\\n\")\n",
    "    \n",
    "    import json\n",
    "    data = json.loads(open(config[\"fpgacommands\"]+\".json\").read())\n",
    "    print(\"Network Operations Count: %d\"%data['ops'])\n",
    "    print(\"DDR Transfers (bytes): %d\"%data['moveops']) \n",
    "    \n",
    "except Exception as e:\n",
    "    print(\"Failed to complete compilation:\",e)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Try it yourself with a different model\n",
    "\n",
    "Now that you have had a chance to see how this works, it's time to get some hands on experience.\n",
    "Change the following from the example above: \n",
    "    1. The Network - From Inception v1 to ResNet50. \n",
    "    2. The Weights - New Network, will require to us to re-extract the weights. \n",
    "    3. The amount of on-chip memory available. \n",
    "    4. The size of the xDNN overlay. \n",
    "\n",
    "Resources: \n",
    "ResNet50 prototxt located here: `\"../models/caffe/resnet/fp32/resnet50_deploy.prototxt\"`  \n",
    "ResNet50 caffemodel located  here:  `\"../models/caffe/resnet/fp32/resnet50.caffemodel\"`  \n",
    "\n",
    "In the last example, you ran through the compiler using a setting of 5 MB. Try `3` for 3 MB of on-chip memory. This will restrict the memory available to the compiler and it will automatically create DDR transfer commands to move intermediate results between the FPGA and DDR. You will see the DDR transfers is no longer 0 bytes.\n",
    "\n",
    "Lastly, 56 or 28 are supported as the DSP argument, so change this to 28. This corresponds to the 28x32 DSP configuration which reduces the number of DSPs compared to 56x32, but allows us to run twice as many CNN processing engines.\n",
    "\n",
    "Note: Because Resnet-50 has more opportunities to optimize the graph, and the model parameters are approximately 2x the size of the GoogLeVet-v1 model, it will take slightly longer to write the optimized weights run compared to the previous GoogLeNet-v1 example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Since we already have an instance of the compiler, you can just update these params:\n",
    "\n",
    "# We have already completed the instructions above for github release\n",
    "\n",
    "compiler.generatefile = \"work/resnet50.cmds\" # Compiler will generate FPGA instructions\n",
    "compiler.networkfile = \"../models/caffe/resnet/fp32/resnet50_deploy.prototxt\"\n",
    "compiler.weights = \"../models/caffe/resnet/fp32/resnet50.caffemodel\"\n",
    "compiler.memory = 3\n",
    "compiler.dsp = 28\n",
    "\n",
    "# Invoke compiler\n",
    "try:\n",
    "    compiler.compile()\n",
    "    \n",
    "    # The compiler extracts the floating point weights from the .caffemodel. \n",
    "    # This weights dir will be stored in the work dir with the appendex '_data'. \n",
    "    # The compiler will name it after the caffemodel, and append _data\n",
    "    config[\"datadir\"] = \"work/\" + config[\"caffemodel\"].split(\"/\")[-1]+\"_data\"\n",
    "        \n",
    "    if os.path.exists(config[\"datadir\"]) and os.path.exists(config[\"fpgacommands\"]+\".json\"):\n",
    "        print(\"Compiler successfully generated JSON and the data directory: %s\" % config[\"datadir\"])\n",
    "    else:\n",
    "        print(\"Compiler failed to generate the JSON or data directory: %s\" % config[\"datadir\"])\n",
    "        raise\n",
    "        \n",
    "    print(\"**********\\nCompilation Successful!\\n\")\n",
    "    \n",
    "    import json\n",
    "    data = json.loads(open(config[\"fpgacommands\"]+\".json\").read())\n",
    "    print(\"Network Operations Count: %d\"%data['ops'])\n",
    "    print(\"DDR Transfers (bytes): %d\"%data['moveops']) \n",
    "    \n",
    "except Exception as e:\n",
    "    print(\"Failed to complete compilation:\",e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen from the op and transfer counts, Resnet-50 takes about 2x the number of ops (multiply + add operations) and because we gave the compiler less on-chip memory, it now enables DDR transfers to move certain intermediate results to and from DDR.\n",
    "\n",
    "Well done! That concludes Part 1. Continue on to Part 2: \n",
    "\n",
    "## [**Next:** Using the xfDNN Quantizer to quantize Models][]   \n",
    "[**Next:** Using the xfDNN Quantizer to quantize Models]: quantizer_caffe.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}