03_autodiff.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook was inspired by neural network & machine learning labs led by [GMUM](https://gmum.net/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See also [A Gentle Introduction to `torch.autograd`](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html) and the [video](https://www.youtube.com/watch?v=tIeHLnjs5U8) from 3Blue1Brown (the whole [playlist](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi) is worth a watch)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic differentiation in PyTorch\n",
    "Training neural networks happens in two stages:\n",
    "- **Forward propagation** -- where the network makes its best guess about what the output should be. This is done by running the input data through the parametrized functions that constitute the network.\n",
    "- **Backward propagation** -- where the network updates its parameters in proportion to the error of its guess. This is done by traversing backward from the output, collecting the derivatives of the error with respect to the parameters of the functions, and optimizing the parameters using gradient descent. \n",
    "\n",
    "We can also remind ourselves of the [chain rule](https://en.wikipedia.org/wiki/Chain_rule) of calculus. If we have $L(x)=g(f(x))$, then:\n",
    "\n",
    "$$\\frac{d L(x)}{d x} = \\frac{d L(x)}{ d f(x)} \\frac{d f(x)}{d x}.$$\n",
    "\n",
    "In the context of automatic differentiation in PyTorch what's important is that to calculate the gradient $\\frac{d L(x)}{d x}$ we don't need to know anything about $g(x)$ if we know $\\frac{d L(x)}{ d f(x)}$. If each module in PyTorch can keep track of its own gradient and be fed in the gradient from the next layer, then calculating the gradient of even a complicated function is possible in this chain-like manner.\n",
    "\n",
    "In PyTorch each function that we use has these two methods:\n",
    "\n",
    "- `forward`, which gets fed $x$ and calculates $f(x)$.\n",
    "- `backward`, which gets fed $\\frac{d L(x)}{ d f(x)}$ and calculates $\\frac{d L(x)}{d x}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1: multiplication\n",
    "\n",
    "Let's say we want to implement a new multiplication function $f(a,b)=a\\cdot b$ in PyTorch and we want this function to be able to calculate its own derivatives.\n",
    "\n",
    "To do that, we have to implement a `torch.autograd.Function` object with the methods:\n",
    "- `forward`:\n",
    "    1. Gets in `a` and `b`.\n",
    "    2. Saves them for later to help with calculating gradients.\n",
    "    3. Returns `a * b`.\n",
    "- `backward`:\n",
    "    1. Gets in `grad_output` (i.e. $\\frac{d L(a,b)}{d f(a,b)}$).\n",
    "    2. Retrieves `a` and `b` from memory.\n",
    "    3. Calculates the derivative $\\frac{d f(a,b)}{d a} = \\frac{d (a \\cdot b)}{d a}=b$.\n",
    "    4. Calculates the derivative $\\frac{d f(a,b)}{d b} = \\frac{d (a \\cdot b)}{d b}=a$.\n",
    "    5. Returns the derivatives $\\frac{d L(a,b)}{d f(a,b)}\\frac{d f(a,b)}{d a}$ and $\\frac{d L(a,b)}{d f(a,b)} \\frac{d f(a,b)}{d b}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "class MyProduct(torch.autograd.Function):\n",
    "    \n",
    "    @staticmethod\n",
    "    def forward(self, a, b):\n",
    "        self.save_for_backward(a, b)\n",
    "        return a * b\n",
    "\n",
    "    @staticmethod\n",
    "    def backward(self, grad_output):\n",
    "        # retrieve a and b from memory\n",
    "        a, b = self.saved_tensors\n",
    "        # calculate the derivative wrt a\n",
    "        a_grad = b\n",
    "        # calculate the derivative wrt b\n",
    "        b_grad = a\n",
    "        # return the derivatives\n",
    "        return grad_output * a_grad, grad_output * b_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.autograd.gradcheck` checks whether the gradients calculated by our function match with numerical estimates from small finite differences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a = torch.randn(20, 30, dtype=torch.double, requires_grad=True) * 2 - 5\n",
    "b = torch.randn(20, 30, dtype=torch.double, requires_grad=True) + 6\n",
    "\n",
    "prod_fn = MyProduct.apply\n",
    "assert torch.autograd.gradcheck(prod_fn, (a, b), eps=1e-3, atol=1e-2, rtol=1e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2: ReLU\n",
    "$\\mathtt{ReLU}(x) = \\max(x,0)$ is currently the most popular activation function in deep learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEGCAYAAACEgjUUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdwElEQVR4nO3dd5iU9dXG8e+hS0dZirQF6ZG2rCASjaBJFCyJXSSJJRYUsCQalRjzmqJGY4kCCVHjm7AgCBbskthijChb6B3pbQGXvmw77x87vCEIMuzuPM+U+3NdXu6Und95dPee35555oy5OyIiktyqhV2AiIjEnsJeRCQFKOxFRFKAwl5EJAUo7EVEUkCNsAs4kqZNm3p6enrYZYiIJIzs7Oyt7p52uNviNuzT09OZPXt22GWIiCQMM1t9pNvUxhERSQEKexGRFKCwFxFJAQp7EZEUoLAXEUkBgYW9mTU2s2lmttjMFpnZgKDWFhFJdUGeevkk8La7X2JmtYC6Aa4tIpLSAtnZm1lD4AzgWQB3L3L3giDWFhFJFJ99sZ1n/rmSWIyeD6qN0wHIB/5iZrlm9oyZ1Tv0TmZ2g5nNNrPZ+fn5AZUmIhK+LbsKuWVSDlmz1rCvuLTKHz+osK8BZADj3b0PsAe4+9A7ufsEd89098y0tMO+41dEJOmUlJYxalIuuwqLGT88g7q1qr7DHlTYrwPWufusyOVplIe/iEjKe/Tdpcz6Yju//X4PurZoGJM1Agl7d98ErDWzLpGrzgIWBrG2iEg8m7lwM3/8cAXD+rfloozWMVsnyLNxRgFZkTNxVgLXBLi2iEjcWb1tD3dMzaNHq0b84rzuMV0rsLB39zwgM6j1RETiWWFxKSMm5lDNjHFXZVCnZvWYrhe3I45FRJLZ/a8uYOHGnTx3dSZtjo/92440LkFEJGBTZ69lyuy1jBzUkcFdmweypsJeRCRACzbs4L5X5nPaSSdw+7c7B7auwl5EJCA79hVzc1YOTerW4g9X9qF6NQtsbfXsRUQC4O7c+eIc1n+5jyk3nkrT+rUDXV87exGRAEz4aCXvLtzMPUO60bfd8YGvr7AXEYmxT1du4+G3FzO0R0uuHZgeSg0KexGRGNqys5CRk3JJP6EeD13cA7Pg+vQHU89eRCRGSkrLGDk5lz37S8j6cX8a1KkZWi0KexGRGHnknSV89sV2Hr+8F11aNAi1FrVxRERi4J0Fm/jTRyu5qn9bvt8ndgPOoqWwFxGpYqu27uGnU+fQs3UjfnF+bAecRUthLyJShQqLSxmRlUO1asbYYRnUrhHbAWfRUs9eRKQK3ffKfBZv2slzV58SyICzaGlnLyJSRaZ8voYXs9cxalBHBnVpFnY5/0VhLyJSBeav38F9ry7g9E5NufXs4AacRUthLyJSSQcGnJ1QrxZPXN470AFn0VLPXkSkEsrKnJ9MncOGgn1MuXEAJwQ84Cxa2tmLiFTCnz5ayd8XbWbM0G70bdck7HKOSGEvIlJB/16xjUfeWczQni25+rT0sMv5Wgp7EZEK2LyzkFGTc2jftB4PX9wztAFn0VLPXkTkGBWXljFyUg57i0qZfP2p1K8d/1Ea/xWKiMSZ3729mM9XfcmTV/SmU/NwB5xFS20cEZFj8Pb8jfz5n1/wwwHtuLB3q7DLiVpgO3szWwXsAkqBEnfPDGptEZGq8MXWPdz54lx6tWnMmKHdwi7nmATdxhnk7lsDXlNEpNL2FZUyYmI2Naob466KnwFn0VLPXkTkKNydMa/MY8nmXTx/TT9aNT4u7JKOWZA9ewfeNbNsM7vhcHcwsxvMbLaZzc7Pzw+wNBGRI5v82VpeylnP6MGd+FbntLDLqZAgw36gu2cA5wK3mNkZh97B3Se4e6a7Z6alJeZ/UBFJLvPW7eCXM8oHnI0+q1PY5VRYYGHv7hsi/94CvAz0C2ptEZGKKNhbxIisbJrWr8WTV/SJywFn0Qok7M2snpk1OPA18B1gfhBri4hURFmZc8fUOWzeWci44X05vl6tsEuqlKBeoG0OvBx5O3ENYJK7vx3Q2iIix2z8hyt4b/EWHrjwG/Ru0zjsciotkLB395VAryDWEhGprE+Wb+X37y7hgl4n8oNT24VdTpXQO2hFRA6yaUchoybn0iGtPg9e1CPuB5xFS+fZi4hEHBhwtq+4lCnDM6iXAAPOopU8RyIiUkkPvbWY2au/5Kkr+9CxWWIMOIuW2jgiIsCb8zby7MdfcPVp6Zzf68Swy6lyCnsRSXkr8ndz17S59GnbmHuHJNaAs2gp7EUkpe0tKmHExGxq1ajG2GEZ1KqRnLGonr2IpCx3Z8zL81m2ZTd/vbYfJybggLNoJedTmIhIFLJmreHl3PXcfnZnTu+U3PO4FPYikpLmrivggdcWcmaXNEYO6hh2OTGnsBeRlFOwt4gRE3NIa1Cbxy/rTbUEHnAWLfXsRSSllJU5t03JI3/Xfl68aQBNEnzAWbS0sxeRlDL2/eV8sCSf+87vTq8kGHAWLYW9iKSMj5dt5bG/L+V7vU9keP+2YZcTKIW9iKSEjTv2MfqFXDo1q89vk2jAWbQU9iKS9IpKyrglK4f9xaWMH96XurVS7+XK1DtiEUk5D761iJw1BYwdlsFJafXDLicU2tmLSFJ7fe4G/vKvVVwzMJ2hPVuGXU5oFPYikrRW5O/mZ9PmktG2Mfecm5wDzqKlsBeRpHRgwFntmtUZe1XyDjiLlnr2IpJ03J17X5rHsi27+du1/WnZKHkHnEUrtZ/qRCQpTZy1hlfyNnDH2Z35ZqemYZcTFxT2IpJU8tYW8MBrCxjUJY1bUmDAWbQU9iKSNL7cU8QtWTk0b1iHxy9PjQFn0VLPXkSSwsEDzqaNGEDjuqkx4Cxage7szay6meWa2etBrisiye+p95bz4dJ87r+gOz1bNw67nLgTdBvnVmBRwGuKSJL757J8nvjHUi7q04ph/VJrwFm0Agt7M2sNDAWeCWpNEUl+Gwr2MXpyLp2bNeDX3z855QacRSvInf0TwF1A2ZHuYGY3mNlsM5udn58fWGEikpiKSsq4OSuH4lJn/PCMlBxwFq1Awt7MzgO2uHv2193P3Se4e6a7Z6alJfeH/4pI5f32zUXkrS3gd5f0pEOKDjiLVlA7+4HABWa2CngBGGxmEwNaW0SS0Iw5G3j+k1Vc9832DOmRugPOohVI2Lv7Pe7e2t3TgSuA99x9eBBri0jyWb5lF3dPn0tmuybcfW7XsMtJCHpTlYgklD37S7hpYg51a1Xn6WEZ1KyuGItG4K9muPsHwAdBrysiic/dueeleazM383E6/rTolGdsEtKGHpKFJGE8bdPVzNjzgZ+8p0unNZRA86OhcJeRBJC7pov+dXrCxnctRkjvnVS2OUkHIW9iMS97QcPOLtMA84qQu9AEJG4Vlrm3PpCLlt3FzF9xGk0qlsz7JISksJeROLaH/6xjH8u28qDF/WgR+tGYZeTsNTGEZG49cGSLfzhvWVcnNGaK05pE3Y5CU1hLyJxaX3BPm6bkkeX5g349fc04KyyFPYiEnf2l5Ryc1YOpaXO+OF9Oa5W9bBLSnjq2YtI3PnNG4uYs7aAPw7PoH3TemGXkxS0sxeRuPJq3nr++u/VXH96e845WQPOqsoxh72Z1TMz/U0lIlVu6eZd3D19HqekN+GuczTgrCodNezNrJqZDTOzN8xsC7AY2GhmC8zsETPrFPsyRSTZ7d5fwk0Ts6lXu4YGnMVANP813wdOAu4BWrh7G3dvBpwOfAo8ZGYaVywiFebu/Gz6XFZt3cNTV/aheUMNOKtq0bxAe7a7Fx96pbtvB6YD081Mb2kTkQp7/pNVvDF3I3ed04UBJ50QdjlJ6ag7+wNBb2ZP2BFOdD3ck4GISDSyV3/Jb95YxNndmnHTGRpwFivH0hTbDcwws3oAZvYdM/tXbMoSkVSwbfd+Rk7KoWXjOvz+Ug04i6Woz7N395+b2TDgAzPbD+wB7o5ZZSKS1MoHnOWxbU8RL2nAWcxFHfZmdhZwPeUh3xK4zt2XxKowEUluT/59KR8v38pDF/Xg5FYacBZrx9LGGQPc5+5nApcAU8xscEyqEpGk9v7iLfzhveVc0rc1l2vAWSCOurM3M/Ny/x/s7j7PzM4FpgHvHbhPLAsVkeSwdvtebpuSR7eWDfnVhRpwFpSozrM3s1Fm1vaQ67dRfo79/wI/qvrSRCTZ7C8p5ZZJOZSVOeOvytCAswBF07M/B7gWmGxm7YEC4DjKnyjeBR5397xYFSgiyeOB1xYyd90O/vSDvqRrwFmgjhr27l4IjAPGRd481RTY5+4FMa5NRJLIy7nryJq1hhvP6MB3v9Ei7HJSTjQ9+4sOucqBIjNb6u7LYlOWiCSTJZt2cc9L8+jX/nju/G6XsMtJSdG0cc4/zHUGdDezF939kaM9gJnVAT4CakfWnObu9x9TpSKSkHYVFjNiYjb1a9fk6Sv7UEMDzkIRTRvnmsNdH2np5ABHDXtgPzDY3XdHvu9jM3vL3T89pmpFJKEcGHC2evtesn7cn2YacBaayjzFRv29kVM3d0cu1oz8o1M1RZLcc/9axZvzNnHnd7twagcNOAtTND371zh8MHcAnop2ocgHnmQDHYGx7j7rMPe5AbgBoG3bQ8/0FJFEMnvVdh58cxHf6d6cG8/oEHY5KS+anv2jh1x2oBhY6e6bo13I3UuB3mbWGHjZzE529/mH3GcCMAEgMzNTO3+RBLV1935umZRDqybH8cilvfTGqTgQTc/+wyPdZmY3ufsfj2VBdy8wsw8oP39//lHuLiIJprTMGT05l4K9xbx8cz8aHacBZ/Ggsi+L3xvNncwsLbKjx8yOA86m/OMNRSTJPDZzCZ+s2Mavvncy3U9sGHY5EhH11MsjiPZvs5bA/0b69tWAqe7+eiXXFpE4849Fmxn7/gouz2zDZZkacBZPKhv2UfXV3X0u0KeSa4lIHFu7fS+3T8mje8uG/M+F3wi7HDlENGfj7OI/oX5gJ++Rr4+LUV0ikkAKi0sZkZWNA+OHZ1CnpgacxZtoXqBtEEQhIpK4/ue1hcxfv5M//zCTdidowFk8OqYXaM3sTDO7NvJ1s8gUTBFJYdOz1zH5szWMOPMkvt29edjlyBFEHfZm9ihwM//53Nk6QFYsihKRxLB4007GvDKPAR1O4Cff7hx2OfI1jmVnP9jdL6P8M2hx9zWA/l4TSVE7C4sZMTGHhnVq8gcNOIt7x3I2TnFkeqUDmFkroDQmVYlIXHN37npxLmu272Xy9aeS1qB22CXJURxL2P8K+BBoY2Z/A06nvK0jIinm2Y+/4O0FmxgzpBv92h8fdjkShajD3t1fN7NZwKmRq+5w9/zYlCUi8erzVdt58K3FnPONFvz4dJ2jkSiiarKZWQsz6wsUuPtrwJvAuWY2J6bViUhcyd+1n1uycmjT5Dh+d2lPDThLIEcNezO7EZgHPA18aGZnASuBS4GbYlueiMSLktIyRk/OZWdhMeOH96VhHQ04SyTRtHFGA93cfauZtQEWAMPdfUZsSxORePL7mUv598ptPHppL7q11ICzRBNNG2e/u28FcPe1wBoFvUhqmblwM+M/WMGV/dpwSd/WYZcjFRDNzr6DmR0c7q0PvuzuF1R9WSISL9Zs28sdU/M4uVVD7j9fA84SVTRhf+Ehl38fi0JEJP4cGHBmwPir+mrAWQKr1CdViUhy++WMBSzYsJNnf5RJm+Prhl2OVILe3ywih/Xi7LW88Plabj7zJM7qpgFniU5hLyJfsXDDTn7+ynwGdDiBOzTgLCko7EXkv+wsLObmrGwa19WAs2RS2Y8lFJEk4u78dOoc1n25jxdu0ICzZKKnbBH5f3/+50reXbiZu8/tSma6BpwlE4W9iAAwa+U2Hn57CUN6tOC6b2rAWbJR2IsIW3YVMnJyLu2Or8vDF2vAWTJSz14kxZWUljFqUi67Cov523X9aKABZ0lJYS+S4h59dymzvtjOY5f1omsLDThLVoG0ccysjZm9b2aLzGyBmd0axLoi8vVmLtzMHz9cwbD+bbkoQwPOkllQO/sS4CfunmNmDYBsM5vp7gsDWl9EDrF62x7umJpHj1aN+MV53cMuR2IskJ29u29095zI17uARUCrINYWka8qLC7lpok5VDNj3FUZGnCWAgI/G8fM0oE+wKzD3HaDmc02s9n5+fp4W5FY+cWr81m0cSePX95LA85SRKBhb2b1genAbe6+89Db3X2Cu2e6e2ZaWlqQpYmkjKmfr2Xq7HWMHNSRwV014CxVBBb2ZlaT8qDPcveXglpXRP5jwYYd3PfqfAZ2PIHbNeAspQR1No4BzwKL3P2xINYUkf+2Y18xIybm0KRuLZ68og/Vq+mNU6kkqJ39QOAHwGAzy4v8MySgtUVSnrvz0xfnsKFgH2Ov6kPT+hpwlmoCOfXS3T8GtI0QCcmfPlrJzIWbue+87vRtpwFnqUizcUSS3L9XbON3by9maI+WXDswPexyJCQKe5EktmVnIaMm55LetB4PX6IBZ6lMs3FEklRxaRkjJ+WyZ38Jk67vT/3a+nVPZfq/L5KkHnlnCZ+t2s4Tl/emc/MGYZcjIVMbRyQJvT1/ExM+WsnwU9vyvT6aTCIKe5Gk88XWPdz54hx6tW7EfRpwJhEKe5Eksq+olBETs6le3Rh7VQa1a2jAmZRTz14kSbg7P39lPos37eIv15xC6yYacCb/oZ29SJJ44fO1TM9Zx+jBHRnUpVnY5UicUdiLJIH563dw/4wFnN6pKbeerQFn8lUKe5EEt2NvMTdNzOaEerV44vLeGnAmh6WevUgCKytz7piax6YdhUy5cQAnaMCZHIF29iIJbPyHK/jH4i2MGdqNvu2ahF2OxDGFvUiC+mTFVn7/7hKG9mzJ1aelh12OxDmFvUgC2rSjkNGTc2nftB4PX6wBZ3J06tmLJJjyAWc57NlfyqTrT9WAM4mKfkpEEszDby1m9uovefIKDTiT6KmNI5JA3pq3kWc+/oIfDmjHhb014Eyip7AXSRAr83dz57S59GrTmDFDu4VdjiQYhb1IAthXVMrNWTnUrG6M04AzqQD17EXinLsz5pV5LNm8i+ev6UerxseFXZIkIO3sReLc5M/W8lLOem49qxPf6pwWdjmSoBT2InFs3rod/HLGAs7onMbowZ3CLkcSmMJeJE4V7C1iRFY2TeuXDzirpgFnUgmBhL2ZPWdmW8xsfhDriSS6sjLn9il5bN5ZyLjhfTm+Xq2wS5IEF9TO/nngnIDWEkl44z5YzvtL8rnvvO70btM47HIkCQQS9u7+EbA9iLVEEt2/lm/lsZlLuaDXifzg1HZhlyNJIq569mZ2g5nNNrPZ+fn5YZcjErgDA846pNXnwYt6aMCZVJm4Cnt3n+Dume6emZamU8wktRSXlnHLpBwKi0v54/C+1NOAM6lC+mkSiRMPvrmY7NVf8vSwPnRsVj/sciTJxNXOXiRVvTF3I8/96wuuPi2d83qeGHY5koSCOvVyMvBvoIuZrTOz64JYVyQRrMjfzV3T5pDRtjH3DtGAM4mNQNo47n5lEOuIJJq9RSWMmJhN7ZrVeXpYBrVq6I9tiQ317EVC4u6MeXk+y7bs5q/X9uNEDTiTGNI2QiQkWbPW8HLuem4/uzOnd9LZZxJbCnuREMxdV8ADry3kzC5pjBzUMexyJAUo7EUC9uWeIkZMzCGtQW0ev0wDziQY6tmLBKiszLl9ah75u/bz4k0DaKIBZxIQ7exFAvT0+8v5YEk+953fnV4acCYBUtiLBOTjZVt5/O9L+V7vExnev23Y5UiKUdiLBGBDwT5Gv5BLp2b1+a0GnEkIFPYiMVZUUj7grKikjPHD+1K3ll4qk+Dpp04kxn775iJy1xQw7qoMTkrTgDMJh3b2IjH02pwNPP/JKq4d2J4hPVqGXY6kMIW9SIws37Kbu6fPpW+7JtwzpGvY5UiKU9iLxMCe/eUDzurUrM7YYRnUrK5fNQmXevYiVczduffleazI383frutPi0Z1wi5JRDt7kao28dPVvJq3gTu+3ZmBHZuGXY4IoLAXqVJ5awt44PWFDOqSxs1nasCZxA+FvUgV2b6niJsnZtO8YR0ev1wDziS+qGcvUgVKy5zbpuSxdXcR00YMoHFdDTiT+KKdvUgVeOq9ZXy0NJ/7L+hOz9aNwy5H5CsU9iKV9OHSfJ78xzIu6tOKYf004Ezik8JepBLWF+zjthdy6dK8Ab/5vgacSfxS2ItUUFFJGbdk5VBc6oy7KoPjalUPuySRI1LYi1TA9j1FjJ6cS97aAh69tCcdNOBM4pzOxhE5Bu7OG/M2cv+rC9ixr5h7h3TlnJM14EziX2Bhb2bnAE8C1YFn3P2hoNYWqQqbdxby81fmM3PhZnq2bkTW9f3p2qJh2GWJRCWQsDez6sBY4NvAOuBzM5vh7guDWF+kMtydqbPX8us3FlFUUsaYId24ZmA6NTTcTBJIUDv7fsByd18JYGYvABcCVR725z/1MYXFpVX9sJLC9hWXsu7LffRvfzwPX9yT9Kb1wi5J5JgFFfatgLUHXV4H9D/0TmZ2A3ADQNu2FTtf+aS0ehSVllXoe0WOZOSgjlyW2UYjECRhBRX2h/sN8a9c4T4BmACQmZn5lduj8cQVfSrybSIiSS2opuM6oM1Bl1sDGwJaW0Qk5QUV9p8DncysvZnVAq4AZgS0tohIygukjePuJWY2EniH8lMvn3P3BUGsLSIiAZ5n7+5vAm8GtZ6IiPyHThQWEUkBCnsRkRSgsBcRSQEKexGRFGDuFXrvUsyZWT6wuoLf3hTYWoXlhClZjiVZjgN0LPEoWY4DKncs7dw97XA3xG3YV4aZzXb3zLDrqArJcizJchygY4lHyXIcELtjURtHRCQFKOxFRFJAsob9hLALqELJcizJchygY4lHyXIcEKNjScqevYiI/Ldk3dmLiMhBFPYiIikgqcPezEaZ2RIzW2Bmvwu7nsows5+amZtZ07BrqSgze8TMFpvZXDN72cwah13TsTCzcyI/T8vN7O6w66koM2tjZu+b2aLI78atYddUWWZW3cxyzez1sGupDDNrbGbTIr8ni8xsQFU9dtKGvZkNovxzbnu6+zeAR0MuqcLMrA3lH9a+JuxaKmkmcLK79wSWAveEXE/UzKw6MBY4F+gOXGlm3cOtqsJKgJ+4ezfgVOCWBD6WA24FFoVdRBV4Enjb3bsCvajCY0rasAdGAA+5+34Ad98Scj2V8ThwF4f5KMdE4u7vuntJ5OKnlH9iWaLoByx395XuXgS8QPlmIuG4+0Z3z4l8vYvyQGkVblUVZ2atgaHAM2HXUhlm1hA4A3gWwN2L3L2gqh4/mcO+M3C6mc0ysw/N7JSwC6oIM7sAWO/uc8KupYpdC7wVdhHHoBWw9qDL60jggDzAzNKBPsCskEupjCco3wyVhVxHZXUA8oG/RFpSz5hZvap68MA+vCQWzOzvQIvD3DSG8mNrQvmfqacAU82sg8fhuaZHOY57ge8EW1HFfd2xuPurkfuMobyVkBVkbZVkh7ku7n6WjoWZ1QemA7e5+86w66kIMzsP2OLu2WZ2ZsjlVFYNIAMY5e6zzOxJ4G7gvqp68ITl7mcf6TYzGwG8FAn3z8ysjPIBQ/lB1RetIx2HmfUA2gNzzAzK2x45ZtbP3TcFWGLUvu7/CYCZ/Qg4DzgrHp94v8Y6oM1Bl1sDG0KqpdLMrCblQZ/l7i+FXU8lDAQuMLMhQB2goZlNdPfhIddVEeuAde5+4K+saZSHfZVI5jbOK8BgADPrDNQiwabiufs8d2/m7ununk75D0NGvAb90ZjZOcDPgAvcfW/Y9Ryjz4FOZtbezGoBVwAzQq6pQqx85/AssMjdHwu7nspw93vcvXXk9+MK4L0EDXoiv9drzaxL5KqzgIVV9fgJvbM/iueA58xsPlAE/CjBdpLJ6GmgNjAz8pfKp+5+U7glRcfdS8xsJPAOUB14zt0XhFxWRQ0EfgDMM7O8yHX3Rj4nWsI1CsiKbChWAtdU1QNrXIKISApI5jaOiIhEKOxFRFKAwl5EJAUo7EVEUoDCXkQkBSjsRURSgMJeRCQFKOxFomBmp0Tm8Ncxs3qROfAnh12XSLT0piqRKJnZrymfv3Ic5TNMHgy5JJGoKexFohR5C/vnQCFwmruXhlySSNTUxhGJ3vFAfaAB5Tt8kYShnb1IlMxsBuWfUNUeaOnuI0MuSSRqyTz1UqTKmNkPgRJ3nxT5PNpPzGywu78Xdm0i0dDOXkQkBahnLyKSAhT2IiIpQGEvIpICFPYiIilAYS8ikgIU9iIiKUBhLyKSAv4PjtYEhbOW+TMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"$\\mathtt{ReLU}(x)$\")\n",
    "_ = plt.plot(np.linspace(-6, 6), np.maximum(np.linspace(-6, 6), 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyReLU(torch.autograd.Function):\n",
    "    @staticmethod\n",
    "    def forward(self, x):\n",
    "        self.save_for_backward(x)\n",
    "        return x.clamp(min=0)\n",
    "    \n",
    "    @staticmethod\n",
    "    def backward(self, grad_output):\n",
    "        x, = self.saved_tensors\n",
    "        grad_input = grad_output.clone()\n",
    "        grad_input[x < 0] = 0\n",
    "        return grad_input\n",
    "\n",
    "relu_fn = MyReLU.apply\n",
    "assert torch.autograd.gradcheck(relu_fn, a, eps=1e-6, atol=1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1 (1p)\n",
    "Implement the following functions:\n",
    "- `MyAdd(a, b): a + b`,\n",
    "- `MyDiv(a, b): a / b`,\n",
    "- `MySigmoid(x): 1 / (1 + exp(-x))`,\n",
    "- `ReQU(x)`: $x^2$ for $x\\geq0$ and $0$ for $x<0$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyAdd(torch.autograd.Function):\n",
    "    @staticmethod\n",
    "    def forward(self, a, b):\n",
    "        self.save_for_backward(a, b)\n",
    "        return ???\n",
    "\n",
    "    @staticmethod\n",
    "    def backward(self, grad_output):\n",
    "        a, b = self.saved_tensors\n",
    "        return ???\n",
    "\n",
    "add_fn = MyAdd.apply\n",
    "assert torch.autograd.gradcheck(add_fn, (a, b), eps=1e-3, atol=1e-2, rtol=1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyDiv(torch.autograd.Function):\n",
    "    @staticmethod\n",
    "    def forward(self, a, b):\n",
    "        self.save_for_backward(a, b)\n",
    "        return ???\n",
    "\n",
    "    @staticmethod\n",
    "    def backward(self, grad_output):\n",
    "        a, b = self.saved_tensors\n",
    "        return ???\n",
    "\n",
    "div_fn = MyDiv.apply\n",
    "assert torch.autograd.gradcheck(div_fn, (a, b), eps=1e-3, atol=1e-2, rtol=1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MySigmoid(torch.autograd.Function):\n",
    "    @staticmethod\n",
    "    def forward(self, x):\n",
    "        self.save_for_backward(x)\n",
    "        return ???\n",
    "    \n",
    "    @staticmethod\n",
    "    def backward(self, grad_output):\n",
    "        x, = self.saved_tensors\n",
    "        return ???\n",
    "\n",
    "sigmoid_fn = MySigmoid.apply\n",
    "assert torch.autograd.gradcheck(sigmoid_fn, a, eps=1e-3, atol=1e-2, rtol=1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReQU(torch.autograd.Function):\n",
    "    @staticmethod\n",
    "    def forward(self, x):\n",
    "        self.save_for_backward(x)\n",
    "        return ???\n",
    "    \n",
    "    @staticmethod\n",
    "    def backward(self, grad_output):\n",
    "        x, = self.saved_tensors\n",
    "        return ???\n",
    "    \n",
    "requ_fn = ReQU.apply\n",
    "assert torch.autograd.gradcheck(requ_fn, a, eps=1e-3, atol=1e-2, rtol=1e-2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}