notebook/08-03-CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "---\n",
    "---\n",
    "# Convolutional Networks\n",
    "---\n",
    "---\n",
    "\n",
    "![image.png](img/chengjun.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl23.png)\n",
    "\n",
    "https://www.deeplearningbook.org/contents/convnets.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/cnn3d.png)\n",
    "\n",
    "https://adamharley.com/nn_vis/\n",
    "\n",
    "A. W. Harley, \"An Interactive Node-Link Visualization of Convolutional Neural Networks,\" in ISVC, pages 867-877, 2015"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Convolutional networks (LeCun, 1989), also known as convolutional neural networks, or CNNs, are a specialized kind of neural network for processing data that has a known *grid-like topology*. \n",
    "\n",
    "- time-series data,  1-D grid \n",
    "- image data, 2-D grid of pixels. \n",
    "\n",
    "The name “convolutional neural network” indicates that the network employs a mathematical operation called `convolution`. \n",
    "\n",
    "- Convolution is a specialized kind of linear operation. \n",
    "- Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center>\n",
    "    \n",
    "<div><img src=img/dl24.png></div>\n",
    "    \n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center>\n",
    "<div><img src=img/cnn.gif></div>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "全连接前馈网络在图像处理中的缺点：图像数据维度太高（一个像素点被视为一个维度），而且由于图像很多信息在高层次才体现，在过低的层次分析导致过拟合以及训练效率过低。\n",
    "- Scale up neural networks to process very large images / video sequences\n",
    "    - Sparse connections\n",
    "    - Parameter sharing\n",
    "- Automatically generalize across spatial translations of inputs\n",
    "- Applicable to any input that is laid out on a grid (1-D, 2-D, 3-D, …)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- Replace matrix multiplication in neural nets with convolution\n",
    "- Everything else stays the same\n",
    "    - Maximum likelihood\n",
    "    - Back-propagation\n",
    "    - etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Convolution** is a mathematical way of combining two signals to form a third signal. \n",
    "\n",
    "$$(f * g)(t) \\triangleq\\ \\int_{-\\infty}^\\infty f(\\tau) g(t - \\tau) \\, d\\tau$$\n",
    "\n",
    "\n",
    "<center>    \n",
    "<div><img src=img/Convolution_of_box_signal_with_itself2.gif align = center width = 1000px></div> \n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### The Convolution Operation\n",
    "\n",
    "Suppose we are tracking the location of a spaceship with a laser tensor.\n",
    "\n",
    "$x(\\tau)$ denote the location of the `spaceship` at time $\\tau$. Assume that our laser sensor is somewhat noisy, we would like to average together several measurements. But we would like to give more weight to recent measurements with a weight function $w(t)$, wheare $\\tau$ is the age of a measurement. We can obtain a new function $F(t)$ providing a smoothed estimate of the position of the spaceship.\n",
    "\n",
    "$$ F(t) = x*w(t) = \\int x(\\tau) w(t-\\tau) d \\tau$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Suppose we are tracking the location of a spaceship with a laser tensor.\n",
    "\n",
    "\n",
    "<center>\n",
    "<div><img src=img/convgaus.gif width = 800px></div></center>\n",
    "\n",
    "https://mathworld.wolfram.com/Convolution.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Convolution network terminology**\n",
    "- input: function $x$\n",
    "- kernal: function $w$\n",
    "- featuremap: output\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In machine learning applications, the input is usually **a multidimensional array** of data, and the kernel is usually a multidimensional array of parameters that are adapted by the learning algorithm. We will refer to these multidimensional arrays as **tensors**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Discrete Convolution**\n",
    "\n",
    "We only measure at discreted timestamp\n",
    "\n",
    "$$ s(t) = \\sum_{\\tau = -\\infty}^{\\infty} x(\\tau) w(t-\\tau)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Finally, we often use convolutions over more than one axis at a time. For example, if we use a two-dimensional image $I$ as our input, we probably also want to use a two-dimensional kernel \n",
    "\n",
    "$$S(i, j) = (I ∗ K)(i, j) =\\sum_m \\sum_n I(m, n)K(i − m, j − n)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Discrete convolution can be viewed as multiplication by a matrix\n",
    "\n",
    "![image.png](img/dl25.png)\n",
    "\n",
    "Matrix (Dot) Product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**2D Convolution**\n",
    "\n",
    "![image.png](img/dl26.png)\n",
    "\n",
    "Discrete convolution can be viewed as multiplication by a matrix\n",
    "本质来说，CNN基于一个假设：即使每个像素只偏移一点点，图片的整体信息仍然不变。从而可以用卷积操作高效降维度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl27.png)\n",
    "\n",
    "**Padding**\n",
    "\n",
    "每次卷积操作都会使图像缩小，角落和边缘区域的像素点使用的很少，即丢失了图像边缘位置的信息。为了解决这两个问题，在卷积操作之前，先对图像周围进行填充，通常为zero-padding，即填充0。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl28.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Three Operations\n",
    "- Convolution: like matrix multiplication\n",
    "    - Take an input, produce an output (hidden layer)\n",
    "- “Deconvolution”: like multiplication by transpose of a matrix\n",
    "    - Used to back-propagate error from output to input\n",
    "    - Reconstruction in autoencoder / RBM\n",
    "- Weight gradient computation\n",
    "    - Used to backpropagate error from output to weights\n",
    "    - Accounts for the parameter sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Convolution leverages three important ideas: \n",
    "- sparse interactions, \n",
    "- parameter sharing,  \n",
    "- equivariant representations.\n",
    "        \n",
    "$$f(g(x)) = g(f(x))$$\n",
    "    \n",
    "    if we move the object in the input, its representation wil move the same amount in the outupt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl29.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl30.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl31.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl32.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Example: Edge Detection by Convolution\n",
    "\n",
    "![image.png](img/dl33.png)\n",
    "\n",
    "[卷积的直观理解](https://github.com/computational-class/summer-school/blob/master/class_03_%E5%8D%B7%E7%A7%AF%E7%9A%84%E7%9B%B4%E8%A7%82%E7%90%86%E8%A7%A3.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl34.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A typical layer of a convolutional network consists of **three stages**\n",
    "\n",
    "- In the ﬁrst stage, the layer performs several **convolutions** in parallel to produce aset of linear activations. \n",
    "- In the second stage, each linear activation is run through a nonlinear activation function, such as the rectiﬁed linear activation function. This stage is sometimes called the **detector** stage. \n",
    "- In the third stage, we use **apooling** function to modify the output of the layer further."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "![image.png](img/dl35.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Pooling 池化\n",
    "\n",
    "**A pooling function** replaces the output of the net at a certain location with a summary statistic of the nearby outputs. \n",
    "\n",
    "For example, the **max pooling** (Zhou and Chellappa, 1988) operation reports the maximum output within a rectangular neighborhood. \n",
    "\n",
    "Other popular pooling functions include \n",
    "- the average of a rectangularneighborhood, \n",
    "- the L2 norm (每个元素的平方的和) of a rectangular neighborhood,  \n",
    "- a weighted average based on the distance from the central pixel.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl36.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl37.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl38.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl39.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The human eye is mostly very low resolution, except for a tiny patch called the fovea. The fovea only observes an area about the size of a thumbnail held at arms length. \n",
    "\n",
    "![image.png](img/dl40.png)\n",
    "\n",
    "Though we feel as if we can see an entire scene in high resolution, this is an illusion created by the subconscious part of our brain, as it stitches together several glimpses of small areas. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Most convolutional networks actually receive large full-resolution photographs as input. \n",
    "\n",
    "The human brain makes several eye movements called saccadesto glimpse the most visually salient or task-relevant parts of a scene. \n",
    "\n",
    "Incorporating similar attention mechanisms into deep learning models is an active research direction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "https://atcold.github.io/pytorch-Deep-Learning/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T03:03:51.733295Z",
     "start_time": "2020-04-06T03:03:51.729210Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 使用PyTorch建立卷积神经网络并处理MNIST数据\n",
    "\n",
    "https://computational-communication.com/pytorch-mnist/\n",
    "\n",
    "https://github.com/computational-class/summer-school/blob/master/class_04_CNN.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](img/dl41.png)\n",
    "\n",
    "https://pytorch.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:50:48.566334Z",
     "start_time": "2024-08-03T05:50:42.986378Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torchvision import datasets, transforms\n",
    "from torch.autograd import Variable\n",
    "#!pip3 install torch torchvision torchaudio\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:50:50.666366Z",
     "start_time": "2024-08-03T05:50:50.579402Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# MNIST Dataset\n",
    "train_dataset = datasets.MNIST(root='../data/',\n",
    "                            train=True,\n",
    "                            transform=transforms.ToTensor(),\n",
    "                            download=True)\n",
    "                            # change it to False after you have downloaded the data\n",
    "\n",
    "test_dataset = datasets.MNIST(root='../data/',\n",
    "                           train=False,\n",
    "                           transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:50:52.729417Z",
     "start_time": "2024-08-03T05:50:52.726721Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Hyper Parameters\n",
    "batch_size = 100 \n",
    "learning_rate = 0.001\n",
    "num_epochs = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:50:54.650227Z",
     "start_time": "2024-08-03T05:50:54.646828Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Data Loader (Input Pipeline)\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset,\n",
    "                                           batch_size=batch_size,\n",
    "                                           shuffle=True)\n",
    "\n",
    "test_loader = torch.utils.data.DataLoader(dataset=test_dataset,\n",
    "                                          batch_size=batch_size,\n",
    "                                          shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:52:26.604275Z",
     "start_time": "2024-08-03T05:52:26.570147Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pylab as plt\n",
    "import torchvision\n",
    "\n",
    "# functions to show an image\n",
    "def imshow(img):\n",
    "    img = img / 2 + 0.5     # unnormalize\n",
    "    npimg = img.numpy()\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "\n",
    "\n",
    "# get some random training images\n",
    "dataiter = iter(train_loader)\n",
    "images, labels = next(dataiter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:52:30.546416Z",
     "start_time": "2024-08-03T05:52:30.540336Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([100, 1, 28, 28])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:52:33.890375Z",
     "start_time": "2024-08-03T05:52:33.747354Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(4) tensor(1) tensor(5) tensor(3) tensor(3)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAABoCAYAAADo66t9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAlBElEQVR4nO2dWWwl2XmYv3P3pe5+STbZJJvdPT2T0TYZS3CcOIiMKEZkx5DykkBGHCiIgXlJEDtwEEsREEBvBhI48UMWCLZjJREsB7YSC4aTWFAiBHkYRR7J9ozUPdM96m7uvOTd9/XkgfzPFDnslWzeqtvnAwiSdbn8VbfqP//5V6W1xmKxWCz+IzBtASwWi8XydFgFbrFYLD7FKnCLxWLxKVaBWywWi0+xCtxisVh8ilXgFovF4lPOpMCVUp9USr2tlLqjlPrceQllsVgslkejnjYPXCkVBN4BfhLYBL4D/KzW+gfnJ57FYrFYHkToDL/7o8AdrfUPAZRSXwU+DTxQgScSCZ3NZs/wLy0Wi+X5Y2dn50BrPXfy+FkU+GVgw/X9JvAXTv6QUuo14DWATCbDa6+9doZ/abFYLM8fX/ziF++fdvwsPnB1yrH3+WO01l/SWn9Ma/2xRCJxhn9nsVgsFjdnUeCbwIrr+2Vg+2ziWCwWi+VxOYsC/w5wQyl1VSkVAT4DfP18xLJYLBbLo3hqH7jWeqSU+ofA/wSCwG9qrb9/bpJZLBaL5aGcJYiJ1voPgT88J1ksFovF8gTYSkyLxWLxKVaBWywWi0+xCtxisVh8ilXgFovF4lOsArdYLBafcqYsFIvF4k+UUqd+PAyt9akflulhFbjF8pwRDAZJpVJEIhEKhQKJRIJsNksqlQI4VZFrranVavR6ParVKo1Gg06nQ6PRsEp8ilgFbrE8ZwQCAeLxOPF4nEuXLpHJZFhcXKRYLD7QEp9MJuzu7tJsNolEIuZYo9G4aPEtLmZSgctNmEqlKBQKaK0ZDAaMRiOq1SrD4fBcrIZYLMb8/DzRaJR8Pk8kEqHf7zMajdjZ2WFnZ+cczub8CIVCOI5DJBJhfn6eWCzG9vY25XKZ8XjMaDSatoiWZ4A8D4lEgmKxSDwe5/Lly8TjcXK5HPF4nEQi8VAXilLK3DuhUIhcLke1WiWTydBut9nd3WU8Hl/gWVlgxhV4NpvlhRdeYDKZ0Gq16Ha7dDodo6jOqsTj8Thra2tkMhlu3LhBMpmk0WjQ6/V444032N3d9dT2MhQKkc/ncRyHD33oQ2SzWb7zne/QarXMAmeZPZRSBINBHMfh6tWrpNNprl+/TjweJxgMPtL3LX8jnU4DkMvlGI/HlMtl0uk0+/v7HBwcWAU+BWZSgScSCZLJJIVCwVjgjuPQarXY2tqi1+sxHo+fWrkGg0FCoRDxeJxUKoXjOASDQQDa7Tb1ep1ut+sp5Q2HW+dYLEYikSAajRKNRnEch0wmQ7PZpNfrTVvEZ4ooskAgQDQaNZ9DoRCDwYB+v2/eW6WU+SyEw2FCoZD5Ga01k8mEXq9HqVRiOBye6b56VsRiMbMbLRQKJJNJQqHDR7/dbjMajej1egwGgwfKrpQiFosRCoWIxWJEo1EikQjJZJJms/lYi4BXUEoRCASO3Q+hUIhAIEAikSCRSJj3eDKZ0O/3mUwmdLtdRqMR7Xabfr8/7dMAZlCBK6UoFossLS2xtLTE2toagUDABGHeffdd2u02k8nkqf9HOBzGcRyy2SwLCws4jkM4HGYymVAqldja2qJarZ7jWZ0PwWCQTCZjAlbJZJJisUiv12NnZ4d6ve455XOeyAIWDofJ5/PEYjGj0KrVKpVKhWg0ahRcMpkkEDjMtFVKkclkiMfjJJNJEokE4/GYwWBAqVTiW9/6Fo1Gg36/7zlLNJPJsLa2RrFY5Pr164TDYQBGoxHlcplms8ne3h7lcvmBmSXBYJD5+Xkcx+Hy5ctcunSJRCJBoVCg0+mY6+QHlFJEIhGjsOW9jkajLC8vs7KyYu6D4XBIpVIxz0i73WZ9fZ39/f1pnwYwQwpcKUU4HCYYDJJMJslkMubN0VozGo0YjUbnoqBCoRCJRIJ4PE4kEiEcDh+zxjqdDsPh8BzO6nwRBRaLxYzlEYvFcBzHHJtMJmda3LxIIBAgGAwSjUYpFApEo1FyuRzRaJRsNks8HkdrTSAQIBwOGwsskUgcU0xyncQClfvppKXuNeTeHA6HNJtNAoGAidXs7++bXWOr1XqoAk+lUsZQgfcsWb8ob7G0w+EwmUzGKG55zyORCOl0mkQiYXZbgNEj2WyWSCRCo9FgOBzS7/fpdrtTPaeZUuCFQgHHcbh27RpXr14lGo2ilKLX67G3t2fSoM66zU0kEiwvL5PP5006Vr/fp9/vUy6X2d3dnfobexqhUIhCoWACrgCFQoFYLEa/32d7e5vhcEiv15spSzwSieA4DsVikY9+9KMkk0nj/5WFbDweMx6Pj22vTyomt79Ygr79fv+hrgcv0O/3qdfrtNttyuUyw+GQ/f19+v2+cfvIx4POQ4yWQCDAYDC44DM4H2TBzmQyvPTSS8bQE2tc3GbhcJjRaESr1SIQCJDJZFBKkc/nGY/HFAoFKpUK9+7d486dO1N9732vwOVBkxtMXAMSVR+NRgwGAzqdjvFhPe0Fl4c3HA4Tj8ePWbKTyYTxeMxwOGQwGHhuGw3v7VLC4bC5YcX6EF+wF+U+K4FAgEgkQiwWI5vNGqtLXGtiRT8I+ZnJZILWmvF4bHyjcl/JMS8q8vF4bBYa+Vwul83C8yiZxcp2L25wmEY4GAw8udt0Iwu1xKzS6TTZbJZEIoHjOOa+d38MBgNarZZZtEWxizEwGo2IxWLTPjX/K/BQKMTc3BzJZJIbN25QLBbJ5XKEw2GazSaVSoVms8nm5qbJRHlaC1zexGQySTabxXEcowREgctnLz7IpxGNRs05JZNJlFKeDMCeBff5yfmORiOTnXQyIKWUMuevtTbuBskwGgwGDAYDer0ezWaTfr9Po9E4NxfdeSOZVycV7+Mob3EdxONxVlZWmJ+fJ5PJAFCtVrlz5w7lctnTGUyFQoH5+XmKxSJXr141i7nktvd6Pba3t6nVaoxGI/Mcj0YjgsGgCfx/5CMfIZ/PEw6Hzc572vhegQcCARzHMVH2YrFoXCfD4ZBGo0Gj0aBer5sb+Wl9vMFg0KzCogjcW2ovW2HCyYdWrBOxzB9mifoVyTaJRCLHXCaj0YhOp0On0zn28/KeysLc7XYZDoeUy2Wj8MX/6YdKRIn/PClicUvgNp1OG5eD1pper0e5XKbRaHg2bqKUIh6Pk8/nmZubY2lpybhV+/0+zWaTZrPJ1taWySRy7yjccaLr16+TTqePxdumja+fVqUU0WiUK1eukM/nTWBKXBq1Wo07d+7Q6XRMAc9ZLIV4PE42myWXy5HNZs1CMRqN2Nvbo16v02w2PWuJDQYDsxN54YUXPGFBXASxWIy5uTlyuRzBYNC8X+12m83NzYdmDEkAXALU4jMWS21WEX9xMpnk2rVrpFIp5ufnSSaTjMdjGo0G1WqVUqlEr9fznAIX3RAKhZifnzeWt7QA2NjYoNPpUCqV6Ha71Ot1BoPB+84jFAqRTqdJpVImacFLRW++VeBSrBOJRLh06ZKpiJTcTa01rVaLzc1N+v3+uQTmYrEYmUzGvKGSOzoej6lWqxwcHNButz37YLuDVysrK9MW58KQrJNUKkUgEDAVudVqlXv37lEqlaYtoueQVMt0Os3q6uqx7IxGo0G73TZKXHaeXiMcDhOLxcjlciwtLdHr9ajVasb102q1qNVqDw3KBgIBksmkcZl4LePItwpcytglGCFBKYBSqcT+/j5bW1vnUmEogRvHcYwPUKz8drttovvlctkzCf6nIS4Br7t5zgtJ+ZNdUzweBw7dXc1m01hdlvcMIvH3ZjIZlpaWcBzHBH07nQ7j8djkjO/v73s27VTy9qXmIRAI0Ol0WF9fp9FomMK1k7LLdYjH42QyGRzHYW1tjUQiYXb1m5ubVCoVTyz8vlXgjuNw48YNMpnMsYCC1pr19XXefPNN2u32uQTkJI84l8uxvLxsCj36/T6VSoV6vX6sp4iXeVS62CyRTCbJ5/PMz8+zsLBg8tyHwyG1Wo2DgwNPL7gXiWSaiKukUCjwwgsvGP+vUsokBLz77rtsbGycKSHgWSMFfZcvXzaus2azya1bt4zL5DRDRq6DtBvIZrO89NJLhMNhdnd32d/f5wc/+AH37t3zhCHkOwUeiUTM6phOp0kmkyaYIBkCnU6HXq93Ln4qaYqVSCRIp9PE43ETvByPx3S7XZOe6IU39FE8KPMgGo2SyWRMQYsXraonRdJLJXAp2SWSheHFqslnjWz/RVHJ7jUWi5lClmKxSDqdJhqNEgwGzbNULpdNK1mJB3j5fhdftdzL7nRKeQ7kvhD3qzT2ymazplpXAtWlUolms2l2Il7Adwo8k8mwurpKoVBgeXnZ3HiTyYT9/X3q9ToHBwe0Wq1zsQ6CwSBXrlxhaWmJxcVFcrmceQiGwyEHBwfU63XfK4NMJsO1a9fY2dlha2tr2uKcC6FQyGQLASaHWyoSZ711wEnEPSB58ZFIhGvXrpHL5cjn82QyGWKxmEknDQQCDIdDdnd3abVavPnmm+zt7ZlMDa9fO0nzFDeZFOdIsFKySSQVOZ1Os7i4yOLiIvF4nHQ6TbPZ5K233qLZbLK9ve2pPijgIwUuSlMS6aX0VUrlx+MxnU7H+LbOQ3mfLBCKxWImaDkcDo317cUo/JMipeZeC9I8LVJ8IQ2Y3OckPk5xDbhfk/zfJ8mV9jpyjmJhSpuASCRCNps1QXnp/SJZVFJpKmX27Xbb3Otev9+11qaqWArrRG5R3tKhUTJuZEcfi8VQStHpdGi1WjSbTZO94rVn3TcKXPKVc7kcV65cMUUZcLjS9no9NjY22N7eNpHxsyDR53g8zsLCAsvLy8bPXq/X2djYoF6vs7m5Sbfb9X0wLBgMmr4ufkcUViaTMT5QQdwGr7zyCp1OxyxaQqvVMg2e7t+/bx5+PyvxUChkpu+8/PLLJBIJU5BycoErlUqsr68zGAxot9sMBgMODg6Ma9Iv10JrbfL2JXYlz6i7582rr75qXEbxeJxer0e322V/f5/bt2+bzBWp6Paam9Q3ClysYWlhKdawWApyw8mW6awXWVIUo9GomV7idp3U63UajYavLPDT5hnKOZ0sk/Yz4t+VjnKSrw/vWeaS4yzdCYV4PM5kMjG/7y6h9yvy7MTjcYrFoulCKU3YANMXpdPpUKlUTJHLYDCgVqt5vlz+JFJBOx6PjRUuOwu5ByS9VCq5I5GIaa3baDTY29vzfK8bXyhwSeFLp9PkcjmTBSLVltVq1WxzWq3WmYOX7vxyedDdfTOkAKDVapnOg35Q4OPx2DyMjUbDdNebBatbkKBzLBajWCwyNzdn7hWtteltkcvlmEwmJsApRKNRUqkUrVaLSCRCs9nkzp07NJvNKZ7V2ZCJUfl83jQvCwaDxyZVra+vG193o9EwBtFZKpenjcheqVRYX1/n4OAArTXhcNj4vDOZzLH+6Nvb23zve98zPW68vnj7QoHDew+W+O4k82QymRhfVbfbPXOAQbbf0jtbqjsF8a2J9e3VxlWnIWXh0u+k3++bEvpZQTIKHMcxOcxu6xswjY1Owx1jEf/vxsaGrxW4NF+TYQWRSMQsaLJ7LZfLbGxsmPiOHBeXibs/jF8Qf3e73aZSqZh2ueL7loXeXZFcr9dZX1/3TezDNwpctn3SQEoYjUaUSiUqlcq5tHANhUImgX9xcZF8Pm/6RUvfhP39fdMTw0/WifuB3d/fRynF5cuXPdFV7bxQSplmY+6FF95LIxuNRqYCT1Li3OXSuVyOQCBgeoevrKyQSCSMEvAbMkke4K233jKuFMnACIfDLC8vk0qlTEKATJ6RJl6i5KWfkFdKyR8HUcqdTsfUAZRKJTqdDsvLy2ZiVzQaZXV1lY9//OOmYKff79NqtTz7nPtKgRcKBVKplLGSxRre399nf3//XEaChcNh0+tEFLisxs1mk52dHQ4ODmg2m76xvAV5OCUwNRqNyGazFIvFaYt2brgV+MmFSXYgvV7P9ITZ2tqiVquZCU7SU14C5slkkuXlZWOR+1GB9/t9arWayZpyHIcXX3yRVCplrtPy8jJXrlwxvyMKXCYOyRQrrbWpe/AL9Xqder1uvpdzarVaxpARn/jq6iqrq6usr6/T7XbNzt4q8GeE5HJKU/anRTryOY5DoVAgk8kcyx+W1qOivP2wvXoQsvCd1rxnFhG3QKvV4v79+3Q6Hfb39+l2u7TbbeMSkziHuM2k85wEsr3Qfe5pkNRIpZSJEW1sbJiyecnIkElDkkYnik1GyQ2HQ4rFIrVazRS0SJGLH5+H8XhMqVRiMBiY2IicvwxtEdeZBDa9lm3mawUuvmrx7zUajaf+W5FIxLSkXVtbI5VKmS249PiuVCrcvXvX9xNrJHh1XtWqXkcKd3Z3d3n99deN0paFGWB3d9d0KJSAuVjiyWSSyWTi21iB+IIlwwJgZ2fHKGkpO89ms2SzWebn50kkEqY1syxkKysrTCYTtra22NvbY2tryzRv8+PzMBgMuH37thkTJ9N2JC/8wx/+MK1Wi2QySaPR4Pbt21aBPy3upHx3+pv0LRiPx9TrdaNcZbt38sY6GdASpOmRzNKUtrTS91hmXYoP1e/ITD8JWEl0XnYz8sD77cF0D9eQXYbkdtdqNZMWdjK7wJ1O12w2TVdLKQYaj8fvm+LjN9xyu3deUrQiVqj0AJdFS2ZCSs64tLLodrvMz88b61QUuV+ujcSEtNbU63XjTpH3Wc45k8mYmaASR/FKWqVvFHij0WB7e5tQKGSaskv3tBdffNGk84l1KelR7gstFrv7szA3N8cHP/hBUqmUST0LBoOMx2N2d3c5ODgwFpqfbtLTEHfQYDAw+exyg0oHN2kX6sfFSkadNRoNYrEY29vb3Lx587FSw9rtNhsbG/T7fa5fv040GjWFHul02uQK+/G6PAittUnBLZVKpmhOyu0lkP/CCy+YOFQ+n2dhYYErV65QKpX47ne/a+ILfno2ZHfy7rvvcv/+fZaXl1ldXT1WMHjjxg2TU14ul9nc3KRcLk9bdMBHClwqwdwWsPTjliIbmXcnVpZY7YL49NxNjoRUKmXSFEOh0LHqPInKz1KjJzknGUwQDAZNRkI8HjctV/2IlIDLIiQfkjX0MAXjLvyQ91naNciiPwvFTieRLpVi8EgHTmncJjsTaXglvuJsNku/3zfJBX6piRDkXhA9IT1ywuEwg8HAFPNJp8bBYHBsdz7txeqRClwptQL8R+ASMAG+pLX+NaVUHvgdYA24B/xtrfWDR5ucARnOMBqNcByHcrlspuNIxVw4HOaVV17hpZdeMlvoZrNJuVw2F1n8maFQyFReCdI2U6wPIRAImCnuUqHXaDQolUpTf/OeFgliuofSivsknU5z9epVarUa9XrdM1vFx2UymbC3t0elUmFzc5NwOHwsFexR75kMK3a7l+R+8Ov7/TSIEp5MJsb91Ol0iMfjXL58mYWFBdO5MBwOE41GqdVqvP7661QqFd9eq3K5bPr7TyYT8zyEQiGWl5cpFAqmL4wMipkmj2OBj4Bf0lp/VymVAt5QSn0D+HvAN7XWv6KU+hzwOeCXn5WgonBkGywrILxnLeTzefPzWmtqtZqxnuBQgafTaTOU9GSBjqQmnhxqKz0jkskkjuMwGAx8Wdjgxv2AytfSpU5iCmJx+uk8Jc3taWsC3EMv5EOqFp833G1YZXSgKGtxr4gxJGl47pGGfrxmEu8KhUImd340Gh0z+qQFgxfSiB+pwLXWO8DO0ddNpdRN4DLwaeAnjn7sy8C3eIYKXLqJlctl3n77bXK5nAkyiNvDrWyUUiQSCS5dunTsmKQbnhzeezK4KUHRfr9v5ibW63WTT+vHm/NxEJ9vMBgkm80C7001fx6QXYjjOGZwrQRD/RakO08k4DeZTNjZ2TGVyBLwTyaTJJNJUxgm8SK/0mq1uHfvHtVq1XSulKyc1dVV4vE49+/f586dO1O9H57IB66UWgNeBb4NLBwpd7TWO0qp+Qf8zmvAa3DYc/ppEWug2WyyubnJYDBgcXERrTWxWOyBQwpOVuM9iJO/L4E+KfaQ4a3n1arWq8gQ1/F4bHYbs5J58zhI++B4PG7iJeJmEiNiVt/7RyF+8kqlQqVSIRqNsrS0ZGJPsViMQqFAIBCgXq/7WoH3ej12d3fpdDqmV3oulyMSiTA/P4/jOGdKWz4vHluBK6Uc4PeAX9RaNx43kKO1/hLwJYClpaUz3/nSMe3g4IBbt26ZsmDZ0kmfAyk8kMCVu5Wk9Dk5rR9Gv9+nWq3S7Xa5d+8ezWbTDCuWgJ+fgjSPwj21RIK0EsQVv/hZCqT8RjgcNq0UxE3X7XbpdDqebCc6TXq9nokxLSwsmFa9g8HgfTtcvzIcDimXywyHQ1ZXV427SDqUxuNxk+02jfvisa6yUirMofL+itb6a0eH95RSi0fW9yJwIRM+ZRqIlMHKihiPx41fe3Fxkfn5edrttul5IY1spOf19evXT1XgvV7PDD69deuWKZmfJaXtRnpDyPlJpo7b1/k8KXBpMZpOp48NrpZBIc/LTuRx6Ha7lEolM14wEAjgOI7Jn58FBoMBu7u7Znh5Op0+1tY6mUya8XLT4HGyUBTwG8BNrfWvul76OvBZ4FeOPv/+M5HwAUiwaTQaHWuwI5klMj1DLGfpwidNa04GINyVatK0SrbNs2xxdTodqtUq0WiUQqHwvtdnMWXuSfGT31umDYVCIePyOK+8dffEdnGXFAoFs9iNRiMqlYopmJoFJGNLPiQrSdKXs9kszWbT1IdcNI+zTP448HeBN5VSf3J07J9xqLj/i1Lq54F14G89EwkfgliPklTvnvnnztOUj3A4zOLiIo7jvO8GE9dMrVZja2vLWFyzannD4c25v7/PcDgkFAqxsrIybZEsZ0B2TwsLCziOYxoxSVHTWf+2PFcLCwssLi6aqe/iumy1Wrzzzjum+dUsMJlM6PV6BINBU6gkwe18Ps+1a9fY29vj4OBgKrricbJQ/i/wIDPsE+crzpMjHfYehvRLcU/XObnFk4GnMvfveWn0JMVOJ62L5xW3opKsJj9MsBeLUPp4pFIps6t0n8vDft9tALmPSRxEFLUE9WQgCBzu5Nrttq8mVD0t7h4y055iNRuOqkeQTqf5wAc+gOM4LCwsmF4ObqrVKt///vdpNBrU6/XnQoHro7FTcJjdI1NoEonElCWbHjLcWSoQZSq7BLK9Sjwe5+WXXyaTyXDlyhVSqRS3bt0y8y0fhWRsSS8cd173/Py8cZnIQAhR6sFgkEajwd27d01bBi+PIHtSZH6mXAt3TEiqU89jDsHTMtMKXG4wadIv/Y/lRhXXimyTGo3GMd/384DbR+quOnzecFtUskWW3Z20nfVyAFOqjCWlT6bMi7/aPd8TOLbLcvu1I5GIaVoVi8WIxWLkcjnTnVA6M8q1kR2cKG+/ldI/Cne7anf7DdndSMXutBasmVXgSinm5uZYW1sjm82yurpq+jfItkfm5VWrVTY2Ntjd3fXViLTzQFIjJcfdXeH6PBGLxYjH4xQKBZMSJw2eDg4OKJfLUy+bfhiSKivusPF4zOXLl8nn83Q6HZrNphnsEAgETCGcIFa1ZCCJG0nSSqVjofQKEXdjvV6n2+1SqVRME7lZQgp3ZH5mLBYzzdJ2d3f54Q9/ONXJXDOpwMUnlUwmWVpaIp1Ok81mj92w7nl50uuh1WrNfNbJSWSxckfZn0fccyPFymw0GrTbbTqdjuerUSVbwt2gTHacUg/RbrdNt8FLly4d6wUkuf+ya3X/XSnkEh/3wcEBlUqFer1uJjv5rQvhw3D7tEOhkGkzLQPApZ1Hs9mcet+XmVTgUoCSyWSM/869ZZTxaN1ul42NDe7evevpuXcXQa/XMzdjoVAwk1i01hwcHExbvGeGO7NibW2Nubk5AoEA/X7fLOx+qL4dDAZsbW2ZnYIob8dxjgXdFhYWCAaDJBKJ9zVtCwQCpu+9zAuVaUVSSyHWpwzF9ttc2EchLijJ9U6lUqytrZn+J9LOY29vz/RKmSYzq8ClsEf6ergRBS49xu/evTslSb2DbK8lNiCNvyT9clYRi7NYLHLjxg2z2A+HQ2Nlej0DBQ53UDs7OwSDQdrttukaODc3dyz76rTnwc1gMKBardJsNtne3qbX61EqlUznPb91p3xSJNFBupM6jsPKyoppySGtBDY3N6nX61Nf1GdSgbsn8sB7WyIJWo5GIzOgwQv9DLzAcDg0lWYn8+dnEbE4ZXCxDPdVShkrVIbhel15u5HS/8lkQqlUotvtHpu0tL29/dA0UemhLi0l3JObZsnSdiOzBOLxOIuLi6ysrJjME7knRqORaeDl3plNm5lU4JJZcdqDJ1PZ79+/z+3bt2feonhcer0e1WoVx3GOjd2axd4fMpEpFAqZ3tZLS0skk0mTGlatVtnd3fXkINuH4W7CVqlU3tdl81E5y+42yu6v3Z9nDXEtzc/Ps7a2xtWrV4+9LgHi9fV1arUaGxsb7O3teeJ6zKQCFyu72WyysbFxrCG/KHBJF5xVq+JJkUBUo9Fgc3PT+PpkALCfkfRASSmVz+Fw2BSlhMNhxuOxmVhfqVSM68QLD+qTMOsK91ng7osvoxRlLKMUJ1WrVWN5e0VvzKQCl2j8O++8w/r6+vtel22mH4f2Piuk+rRWq3H//v1j/Z/9ZIGehsx3TKVSXL16lWQySbFYNAVd0rmy1Wqxvr7OG2+8QbfbfS6zkp5XxHiRbBpZyNvtNtvb23Q6HTY3Nz1XDzCTCly2fxIltzwasUCkOGFWcbsQpExeKWWsLCnmcg9wsMw2koIpqZaS5y5j5CTXXTJvvMRMKnCLxc14PDa5zDdv3jR9PcStEggETMtgCWDOou/fcjoyflGU9s2bNxmNRqadhrhMvKa8wSpwy3OAxD7Er2mxnESqSGVYjF94ftvOWSwWi8+xCtxisVh8ilXgFovF4lOsArdYLBafYhW4xWKx+BSrwC0Wi8WnWAVusVgsPkVdZLGCUmofaAP+SbSEIlbeZ4mV99li5X22XJS8V7TWcycPXqgCB1BK/bHW+mMX+k/PgJX32WLlfbZYeZ8t05bXulAsFovFp1gFbrFYLD5lGgr8S1P4n2fByvtssfI+W6y8z5apynvhPnCLxWKxnA/WhWKxWCw+xSpwi8Vi8SkXpsCVUp9USr2tlLqjlPrcRf3fx0UptaKU+t9KqZtKqe8rpX7h6HheKfUNpdTto8+5acvqRikVVEp9Tyn1B0ffe13erFLqd5VSt46u9V/0ssxKqX98dD+8pZT6baVUzEvyKqV+UylVUkq95Tr2QPmUUp8/egbfVkr9dY/I+y+O7oc/U0r9V6VU1svyul77J0oprZQquo5dqLwXosCVUkHg3wA/BXwA+Fml1Acu4n8/ASPgl7TWLwM/BvyDIxk/B3xTa30D+ObR917iF4Cbru+9Lu+vAf9Da/3ngFc4lN2TMiulLgP/CPiY1vpDQBD4DN6S97eAT544dqp8R/fzZ4APHv3Ovz16Ni+S3+L98n4D+JDW+iPAO8DnwdPyopRaAX4SWHcdu3B5L8oC/1Hgjtb6h1rrAfBV4NMX9L8fC631jtb6u0dfNzlULJc5lPPLRz/2ZeBvTkXAU1BKLQN/A/h112Evy5sG/grwGwBa64HWuoaHZeZwalVcKRUCEsA2HpJXa/1/gMqJww+S79PAV7XWfa31XeAOh8/mhXGavFrrP9Jay6Tg14Hlo689Ke8R/wr4p4A7C+TC5b0oBX4Z2HB9v3l0zJMopdaAV4FvAwta6x04VPLA/BRFO8m/5vAmmriOeVnea8A+8B+O3D6/rpRK4lGZtdZbwL/k0MraAepa6z/Co/K6eJB8fngO/z7w34++9qS8SqlPAVta6z898dKFy3tRClydcsyT+YtKKQf4PeAXtdaNacvzIJRSPwOUtNZvTFuWJyAE/Ajw77TWr3LYF8cT7pLTOPIdfxq4CiwBSaXUz01XqjPh6edQKfUFDl2ZX5FDp/zYVOVVSiWALwD//LSXTzn2TOW9KAW+Cay4vl/mcCvqKZRSYQ6V91e01l87OrynlFo8en0RKE1LvhP8OPAppdQ9Dl1Sf1Up9Z/xrrxweB9saq2/ffT973Ko0L0q818D7mqt97XWQ+BrwF/Cu/IKD5LPs8+hUuqzwM8Af0e/V5ziRXmvc7ig/+nRs7cMfFcpdYkpyHtRCvw7wA2l1FWlVIRDR//XL+h/PxZKKcWhb/am1vpXXS99Hfjs0defBX7/omU7Da3157XWy1rrNQ6v5//SWv8cHpUXQGu9C2wopV46OvQJ4Ad4V+Z14MeUUomj++MTHMZGvCqv8CD5vg58RikVVUpdBW4A/28K8h1DKfVJ4JeBT2mtO66XPCev1vpNrfW81nrt6NnbBH7k6N6+eHm11hfyAfw0hxHmd4EvXNT/fQL5/jKH250/A/7k6OOngQKHkfzbR5/z05b1FNl/AviDo689LS/w54E/PrrO/w3IeVlm4IvALeAt4D8BUS/JC/w2h/75IYfK5OcfJh+H2/93gbeBn/KIvHc49B3Lc/fvvSzvidfvAcVpyWtL6S0Wi8Wn2EpMi8Vi8SlWgVssFotPsQrcYrFYfIpV4BaLxeJTrAK3WCwWn2IVuMVisfgUq8AtFovFp/x/tCtVtAGppXQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show images\n",
    "imshow(torchvision.utils.make_grid(images[:5]))\n",
    "# print labels\n",
    "print(' '.join('%5s' % labels[j] for j in range(5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:52:36.851812Z",
     "start_time": "2024-08-03T05:52:36.847810Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:52:41.084396Z",
     "start_time": "2024-08-03T05:52:41.074126Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# CNN Model (2 conv layer)\n",
    "class CNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        self.layer1 = nn.Sequential(\n",
    "            # in_channels=1 (图片只有一种颜色), out_channels=16(16个不同的卷积核，产生16个卷积结果)\n",
    "            # kernel_size: 每个卷积核的边长， padding：图片周围加两圈0\n",
    "            nn.Conv2d(1, 16, kernel_size=5, padding=2),\n",
    "            # apply batch normalization\n",
    "            nn.BatchNorm2d(16),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(2))\n",
    "            # 每个kernel take all channels from input and comebine them into one channel\n",
    "        self.layer2 = nn.Sequential(\n",
    "            nn.Conv2d(16, 32, kernel_size=5, padding=2),\n",
    "            nn.BatchNorm2d(32),\n",
    "            nn.ReLU(),\n",
    "            nn.MaxPool2d(2))\n",
    "        self.fc = nn.Linear(7*7*32, 10) \n",
    "        # a linear transformation AX +b: 把7*7*32维的张量 reduce 成 (x1,..,x10)的10维向量\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.layer1(x)\n",
    "        out = self.layer2(out)\n",
    "        out = out.view(out.size(0), -1)\n",
    "        out = self.fc(out)\n",
    "        return out\n",
    "\n",
    "cnn = CNN()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:52:42.210173Z",
     "start_time": "2024-08-03T05:52:42.206381Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Loss and Optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(cnn.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:52:44.942867Z",
     "start_time": "2024-08-03T05:52:44.934975Z"
    },
    "jupyter": {
     "outputs_hidden": true
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0275, 0.5725, 0.4314, 0.0157, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5529, 1.0000,\n",
       "          0.8157, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.4824, 0.9922, 0.9922, 0.0824, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1765, 0.8667, 0.9922,\n",
       "          0.3804, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.3373, 0.9373, 0.9922, 0.6588, 0.0471, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5333, 0.9922, 0.8745,\n",
       "          0.0863, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.3569, 0.8941, 0.9922, 0.6235, 0.2667, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.1098, 0.7294, 0.9922, 0.8275,\n",
       "          0.0588, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0196,\n",
       "          0.8941, 0.9922, 0.9608, 0.0980, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.3765, 0.9529, 0.9922, 0.4039,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.6000,\n",
       "          0.9922, 0.9922, 0.1922, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0510, 0.7843, 0.9922, 0.6980, 0.0314,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.4510, 0.9451,\n",
       "          0.9922, 0.6157, 0.0392, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.6314, 0.9922, 0.9922, 0.4235, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0667, 0.8392, 0.9922,\n",
       "          0.8627, 0.1569, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.1647, 0.9725, 0.9922, 0.5020, 0.1059, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3490, 0.9725, 0.9922,\n",
       "          0.5882, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0157, 0.5725, 0.9922, 0.9765, 0.1922, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.6627, 0.9922, 0.9804,\n",
       "          0.1725, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.3255, 0.9922, 0.9725, 0.5804, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.6627, 0.9922, 0.9882,\n",
       "          0.3176, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.1882, 0.7804, 0.9922, 0.9333, 0.1373, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3451, 0.9686, 0.9922,\n",
       "          0.8863, 0.1961, 0.0824, 0.0000, 0.0000, 0.0431, 0.0863, 0.0863,\n",
       "          0.7333, 0.9922, 0.9922, 0.4902, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0667, 0.7373, 0.9922,\n",
       "          0.9922, 0.9922, 0.9804, 0.6588, 0.6588, 0.8235, 0.9922, 0.9922,\n",
       "          0.9922, 0.9922, 0.9922, 0.8157, 0.3922, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1843, 0.7843,\n",
       "          0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922,\n",
       "          0.9922, 0.9922, 0.6824, 0.1882, 0.0627, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0196,\n",
       "          0.2941, 0.4667, 0.8431, 0.6235, 0.4667, 0.5412, 0.9922, 0.9922,\n",
       "          0.9843, 0.5059, 0.0275, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3098, 0.9922, 0.9922,\n",
       "          0.8353, 0.1451, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.6353, 0.9098, 0.9922, 0.9922,\n",
       "          0.3059, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0078, 0.6549, 0.9882, 0.9922, 0.9647, 0.5569,\n",
       "          0.0118, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.1608, 0.9922, 0.9922, 0.9922, 0.6627, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.1059, 0.9922, 0.9922, 0.9412, 0.0549, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000]]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:52:48.450913Z",
     "start_time": "2024-08-03T05:52:48.445965Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 28, 28])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:56:09.596096Z",
     "start_time": "2024-08-03T05:53:16.727180Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/5], Iter [200/600] Loss: 0.1033\n",
      "Epoch [1/5], Iter [400/600] Loss: 0.1002\n",
      "Epoch [1/5], Iter [600/600] Loss: 0.0250\n",
      "Epoch [2/5], Iter [200/600] Loss: 0.0394\n",
      "Epoch [2/5], Iter [400/600] Loss: 0.0169\n",
      "Epoch [2/5], Iter [600/600] Loss: 0.0215\n",
      "Epoch [3/5], Iter [200/600] Loss: 0.0244\n",
      "Epoch [3/5], Iter [400/600] Loss: 0.0213\n",
      "Epoch [3/5], Iter [600/600] Loss: 0.0270\n",
      "Epoch [4/5], Iter [200/600] Loss: 0.0183\n",
      "Epoch [4/5], Iter [400/600] Loss: 0.0212\n",
      "Epoch [4/5], Iter [600/600] Loss: 0.0315\n",
      "Epoch [5/5], Iter [200/600] Loss: 0.0025\n",
      "Epoch [5/5], Iter [400/600] Loss: 0.0186\n",
      "Epoch [5/5], Iter [600/600] Loss: 0.0563\n"
     ]
    }
   ],
   "source": [
    "# Train the Model\n",
    "for epoch in range(num_epochs):\n",
    "    for i, (images, labels) in enumerate(train_loader):\n",
    "        images = Variable(images)\n",
    "        labels = Variable(labels)\n",
    "\n",
    "        # Forward + Backward + Optimize\n",
    "        optimizer.zero_grad()\n",
    "        outputs = cnn(images)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        if (i+1) % 200 == 0:\n",
    "            print ('Epoch [%d/%d], Iter [%d/%d] Loss: %.4f'\n",
    "                   %(epoch+1, num_epochs, i+1, len(train_dataset)//batch_size, loss.data.item()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T05:57:07.803709Z",
     "start_time": "2024-08-03T05:57:05.161085Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy of the model on the 10000 test images: 98 %\n"
     ]
    }
   ],
   "source": [
    "# Test the Model\n",
    "cnn.eval()  # Change model to 'eval' mode (BN uses moving mean/var).\n",
    "correct = 0\n",
    "total = 0\n",
    "for images, labels in test_loader:\n",
    "    images = Variable(images)\n",
    "    outputs = cnn(images)\n",
    "    _, predicted = torch.max(outputs.data, 1) # 自动返回数列中最大的value和对应的index\n",
    "    total += labels.size(0)\n",
    "    correct += (predicted == labels).sum()\n",
    "\n",
    "print('Test Accuracy of the model on the 10000 test images: %d %%' % (100 * correct / total))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T06:06:36.370528Z",
     "start_time": "2024-08-03T06:06:36.241169Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAANh0lEQVR4nO3df6zddX3H8dfL/sJeYFKwtSuVKqKxOsHlCppuSw3DAYYUo2w0GekSZskGCSxmG2ExkmxxjIiETWdSR2clCFOBQLRzksaNkLHKhZRSKFuRdVh71wvUrUXgtqXv/XG/LJdyz+dezvd7zve07+cjuTnnfN/ne77vfHtf/X7v+XzP+TgiBODY95a2GwDQH4QdSIKwA0kQdiAJwg4kMbufG5vreXGchvq5SSCVV/QLHYhxT1WrFXbb50u6RdIsSX8XETeUnn+chnSOz62zSQAFm2NTx1rXp/G2Z0n6qqQLJC2XtNr28m5fD0Bv1fmb/WxJT0fEMxFxQNKdklY10xaAptUJ+xJJP530eFe17HVsr7U9YnvkoMZrbA5AHXXCPtWbAG+49jYi1kXEcEQMz9G8GpsDUEedsO+StHTS41Ml7a7XDoBeqRP2hyWdYftdtudKulTSfc20BaBpXQ+9RcQh21dJ+idNDL2tj4gnGusMQKNqjbNHxEZJGxvqBUAPcbkskARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IIlaUzbb3ilpv6RXJR2KiOEmmgLQvFphr3w8Ip5v4HUA9BCn8UASdcMekn5o+xHba6d6gu21tkdsjxzUeM3NAehW3dP4FRGx2/ZCSffbfioiHpj8hIhYJ2mdJJ3oBVFzewC6VOvIHhG7q9sxSfdIOruJpgA0r+uw2x6yfcJr9yV9QtK2phoD0Kw6p/GLJN1j+7XX+VZE/KCRrgA0ruuwR8Qzks5ssBcAPcTQG5AEYQeSIOxAEoQdSIKwA0k08UGYFF747Mc61t552dPFdZ8aW1SsHxifU6wvuaNcn7/rxY61w1ueLK6LPDiyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLPP0J/88bc61j499PPyyqfX3PjKcnnnoZc61m557uM1N370+vHYaR1rQzf9UnHd2Zseabqd1nFkB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkHNG/SVpO9II4x+f2bXtN+sVnzulYe/5D5f8zT9pe3sc/f7+L9bkf+p9i/cYP3t2xdt5bXy6u+/2Xji/WPzm/82fl63o5DhTrm8eHivWVxx3setvv+f4Vxfp71z7c9Wu3aXNs0r7YO+UvFEd2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCz7PP0NB3Nxdq9V77xHqr62/esbJj7S9WLCtv+1/K33l/48r3dNHRzMx++XCxPrR1tFg/+YG7ivVfmdv5+/bn7yx/F/+xaNoju+31tsdsb5u0bIHt+23vqG5P6m2bAOqayWn8NySdf8SyayVtiogzJG2qHgMYYNOGPSIekLT3iMWrJG2o7m+QdHGzbQFoWrdv0C2KiFFJqm4Xdnqi7bW2R2yPHNR4l5sDUFfP342PiHURMRwRw3M0r9ebA9BBt2HfY3uxJFW3Y821BKAXug37fZLWVPfXSLq3mXYA9Mq04+y279DEN5efYnuXpC9IukHSt21fLulZSZf0skmUHfrvPR1rQ3d1rknSq9O89tB3X+iio2bs+f2PFesfmFv+9f3S3vd1rC37+2eK6x4qVo9O04Y9IlZ3KB2d30IBJMXlskAShB1IgrADSRB2IAnCDiTBR1zRmtmnLS3Wv3LdV4r1OZ5VrH/nlt/sWDt59KHiuscijuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7GjNU3+0pFj/yLzyVNZPHChPR73gyZfedE/HMo7sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+zoqfFPfqRj7dHP3DzN2uUZhP7g6quL9bf+64+nef1cOLIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs6Onnr2g8/HkeJfH0Vf/53nF+vwfPFasR7Gaz7RHdtvrbY/Z3jZp2fW2f2Z7S/VzYW/bBFDXTE7jvyHp/CmW3xwRZ1U/G5ttC0DTpg17RDwgaW8fegHQQ3XeoLvK9tbqNP+kTk+yvdb2iO2RgxqvsTkAdXQb9q9JOl3SWZJGJd3U6YkRsS4ihiNieM40H2wA0DtdhT0i9kTEqxFxWNLXJZ3dbFsAmtZV2G0vnvTwU5K2dXougMEw7Ti77TskrZR0iu1dkr4gaaXtszQxlLlT0hW9axGD7C0nnFCsX/brD3as7Tv8SnHdsS++u1ifN/5wsY7XmzbsEbF6isW39qAXAD3E5bJAEoQdSIKwA0kQdiAJwg4kwUdcUcuO6z9QrH/vlL/tWFu149PFdedtZGitSRzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtlR9L+/+9Fifevv/HWx/pNDBzvWXvyrU4vrztNosY43hyM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOHtys5f8crF+zef/oVif5/Kv0KWPXdax9vZ/5PPq/cSRHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJz9GOfZ5X/iM7+3q1i/5PgXivXb9y8s1hd9vvPx5HBxTTRt2iO77aW2f2R7u+0nbF9dLV9g+37bO6rbk3rfLoBuzeQ0/pCkz0XE+yV9VNKVtpdLulbSpog4Q9Km6jGAATVt2CNiNCIere7vl7Rd0hJJqyRtqJ62QdLFPeoRQAPe1Bt0tpdJ+rCkzZIWRcSoNPEfgqQp/3izvdb2iO2Rgxqv2S6Abs047LaPl3SXpGsiYt9M14uIdRExHBHDczSvmx4BNGBGYbc9RxNBvz0i7q4W77G9uKovljTWmxYBNGHaoTfblnSrpO0R8eVJpfskrZF0Q3V7b086RD1nvq9Y/vOFt9V6+a9+8ZJi/W2PPVTr9dGcmYyzr5B0maTHbW+pll2niZB/2/blkp6VVP5XB9CqacMeEQ9Kcofyuc22A6BXuFwWSIKwA0kQdiAJwg4kQdiBJPiI6zFg1vL3dqytvbPe5Q/L119ZrC+77d9qvT76hyM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOPsx4Kk/7PzFvhfNn/GXCk3p1H8+UH5CRK3XR/9wZAeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBhnPwq8ctHZxfqmi24qVOc32wyOWhzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJmczPvlTSNyW9Q9JhSesi4hbb10v6rKTnqqdeFxEbe9VoZrtXzCrW3zm7+7H02/cvLNbn7Ct/np1Psx89ZnJRzSFJn4uIR22fIOkR2/dXtZsj4ku9aw9AU2YyP/uopNHq/n7b2yUt6XVjAJr1pv5mt71M0oclba4WXWV7q+31tqf8biTba22P2B45qPF63QLo2ozDbvt4SXdJuiYi9kn6mqTTJZ2liSP/lBdoR8S6iBiOiOE5mle/YwBdmVHYbc/RRNBvj4i7JSki9kTEqxFxWNLXJZU/rQGgVdOG3bYl3Sppe0R8edLyxZOe9ilJ25pvD0BTZvJu/ApJl0l63PaWatl1klbbPksToy87JV3Rg/5Q01++sLxYf+i3lhXrMfp4g92gTTN5N/5BSZ6ixJg6cBThCjogCcIOJEHYgSQIO5AEYQeSIOxAEo4+Trl7ohfEOT63b9sDstkcm7Qv9k41VM6RHciCsANJEHYgCcIOJEHYgSQIO5AEYQeS6Os4u+3nJP3XpEWnSHq+bw28OYPa26D2JdFbt5rs7bSIePtUhb6G/Q0bt0ciYri1BgoGtbdB7Uuit271qzdO44EkCDuQRNthX9fy9ksGtbdB7Uuit271pbdW/2YH0D9tH9kB9AlhB5JoJey2z7f977aftn1tGz10Ynun7cdtb7E90nIv622P2d42adkC2/fb3lHdTjnHXku9XW/7Z9W+22L7wpZ6W2r7R7a3237C9tXV8lb3XaGvvuy3vv/NbnuWpP+QdJ6kXZIelrQ6Ip7sayMd2N4paTgiWr8Aw/ZvSHpR0jcj4oPVshsl7Y2IG6r/KE+KiD8dkN6ul/Ri29N4V7MVLZ48zbikiyX9nlrcd4W+flt92G9tHNnPlvR0RDwTEQck3SlpVQt9DLyIeEDS3iMWr5K0obq/QRO/LH3XobeBEBGjEfFodX+/pNemGW913xX66os2wr5E0k8nPd6lwZrvPST90PYjtte23cwUFkXEqDTxyyNpYcv9HGnaabz76Yhpxgdm33Uz/XldbYR9qu/HGqTxvxUR8auSLpB0ZXW6ipmZ0TTe/TLFNOMDodvpz+tqI+y7JC2d9PhUSbtb6GNKEbG7uh2TdI8GbyrqPa/NoFvdjrXcz/8bpGm8p5pmXAOw79qc/ryNsD8s6Qzb77I9V9Klku5roY83sD1UvXEi20OSPqHBm4r6PklrqvtrJN3bYi+vMyjTeHeaZlwt77vWpz+PiL7/SLpQE+/I/0TSn7XRQ4e+3i3psernibZ7k3SHJk7rDmrijOhySSdL2iRpR3W7YIB6u03S45K2aiJYi1vq7dc08afhVklbqp8L2953hb76st+4XBZIgivogCQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJ/wNGNvRI2D7VDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "case=test_dataset[0][0]\n",
    "case=case.reshape(1, 1, 28, 28)\n",
    "plt.imshow(case[0][0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-03T06:06:37.178329Z",
     "start_time": "2024-08-03T06:06:37.167624Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[7]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = cnn(case)\n",
    "pred = output.argmax(dim=1, keepdim=True)\n",
    "pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Chapter 9, Deep Learning. Ian Goodfellow et al. 2016 \n",
    "http://www.deeplearningbook.org/contents/convnets.html\n",
    "- CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization https://poloclub.github.io/cnn-explainer/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![image.png](img/chengjun2.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}