{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 前言\n",
    "\n",
    "原文翻译自：[Deep Learning with PyTorch: A 60 Minute Blitz](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)\n",
    "\n",
    "翻译：[林不清](https://www.zhihu.com/people/lu-guo-92-42-88)\n",
    "\n",
    "整理：机器学习初学者公众号 ![gongzhong](images/gongzhong.jpg)\n",
    "\n",
    "## 目录\n",
    "\n",
    "[60分钟入门PyTorch（一）——Tensors](https://zhuanlan.zhihu.com/p/347676809)\n",
    "\n",
    "[60分钟入门PyTorch（二）——Autograd自动求导](https://zhuanlan.zhihu.com/p/347672836)\n",
    "\n",
    "[60分钟入门Pytorch（三）——神经网络](https://zhuanlan.zhihu.com/p/347678492)\n",
    "\n",
    "[60分钟入门PyTorch（四）——训练一个分类器](https://zhuanlan.zhihu.com/p/347681137)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autograd：自动求导\n",
    "\n",
    "torch.autograd是pytorch自动求导的工具，也是所有神经网络的核心。我们首先先简单了解一下这个包如何训练神经网络。\n",
    "\n",
    "**背景介绍**\n",
    "\n",
    "    神经网络(NNs)是作用在输入数据上的一系列嵌套函数的集合，这些函数由权重和误差来定义，被存储在PyTorch中的tensors中。\n",
    "    神经网络训练的两个步骤：\n",
    "    前向传播：在前向传播中，神经网络通过将接收到的数据与每一层对应的权重和误差进行运算来对正确的输出做出最好的预测。\n",
    "    反向传播：在反向传播中，神经网络调整其参数使得其与输出误差成比例。反向传播基于梯度下降策略，是链式求导法则的一个应用，以目标的负梯度方向对参数进行调整。\n",
    "    更加详细的介绍可以参照下述地址：\n",
    "[3Blue1Brown](https://www.youtube.com/watch?v=tIeHLnjs5U8)\n",
    "\n",
    "**Pytorch应用**\n",
    "\n",
    "来看一个简单的示例，我们从torchvision加载一个预先训练好的resnet18模型，接着创建一个随机数据tensor来表示一有3个通道、高度和宽度为64的图像，其对应的标签初始化为一些随机值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: \"https://download.pytorch.org/models/resnet18-5c106cde.pth\" to C:\\Users\\hai_g/.cache\\torch\\checkpoints\\resnet18-5c106cde.pth\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eecbf11d7b4a414a9e3032de9e86c5c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=46827520.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import torch, torchvision\n",
    "model = torchvision.models.resnet18(pretrained=True)\n",
    "data = torch.rand(1, 3, 64, 64)\n",
    "labels = torch.rand(1, 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们将输入数据向输出方向传播到模型的每一层中来预测输出，这就是前向传播。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = model(data) # 前向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们利用模型的预测输出和对应的权重来计算误差，然后反向传播误差。完成计算后，您可以调用.backward()并自动计算所有梯度。此张量的梯度将累积到.grad属性中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = (prediction - labels).sum()\n",
    "loss.backward() # 反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，我们加载一个优化器，在本例中，SGD的学习率为0.01，momentum 为0.9。我们在优化器中注册模型的所有参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "optim = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们调用`.step()`来执行梯度下降，优化器通过存储在`.grad`中的梯度来调整每个参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "optim.step() #梯度下降"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，你已经具备了训练神经网络所需所有条件。下面几节详细介绍了Autograd包的工作原理——可以跳过它们。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "********************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Autograd中的求导**\n",
    "\n",
    "先来看一下`autograd`是如何收集梯度的。我们创建两个张量a和b并设置requires_grad = True以跟踪它的计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "a = torch.tensor([2., 3.], requires_grad=True)\n",
    "b = torch.tensor([6., 4.], requires_grad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着在``a``和``b``的基础上创建张量``Q``\n",
    "\\begin{align}Q = 3a^3 - b^2\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q = 3*a**3 - b**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设``a``和``b``是一个神经网络的权重，``Q``是它的误差，在神经网络训练中，我们需要w.r.t参数的误差梯度，即\n",
    "\n",
    "\\begin{align}\\frac{\\partial Q}{\\partial a} = 9a^2\\end{align}\n",
    "\n",
    "\\begin{align}\\frac{\\partial Q}{\\partial b} = -2b\\end{align}\n",
    "当我们调用``Q``的``.backward()``时，autograd计算这些梯度并把它们存储在张量的 ``.grad``属性中。我们需要在``Q.backward()``中显式传递``gradient``，``gradient``是一个与`` Q ``相同形状的张量，它表示Q w.r.t本身的梯度，即\n",
    "\\begin{align}\\frac{dQ}{dQ} = 1\\end{align}\n",
    "同样，我们也可以将``Q``聚合为一个标量并隐式向后调用，如``Q.sum().backward()``。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "external_grad = torch.tensor([1., 1.])\n",
    "Q.backward(gradient=external_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在梯度都被存放在``a.grad``和``b.grad``中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([True, True])\n",
      "tensor([True, True])\n"
     ]
    }
   ],
   "source": [
    "# 检查一下存储的梯度是否正确\n",
    "print(9*a**2 == a.grad)\n",
    "print(-2*b == b.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**可选阅读----用autograd进行向量计算**\n",
    "\n",
    "在数学上，如果你有一个向量值函数𝑦⃗ =𝑓(𝑥⃗ ) ，则𝑦⃗ 相对于𝑥⃗ 的梯度是雅可比矩阵：\n",
    "\\begin{align}J\n",
    "     =\n",
    "      \\left(\\begin{array}{cc}\n",
    "      \\frac{\\partial \\bf{y}}{\\partial x_{1}} &\n",
    "      ... &\n",
    "      \\frac{\\partial \\bf{y}}{\\partial x_{n}}\n",
    "      \\end{array}\\right)\n",
    "     =\n",
    "     \\left(\\begin{array}{ccc}\n",
    "      \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{1}}{\\partial x_{n}}\\\\\n",
    "      \\vdots & \\ddots & \\vdots\\\\\n",
    "      \\frac{\\partial y_{m}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
    "      \\end{array}\\right)\\end{align}\n",
    "      \n",
    "一般来说，torch.autograd是一个计算雅可比向量积的引擎。 也就是说，给定任何向量𝑣=(𝑣1𝑣2...𝑣𝑚)𝑇，计算乘积𝐽⋅𝑣。如果𝑣恰好是标量函数的梯度𝑙=𝑔(𝑦⃗ )，即$v={{(\\frac{\\partial l}{\\partial {{y}_{1}}}\\cdots \\frac{\\partial l}{\\partial {{y}_{m}}})}^{T}}$ 然后根据链式法则，雅可比向量乘积将是𝑙相对于𝑥⃗ 的梯度\n",
    "\n",
    "\\begin{align}J^{T}\\cdot \\vec{v}=\\left(\\begin{array}{ccc}\n",
    "      \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{1}}\\\\\n",
    "      \\vdots & \\ddots & \\vdots\\\\\n",
    "      \\frac{\\partial y_{1}}{\\partial x_{n}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
    "      \\end{array}\\right)\\left(\\begin{array}{c}\n",
    "      \\frac{\\partial l}{\\partial y_{1}}\\\\\n",
    "      \\vdots\\\\\n",
    "      \\frac{\\partial l}{\\partial y_{m}}\n",
    "      \\end{array}\\right)=\\left(\\begin{array}{c}\n",
    "      \\frac{\\partial l}{\\partial x_{1}}\\\\\n",
    "      \\vdots\\\\\n",
    "      \\frac{\\partial l}{\\partial x_{n}}\n",
    "      \\end{array}\\right)\\end{align}\n",
    "\n",
    "雅可比向量积的这种特性使得将外部梯度馈送到具有非标量输出的模型中非常方便。``external_grad`` 代表$\\vec{v}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**图计算**\n",
    "\n",
    "从概念上讲，autograd在由[函数](https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)对象组成的有向无环图(DAG)中保存数据(tensor)和所有执行的操作(以及产生的新tensor)的记录，在这个DAG中，叶节点是输入数据，根节点是输出数据，通过从根节点到叶节点跟踪这个图，您可以使用链式法则自动计算梯度。\n",
    "\n",
    "在前向传播中，autograd同时完成两件事情：\n",
    "* 运行所请求的操作来计算结果tensor\n",
    "* 保持DAG中操作的梯度\n",
    "\n",
    "在反向传播中，当在DAG根节点上调用``.backward()``时，反向传播启动，``autograd``接下来完成：\n",
    "* 计算每一个``.grad_fn``的梯度\n",
    "* 将它们累加到各自张量的.grad属性中\n",
    "* 利用链式法则，一直传播到叶节点\n",
    "\n",
    "下面是DAG的可视化表示的示例。图中，箭头表示前向传播的方向，节点表示向前传递中每个操作的向后函数。蓝色标记的叶节点代表叶张量 ``a``和``b``\n",
    "\n",
    "![autograd](images/autograd.jpg)\n",
    "\n",
    "<div class=\"alert alert-info\"><h4>注意</h4><p>**DAG在PyTorch中是动态的**\n",
    " 值得注意的是图是重新开始创建的; 在调用每一个``.backward()``后，autograd开始填充一个新图，这就是能够在模型中使用控制流语句的原因。你可以根据需求在每次迭代时更改形状、大小和操作。\n",
    "  </p></div>\n",
    "\n",
    "\n",
    " \n",
    "``torch.autograd``追踪所有``requires_grad``为``True``的张量的相关操作。对于不需要梯度的张量，将此属性设置为False将其从梯度计算DAG中排除。\n",
    "操作的输出张量将需要梯度，即使只有一个输入张量``requires_grad=True``。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Does `a` require gradients? : False\n",
      "Does `b` require gradients?: True\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(5, 5)\n",
    "y = torch.rand(5, 5)\n",
    "z = torch.rand((5, 5), requires_grad=True)\n",
    "\n",
    "a = x + y\n",
    "print(f\"Does `a` require gradients? : {a.requires_grad}\")\n",
    "b = x + z\n",
    "print(f\"Does `b` require gradients?: {b.requires_grad}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在神经网络中，不计算梯度的参数通常称为冻结参数。如果您事先知道您不需要这些参数的梯度，那么“冻结”部分模型是很有用的(这通过减少autograd计算带来一些性能好处)。\n",
    "另外一个常见的用法是微调一个[预训练好的网络](https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html)，在微调的过程中，我们冻结大部分模型——通常，只修改分类器来对新的<标签>做出预测,让我们通过一个小示例来演示这一点。与前面一样，我们加载一个预先训练好的resnet18模型，并冻结所有参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn, optim\n",
    "\n",
    "model = torchvision.models.resnet18(pretrained=True)\n",
    "\n",
    "# 冻结网络中所有的参数\n",
    "for param in model.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设我们想在一个有10个标签的新数据集上微调模型。在resnet中，分类器是最后一个线性层模型``model.fc``。我们可以简单地用一个新的线性层(默认未冻结)代替它作为我们的分类器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fc = nn.Linear(512, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在除了``model.fc``的参数外，模型的其他参数均被冻结，参与计算的参数是``model.fc``的权值和偏置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 只优化分类器\n",
    "optimizer = optim.SGD(model.fc.parameters(), lr=1e-2, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意，尽管我们注册了优化器中所有参数，但唯一参与梯度计算(并因此在梯度下降中更新)的参数是分类器的权值和偏差。\n",
    "[torch.no_grad()](https://pytorch.org/docs/stable/generated/torch.no_grad.html)中也具有相同的功能。\n",
    "\n",
    "**拓展阅读**\n",
    "* [就地修改操作以及多线程Autograd](https://pytorch.org/docs/stable/notes/autograd.html)\n",
    "* [反向模式autodiff的示例](https://colab.research.google.com/drive/1VpeE6UvEPRz9HmsHh1KS0XxXjYu533EC)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}