暂无图片
暂无图片
暂无图片
暂无图片
暂无图片
首页 / 基于Pytorch构建ResNet网络对cifar-10进行分类

基于Pytorch构建ResNet网络对cifar-10进行分类

追梦IT人 2023-08-21
260

何凯明等人在2015年提出的ResNet，在ImageNet比赛classification任务上获得第一名，获评CVPR2016最佳论文。

自从深度神经网络在ImageNet大放异彩之后，后来问世的深度神经网络就朝着网络层数越来越深的方向发展，从LeNet、AlexNet、VGG-Net、GoogLeNet。直觉上我们不难得出结论：增加网络深度后，网络可以进行更加复杂的特征提取，因此更深的模型可以取得更好的结果。

但事实并非如此，人们发现随着网络深度的增加，模型精度并不总是提升，并且这个问题显然不是由过拟合（overfitting）造成的，因为网络加深后不仅测试误差变高了，它的训练误差竟然也变高了。作者提出，这可能是因为更深的网络会伴随梯度消失/爆炸问题，从而阻碍网络的收敛。将这种加深网络深度但网络性能却下降的现象称为退化问题。例如传统神经网络的层数从20增加为56时，网络的训练误差和测试误差均出现了明显的增长。

何博士灵感爆发，他提出了残差学习来解决退化问题。对于一个堆积层结构（几层堆积而成）当输入为  时其学习到的特征记为 H(x) ，现在我们希望其可以学习到残差 F(x)=H(x)-x ，这样其实原始的学习特征是 F(x)+x 。之所以这样是因为残差学习相比原始特征直接学习更容易。当残差为0时，此时堆积层仅仅做了恒等映射，至少网络性能不会下降，实际上残差不会为0，这也会使得堆积层在输入特征基础上学习到新的特征，从而拥有更好的性能。残差学习的结构如图4所示。这有点类似与电路中的“短路”，所以是一种短路连接

而且网络的训练，也并不是越多轮次越好，这一点可以参考一下GoogLeNet网络的训练和验证数据，基本上到了60轮次，就出现上下波动了。

ResNet主要有五种主要形式：Res18，Res34，Res50，Res101，Res152；

如下图所示，每个网络都包括三个主要部分：输入部分、输出部分和中间卷积部分（中间卷积部分包括如图所示的Stage1到Stage4共计四个stage）。尽管ResNet的变种形式丰富，但都遵循上述的结构特点，网络之间的不同主要在于中间卷积部分的block参数和个数存在差异。

具体代码如下：

  1. #定义ResNet网络
  2. # 3x3卷积定义
  3. def conv3x3(in_channels, out_channels, kernel_size = 3,stride=1, padding=1):
  4. return nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size,
  5.                      stride=stride, padding=padding, bias=False)


  8. # Resnet_50中的残差块
  9. classResidualBlock(nn.Module):
  10.     def __init__(self, in_channels, out_channels, stride=1, downsample=None):
  11. super(ResidualBlock, self).__init__()
  12.         self.mid_channels = out_channels//4
  13.         self.conv1 = conv3x3(in_channels, self.mid_channels, kernel_size=1, stride=stride, padding=0)#Resnet50中，从第二个残差块开始每个layer的第一个残差块都需要一次downsample
  14.         self.bn1 = nn.BatchNorm2d(self.mid_channels)
  15.         self.relu = nn.ReLU(inplace=True)
  16.         self.conv2 = conv3x3(self.mid_channels, self.mid_channels)
  17.         self.bn2 = nn.BatchNorm2d(self.mid_channels)
  18.         self.conv3 = conv3x3(self.mid_channels, out_channels,kernel_size=1,padding=0)
  19.         self.bn3 = nn.BatchNorm2d(out_channels)
  20.         self.downsample_0 = conv3x3(in_channels,out_channels,kernel_size=1,stride=1,padding=0)
  21.         self.downsample = downsample

  23.     def forward(self, x):
  24.         residual = x
  25.         out = self.conv1(x)
  26.         out = self.bn1(out)
  27.         out = self.relu(out)
  28.         out = self.conv2(out)
  29.         out = self.bn2(out)
  30.         out = self.conv3(out)
  31.         out = self.bn3(out)
  32. if self.downsample:
  33.             residual = self.downsample(x)
  34. else:
  35.             residual = self.downsample_0(x)

  37.         out += residual
  38.         out = self.bn3(out)
  39.         out = self.relu(out)
  40. return out


  43. # ResNet定义
  44. classResNet(nn.Module):
  45.     def __init__(self, block, layers, num_classes=10):
  46. super(ResNet, self).__init__()
  47.         self.conv = conv3x3(3, 64,kernel_size=7,stride=2)
  48.         self.bn = nn.BatchNorm2d(64)
  49.         self.max_pool = nn.MaxPool2d(3,2,padding=1)
  50.         self.layer1 = self.make_layer(block, 64, 256, layers[0])
  51.         self.layer2 = self.make_layer(block, 256, 512, layers[1], 2)
  52.         self.layer3 = self.make_layer(block, 512, 1024, layers[2], 2)
  53.         self.layer4 = self.make_layer(block, 1024, 2048, layers[3], 2)
  54.         self.avg_pool = nn.AvgPool2d(3,stride=1,padding=1)
  55.         self.fc = nn.Linear(math.ceil(img_height/32)*math.ceil(img_width/32)*2048, num_classes)

  57.     def make_layer(self, block, in_channels, out_channels, blocks, stride=1):
  58.         downsample = None
  59. if(stride != 1):
  60.             downsample = nn.Sequential(
  61.                 conv3x3(in_channels, out_channels, kernel_size=3,stride=stride),
  62.                 nn.BatchNorm2d(out_channels))
  63.         layers = []
  64.         layers.append(block(in_channels, out_channels, stride, downsample))

  66. for i in range(1, blocks):
  67.             layers.append(block(out_channels, out_channels))
  68. return nn.Sequential(*layers)

  70.     def forward(self, x):
  71.         out = self.conv(x)
  72.         out = self.bn(out)
  73.         out = self.max_pool(out)
  74.         out = self.layer1(out)
  75.         out = self.layer2(out)
  76.         out = self.layer3(out)
  77.         out = self.layer4(out)
  78.         out = self.avg_pool(out)
  79.         out = out.view( -1,math.ceil(img_height/32)*math.ceil(img_width/32)*2048)
  80. return out

  82. # 定义模型输出模式，GPU和CPU均可
  83. #Resnet-503-4-6-3总计(3+4+6+3)*3=48个conv层 加上开头的两个Conv一共50层
  84. model = ResNet(ResidualBlock, [3, 4, 6, 3]).to(DEVICE)


输出ResNet网络架构

  1. ResNet(
  2. (conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(1, 1), bias=False)
  3. (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  4. (max_pool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  5. (layer1): Sequential(
  6. (0): ResidualBlock(
  7. (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
  8. (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  9. (relu): ReLU(inplace=True)
  10. (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  11. (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  12. (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  13. (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  14. (downsample_0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  15. )
  16. (1): ResidualBlock(
  17. (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
  18. (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  19. (relu): ReLU(inplace=True)
  20. (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  21. (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  22. (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  23. (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  24. (downsample_0): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  25. )
  26. (2): ResidualBlock(
  27. (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
  28. (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  29. (relu): ReLU(inplace=True)
  30. (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  31. (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  32. (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  33. (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  34. (downsample_0): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  35. )
  36. )
  37. (layer2): Sequential(
  38. (0): ResidualBlock(
  39. (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
  40. (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  41. (relu): ReLU(inplace=True)
  42. (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  43. (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  44. (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  45. (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  46. (downsample_0): Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  47. (downsample): Sequential(
  48. (0): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  49. (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  50. )
  51. )
  52. (1): ResidualBlock(
  53. (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
  54. (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  55. (relu): ReLU(inplace=True)
  56. (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  57. (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  58. (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  59. (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  60. (downsample_0): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  61. )
  62. (2): ResidualBlock(
  63. (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
  64. (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  65. (relu): ReLU(inplace=True)
  66. (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  67. (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  68. (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  69. (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  70. (downsample_0): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  71. )
  72. (3): ResidualBlock(
  73. (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
  74. (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  75. (relu): ReLU(inplace=True)
  76. (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  77. (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  78. (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  79. (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  80. (downsample_0): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  81. )
  82. )
  83. (layer3): Sequential(
  84. (0): ResidualBlock(
  85. (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
  86. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  87. (relu): ReLU(inplace=True)
  88. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  89. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  90. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  91. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  92. (downsample_0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  93. (downsample): Sequential(
  94. (0): Conv2d(512, 1024, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  95. (1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  96. )
  97. )
  98. (1): ResidualBlock(
  99. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  100. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  101. (relu): ReLU(inplace=True)
  102. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  103. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  104. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  105. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  106. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  107. )
  108. (2): ResidualBlock(
  109. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  110. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  111. (relu): ReLU(inplace=True)
  112. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  113. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  114. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  115. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  116. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  117. )
  118. (3): ResidualBlock(
  119. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  120. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  121. (relu): ReLU(inplace=True)
  122. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  123. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  124. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  125. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  126. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  127. )
  128. (4): ResidualBlock(
  129. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  130. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  131. (relu): ReLU(inplace=True)
  132. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  133. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  134. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  135. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  136. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  137. )
  138. (5): ResidualBlock(
  139. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  140. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  141. (relu): ReLU(inplace=True)
  142. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  143. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  144. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  145. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  146. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  147. )
  148. )
  149. (layer4): Sequential(
  150. (0): ResidualBlock(
  151. (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
  152. (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  153. (relu): ReLU(inplace=True)
  154. (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  155. (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  156. (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
  157. (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  158. (downsample_0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
  159. (downsample): Sequential(
  160. (0): Conv2d(1024, 2048, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  161. (1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  162. )
  163. )
  164. (1): ResidualBlock(
  165. (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  166. (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  167. (relu): ReLU(inplace=True)
  168. (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  169. (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  170. (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
  171. (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  172. (downsample_0): Conv2d(2048, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
  173. )
  174. (2): ResidualBlock(
  175. (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  176. (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  177. (relu): ReLU(inplace=True)
  178. (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  179. (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  180. (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
  181. (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  182. (downsample_0): Conv2d(2048, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
  183. )
  184. )
  185. (avg_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
  186. (fc): Linear(in_features=2048, out_features=10, bias=True)
  187. )

ResNet的损失率和准确率变化

说实话，到了ResNet基本上已经无法理解其数学原理了，只知道增加了残差网络，降低深度网络学习退化问题。

最后欢迎关注公众号：python与大数据分析


paddingresnetrelupytorch
文章转载自追梦IT人，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

评论