devbox
小丑西瓜9
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

卷积神经网络(CNN)代码实现(MNIST)解析_cnn超参数训练代码(三维图像)

作者：小丑西瓜9 | 2024-02-25 11:23:06

赞

踩

cnn超参数训练代码(三维图像)

在http://blog.csdn.net/fengbingchun/article/details/50814710中给出了CNN的简单实现，这里对每一步的实现作个说明：

共7层：依次为输入层、C1层、S2层、C3层、S4层、C5层、输出层，C代表卷积层(特征提取)，S代表降采样层或池化层(Pooling)，输出层为全连接层。

1.        各层权值、偏置(阈值)初始化：

各层权值、偏置个数计算如下：

(1)、输入层：预处理后的32*32图像数据，无权值和偏置；

(2)、C1层：卷积窗大小5*5，输出特征图数量6，卷积窗种类1*6=6，输出特征图大小28*28，因此可训练参数(权值+偏置)：(5*5*1)*6+6=150+6；

         (3)、S2层：卷积窗大小2*2，输出下采样图数量6，卷积窗种类6，输出下采样图大小14*14，因此可训练参数(权值+偏置)：1*6+6=6+6；

         (4)、C3层：卷积窗大小5*5，输出特征图数量16，卷积窗种类6*16=96，输出特征图大小10*10，因此可训练参数(权值+偏置)：(5*5*6)*16+16=2400+16；

         (5)、S4层：卷积窗大小2*2，输出下采样图数量16，卷积窗种类16，输出下采样图大小5*5，因此可训练参数(权值+偏置)：1*16+16=16+16；

         (6)、C5层：卷积窗大小5*5，输出特征图数量120，卷积窗种类16*120=1920，输出特征图大小1*1，因此可训练参数(权值+偏置)：(5*5*16)*120+120=48000+120；

         (7)、输出层：卷积窗大小1*1，输出特征图数量10，卷积窗种类120*10=1200，输出特征图大小1*1，因此可训练参数(权值+偏置)：(1*120)*10+10=1200+10.

         代码段如下：

  1. #define num_map_input_CNN		1 //输入层map个数
  2. #define num_map_C1_CNN			6 //C1层map个数
  3. #define num_map_S2_CNN			6 //S2层map个数
  4. #define num_map_C3_CNN			16 //C3层map个数
  5. #define num_map_S4_CNN			16 //S4层map个数
  6. #define num_map_C5_CNN			120 //C5层map个数
  7. #define num_map_output_CNN		10 //输出层map个数
  8.  
  9. #define len_weight_C1_CNN		150 //C1层权值数，(5*5*1)*6=150
  10. #define len_bias_C1_CNN			6 //C1层阈值数，6
  11. #define len_weight_S2_CNN		6 //S2层权值数,1*6=6
  12. #define len_bias_S2_CNN			6 //S2层阈值数,6
  13. #define len_weight_C3_CNN		2400 //C3层权值数，(5*5*6)*16=2400
  14. #define len_bias_C3_CNN			16 //C3层阈值数,16
  15. #define len_weight_S4_CNN		16 //S4层权值数，1*16=16
  16. #define len_bias_S4_CNN			16 //S4层阈值数，16
  17. #define len_weight_C5_CNN		48000 //C5层权值数，(5*5*16)*120=48000
  18. #define len_bias_C5_CNN			120 //C5层阈值数，120
  19. #define len_weight_output_CNN	1200 //输出层权值数，(1*120)*10=1200
  20. #define len_bias_output_CNN		10 //输出层阈值数，10
  21.  
  22. #define num_neuron_input_CNN	1024 //输入层神经元数，(32*32)*1=1024
  23. #define num_neuron_C1_CNN		4704 //C1层神经元数，(28*28)*6=4704
  24. #define num_neuron_S2_CNN		1176 //S2层神经元数，(14*14)*6=1176
  25. #define num_neuron_C3_CNN		1600 //C3层神经元数，(10*10)*16=1600
  26. #define num_neuron_S4_CNN		400 //S4层神经元数，(5*5)*16=400
  27. #define num_neuron_C5_CNN		120 //C5层神经元数，(1*1)*120=120
  28. #define num_neuron_output_CNN	10 //输出层神经元数，(1*1)*10=10

         权值、偏置初始化：

(1)、权值使用函数uniform_real_distribution均匀分布初始化，tiny-cnn中每次初始化权值数值都相同，这里作了调整，使每次初始化的权值均不同。每层权值初始化大小范围都不一样；

(2)、所有层的偏置均初始化为0.

         代码段如下：

  1. double CNN::uniform_rand(double min, double max)
  2. {
  3. 	//static std::mt19937 gen(1);
  4. 	std::random_device rd;
  5. 	std::mt19937 gen(rd());
  6. 	std::uniform_real_distribution<double> dst(min, max);
  7. 	return dst(gen);
  8. }
  9.  
  10. bool CNN::uniform_rand(double* src, int len, double min, double max)
  11. {
  12. 	for (int i = 0; i < len; i++) {
  13. 		src[i] = uniform_rand(min, max);
  14. 	}
  15.  
  16. 	return true;
  17. }
  18.  
  19. bool CNN::initWeightThreshold()
  20. {
  21. 	srand(time(0) + rand());
  22. 	const double scale = 6.0;
  23.  
  24. 	double min_ = -std::sqrt(scale / (25.0 + 150.0));
  25. 	double max_ = std::sqrt(scale / (25.0 + 150.0));
  26. 	uniform_rand(weight_C1, len_weight_C1_CNN, min_, max_);
  27. 	for (int i = 0; i < len_bias_C1_CNN; i++) {
  28. 		bias_C1[i] = 0.0;
  29. 	}
  30.  
  31. 	min_ = -std::sqrt(scale / (4.0 + 1.0));
  32. 	max_ = std::sqrt(scale / (4.0 + 1.0));
  33. 	uniform_rand(weight_S2, len_weight_S2_CNN, min_, max_);
  34. 	for (int i = 0; i < len_bias_S2_CNN; i++) {
  35. 		bias_S2[i] = 0.0;
  36. 	}
  37.  
  38. 	min_ = -std::sqrt(scale / (150.0 + 400.0));
  39. 	max_ = std::sqrt(scale / (150.0 + 400.0));
  40. 	uniform_rand(weight_C3, len_weight_C3_CNN, min_, max_);
  41. 	for (int i = 0; i < len_bias_C3_CNN; i++) {
  42. 		bias_C3[i] = 0.0;
  43. 	}
  44.  
  45. 	min_ = -std::sqrt(scale / (4.0 + 1.0));
  46. 	max_ = std::sqrt(scale / (4.0 + 1.0));
  47. 	uniform_rand(weight_S4, len_weight_S4_CNN, min_, max_);
  48. 	for (int i = 0; i < len_bias_S4_CNN; i++) {
  49. 		bias_S4[i] = 0.0;
  50. 	}
  51.  
  52. 	min_ = -std::sqrt(scale / (400.0 + 3000.0));
  53. 	max_ = std::sqrt(scale / (400.0 + 3000.0));
  54. 	uniform_rand(weight_C5, len_weight_C5_CNN, min_, max_);
  55. 	for (int i = 0; i < len_bias_C5_CNN; i++) {
  56. 		bias_C5[i] = 0.0;
  57. 	}
  58.  
  59. 	min_ = -std::sqrt(scale / (120.0 + 10.0));
  60. 	max_ = std::sqrt(scale / (120.0 + 10.0));
  61. 	uniform_rand(weight_output, len_weight_output_CNN, min_, max_);
  62. 	for (int i = 0; i < len_bias_output_CNN; i++) {
  63. 		bias_output[i] = 0.0;
  64. 	}
  65.  
  66. 	return true;
  67. }

2.        加载MNIST数据：

关于MNIST的介绍可以参考：http://blog.csdn.net/fengbingchun/article/details/49611549

使用MNIST库作为训练集和测试集，训练样本集为60000个，测试样本集为10000个。

(1)、MNIST库中图像原始大小为28*28，这里缩放为32*32，数据取值范围为[-1,1]，扩充值均取-1，作为输入层输入数据。

代码段如下：

  1. static void readMnistImages(std::string filename, double* data_dst, int num_image)
  2. {
  3. 	const int width_src_image = 28;
  4. 	const int height_src_image = 28;
  5. 	const int x_padding = 2;
  6. 	const int y_padding = 2;
  7. 	const double scale_min = -1;
  8. 	const double scale_max = 1;
  9.  
  10. 	std::ifstream file(filename, std::ios::binary);
  11. 	assert(file.is_open());
  12.  
  13. 	int magic_number = 0;
  14. 	int number_of_images = 0;
  15. 	int n_rows = 0;
  16. 	int n_cols = 0;
  17. 	file.read((char*)&magic_number, sizeof(magic_number));
  18. 	magic_number = reverseInt(magic_number);
  19. 	file.read((char*)&number_of_images, sizeof(number_of_images));
  20. 	number_of_images = reverseInt(number_of_images);
  21. 	assert(number_of_images == num_image);
  22. 	file.read((char*)&n_rows, sizeof(n_rows));
  23. 	n_rows = reverseInt(n_rows);
  24. 	file.read((char*)&n_cols, sizeof(n_cols));
  25. 	n_cols = reverseInt(n_cols);
  26. 	assert(n_rows == height_src_image && n_cols == width_src_image);
  27.  
  28. 	int size_single_image = width_image_input_CNN * height_image_input_CNN;
  29.  
  30. 	for (int i = 0; i < number_of_images; ++i) {
  31. 		int addr = size_single_image * i;
  32.  
  33. 		for (int r = 0; r < n_rows; ++r) {
  34. 			for (int c = 0; c < n_cols; ++c) {
  35. 				unsigned char temp = 0;
  36. 				file.read((char*)&temp, sizeof(temp));
  37. 				data_dst[addr + width_image_input_CNN * (r + y_padding) + c + x_padding] = (temp / 255.0) * (scale_max - scale_min) + scale_min;
  38. 			}
  39. 		}
  40. 	}
  41. }

(2)、对于Label,输出层有10个节点，对应位置的节点值设为0.8，其它节点设为-0.8，作为输出层数据。

代码段如下：

  1. static void readMnistLabels(std::string filename, double* data_dst, int num_image)
  2. {
  3. 	const double scale_max = 0.8;
  4.  
  5. 	std::ifstream file(filename, std::ios::binary);
  6. 	assert(file.is_open());
  7.  
  8. 	int magic_number = 0;
  9. 	int number_of_images = 0;
  10. 	file.read((char*)&magic_number, sizeof(magic_number));
  11. 	magic_number = reverseInt(magic_number);
  12. 	file.read((char*)&number_of_images, sizeof(number_of_images));
  13. 	number_of_images = reverseInt(number_of_images);
  14. 	assert(number_of_images == num_image);
  15.  
  16. 	for (int i = 0; i < number_of_images; ++i) {
  17. 		unsigned char temp = 0;
  18. 		file.read((char*)&temp, sizeof(temp));
  19. 		data_dst[i * num_map_output_CNN + temp] = scale_max;
  20. 	}
  21. }static void readMnistLabels(std::string filename, double* data_dst, int num_image)
  22. {
  23. 	const double scale_max = 0.8;
  24.  
  25. 	std::ifstream file(filename, std::ios::binary);
  26. 	assert(file.is_open());
  27.  
  28. 	int magic_number = 0;
  29. 	int number_of_images = 0;
  30. 	file.read((char*)&magic_number, sizeof(magic_number));
  31. 	magic_number = reverseInt(magic_number);
  32. 	file.read((char*)&number_of_images, sizeof(number_of_images));
  33. 	number_of_images = reverseInt(number_of_images);
  34. 	assert(number_of_images == num_image);
  35.  
  36. 	for (int i = 0; i < number_of_images; ++i) {
  37. 		unsigned char temp = 0;
  38. 		file.read((char*)&temp, sizeof(temp));
  39. 		data_dst[i * num_map_output_CNN + temp] = scale_max;
  40. 	}
  41. }

3.        前向传播：主要计算每层的神经元值；其中C1层、C3层、C5层操作过程相同；S2层、S4层操作过程相同。

(1)、输入层：神经元数为(32*32)*1=1024。

(2)、C1层：神经元数为(28*28)*6=4704，分别用每一个5*5的卷积图像去乘以32*32的图像，获得一个28*28的图像，即对应位置相加再求和，stride长度为1；一共6个5*5的卷积图像，然后对每一个神经元加上一个阈值，最后再通过tanh激活函数对每一神经元进行运算得到最终每一个神经元的结果。

激活函数的作用：它是用来加入非线性因素的，解决线性模型所不能解决的问题，提供网络的非线性建模能力。如果没有激活函数，那么该网络仅能够表达线性映射，此时即便有再多的隐藏层，其整个网络跟单层神经网络也是等价的。因此也可以认为，只有加入了激活函数之后，深度神经网络才具备了分层的非线性映射学习能力。

代码段如下：

  1. double CNN::activation_function_tanh(double x)
  2. {
  3. 	double ep = std::exp(x);
  4. 	double em = std::exp(-x);
  5.  
  6. 	return (ep - em) / (ep + em);
  7. }
  8.  
  9. bool CNN::Forward_C1()
  10. {
  11. 	init_variable(neuron_C1, 0.0, num_neuron_C1_CNN);
  12.  
  13. 	for (int o = 0; o < num_map_C1_CNN; o++) {
  14. 		for (int inc = 0; inc < num_map_input_CNN; inc++) {
  15. 			int addr1 = get_index(0, 0, num_map_input_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN * num_map_input_CNN);
  16. 			int addr2 = get_index(0, 0, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN);
  17. 			int addr3 = get_index(0, 0, o, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
  18.  
  19. 			const double* pw = &weight_C1[0] + addr1;
  20. 			const double* pi = data_single_image + addr2;
  21. 			double* pa = &neuron_C1[0] + addr3;
  22.  
  23. 			for (int y = 0; y < height_image_C1_CNN; y++) {
  24. 				for (int x = 0; x < width_image_C1_CNN; x++) {
  25. 					const double* ppw = pw;
  26. 					const double* ppi = pi + y * width_image_input_CNN + x;
  27. 					double sum = 0.0;
  28.  
  29. 					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  30. 						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  31. 							sum += *ppw++ * ppi[wy * width_image_input_CNN + wx];
  32. 						}
  33. 					}
  34.  
  35. 					pa[y * width_image_C1_CNN + x] += sum;
  36. 				}
  37. 			}
  38. 		}
  39.  
  40. 		int addr3 = get_index(0, 0, o, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
  41. 		double* pa = &neuron_C1[0] + addr3;
  42. 		double b = bias_C1[o];
  43. 		for (int y = 0; y < height_image_C1_CNN; y++) {
  44. 			for (int x = 0; x < width_image_C1_CNN; x++) {
  45. 				pa[y * width_image_C1_CNN + x] += b;
  46. 			}
  47. 		}
  48. 	}
  49.  
  50. 	for (int i = 0; i < num_neuron_C1_CNN; i++) {
  51. 		neuron_C1[i] = activation_function_tanh(neuron_C1[i]);
  52. 	}
  53.  
  54. 	return true;
  55. }

(3)、S2层：神经元数为(14*14)*6=1176，对C1中6个28*28的特征图生成6个14*14的下采样图，相邻四个神经元分别乘以同一个权值再进行相加求和，再求均值即除以4，然后再加上一个阈值，最后再通过tanh激活函数对每一神经元进行运算得到最终每一个神经元的结果。

代码段如下：

  1. bool CNN::Forward_S2()
  2. {
  3. 	init_variable(neuron_S2, 0.0, num_neuron_S2_CNN);
  4. 	double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN);
  5.  
  6. 	assert(out2wi_S2.size() == num_neuron_S2_CNN);
  7. 	assert(out2bias_S2.size() == num_neuron_S2_CNN);
  8.  
  9. 	for (int i = 0; i < num_neuron_S2_CNN; i++) {
  10. 		const wi_connections& connections = out2wi_S2[i];
  11. 		neuron_S2[i] = 0;
  12.  
  13. 		for (int index = 0; index < connections.size(); index++) {
  14. 			neuron_S2[i] += weight_S2[connections[index].first] * neuron_C1[connections[index].second];
  15. 		}
  16.  
  17. 		neuron_S2[i] *= scale_factor;
  18. 		neuron_S2[i] += bias_S2[out2bias_S2[i]];
  19. 	}
  20.  
  21. 	for (int i = 0; i < num_neuron_S2_CNN; i++) {
  22. 		neuron_S2[i] = activation_function_tanh(neuron_S2[i]);
  23. 	}
  24.  
  25. 	return true;
  26. }

(4)、C3层：神经元数为(10*10)*16=1600，C3层实现方式与C1层完全相同，由S2中的6个14*14下采样图生成16个10*10特征图，对于生成的每一个10*10的特征图，是由6个5*5的卷积图像去乘以6个14*14的下采样图，然后对应位置相加求和，然后对每一个神经元加上一个阈值，最后再通过tanh激活函数对每一神经元进行运算得到最终每一个神经元的结果。

也可按照Y.Lecun给出的表进行计算，即对于生成的每一个10*10的特征图，是由n个5*5的卷积图像去乘以n个14*14的下采样图，其中n是小于6的，即不完全连接。这样做的原因：第一，不完全的连接机制将连接的数量保持在合理的范围内。第二，也是最重要的，其破坏了网络的对称性。由于不同的特征图有不同的输入，所以迫使他们抽取不同的特征。

代码段如下：

  1. // connection table [Y.Lecun, 1998 Table.1]
  2. #define O true
  3. #define X false
  4. static const bool tbl[6][16] = {
  5. 	O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O,
  6. 	O, O, X, X, X, O, O, O, X, X, O, O, O, O, X, O,
  7. 	O, O, O, X, X, X, O, O, O, X, X, O, X, O, O, O,
  8. 	X, O, O, O, X, X, O, O, O, O, X, X, O, X, O, O,
  9. 	X, X, O, O, O, X, X, O, O, O, O, X, O, O, X, O,
  10. 	X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O
  11. };
  12. #undef O
  13. #undef X
  14.  
  15. bool CNN::Forward_C3()
  16. {
  17. 	init_variable(neuron_C3, 0.0, num_neuron_C3_CNN);
  18.  
  19. 	for (int o = 0; o < num_map_C3_CNN; o++) {
  20. 		for (int inc = 0; inc < num_map_S2_CNN; inc++) {
  21. 			if (!tbl[inc][o]) continue;
  22.  
  23. 			int addr1 = get_index(0, 0, num_map_S2_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C3_CNN * num_map_S2_CNN);
  24. 			int addr2 = get_index(0, 0, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN);
  25. 			int addr3 = get_index(0, 0, o, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
  26.  
  27. 			const double* pw = &weight_C3[0] + addr1;
  28. 			const double* pi = &neuron_S2[0] + addr2;
  29. 			double* pa = &neuron_C3[0] + addr3;
  30.  
  31. 			for (int y = 0; y < height_image_C3_CNN; y++) {
  32. 				for (int x = 0; x < width_image_C3_CNN; x++) {
  33. 					const double* ppw = pw;
  34. 					const double* ppi = pi + y * width_image_S2_CNN + x;
  35. 					double sum = 0.0;
  36.  
  37. 					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  38. 						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  39. 							sum += *ppw++ * ppi[wy * width_image_S2_CNN + wx];
  40. 						}
  41. 					}
  42.  
  43. 					pa[y * width_image_C3_CNN + x] += sum;
  44. 				}
  45. 			}
  46. 		}
  47.  
  48. 		int addr3 = get_index(0, 0, o, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
  49. 		double* pa = &neuron_C3[0] + addr3;
  50. 		double b = bias_C3[o];
  51. 		for (int y = 0; y < height_image_C3_CNN; y++) {
  52. 			for (int x = 0; x < width_image_C3_CNN; x++) {
  53. 				pa[y * width_image_C3_CNN + x] += b;
  54. 			}
  55. 		}
  56. 	}
  57.  
  58. 	for (int i = 0; i < num_neuron_C3_CNN; i++) {
  59. 		neuron_C3[i] = activation_function_tanh(neuron_C3[i]);
  60. 	}
  61.  
  62. 	return true;
  63. }

(5)、S4层：神经元数为(5*5)*16=400，S4层实现方式与S2层完全相同，由C3中16个10*10的特征图生成16个5*5下采样图，相邻四个神经元分别乘以同一个权值再进行相加求和，再求均值即除以4，然后再加上一个阈值，最后再通过tanh激活函数对每一神经元进行运算得到最终每一个神经元的结果。

代码段如下：

  1. bool CNN::Forward_S4()
  2. {
  3. 	double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN);
  4. 	init_variable(neuron_S4, 0.0, num_neuron_S4_CNN);
  5.  
  6. 	assert(out2wi_S4.size() == num_neuron_S4_CNN);
  7. 	assert(out2bias_S4.size() == num_neuron_S4_CNN);
  8.  
  9. 	for (int i = 0; i < num_neuron_S4_CNN; i++) {
  10. 		const wi_connections& connections = out2wi_S4[i];
  11. 		neuron_S4[i] = 0.0;
  12.  
  13. 		for (int index = 0; index < connections.size(); index++) {
  14. 			neuron_S4[i] += weight_S4[connections[index].first] * neuron_C3[connections[index].second];
  15. 		}
  16.  
  17. 		neuron_S4[i] *= scale_factor;
  18. 		neuron_S4[i] += bias_S4[out2bias_S4[i]];
  19. 	}
  20.  
  21. 	for (int i = 0; i < num_neuron_S4_CNN; i++) {
  22. 		neuron_S4[i] = activation_function_tanh(neuron_S4[i]);
  23. 	}
  24.  
  25. 	return true;
  26. }

(6)、C5层：神经元数为(1*1)*120=120，也可看为全连接层，C5层实现方式与C1、C3层完全相同，由S4中16个5*5下采样图生成120个1*1特征图，对于生成的每一个1*1的特征图，是由16个5*5的卷积图像去乘以16个5*5的下采用图，然后相加求和，然后对每一个神经元加上一个阈值，最后再通过tanh激活函数对每一神经元进行运算得到最终每一个神经元的结果。

代码段如下：

  1. bool CNN::Forward_C5()
  2. {
  3. 	init_variable(neuron_C5, 0.0, num_neuron_C5_CNN);
  4.  
  5. 	for (int o = 0; o < num_map_C5_CNN; o++) {
  6. 		for (int inc = 0; inc < num_map_S4_CNN; inc++) {
  7. 			int addr1 = get_index(0, 0, num_map_S4_CNN * o + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C5_CNN * num_map_S4_CNN);
  8. 			int addr2 = get_index(0, 0, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN);
  9. 			int addr3 = get_index(0, 0, o, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
  10.  
  11. 			const double *pw = &weight_C5[0] + addr1;
  12. 			const double *pi = &neuron_S4[0] + addr2;
  13. 			double *pa = &neuron_C5[0] + addr3;
  14.  
  15. 			for (int y = 0; y < height_image_C5_CNN; y++) {
  16. 				for (int x = 0; x < width_image_C5_CNN; x++) {
  17. 					const double *ppw = pw;
  18. 					const double *ppi = pi + y * width_image_S4_CNN + x;
  19. 					double sum = 0.0;
  20.  
  21. 					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  22. 						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  23. 							sum += *ppw++ * ppi[wy * width_image_S4_CNN + wx];
  24. 						}
  25. 					}
  26.  
  27. 					pa[y * width_image_C5_CNN + x] += sum;
  28. 				}
  29. 			}
  30. 		}
  31.  
  32. 		int addr3 = get_index(0, 0, o, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
  33. 		double *pa = &neuron_C5[0] + addr3;
  34. 		double b = bias_C5[o];
  35. 		for (int y = 0; y < height_image_C5_CNN; y++) {
  36. 			for (int x = 0; x < width_image_C5_CNN; x++) {
  37. 				pa[y * width_image_C5_CNN + x] += b;
  38. 			}
  39. 		}
  40. 	}
  41.  
  42. 	for (int i = 0; i < num_neuron_C5_CNN; i++) {
  43. 		neuron_C5[i] = activation_function_tanh(neuron_C5[i]);
  44. 	}
  45.  
  46. 	return true;
  47. }

(7)、输出层：神经元数为(1*1)*10=10，为全连接层，输出层中的每一个神经元均是由C5层中的120个神经元乘以相对应的权值，然后相加求和；然后对每一个神经元加上一个阈值，最后再通过tanh激活函数对每一神经元进行运算得到最终每一个神经元的结果。

代码段如下：

  1. bool CNN::Forward_output()
  2. {
  3. 	init_variable(neuron_output, 0.0, num_neuron_output_CNN);
  4.  
  5. 	for (int i = 0; i < num_neuron_output_CNN; i++) {
  6. 		neuron_output[i] = 0.0;
  7.  
  8. 		for (int c = 0; c < num_neuron_C5_CNN; c++) {
  9. 			neuron_output[i] += weight_output[c * num_neuron_output_CNN + i] * neuron_C5[c];
  10. 		}
  11.  
  12. 		neuron_output[i] += bias_output[i];
  13. 	}
  14.  
  15. 	for (int i = 0; i < num_neuron_output_CNN; i++) {
  16. 		neuron_output[i] = activation_function_tanh(neuron_output[i]);
  17. 	}
  18.  
  19. 	return true;
  20. }

4.        反向传播：主要计算每层权值和偏置的误差以及每层神经元的误差；其中输入层、S2层、S4层操作过程相同；C1层、C3层操作过程相同。

(1)、输出层：计算输出层神经元误差；通过mse损失函数的导数函数和tanh激活函数的导数函数来计算输出层神经元误差，即a、已计算出的输出层神经元值减去对应label值，b、1.0减去输出层神经元值的平方，c、a与c的乘积和。

损失函数作用：在统计学中损失函数是一种衡量损失和错误(这种损失与”错误地”估计有关)程度的函数。损失函数在实践中最重要的运用，在于协助我们通过过程的改善而持续减少目标值的变异，并非仅仅追求符合逻辑。在深度学习中，对于损失函数的收敛特性，我们期望是当误差越大的时候，收敛(学习)速度应该越快。成为损失函数需要满足两点要求：非负性；预测值和期望值接近时，函数值趋于0.

代码段如下：

  1. double CNN::loss_function_mse_derivative(double y, double t)
  2. {
  3. 	return (y - t);
  4. }
  5.  
  6. void CNN::loss_function_gradient(const double* y, const double* t, double* dst, int len)
  7. {
  8. 	for (int i = 0; i < len; i++) {
  9. 		dst[i] = loss_function_mse_derivative(y[i], t[i]);
  10. 	}
  11. }
  12.  
  13. double CNN::activation_function_tanh_derivative(double x)
  14. {
  15. 	return (1.0 - x * x);
  16. }
  17.  
  18. double CNN::dot_product(const double* s1, const double* s2, int len)
  19. {
  20. 	double result = 0.0;
  21.  
  22. 	for (int i = 0; i < len; i++) {
  23. 		result += s1[i] * s2[i];
  24. 	}
  25.  
  26. 	return result;
  27. }
  28.  
  29. bool CNN::Backward_output()
  30. {
  31. 	init_variable(delta_neuron_output, 0.0, num_neuron_output_CNN);
  32.  
  33. 	double dE_dy[num_neuron_output_CNN];
  34. 	init_variable(dE_dy, 0.0, num_neuron_output_CNN);
  35. 	loss_function_gradient(neuron_output, data_single_label, dE_dy, num_neuron_output_CNN); // 损失函数: mean squared error(均方差)
  36. 	
  37. 	// delta = dE/da = (dE/dy) * (dy/da)
  38. 	for (int i = 0; i < num_neuron_output_CNN; i++) {
  39. 		double dy_da[num_neuron_output_CNN];
  40. 		init_variable(dy_da, 0.0, num_neuron_output_CNN);
  41.  
  42. 		dy_da[i] = activation_function_tanh_derivative(neuron_output[i]);
  43. 		delta_neuron_output[i] = dot_product(dE_dy, dy_da, num_neuron_output_CNN);
  44. 	}
  45.  
  46. 	return true;
  47. }

(2)、C5层：计算C5层神经元误差、输出层权值误差、输出层偏置误差；通过输出层神经元误差乘以输出层权值，求和，结果再乘以C5层神经元的tanh激活函数的导数(即1-C5层神经元值的平方)，获得C5层每一个神经元误差；通过输出层神经元误差乘以C5层神经元获得输出层权值误差；输出层偏置误差即为输出层神经元误差。

代码段如下：

  1. bool CNN::muladd(const double* src, double c, int len, double* dst)
  2. {
  3. 	for (int i = 0; i < len; i++) {
  4. 		dst[i] += (src[i] * c);
  5. 	}
  6.  
  7. 	return true;
  8. }
  9.  
  10. bool CNN::Backward_C5()
  11. {
  12. 	init_variable(delta_neuron_C5, 0.0, num_neuron_C5_CNN);
  13. 	init_variable(delta_weight_output, 0.0, len_weight_output_CNN);
  14. 	init_variable(delta_bias_output, 0.0, len_bias_output_CNN);
  15.  
  16. 	for (int c = 0; c < num_neuron_C5_CNN; c++) {
  17. 		// propagate delta to previous layer
  18. 		// prev_delta[c] += current_delta[r] * W_[c * out_size_ + r]
  19. 		delta_neuron_C5[c] = dot_product(&delta_neuron_output[0], &weight_output[c * num_neuron_output_CNN], num_neuron_output_CNN);
  20. 		delta_neuron_C5[c] *= activation_function_tanh_derivative(neuron_C5[c]);
  21. 	}
  22.  
  23. 	// accumulate weight-step using delta
  24. 	// dW[c * out_size + i] += current_delta[i] * prev_out[c]
  25. 	for (int c = 0; c < num_neuron_C5_CNN; c++) {
  26. 		muladd(&delta_neuron_output[0], neuron_C5[c], num_neuron_output_CNN, &delta_weight_output[0] + c * num_neuron_output_CNN);
  27. 	}
  28.  
  29. 	for (int i = 0; i < len_bias_output_CNN; i++) {
  30. 		delta_bias_output[i] += delta_neuron_output[i];
  31. 	}
  32.  
  33. 	return true;
  34. }

(3)、S4层：计算S4层神经元误差、C5层权值误差、C5层偏置误差；通过C5层权值乘以C5层神经元误差，求和，结果再乘以S4层神经元的tanh激活函数的导数(即1-S4神经元的平方)，获得S4层每一个神经元误差；通过S4层神经元乘以C5层神经元误差，求和，获得C5层权值误差；C5层偏置误差即为C5层神经元误差。

代码段如下：

  1. bool CNN::Backward_S4()
  2. {
  3. 	init_variable(delta_neuron_S4, 0.0, num_neuron_S4_CNN);
  4. 	init_variable(delta_weight_C5, 0.0, len_weight_C5_CNN);
  5. 	init_variable(delta_bias_C5, 0.0, len_bias_C5_CNN);
  6.  
  7. 	// propagate delta to previous layer
  8. 	for (int inc = 0; inc < num_map_S4_CNN; inc++) {
  9. 		for (int outc = 0; outc < num_map_C5_CNN; outc++) {
  10. 			int addr1 = get_index(0, 0, num_map_S4_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S4_CNN * num_map_C5_CNN);
  11. 			int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
  12. 			int addr3 = get_index(0, 0, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN);
  13.  
  14. 			const double* pw = &weight_C5[0] + addr1;
  15. 			const double* pdelta_src = &delta_neuron_C5[0] + addr2;
  16. 			double* pdelta_dst = &delta_neuron_S4[0] + addr3;
  17.  
  18. 			for (int y = 0; y < height_image_C5_CNN; y++) {
  19. 				for (int x = 0; x < width_image_C5_CNN; x++) {
  20. 					const double* ppw = pw;
  21. 					const double ppdelta_src = pdelta_src[y * width_image_C5_CNN + x];
  22. 					double* ppdelta_dst = pdelta_dst + y * width_image_S4_CNN + x;
  23.  
  24. 					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  25. 						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  26. 							ppdelta_dst[wy * width_image_S4_CNN + wx] += *ppw++ * ppdelta_src;
  27. 						}
  28. 					}
  29. 				}
  30. 			}
  31. 		}
  32. 	}
  33.  
  34. 	for (int i = 0; i < num_neuron_S4_CNN; i++) {
  35. 		delta_neuron_S4[i] *= activation_function_tanh_derivative(neuron_S4[i]);
  36. 	}
  37.  
  38. 	// accumulate dw
  39. 	for (int inc = 0; inc < num_map_S4_CNN; inc++) {
  40. 		for (int outc = 0; outc < num_map_C5_CNN; outc++) {
  41. 			for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  42. 				for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  43. 					int addr1 = get_index(wx, wy, inc, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN);
  44. 					int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
  45. 					int addr3 = get_index(wx, wy, num_map_S4_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S4_CNN * num_map_C5_CNN);
  46.  
  47. 					double dst = 0.0;
  48. 					const double* prevo = &neuron_S4[0] + addr1;
  49. 					const double* delta = &delta_neuron_C5[0] + addr2;
  50.  
  51. 					for (int y = 0; y < height_image_C5_CNN; y++) {
  52. 						dst += dot_product(prevo + y * width_image_S4_CNN, delta + y * width_image_C5_CNN, width_image_C5_CNN);
  53. 					}
  54.  
  55. 					delta_weight_C5[addr3] += dst;
  56. 				}
  57. 			}
  58. 		}
  59. 	}
  60.  
  61. 	// accumulate db
  62. 	for (int outc = 0; outc < num_map_C5_CNN; outc++) {
  63. 		int addr2 = get_index(0, 0, outc, width_image_C5_CNN, height_image_C5_CNN, num_map_C5_CNN);
  64. 		const double* delta = &delta_neuron_C5[0] + addr2;
  65.  
  66. 		for (int y = 0; y < height_image_C5_CNN; y++) {
  67. 			for (int x = 0; x < width_image_C5_CNN; x++) {
  68. 				delta_bias_C5[outc] += delta[y * width_image_C5_CNN + x];
  69. 			}
  70. 		}
  71. 	}
  72.  
  73. 	return true;
  74. }

(4)、C3层：计算C3层神经元误差、S4层权值误差、S4层偏置误差；通过S4层权值乘以S4层神经元误差，求和，结果再乘以C3层神经元的tanh激活函数的导数(即1-S4神经元的平方)，然后再乘以1/4，获得C3层每一个神经元误差；通过C3层神经元乘以S4神经元误差，求和，再乘以1/4，获得S4层权值误差；通过S4层神经元误差求和，来获得S4层偏置误差。

代码段如下：

  1. bool CNN::Backward_C3()
  2. {
  3. 	init_variable(delta_neuron_C3, 0.0, num_neuron_C3_CNN);
  4. 	init_variable(delta_weight_S4, 0.0, len_weight_S4_CNN);
  5. 	init_variable(delta_bias_S4, 0.0, len_bias_S4_CNN);
  6.  
  7. 	double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN);
  8.  
  9. 	assert(in2wo_C3.size() == num_neuron_C3_CNN);
  10. 	assert(weight2io_C3.size() == len_weight_S4_CNN);
  11. 	assert(bias2out_C3.size() == len_bias_S4_CNN);
  12.  
  13. 	for (int i = 0; i < num_neuron_C3_CNN; i++) {
  14. 		const wo_connections& connections = in2wo_C3[i];
  15. 		double delta = 0.0;
  16.  
  17. 		for (int j = 0; j < connections.size(); j++) {
  18. 			delta += weight_S4[connections[j].first] * delta_neuron_S4[connections[j].second];
  19. 		}
  20.  
  21. 		delta_neuron_C3[i] = delta * scale_factor * activation_function_tanh_derivative(neuron_C3[i]);
  22. 	}
  23.  
  24. 	for (int i = 0; i < len_weight_S4_CNN; i++) {
  25. 		const io_connections& connections = weight2io_C3[i];
  26. 		double diff = 0;
  27.  
  28. 		for (int j = 0; j < connections.size(); j++) {
  29. 			diff += neuron_C3[connections[j].first] * delta_neuron_S4[connections[j].second];
  30. 		}
  31.  
  32. 		delta_weight_S4[i] += diff * scale_factor;
  33. 	}
  34.  
  35. 	for (int i = 0; i < len_bias_S4_CNN; i++) {
  36. 		const std::vector<int>& outs = bias2out_C3[i];
  37. 		double diff = 0;
  38.  
  39. 		for (int o = 0; o < outs.size(); o++) {
  40. 			diff += delta_neuron_S4[outs[o]];
  41. 		}
  42.  
  43. 		delta_bias_S4[i] += diff;
  44. 	}
  45.  
  46. 	return true;
  47. }

(5)、S2层：计算S2层神经元误差、C3层权值误差、C3层偏置误差；通过C3层权值乘以C3层神经元误差，求和，结果再乘以S2层神经元的tanh激活函数的导数(即1-S2神经元的平方)，获得S2层每一个神经元误差；通过S2层神经元乘以C3层神经元误差，求和，获得C3层权值误差；C3层偏置误差即为C3层神经元误差和。

代码段如下：

  1. bool CNN::Backward_S2()
  2. {
  3. 	init_variable(delta_neuron_S2, 0.0, num_neuron_S2_CNN);
  4. 	init_variable(delta_weight_C3, 0.0, len_weight_C3_CNN);
  5. 	init_variable(delta_bias_C3, 0.0, len_bias_C3_CNN);
  6.  
  7. 	// propagate delta to previous layer
  8. 	for (int inc = 0; inc < num_map_S2_CNN; inc++) {
  9. 		for (int outc = 0; outc < num_map_C3_CNN; outc++) {
  10. 			if (!tbl[inc][outc]) continue;
  11.  
  12. 			int addr1 = get_index(0, 0, num_map_S2_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S2_CNN * num_map_C3_CNN);
  13. 			int addr2 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
  14. 			int addr3 = get_index(0, 0, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN);
  15.  
  16. 			const double *pw = &weight_C3[0] + addr1;
  17. 			const double *pdelta_src = &delta_neuron_C3[0] + addr2;;
  18. 			double* pdelta_dst = &delta_neuron_S2[0] + addr3;
  19.  
  20. 			for (int y = 0; y < height_image_C3_CNN; y++) {
  21. 				for (int x = 0; x < width_image_C3_CNN; x++) {
  22. 					const double* ppw = pw;
  23. 					const double ppdelta_src = pdelta_src[y * width_image_C3_CNN + x];
  24. 					double* ppdelta_dst = pdelta_dst + y * width_image_S2_CNN + x;
  25.  
  26. 					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  27. 						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  28. 							ppdelta_dst[wy * width_image_S2_CNN + wx] += *ppw++ * ppdelta_src;
  29. 						}
  30. 					}
  31. 				}
  32. 			}
  33. 		}
  34. 	}
  35.  
  36. 	for (int i = 0; i < num_neuron_S2_CNN; i++) {
  37. 		delta_neuron_S2[i] *= activation_function_tanh_derivative(neuron_S2[i]);
  38. 	}
  39.  
  40. 	// accumulate dw
  41. 	for (int inc = 0; inc < num_map_S2_CNN; inc++) {
  42. 		for (int outc = 0; outc < num_map_C3_CNN; outc++) {
  43. 			if (!tbl[inc][outc]) continue;
  44.  
  45. 			for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  46. 				for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  47. 					int addr1 = get_index(wx, wy, inc, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN);
  48. 					int addr2 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
  49. 					int addr3 = get_index(wx, wy, num_map_S2_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_S2_CNN * num_map_C3_CNN);
  50. 					
  51. 					double dst = 0.0;
  52. 					const double* prevo = &neuron_S2[0] + addr1;
  53. 					const double* delta = &delta_neuron_C3[0] + addr2;
  54.  
  55. 					for (int y = 0; y < height_image_C3_CNN; y++) {
  56. 						dst += dot_product(prevo + y * width_image_S2_CNN, delta + y * width_image_C3_CNN, width_image_C3_CNN);
  57. 					}
  58.  
  59. 					delta_weight_C3[addr3] += dst;
  60. 				}
  61. 			}
  62. 		}
  63. 	}
  64.  
  65. 	// accumulate db
  66. 	for (int outc = 0; outc < len_bias_C3_CNN; outc++) {
  67. 		int addr1 = get_index(0, 0, outc, width_image_C3_CNN, height_image_C3_CNN, num_map_C3_CNN);
  68. 		const double* delta = &delta_neuron_C3[0] + addr1;
  69.  
  70. 		for (int y = 0; y < height_image_C3_CNN; y++) {
  71. 			for (int x = 0; x < width_image_C3_CNN; x++) {
  72. 				delta_bias_C3[outc] += delta[y * width_image_C3_CNN + x];
  73. 			}
  74. 		}
  75. 	}
  76.  
  77. 	return true;
  78. }

(6)、C1层：计算C1层神经元误差、S2层权值误差、S2层偏置误差；通过S2层权值乘以S2层神经元误差，求和，结果再乘以C1层神经元的tanh激活函数的导数(即1-C1神经元的平方)，然后再乘以1/4，获得C1层每一个神经元误差；通过C1层神经元乘以S2神经元误差，求和，再乘以1/4，获得S2层权值误差；通过S2层神经元误差求和，来获得S4层偏置误差。

代码段如下：

  1. bool CNN::Backward_C1()
  2. {
  3. 	init_variable(delta_neuron_C1, 0.0, num_neuron_C1_CNN);
  4. 	init_variable(delta_weight_S2, 0.0, len_weight_S2_CNN);
  5. 	init_variable(delta_bias_S2, 0.0, len_bias_S2_CNN);
  6.  
  7. 	double scale_factor = 1.0 / (width_kernel_pooling_CNN * height_kernel_pooling_CNN);
  8.  
  9. 	assert(in2wo_C1.size() == num_neuron_C1_CNN);
  10. 	assert(weight2io_C1.size() == len_weight_S2_CNN);
  11. 	assert(bias2out_C1.size() == len_bias_S2_CNN);
  12.  
  13. 	for (int i = 0; i < num_neuron_C1_CNN; i++) {
  14. 		const wo_connections& connections = in2wo_C1[i];
  15. 		double delta = 0.0;
  16.  
  17. 		for (int j = 0; j < connections.size(); j++) {
  18. 			delta += weight_S2[connections[j].first] * delta_neuron_S2[connections[j].second];
  19. 		}
  20.  
  21. 		delta_neuron_C1[i] = delta * scale_factor * activation_function_tanh_derivative(neuron_C1[i]);
  22. 	}
  23.  
  24. 	for (int i = 0; i < len_weight_S2_CNN; i++) {
  25. 		const io_connections& connections = weight2io_C1[i];
  26. 		double diff = 0.0;
  27.  
  28. 		for (int j = 0; j < connections.size(); j++) {
  29. 			diff += neuron_C1[connections[j].first] * delta_neuron_S2[connections[j].second];
  30. 		}
  31.  
  32. 		delta_weight_S2[i] += diff * scale_factor;
  33. 	}
  34.  
  35. 	for (int i = 0; i < len_bias_S2_CNN; i++) {
  36. 		const std::vector<int>& outs = bias2out_C1[i];
  37. 		double diff = 0;
  38.  
  39. 		for (int o = 0; o < outs.size(); o++) {
  40. 			diff += delta_neuron_S2[outs[o]];
  41. 		}
  42.  
  43. 		delta_bias_S2[i] += diff;
  44. 	}
  45.  
  46. 	return true;
  47. }

(7)、输入层：计算输入层神经元误差、C1层权值误差、C1层偏置误差；通过C1层权值乘以C1层神经元误差，求和，结果再乘以输入层神经元的tanh激活函数的导数(即1-输入层神经元的平方)，获得输入层每一个神经元误差；通过输入层层神经元乘以C1层神经元误差，求和，获得C1层权值误差；C1层偏置误差即为C1层神经元误差和。

  1. bool CNN::Backward_input()
  2. {
  3. 	init_variable(delta_neuron_input, 0.0, num_neuron_input_CNN);
  4. 	init_variable(delta_weight_C1, 0.0, len_weight_C1_CNN);
  5. 	init_variable(delta_bias_C1, 0.0, len_bias_C1_CNN);
  6.  
  7. 	// propagate delta to previous layer
  8. 	for (int inc = 0; inc < num_map_input_CNN; inc++) {
  9. 		for (int outc = 0; outc < num_map_C1_CNN; outc++) {
  10. 			int addr1 = get_index(0, 0, num_map_input_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN);
  11. 			int addr2 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
  12. 			int addr3 = get_index(0, 0, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN);
  13.  
  14. 			const double* pw = &weight_C1[0] + addr1;
  15. 			const double* pdelta_src = &delta_neuron_C1[0] + addr2;
  16. 			double* pdelta_dst = &delta_neuron_input[0] + addr3;
  17.  
  18. 			for (int y = 0; y < height_image_C1_CNN; y++) {
  19. 				for (int x = 0; x < width_image_C1_CNN; x++) {
  20. 					const double* ppw = pw;
  21. 					const double ppdelta_src = pdelta_src[y * width_image_C1_CNN + x];
  22. 					double* ppdelta_dst = pdelta_dst + y * width_image_input_CNN + x;
  23.  
  24. 					for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  25. 						for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  26. 							ppdelta_dst[wy * width_image_input_CNN + wx] += *ppw++ * ppdelta_src;
  27. 						}
  28. 					}
  29. 				}
  30. 			}
  31. 		}
  32. 	}
  33.  
  34. 	for (int i = 0; i < num_neuron_input_CNN; i++) {
  35. 		delta_neuron_input[i] *= activation_function_identity_derivative(data_single_image[i]/*neuron_input[i]*/);
  36. 	}
  37.  
  38. 	// accumulate dw
  39. 	for (int inc = 0; inc < num_map_input_CNN; inc++) {
  40. 		for (int outc = 0; outc < num_map_C1_CNN; outc++) {
  41. 			for (int wy = 0; wy < height_kernel_conv_CNN; wy++) {
  42. 				for (int wx = 0; wx < width_kernel_conv_CNN; wx++) {
  43. 					int addr1 = get_index(wx, wy, inc, width_image_input_CNN, height_image_input_CNN, num_map_input_CNN);
  44. 					int addr2 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
  45. 					int addr3 = get_index(wx, wy, num_map_input_CNN * outc + inc, width_kernel_conv_CNN, height_kernel_conv_CNN, num_map_C1_CNN);
  46.  
  47. 					double dst = 0.0;
  48. 					const double* prevo = data_single_image + addr1;//&neuron_input[0]
  49. 					const double* delta = &delta_neuron_C1[0] + addr2;
  50.  
  51. 					for (int y = 0; y < height_image_C1_CNN; y++) {
  52. 						dst += dot_product(prevo + y * width_image_input_CNN, delta + y * width_image_C1_CNN, width_image_C1_CNN);
  53. 					}
  54.  
  55. 					delta_weight_C1[addr3] += dst;
  56. 				}
  57. 			}
  58. 		}
  59. 	}
  60.  
  61. 	// accumulate db
  62. 	for (int outc = 0; outc < len_bias_C1_CNN; outc++) {
  63. 		int addr1 = get_index(0, 0, outc, width_image_C1_CNN, height_image_C1_CNN, num_map_C1_CNN);
  64. 		const double* delta = &delta_neuron_C1[0] + addr1;
  65.  
  66. 		for (int y = 0; y < height_image_C1_CNN; y++) {
  67. 			for (int x = 0; x < width_image_C1_CNN; x++) {
  68. 				delta_bias_C1[outc] += delta[y * width_image_C1_CNN + x];
  69. 			}
  70. 		}
  71. 	}
  72.  
  73. 	return true;
  74. }

5.        更新各层权值、偏置：通过之前计算的各层权值、各层权值误差；各层偏置、各层偏置误差以及学习率来更新各层权值和偏置。

代码段如下：

  1. void CNN::update_weights_bias(const double* delta, double* e_weight, double* weight, int len)
  2. {
  3. 	for (int i = 0; i < len; i++) {
  4. 		e_weight[i] += delta[i] * delta[i];
  5. 		weight[i] -= learning_rate_CNN * delta[i] / (std::sqrt(e_weight[i]) + eps_CNN);
  6. 	}
  7. }
  8.  
  9. bool CNN::UpdateWeights()
  10. {
  11. 	update_weights_bias(delta_weight_C1, E_weight_C1, weight_C1, len_weight_C1_CNN);
  12. 	update_weights_bias(delta_bias_C1, E_bias_C1, bias_C1, len_bias_C1_CNN);
  13.  
  14. 	update_weights_bias(delta_weight_S2, E_weight_S2, weight_S2, len_weight_S2_CNN);
  15. 	update_weights_bias(delta_bias_S2, E_bias_S2, bias_S2, len_bias_S2_CNN);
  16.  
  17. 	update_weights_bias(delta_weight_C3, E_weight_C3, weight_C3, len_weight_C3_CNN);
  18. 	update_weights_bias(delta_bias_C3, E_bias_C3, bias_C3, len_bias_C3_CNN);
  19.  
  20. 	update_weights_bias(delta_weight_S4, E_weight_S4, weight_S4, len_weight_S4_CNN);
  21. 	update_weights_bias(delta_bias_S4, E_bias_S4, bias_S4, len_bias_S4_CNN);
  22.  
  23. 	update_weights_bias(delta_weight_C5, E_weight_C5, weight_C5, len_weight_C5_CNN);
  24. 	update_weights_bias(delta_bias_C5, E_bias_C5, bias_C5, len_bias_C5_CNN);
  25.  
  26. 	update_weights_bias(delta_weight_output, E_weight_output, weight_output, len_weight_output_CNN);
  27. 	update_weights_bias(delta_bias_output, E_bias_output, bias_output, len_bias_output_CNN);
  28.  
  29. 	return true;
  30. }

6.        测试准确率是否达到要求或已达到循环次数：依次循环3至5中操作，根据训练集数量，每循环60000次时，通过计算的权值和偏置，来对10000个测试集进行测试，如果准确率达到0.985或者达到迭代次数上限100次时，保存权值和偏置。

代码段如下：

  1. bool CNN::train()
  2. {
  3. 	out2wi_S2.clear();
  4. 	out2bias_S2.clear();
  5. 	out2wi_S4.clear();
  6. 	out2bias_S4.clear();
  7. 	in2wo_C3.clear();
  8. 	weight2io_C3.clear();
  9. 	bias2out_C3.clear();
  10. 	in2wo_C1.clear();
  11. 	weight2io_C1.clear();
  12. 	bias2out_C1.clear();
  13.  
  14. 	calc_out2wi(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN, out2wi_S2);
  15. 	calc_out2bias(width_image_S2_CNN, height_image_S2_CNN, num_map_S2_CNN, out2bias_S2);
  16. 	calc_out2wi(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN, out2wi_S4);
  17. 	calc_out2bias(width_image_S4_CNN, height_image_S4_CNN, num_map_S4_CNN, out2bias_S4);
  18. 	calc_in2wo(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, in2wo_C3);
  19. 	calc_weight2io(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, weight2io_C3);
  20. 	calc_bias2out(width_image_C3_CNN, height_image_C3_CNN, width_image_S4_CNN, height_image_S4_CNN, num_map_C3_CNN, num_map_S4_CNN, bias2out_C3);
  21. 	calc_in2wo(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, in2wo_C1);
  22. 	calc_weight2io(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, weight2io_C1);
  23. 	calc_bias2out(width_image_C1_CNN, height_image_C1_CNN, width_image_S2_CNN, height_image_S2_CNN, num_map_C1_CNN, num_map_C3_CNN, bias2out_C1);
  24.  
  25. 	int iter = 0;
  26. 	for (iter = 0; iter < num_epochs_CNN; iter++) {
  27. 		std::cout << "epoch: " << iter + 1;
  28.  
  29. 		for (int i = 0; i < num_patterns_train_CNN; i++) {
  30. 			data_single_image = data_input_train + i * num_neuron_input_CNN;
  31. 			data_single_label = data_output_train + i * num_neuron_output_CNN;
  32.  
  33. 			Forward_C1();
  34. 			Forward_S2();
  35. 			Forward_C3();
  36. 			Forward_S4();
  37. 			Forward_C5();
  38. 			Forward_output();
  39.  
  40. 			Backward_output();
  41. 			Backward_C5();
  42. 			Backward_S4();
  43. 			Backward_C3();
  44. 			Backward_S2();
  45. 			Backward_C1();
  46. 			Backward_input();
  47.  
  48. 			UpdateWeights();
  49. 		}
  50.  
  51. 		double accuracyRate = test();
  52. 		std::cout << ",    accuray rate: " << accuracyRate << std::endl;
  53. 		if (accuracyRate > accuracy_rate_CNN) {
  54. 			saveModelFile("E:/GitCode/NN_Test/data/cnn.model");
  55. 			std::cout << "generate cnn model" << std::endl;
  56. 			break;
  57. 		}
  58. 	}
  59.  
  60. 	if (iter == num_epochs_CNN) {
  61. 		saveModelFile("E:/GitCode/NN_Test/data/cnn.model");
  62. 		std::cout << "generate cnn model" << std::endl;
  63. 	}
  64.  
  65. 	return true;
  66. }
  67.  
  68. double CNN::test()
  69. {
  70. 	int count_accuracy = 0;
  71.  
  72. 	for (int num = 0; num < num_patterns_test_CNN; num++) {
  73. 		data_single_image = data_input_test + num * num_neuron_input_CNN;
  74. 		data_single_label = data_output_test + num * num_neuron_output_CNN;
  75.  
  76. 		Forward_C1();
  77. 		Forward_S2();
  78. 		Forward_C3();
  79. 		Forward_S4();
  80. 		Forward_C5();
  81. 		Forward_output();
  82.  
  83. 		int pos_t = -1;
  84. 		int pos_y = -2;
  85. 		double max_value_t = -9999.0;
  86. 		double max_value_y = -9999.0;
  87.  
  88. 		for (int i = 0; i < num_neuron_output_CNN; i++) {
  89. 			if (neuron_output[i] > max_value_y) {
  90. 				max_value_y = neuron_output[i];
  91. 				pos_y = i;
  92. 			}
  93.  
  94. 			if (data_single_label[i] > max_value_t) {
  95. 				max_value_t = data_single_label[i];
  96. 				pos_t = i;
  97. 			}
  98. 		}
  99.  
  100. 		if (pos_y == pos_t) {
  101. 			++count_accuracy;
  102. 		}
  103.  
  104. 		Sleep(1);
  105. 	}
  106.  
  107. 	return (count_accuracy * 1.0 / num_patterns_test_CNN);
  108. }

7.        对输入的图像数据进行识别：载入已保存的权值和偏置，对输入的数据进行识别，过程相当于前向传播。

代码段如下：

  1. int CNN::predict(const unsigned char* data, int width, int height)
  2. {
  3. 	assert(data && width == width_image_input_CNN && height == height_image_input_CNN);
  4.  
  5. 	const double scale_min = -1;
  6. 	const double scale_max = 1;
  7.  
  8. 	double tmp[width_image_input_CNN * height_image_input_CNN];
  9. 	for (int y = 0; y < height; y++) {
  10. 		for (int x = 0; x < width; x++) {
  11. 			tmp[y * width + x] = (data[y * width + x] / 255.0) * (scale_max - scale_min) + scale_min;
  12. 		}
  13. 	}
  14.  
  15. 	data_single_image = &tmp[0];
  16.  
  17. 	Forward_C1();
  18. 	Forward_S2();
  19. 	Forward_C3();
  20. 	Forward_S4();
  21. 	Forward_C5();
  22. 	Forward_output();
  23.  
  24. 	int pos = -1;
  25. 	double max_value = -9999.0;
  26.  
  27. 	for (int i = 0; i < num_neuron_output_CNN; i++) {
  28. 		if (neuron_output[i] > max_value) {
  29. 			max_value = neuron_output[i];
  30. 			pos = i;
  31. 		}
  32. 	}
  33.  
  34. 	return pos;
  35. }

GitHub： https://github.com/fengbingchun/NN_Test

本文内容由网友自发贡献，转载请注明出处：https://www.wpsshop.cn/w/小丑西瓜9/article/detail/140567
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号