神经网络之反向传播算法实现

1神经网络模型以下面神经网络模型为例,说明神经网络中正向传播和反向传播过程及代码实现1.1正向传播(1)输入层神经元$i_1,i_2$,输入层到隐藏层处理过程$$HiddenNeth_1

大家好,又见面了,我是全栈君,今天给大家准备了Idea注册码。

1 神经网络模型

以下面神经网络模型为例,说明神经网络中正向传播和反向传播过程及代码实现
神经网络之反向传播算法实现

1.1 正向传播

(1)输入层神经元\(i_1,i_2\),输入层到隐藏层处理过程

\[HiddenNeth_1 = w_1i_1+w_2i_2 + b_1 \]

\[HiddenNeth_2 = w_3i_1+w_4i_2 + b_1 \]

\[h_1 = sigmoid(HiddenNeth_1) \]

\[h_2 = sigmoid(HiddenNeth_2) \]

(2)隐藏层:神经元\(h_1,h_2\),隐藏层到输出层处理过程

\[OutputNeto_1 = w_5h_1+w_6h_2 \]

\[OutputNeto_2 = w_7h_1+w_8h_2 \]

\[o_1 = sigmoid(OutputNeto_1) \]

\[o_2 = sigmoid(OutputNeto_2) \]

1.2 反向传播

反向传播是根据每次前向传播得到的误差来评估每个神经网络层中的权重对其的影响程度,再根据链式规则和梯度下降对权重进行更新的过程,下面以对更新权重\(w_5和w_1\)为例进行说明

  • totalo1表示参考值0.01
  • totalo2参考值0.99

1.2.1 更新权重w_5

\[\frac{\partial totalo1}{\partial w_5} = \frac{\partial totalo1}{\partial o_1} * \frac{\partial o_1}{\partial outputneto_1} * \frac{\partial outputneto_1}{\partial w_5} \]

其中:$$\frac{\partial totalo1}{\partial o_1} = \frac{\partial \frac{1}{2}{(totalo_1 – o_1)}^2}{\partial o_1} = -(totalo_1 -o_1)$$

\[\frac{\partial o_1}{\partial outputneto_1} = o_1(1-o_1) \]

\[\frac{\partial outputneto_1}{\partial w_5} = h_1 \]

则:$$\frac{\partial totalo1}{\partial w_5} = -(totalo_1 -o_1)o_1(1-o_1)h_1$$
更新w_5的值:$$w_5new = w_5 – learningrate * \frac{\partial totalo1}{\partial w_5}$$

1.2.2 更新权重w_1

\(求w_1的偏导与求w_5偏导唯一的区别在于,w_1的值影响了o_1和o_2\),所以

\[\frac{\partial total}{\partial w_1} = \frac{\partial totalol}{\partial w_1} + \frac{\partial totalo2}{\partial w_1} \]

接下来的求导思路和上面一样:

\[\frac{\partial totalol}{\partial w_1} = \frac{\partial totalol}{\partial h1} * \frac{\partial h1}{\partial hiddenneth1} * \frac{\partial hiddenneth1}{\partial w_1} \]

\[\frac{\partial total}{\partial w_1} = -(totalo1 – o1)o1(1-o1)w_5*h1(1-h1)i1 + -(totalo2 – o2)o2(1-o2)w_7*h1(1-h1)i1 \]

更新w_1的值:$$w_1new = w_1 – learningrate * \frac{\partial total}{\partial w_1}$$

2. 代码实现正向和方向传播算法

import random
import math

class NeuralNetwork:
    learning_rate = 0.5
    def __init__(self,input_num,hidden_num,output_num,input_hidden_weights=None,
                 input_hidden_bias=None,hidden_output_weights=None,hidden_output_bias=None):
        
        self.input_num = input_num
        
        # 构建掩藏层
        self.hidden_layer = NeuralLayer(hidden_num,input_hidden_bias)
        # 构建输出层
        self.output_layer = NeuralLayer(output_num,hidden_output_bias)    
        # 初始化输入层到隐藏层权重
        self.init_input_to_hidden_weights(input_hidden_weights)
        # 初始化隐藏层到输出层权重
        self.init_hidden_to_output_weights(hidden_output_weights)
        
    def init_input_to_hidden_weights(self,weights):
        weight_num = 0
        for i_num in range(len(self.hidden_layer.neurons)):
            for o_num in range(self.input_num):
                if weights is None:
                    self.hidden_layer.neurons[i_num].weights.append(random.random())
                else:
                    self.hidden_layer.neurons[i_num].weights.append(weights[weight_num])
                weight_num += 1
                
    def init_hidden_to_output_weights(self,weights):
        weight_num = 0
        for i_num in range(len(self.output_layer.neurons)):
            for o_num in range(len(self.hidden_layer.neurons)):
                if weights is None:
                    self.output_layer.neurons[i_num].weights.append(random.random())
                else:
                    self.output_layer.neurons[i_num].weights.append(weights[weight_num])
                weight_num += 1
    
    def inspect(self):
        print('..................')
        print('input inspect:',[i for i in self.inputs])
        print('..................')
        print('hidden inspect:')
        self.hidden_layer.inspect()
        print('..................')
        print('output inspect:')
        self.output_layer.inspect()
        print('..................')
        
    def forward(self,inputs):
        hidden_layer_outout = self.hidden_layer.forward(inputs)
        print('hidden_layer_outout',hidden_layer_outout)
        ouput_layer_ouput = self.output_layer.forward(hidden_layer_outout)
        print('ouput_layer_ouput',ouput_layer_ouput)
        return ouput_layer_ouput
        
    def train(self,x,y):
        ouput_layer_ouput = self.forward(x)
        
        # 求total / neto的偏导
        total_o_pd = [0] * len(self.output_layer.neurons)
        for o in range(len(self.output_layer.neurons)):
            total_o_pd[o] = self.output_layer.neurons[o].calculate_total_net_pd(y[o]) 
            
        # 求total / h的偏导 = total.1 / h的偏导 + total.2 / h的偏导
        total_neth_pd = [0] * len(self.hidden_layer.neurons)
        for h in range(len(self.hidden_layer.neurons)):
            total_h_pd = 0
            for o in range(len(self.output_layer.neurons)):
                total_h_pd += total_o_pd[o] * self.hidden_layer.neurons[h].weights[o]
            total_neth_pd[h] = total_h_pd * self.output_layer.neurons[h].calculate_output_net_pd()
            
        # 更新输出层神经元权重
        for o in range(len(self.output_layer.neurons)):
            for ho_w in range(len(self.output_layer.neurons[o].weights)):
                ho_w_gradient = total_o_pd[o] * self.output_layer.neurons[o].calculate_net_linear_pd(ho_w)
                self.output_layer.neurons[o].weights[ho_w] -= self.learning_rate * ho_w_gradient
                
        # 更新隐藏层神经元权重
        for h in range(len(self.hidden_layer.neurons)):
            for ih_w in range(len(self.hidden_layer.neurons[h].weights)):
                ih_w_gradient = total_neth_pd[h] * self.hidden_layer.neurons[h].calculate_net_linear_pd(ih_w)
                self.hidden_layer.neurons[h].weights[ih_w] -= self.learning_rate * ih_w_gradient
                
    def calculate_total_error(self, training_sets):
        total_error = 0
        for t in range(len(training_sets)):
            training_inputs, training_outputs = training_sets[t]
            self.forward(training_inputs)
            for o in range(len(training_outputs)):
                total_error += self.output_layer.neurons[o].calculate_error(training_outputs[o])
        return total_error
    
class NeuralLayer:
    def __init__(self,neural_num,bias):
        self.bias = bias if bias else random.random()
        
        self.neurons = []
        
        for i in range(neural_num):
            self.neurons.append(Neuron(self.bias))
            
    def inspect(self):
        print('weights:',[neuron.weights for neuron in self.neurons])
        print('bias:',[neuron.bias for neuron in self.neurons])
      
    def get_output(self,inputs):
        outputs = []
        for neuron in self.neurons:
            outputs.append(neuron.output)
        return outputs
    
    def forward(self,inputs):
        outputs = []
        for neuron in self.neurons:
            outputs.append(neuron.calculate_output(inputs))
        return outputs
    
class Neuron:
    def __init__(self,bias):
        self.bias = bias        
        self.weights = []
        
    def calculate_output(self,inputs):
        self.inputs = inputs 
        total_net_outputs = self.calculate_total_net_output()
        self.output = self.sigmoid(total_net_outputs)
        return self.output
    
    def calculate_total_net_output(self):
        total = 0
        for i in range(len(self.inputs)):
            total += self.inputs[i] * self.weights[i]
        return total + self.bias
        
    def sigmoid(self,total_net_input):
        return 1 / (1 + math.exp(-total_net_input))
    
    def calculate_total_output_pd(self,total_output):
        return -(total_output - self.output)
    
    def calculate_output_net_pd(self):
        return self.output * (1 - self.output)
    
    def calculate_total_net_pd(self,total_output):
        return self.calculate_total_output_pd(total_output) * self.calculate_output_net_pd()
        
    def calculate_net_linear_pd(self,index):
        return self.inputs[index]
    
    def calculate_error(self, target_output):
        return 0.5 * (target_output - self.output) ** 2
nn = NeuralNetwork(2, 2, 2, input_hidden_weights=[0.15, 0.2, 0.25, 0.3], input_hidden_bias=0.35, 
                   hidden_output_weights=[0.4, 0.45, 0.5, 0.55], hidden_output_bias=0.6)
for i in range(10000):
    nn.train([0.05, 0.1], [0.01, 0.99])
    print(i, round(nn.calculate_total_error([[[0.05, 0.1], [0.01, 0.99]]]), 9))
hidden_layer_outout [0.5932699921071872, 0.596884378259767]
ouput_layer_ouput [0.7513650695523157, 0.7729284653214625]
hidden_layer_outout [0.5932662857458805, 0.5968782561589732]
ouput_layer_ouput [0.7420894573166411, 0.775286681791022]
迭代1次 误差0.291028391
hidden_layer_outout [0.5932662857458805, 0.5968782561589732]
ouput_layer_ouput [0.7420894573166411, 0.775286681791022]
hidden_layer_outout [0.5932623797502049, 0.5968720205376407]
ouput_layer_ouput [0.7324789213021788, 0.7775850216973027]
迭代2次 误差 0.283547957
hidden_layer_outout [0.5932623797502049, 0.5968720205376407]
ouput_layer_ouput [0.7324789213021788, 0.7775850216973027]
hidden_layer_outout [0.5932582778661456, 0.5968656853189723]
ouput_layer_ouput [0.7225410209751278, 0.7798256906644981]
迭代3次 误差 0.275943973
hidden_layer_outout [0.5932582778661456, 0.5968656853189723]
ouput_layer_ouput [0.7225410209751278, 0.7798256906644981]
hidden_layer_outout [0.5932539857457403, 0.5968592652405116]
ouput_layer_ouput [0.7122866732919141, 0.7820107943242337]
迭代4次 误差0.268233041
...
...
...
hidden_layer_outout [0.5930355856011269, 0.5965960242813179]
ouput_layer_ouput [0.016365976874939202, 0.9836751989707726]
hidden_layer_outout [0.5930355857116121, 0.5965960243868048]
ouput_layer_ouput [0.016365393177895763, 0.9836757760229975]
迭代10000次 误差 4.0257e-05

迭代10000次后结果:

hidden_layer_outout [0.5930355856011269, 0.5965960242813179]

ouput_layer_ouput [0.016365976874939202, 0.9836751989707726]

hidden_layer_outout [0.5930355857116121, 0.5965960243868048]

ouput_layer_ouput [0.016365393177895763, 0.9836757760229975]

迭代10000次 误差 4.0257e-05

参考:https://www.cnblogs.com/charlotte77/p/5629865.html,感谢博主的分析

版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员-用户IM,转载请注明出处:https://javaforall.cn/120014.html原文链接:https://javaforall.cn

【正版授权,激活自己账号】: Jetbrains全家桶Ide使用,1年售后保障,每天仅需1毛

【官方授权 正版激活】: 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...

(0)
blank

相关推荐

  • 简述CloudSim的仿真步骤_进程调度的两种方式

    简述CloudSim的仿真步骤_进程调度的两种方式Cloudsim3.0.3中VM调度策略系列类解析(带迁移的策略)注:本文为旧文的markdown重制版Cloudsim中VM调度策略类在DataCenter(或PowerDataCenter)创建时需要制定,更是我们在Cloudsim上试验我们的调度算法的核心所在。CloudSim中给出了VmAllocationPolicy抽象类,还有提供了一个VmAllocationPolicy…

    2022年10月11日
  • npm换源问题

    npm换源问题点击查看解决方案

    2022年10月23日
  • Java内存管理-掌握自定义类加载器的实现(七)

    勿在流沙筑高台,出来混迟早要还的。做一个积极的人编码、改bug、提升自己我有一个乐园,面向编程,春暖花开!上一篇分析了ClassLoader的类加载相关的核心源码,也简单介绍了ClassLoader的设计思想,读源码相对来说是比较枯燥的,还是这个是必须要走的过程,学习源码中的一些思想,一些精髓,看一下大神级人物是怎么写出那么牛逼的代码。我们能够从中学到一点点东西,那也是一种进步和成长了…

  • MySQL按天,按周,按月,按时间段统计【转载】

    MySQL按天,按周,按月,按时间段统计【转载】

  • 【信息学奥赛一本通】题解目录「建议收藏」

    【信息学奥赛一本通】题解目录「建议收藏」OJ网站:点击这里【语言及算法基础篇】第一部分:C++语言第一章:C++语言入门Hello,World!(信息学奥赛一本通-T1001):点击这里 输出第二个整数(信息学奥赛一本通-T1002):点击这里 对齐输出(信息学奥赛一本通-T1003):点击这里 字符三角形(信息学奥赛一本通-T1004):点击这里 地球人口承载力估计(信息学奥赛一本通-T1005):点击…

  • 几种常用的矩阵范数表示_向量范数怎么求

    几种常用的矩阵范数表示_向量范数怎么求按道理讲,这些东西应该熟记于心的。但是自己真心不喜欢记这种东西,看到一个总结不错的博客,转载过来以便于自己查看把!原文1.几种范数矩阵X∈Rm×nX∈Rm×n,σi(X)σi(X)表示XX的第ii大奇异值(即XX′XX′的第ii大特征值的均方根){citerecht2010guaranteed}。rr表示矩阵XX的秩(R

发表回复

您的电子邮箱地址不会被公开。

关注全栈程序员社区公众号