神经网络之反向传播算法实现

1神经网络模型以下面神经网络模型为例,说明神经网络中正向传播和反向传播过程及代码实现1.1正向传播(1)输入层神经元$i_1,i_2$,输入层到隐藏层处理过程$$HiddenNeth_1

大家好,又见面了,我是全栈君,今天给大家准备了Idea注册码。

1 神经网络模型

以下面神经网络模型为例,说明神经网络中正向传播和反向传播过程及代码实现
神经网络之反向传播算法实现

1.1 正向传播

(1)输入层神经元\(i_1,i_2\),输入层到隐藏层处理过程

\[HiddenNeth_1 = w_1i_1+w_2i_2 + b_1 \]

\[HiddenNeth_2 = w_3i_1+w_4i_2 + b_1 \]

\[h_1 = sigmoid(HiddenNeth_1) \]

\[h_2 = sigmoid(HiddenNeth_2) \]

(2)隐藏层:神经元\(h_1,h_2\),隐藏层到输出层处理过程

\[OutputNeto_1 = w_5h_1+w_6h_2 \]

\[OutputNeto_2 = w_7h_1+w_8h_2 \]

\[o_1 = sigmoid(OutputNeto_1) \]

\[o_2 = sigmoid(OutputNeto_2) \]

1.2 反向传播

反向传播是根据每次前向传播得到的误差来评估每个神经网络层中的权重对其的影响程度,再根据链式规则和梯度下降对权重进行更新的过程,下面以对更新权重\(w_5和w_1\)为例进行说明

  • totalo1表示参考值0.01
  • totalo2参考值0.99

1.2.1 更新权重w_5

\[\frac{\partial totalo1}{\partial w_5} = \frac{\partial totalo1}{\partial o_1} * \frac{\partial o_1}{\partial outputneto_1} * \frac{\partial outputneto_1}{\partial w_5} \]

其中:$$\frac{\partial totalo1}{\partial o_1} = \frac{\partial \frac{1}{2}{(totalo_1 – o_1)}^2}{\partial o_1} = -(totalo_1 -o_1)$$

\[\frac{\partial o_1}{\partial outputneto_1} = o_1(1-o_1) \]

\[\frac{\partial outputneto_1}{\partial w_5} = h_1 \]

则:$$\frac{\partial totalo1}{\partial w_5} = -(totalo_1 -o_1)o_1(1-o_1)h_1$$
更新w_5的值:$$w_5new = w_5 – learningrate * \frac{\partial totalo1}{\partial w_5}$$

1.2.2 更新权重w_1

\(求w_1的偏导与求w_5偏导唯一的区别在于,w_1的值影响了o_1和o_2\),所以

\[\frac{\partial total}{\partial w_1} = \frac{\partial totalol}{\partial w_1} + \frac{\partial totalo2}{\partial w_1} \]

接下来的求导思路和上面一样:

\[\frac{\partial totalol}{\partial w_1} = \frac{\partial totalol}{\partial h1} * \frac{\partial h1}{\partial hiddenneth1} * \frac{\partial hiddenneth1}{\partial w_1} \]

\[\frac{\partial total}{\partial w_1} = -(totalo1 – o1)o1(1-o1)w_5*h1(1-h1)i1 + -(totalo2 – o2)o2(1-o2)w_7*h1(1-h1)i1 \]

更新w_1的值:$$w_1new = w_1 – learningrate * \frac{\partial total}{\partial w_1}$$

2. 代码实现正向和方向传播算法

import random
import math

class NeuralNetwork:
    learning_rate = 0.5
    def __init__(self,input_num,hidden_num,output_num,input_hidden_weights=None,
                 input_hidden_bias=None,hidden_output_weights=None,hidden_output_bias=None):
        
        self.input_num = input_num
        
        # 构建掩藏层
        self.hidden_layer = NeuralLayer(hidden_num,input_hidden_bias)
        # 构建输出层
        self.output_layer = NeuralLayer(output_num,hidden_output_bias)    
        # 初始化输入层到隐藏层权重
        self.init_input_to_hidden_weights(input_hidden_weights)
        # 初始化隐藏层到输出层权重
        self.init_hidden_to_output_weights(hidden_output_weights)
        
    def init_input_to_hidden_weights(self,weights):
        weight_num = 0
        for i_num in range(len(self.hidden_layer.neurons)):
            for o_num in range(self.input_num):
                if weights is None:
                    self.hidden_layer.neurons[i_num].weights.append(random.random())
                else:
                    self.hidden_layer.neurons[i_num].weights.append(weights[weight_num])
                weight_num += 1
                
    def init_hidden_to_output_weights(self,weights):
        weight_num = 0
        for i_num in range(len(self.output_layer.neurons)):
            for o_num in range(len(self.hidden_layer.neurons)):
                if weights is None:
                    self.output_layer.neurons[i_num].weights.append(random.random())
                else:
                    self.output_layer.neurons[i_num].weights.append(weights[weight_num])
                weight_num += 1
    
    def inspect(self):
        print('..................')
        print('input inspect:',[i for i in self.inputs])
        print('..................')
        print('hidden inspect:')
        self.hidden_layer.inspect()
        print('..................')
        print('output inspect:')
        self.output_layer.inspect()
        print('..................')
        
    def forward(self,inputs):
        hidden_layer_outout = self.hidden_layer.forward(inputs)
        print('hidden_layer_outout',hidden_layer_outout)
        ouput_layer_ouput = self.output_layer.forward(hidden_layer_outout)
        print('ouput_layer_ouput',ouput_layer_ouput)
        return ouput_layer_ouput
        
    def train(self,x,y):
        ouput_layer_ouput = self.forward(x)
        
        # 求total / neto的偏导
        total_o_pd = [0] * len(self.output_layer.neurons)
        for o in range(len(self.output_layer.neurons)):
            total_o_pd[o] = self.output_layer.neurons[o].calculate_total_net_pd(y[o]) 
            
        # 求total / h的偏导 = total.1 / h的偏导 + total.2 / h的偏导
        total_neth_pd = [0] * len(self.hidden_layer.neurons)
        for h in range(len(self.hidden_layer.neurons)):
            total_h_pd = 0
            for o in range(len(self.output_layer.neurons)):
                total_h_pd += total_o_pd[o] * self.hidden_layer.neurons[h].weights[o]
            total_neth_pd[h] = total_h_pd * self.output_layer.neurons[h].calculate_output_net_pd()
            
        # 更新输出层神经元权重
        for o in range(len(self.output_layer.neurons)):
            for ho_w in range(len(self.output_layer.neurons[o].weights)):
                ho_w_gradient = total_o_pd[o] * self.output_layer.neurons[o].calculate_net_linear_pd(ho_w)
                self.output_layer.neurons[o].weights[ho_w] -= self.learning_rate * ho_w_gradient
                
        # 更新隐藏层神经元权重
        for h in range(len(self.hidden_layer.neurons)):
            for ih_w in range(len(self.hidden_layer.neurons[h].weights)):
                ih_w_gradient = total_neth_pd[h] * self.hidden_layer.neurons[h].calculate_net_linear_pd(ih_w)
                self.hidden_layer.neurons[h].weights[ih_w] -= self.learning_rate * ih_w_gradient
                
    def calculate_total_error(self, training_sets):
        total_error = 0
        for t in range(len(training_sets)):
            training_inputs, training_outputs = training_sets[t]
            self.forward(training_inputs)
            for o in range(len(training_outputs)):
                total_error += self.output_layer.neurons[o].calculate_error(training_outputs[o])
        return total_error
    
class NeuralLayer:
    def __init__(self,neural_num,bias):
        self.bias = bias if bias else random.random()
        
        self.neurons = []
        
        for i in range(neural_num):
            self.neurons.append(Neuron(self.bias))
            
    def inspect(self):
        print('weights:',[neuron.weights for neuron in self.neurons])
        print('bias:',[neuron.bias for neuron in self.neurons])
      
    def get_output(self,inputs):
        outputs = []
        for neuron in self.neurons:
            outputs.append(neuron.output)
        return outputs
    
    def forward(self,inputs):
        outputs = []
        for neuron in self.neurons:
            outputs.append(neuron.calculate_output(inputs))
        return outputs
    
class Neuron:
    def __init__(self,bias):
        self.bias = bias        
        self.weights = []
        
    def calculate_output(self,inputs):
        self.inputs = inputs 
        total_net_outputs = self.calculate_total_net_output()
        self.output = self.sigmoid(total_net_outputs)
        return self.output
    
    def calculate_total_net_output(self):
        total = 0
        for i in range(len(self.inputs)):
            total += self.inputs[i] * self.weights[i]
        return total + self.bias
        
    def sigmoid(self,total_net_input):
        return 1 / (1 + math.exp(-total_net_input))
    
    def calculate_total_output_pd(self,total_output):
        return -(total_output - self.output)
    
    def calculate_output_net_pd(self):
        return self.output * (1 - self.output)
    
    def calculate_total_net_pd(self,total_output):
        return self.calculate_total_output_pd(total_output) * self.calculate_output_net_pd()
        
    def calculate_net_linear_pd(self,index):
        return self.inputs[index]
    
    def calculate_error(self, target_output):
        return 0.5 * (target_output - self.output) ** 2
nn = NeuralNetwork(2, 2, 2, input_hidden_weights=[0.15, 0.2, 0.25, 0.3], input_hidden_bias=0.35, 
                   hidden_output_weights=[0.4, 0.45, 0.5, 0.55], hidden_output_bias=0.6)
for i in range(10000):
    nn.train([0.05, 0.1], [0.01, 0.99])
    print(i, round(nn.calculate_total_error([[[0.05, 0.1], [0.01, 0.99]]]), 9))
hidden_layer_outout [0.5932699921071872, 0.596884378259767]
ouput_layer_ouput [0.7513650695523157, 0.7729284653214625]
hidden_layer_outout [0.5932662857458805, 0.5968782561589732]
ouput_layer_ouput [0.7420894573166411, 0.775286681791022]
迭代1次 误差0.291028391
hidden_layer_outout [0.5932662857458805, 0.5968782561589732]
ouput_layer_ouput [0.7420894573166411, 0.775286681791022]
hidden_layer_outout [0.5932623797502049, 0.5968720205376407]
ouput_layer_ouput [0.7324789213021788, 0.7775850216973027]
迭代2次 误差 0.283547957
hidden_layer_outout [0.5932623797502049, 0.5968720205376407]
ouput_layer_ouput [0.7324789213021788, 0.7775850216973027]
hidden_layer_outout [0.5932582778661456, 0.5968656853189723]
ouput_layer_ouput [0.7225410209751278, 0.7798256906644981]
迭代3次 误差 0.275943973
hidden_layer_outout [0.5932582778661456, 0.5968656853189723]
ouput_layer_ouput [0.7225410209751278, 0.7798256906644981]
hidden_layer_outout [0.5932539857457403, 0.5968592652405116]
ouput_layer_ouput [0.7122866732919141, 0.7820107943242337]
迭代4次 误差0.268233041
...
...
...
hidden_layer_outout [0.5930355856011269, 0.5965960242813179]
ouput_layer_ouput [0.016365976874939202, 0.9836751989707726]
hidden_layer_outout [0.5930355857116121, 0.5965960243868048]
ouput_layer_ouput [0.016365393177895763, 0.9836757760229975]
迭代10000次 误差 4.0257e-05

迭代10000次后结果:

hidden_layer_outout [0.5930355856011269, 0.5965960242813179]

ouput_layer_ouput [0.016365976874939202, 0.9836751989707726]

hidden_layer_outout [0.5930355857116121, 0.5965960243868048]

ouput_layer_ouput [0.016365393177895763, 0.9836757760229975]

迭代10000次 误差 4.0257e-05

参考:https://www.cnblogs.com/charlotte77/p/5629865.html,感谢博主的分析

版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员-用户IM,转载请注明出处:https://javaforall.cn/120014.html原文链接:https://javaforall.cn

【正版授权,激活自己账号】: Jetbrains全家桶Ide使用,1年售后保障,每天仅需1毛

【官方授权 正版激活】: 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...

(0)


相关推荐

  • com组件与dll的区别_组件对象模型

    com组件与dll的区别_组件对象模型这阵子在想一个需要利用com组件的小程序怎么做,突然想起上次去面试的时候考官问过autocad开发时为什么要利用com,而不采用一般的dll呢?   到google上查了一下,许多人也问了一样的问题:)   用com来写程序要比普通的dll麻烦一些,但是带来的好处也大很多,尤其是在开发像autocad这样大型软件的时候,需要跨区域来协同工作。 “学习COM,首先要知道COM的目的是什么,它

    2022年10月23日
  • Git rebase使用详解

    Git rebase使用详解

  • Android中构建数据业务应用

    Android中构建数据业务应用

  • css3实现遮罩层_div遮罩层

    css3实现遮罩层_div遮罩层JS+CSS简单实现DIV遮罩层显示隐藏

  • Oracle创建用户、角色、授权、建表

    Oracle创建用户、角色、授权、建表oracle数据库的权限系统分为系统权限与对象权限。系统权限(databasesystemprivilege)可以让用户执行特定的命令集。例如,createtable权限允许用户创建表,grantanyprivilege权限允许用户授予任何系统权限。对象权限(databaseobjectprivilege)可以让用户能够对各个对象进行某些操作。例如delete权限允许用

  • 在系统设计中应用迪米特法则_软件设计原则口诀

    在系统设计中应用迪米特法则_软件设计原则口诀  迪米特原则又叫 迪米特法则/最少知道原则。现在我们给出迪米特原则的定义:一个对象应该对其他对象保持最少的了解。它的优点是降低了类之间的耦合。它强调只和朋友交流,不和陌生人说话。其中朋友是指出现在成员变量、方法的输入、输出参数中的类称为成员朋友类,而出现在方法体内部的类不属于朋友类。 下面我们通过代码,更好的解释迪米特原则。比如我是一个公司的老板,我开了一家网上课程网站,我让我的…

发表回复

您的电子邮箱地址不会被公开。

关注全栈程序员社区公众号