<jupyter_start><jupyter_text>神经网络算法实现的核心之一是对代价函数的反向求导，Theano和Tensorflow中都定义了求导的符号函数，同样地，作为深度学习平台，自动求导（``autograd``）功能在pytorch中也扮演着核心功能，不同的是，pytorch的动态图功能使其更灵活（define by run）， 比如甚至在每次迭代中都可以通过改变pytorch中Variable的属性，从而使其加入亦或退出反向求导图，这个功能在某些应用中会特别使用，比如在训练的后半段，我们需要更新的只有后层的参数，那么只需要在前层参数的Variable设置成不要求求导。  变量``autograd.Variable``是pytorch设计理念的核心之一，它将tensor封装成Variable，并支持绝大多数tensor上的操作，同时赋予其两个极其重要的属性：``requires_grad``和``volatile``，Variable还具有三个属性： ``.data``用于存储Variable的数值，``.grad``也为变量，用于存储Variable的导数值,``.grad_fn `` (creator) 是生成该Variable的函数，当Variable为用户自定义时，其值为"None". 细节参见源码[Variable](http://pytorch.org/docs/0.1.12/_modules/torch/autograd/variable.html)<jupyter_code>x = Variable(T.ones(2,2), requires_grad=True)
print x
y = T.exp(x + 2)
yy = T.exp(-x-2)
print y
z = (y + yy)/2
out = z.mean()
print z, out
make_dot(out)
out.backward(T.FloatTensor(1), retain_graph=True)
x.grad
T.randn(1,1)
from __future__ import print_function
xx = Variable(torch.randn(1,1), requires_grad = True)
print(xx)
yy = 3*xx
zz = yy**2

#yy.register_hook(print)
zz.backward(T.FloatTensor([0.1]))
print(xx.grad)<jupyter_output>Variable containing:
 1.1988
[torch.FloatTensor of size 1x1]

Variable containing:
 2.1578
[torch.FloatTensor of size 1x1]<jupyter_text>A simple numpy implementation of one hidden layer neural network. In this implementation, for each update of $w_i$, both the forward and backward passes need to computed.<jupyter_code># y_pred = w2*(relu(w1*x))
# loss = 0.5*sum (y_pred - y)^2
import numpy as np

N, D_in, D_hidden, D_out = 50, 40, 100, 10

x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

w1 = np.random.randn(D_in, D_hidden)
w2 = np.random.randn(D_hidden, D_out)

learning_rate = 0.0001
for t in range(100):
    ### 前向通道
    h = x.dot(w1) #50x40 and 40x100 produce 50x100
    h_relu = np.maximum(h, 0)  #this has to be np.maximum as it takes two input arrays and do element-wise max, 50x100
    y_pred = h_relu.dot(w2) #50x100 and 100x10 produce 50x10
    #print y_pred.shape
    
    ### 误差函数
    loss = 0.5 * np.sum(np.square(y_pred - y))
    
    
    ### 反向通道
    grad_y_pred = y_pred - y #50x10
    grad_w2 = h_relu.T.dot(grad_y_pred) #50x100 and 50x10 should produce 100x10, so transpose h_relu
    grad_h_relu = grad_y_pred.dot(w2.T) #50x10 and 100x10 should produce 50x100, so transpose w2
    grad_h = grad_h_relu.copy() #make a copy of 
    grad_h[grad_h < 0] = 0      #
    grad_w1 = x.T.dot(grad_h)     #50x100 and 50x40 should produce 40x100
    
    w1 = w1 - learning_rate * grad_w1
    w2 = w2 - learning_rate * grad_w2<jupyter_output><empty_output><jupyter_text>with very slight modifications, we could end up with the implementation of the same algorithm in PyTorch<jupyter_code>import torch

N, D_in, D_hidden, D_out = 50, 40, 100, 10

x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

w1 = torch.randn(D_in, D_hidden)
w2 = torch.randn(D_hidden, D_out)

learning_rate = 0.0001
for t in range(100):
    h = x.mm(w1) #50x40 and 40x100 produce 50x100
    #h = x.matmul(w1) #50x40 and 40x100 produce 50x100, matmul for checking
    h_relu = h.clamp(min=0)  #this has to be np.maximum as it takes two input arrays and do element-wise max, 50x100
    y_pred = h_relu.mm(w2) #50x100 and 100x10 produce 50x10
    #print y_pred.shape
    
    loss = 0.5 * (y_pred - y).pow(2).sum()
    
    grad_y_pred = y_pred - y #50x10
    grad_w2 = h_relu.t().mm(grad_y_pred) #50x100 and 50x10 should produce 100x10, so transpose h_relu
    grad_h_relu = grad_y_pred.mm(w2.t()) #50x10 and 100x10 should produce 50x100, so transpose w2
    grad_h = grad_h_relu.clone() #make a copy
    grad_h[grad_h < 0] = 0      #
    grad_w1 = x.t().mm(grad_h)     #50x100 and 50x40 should produce 40x100
    
    w1 = w1 - learning_rate * grad_w1
    w2 = w2 - learning_rate * grad_w2<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11740041928721175, 'fraction_numerical': 0.07416142557651992, 'mean_word_length': 4.316155988857939, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '532366', 'license': 'MIT', 'n_tokens_mistral': 1912, 'n_tokens_neox': 1807, 'n_words': 358, 'path': '_drafts/.ipynb_checkpoints/pytorch-自动求导-checkpoint.ipynb', 'repo_name': 'linlinzhao/linlinzhao.github.io', 'size': '3816'}