<jupyter_start><jupyter_text>Stochastic Gradient Descent  - 上节梯度下降法如图所示[![8mATJK.png](https://s1.ax1x.com/2020/03/12/8mATJK.png)](https://imgchr.com/i/8mATJK)- 我们每次都把所有的梯度算出来，称为**批量梯度下降法**- 但是这样在样本容量很大时，也是比较耗时的，解决方法是**随机梯度下降法**[![8mALsH.png](https://s1.ax1x.com/2020/03/12/8mALsH.png)](https://imgchr.com/i/8mALsH)- 我们随机的取一个 $i$ ，然后用这个 $i$ 得到一个向量，然后向这个方向搜索迭代[![8mAHzD.png](https://s1.ax1x.com/2020/03/12/8mAHzD.png)](https://imgchr.com/i/8mAHzD)- 在随机梯度下降法中，我们不能保证寻找的方向就是损失函数减小的方向- 更不能保证时减小的最快的方向- 我们希望 $\eta$ 随着迭代次数增大越来越小，于是 $\eta$ 就有右边表示形式- 其中 a 和 b 是两个超参数  1. 批量梯度下降算法<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
m = 100000

x = np.random.normal(size = m)
X = x.reshape(-1, 1)
y = 4. * x + 3. + np.random.normal(0, 3, size=m)
def J(theta, X_b, y):
    try:
        return np.sum((y - X_b.dot(theta))**2) / len(y)
    except:
        return float('inf')
def dJ(theta, X_b, y):
    return X_b.T.dot(X_b.dot(theta) - y) * 2. / len(y)

def gradient_descent(X_b, y, initial_theta, eta, n_iters = 1e4, epsilon=1e-8):
    theta = initial_theta
    i_iter = 0
    
    while i_iter < n_iters:
        gradient = dJ(theta, X_b, y)
        last_theta = theta
        theta = theta - eta * gradient

        if np.abs(J(theta, X_b, y) - J(last_theta, X_b, y)) < epsilon:
            break
        
        i_iter += 1
    
    return theta

X_b = np.hstack([np.ones([len(X), 1]), X])
initial_theta = np.zeros(X_b.shape[1])
eta = 0.01
theta = gradient_descent(X_b, y, initial_theta, eta)
theta<jupyter_output><empty_output><jupyter_text>2. 随机梯度下降法<jupyter_code># 传入具体的某一行
def dJ(theta, X_b_i, y_i):
    return X_b_i.T.dot(X_b_i.dot(theta) - y_i) * 2. 
def sgd(X_b, y, initial_theta, n_iters):
    
    t0 = 5
    t1 = 50
    
    def learning_rate(cur_iter):
        return t0 / (cur_iter + t1)
    
    theta = initial_theta
    for cur_iter in range(n_iters):
        # 随机取一个 i
        rand_i = np.random.randint(len(X_b))
        gradient = dJ(theta, X_b[rand_i], y[rand_i])
        # 迭代 theta
        theta = theta - learning_rate(cur_iter) * gradient

    return theta

X_b = np.hstack([np.ones([len(X), 1]), X])
initial_theta = np.zeros(X_b.shape[1])
theta = sgd(X_b, y, initial_theta, n_iters=len(X_b)//3)
# 可以看出，我们只使用了三分之一的样本，就达到了很好的效果
theta
# theta 值和批量梯度下降算法几乎一致<jupyter_output><empty_output><jupyter_text>3. 使用我们自己的SGD<jupyter_code>from LR.LinearRegression import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit_sgd(X, y, n_iters=2)
lin_reg.coef_
lin_reg.intercept_<jupyter_output><empty_output><jupyter_text>使用真实数据<jupyter_code>from sklearn import datasets
boston = datasets.load_boston()
X = boston.data
y = boston.target

X = X[y < 50.0]
y = y[y < 50.0]
from LR.model_selection import train_test_split
# 数据集分割
X_train, X_test, y_train, y_test = train_test_split(X, y, seed=333)
# 归一化处理
from sklearn.preprocessing import StandardScaler

standardScaler = StandardScaler()
standardScaler.fit(X_train)
X_train_standard = standardScaler.transform(X_train)
X_test_standard = standardScaler.transform(X_test)
lin_reg2 = LinearRegression()
%time lin_reg2.fit_sgd(X_train_standard, y_train)
lin_reg2.score(X_test_standard, y_test)<jupyter_output><empty_output><jupyter_text>4. scikit-learn中的SGD<jupyter_code>from sklearn.linear_model import SGDRegressor
sgd_reg = SGDRegressor()
%time sgd_reg.fit(X_train_standard, y_train)
sgd_reg.score(X_test_standard, y_test)
SGDRegressor?
sgd_reg = SGDRegressor(n_iter=100)
%time sgd_reg.fit(X_train_standard, y_train)
sgd_reg.score(X_test_standard, y_test)<jupyter_output>Wall time: 7 ms
{'fraction_non_alphanumeric': 0.137426077161363, 'fraction_numerical': 0.027034638130104195, 'mean_word_length': 4.317365269461078, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 20, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '23028', 'license': 'MIT', 'n_tokens_mistral': 1792, 'n_tokens_neox': 1769, 'n_words': 261, 'path': 'ML-Base-MOOC/chapt-3 Linear Regression/07-Stochastic Gradient Descent.ipynb', 'repo_name': 'NovemberChopin/machine-learning', 'size': '3551'}