<jupyter_start><jupyter_text>Gradient DescendWe will construct a simple linear model and see how Gradient Descend algorithm helps us to find the best parameters.<jupyter_code>import torch

torch.manual_seed(42) # fix the seed; easy to debug<jupyter_output><empty_output><jupyter_text>Vectorizes Operation<jupyter_code>u = torch.randint(10, (5, 1))
v = torch.randint(10, (5, 1))
print(u)
print(v)
u.size()
u * v
u.pow(v)<jupyter_output><empty_output><jupyter_text>Ground Truth Model<jupyter_code>n = 5
a = 2.0
b = 3.0
epsilon = 0.02

# Specify our ground truth model
X = torch.randint(10, (n, 1))
ps = torch.ones(n)/2.0
exponents = torch.bernoulli(ps) * epsilon + (torch.ones(n) - epsilon/2)
e = exponents.reshape(5, 1)
T = a * (X.pow(e)) + b

print(X)
print(e)
print(T)
# verify that our implementation is correct. 
print( a * X[0][0]**e[0][0] + b)<jupyter_output>tensor(19.3362)<jupyter_text>The above output shows that we can do vectorized operations on tensors easily. You should be able to write down the ground truth model mathematically.   Fitting a Linear Model$$\hat{\mathbf{y}} = \mathbf{x}\mathbf{w} + \mathbf{b}$$<jupyter_code># note that all operations here are vectorized operation. 

def forward(x): # dangerous: assuming w and b are global variables. 
    return x.mm(w) + b

# Loss function
def SSEloss(y_pred, y): 
    return (y_pred - y) * (y_pred - y) /2.0

def loss(x, t):
    y_pred = forward(x)
    return (y_pred - t).pow(2).sum() /2.0

def gradient_w(x, t):  
    return  torch.t(forward(x) - t).mm(x)

def gradient_b(x, t):  
    return  (forward(x) - t).sum() # need to add sum(); otherwise, each b_i will be different due to different gradients it receives. 
# Training loop
MAX_EPOCHES = 500
learning_rate = 0.01 # it is tricky to find a good learning rate. 


w = torch.randn(1, 1)
b = torch.ones(n, 1)

for epoch in range(MAX_EPOCHES):
    grad_w = gradient_w(X, T)
    w = w - learning_rate * grad_w
    grad_b = gradient_b(X, T)
    b = b - learning_rate * grad_b
    
    print(".. grad: ", w.item(), b[0].item())
    l = loss(X, T)
    print("epoch = ", epoch, ", loss=", l.item(), "learning rate = ", learning_rate)
    
    # Adaptive learning rate on: 500 epoches => loss = 0.04446
    #                       off: 500 epoches => loss = 0.04179                                 
    #learning_rate = 1.0 / (100 + epoch)  
    
# After training
print(forward(X))
print(T)

print(w)
print(b)<jupyter_output>tensor([[ 2.0361]])
tensor([[ 2.9004],
        [ 2.9004],
        [ 2.9004],
        [ 2.9004],
        [ 2.9004]])
{'fraction_non_alphanumeric': 0.14397496087636932, 'fraction_numerical': 0.0406885758998435, 'mean_word_length': 3.2545757071547423, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '280287', 'license': 'MIT', 'n_tokens_mistral': 1034, 'n_tokens_neox': 959, 'n_words': 270, 'path': 'GradientDescend.ipynb', 'repo_name': 'LawranceYRF/6714_projecct1', 'size': '2556'}