<jupyter_start><jupyter_text>Mock Quiz 2 You are strongly encourged to do the mock quiz first before reading the following solutions.  Scroll down for solutions if you are sure.   Q1**In Bayesian regression, what do we need to specify instead of to compute?****Answer**: We need to specify the prior. Placing prior on model parameters is the definition of Bayesian inference approach. Q2**An eigenvector of a covariance matrix tells you what?****Answer**: An eigenvector defines the direction of variability and the corresponding eigenvalue is the corresponding amount/magnitude of variability (the variance captured). Q3**Consider the following dataset: A = (0, 2), B = (0, 1) and C = (1, 0). The k-means algorithm is initialized with centers at A and B. Upon convergence, the two centers will be at****Answer**: Get a piece of paper to draw the three points. Start with centers A and B. Iterate: 1) assign closest points; 2) update centers. You will find it converges very soon at A and the midpoint of BC. Q4**We have a 2x1 feature vector. The first feature is X1 and the second feature is X2. The corresponding 2x2 covariance matrix is C = [9, 2; 2, 1], i.e., the first row is [9 2] and the second row is [2, 1]. What is the standard deviation of X1?****Answer**: The variance of X1 is C(1,1)=9 so the std is $\sqrt{9}=3$. Q5**We model a 3x1 feature vector using independent multivariate Gaussian distribution. How many parameters do we need to specify this multivariate distribution?****Answer**: The mean of this vector needs three parameters, one for each feature. Due to the independence, the covariance matrix is diagonal, with the variance of each feature on the diagonal and three variance parameters in total. Therefore, the total is 6.  Q6**Please modify Lab 6B striclty following the following requirements from a client:...****Answer**: Please see below.  B2. Linear Regression using PyTorch Let us start right away with implementing linear regression in PyTorch to study PyTorch concepts closely. This part follows the [PyTorch Linear regression example](https://github.com/pytorch/examples/tree/master/regression) that trains a **single fully-connected layer** to fit a 4th degree polynomial. A synthetic linear regression problem* Generate model parameters, weight and bias. The weight vector and bias are both tensors, 1D and 0D, respectively.<jupyter_code>import sys
print(sys.version)
import torch
import torch.nn.functional as F

POLY_DEGREE = 6
#Set the random seed for reproducibility 
torch.manual_seed(2019) 
W_target = torch.randn(POLY_DEGREE, 1) * 5
b_target = torch.randn(1) * 5
print(W_target)
print(b_target)<jupyter_output>tensor([[-0.5937],
        [ 1.0550],
        [ 3.7313],
        [-3.0678],
        [-0.5928],
        [ 7.7825]])
tensor([6.8310])<jupyter_text>* Next, define a number of functions to generate the input (variables) and output (target/response).<jupyter_code>def make_features(x):
    """Builds features i.e. a matrix with columns [x, x^2, x^3, x^4]."""
    x = x.unsqueeze(1)
    return torch.cat([x ** i for i in range(1, POLY_DEGREE+1)], 1)

def f(x):
    """Approximated function."""
    return x.mm(W_target) + b_target.item()

def poly_desc(W, b):
    """Creates a string description of a polynomial."""
    result = 'y = '
    for i, w in enumerate(W):
        result += '{:+.2f} x^{} '.format(w, len(W) - i)
    result += '{:+.2f}'.format(b[0])
    return result

def get_batch(batch_size=32):
    """Builds a batch i.e. (x, f(x)) pair."""
    random = torch.randn(batch_size)
    x = make_features(random)
    y = f(x)
    return x, y<jupyter_output><empty_output><jupyter_text>* Define a simple(st) neural network, which is a **single fully connected** (FC) layer. See [`torch.nn.Linear`](https://pytorch.org/docs/master/nn.htmltorch.nn.Linear)<jupyter_code>fc = torch.nn.Linear(W_target.size(0), 1)
print(fc)<jupyter_output>Linear(in_features=6, out_features=1, bias=True)<jupyter_text>This is a *network* with four input units, one output unit, with a bias term.    * Now generate the data. Let us try to get five pairs of (x,y) first to inspect.<jupyter_code>sample_x, sample_y = get_batch(10)
print(sample_x)
print(sample_y)<jupyter_output>tensor([[ 1.0199e+00,  1.0401e+00,  1.0608e+00,  1.0819e+00,  1.1034e+00,
          1.1253e+00],
        [-1.0375e+00,  1.0763e+00, -1.1166e+00,  1.1585e+00, -1.2019e+00,
          1.2469e+00],
        [-7.2291e-01,  5.2260e-01, -3.7779e-01,  2.7311e-01, -1.9743e-01,
          1.4273e-01],
        [-1.0365e+00,  1.0743e+00, -1.1135e+00,  1.1541e+00, -1.1962e+00,
          1.2399e+00],
        [-1.8514e-01,  3.4276e-02, -6.3457e-03,  1.1748e-03, -2.1750e-04,
          4.0268e-05],
        [-1.5111e+00,  2.2835e+00, -3.4506e+00,  5.2143e+00, -7.8794e+00,
          1.1907e+01],
        [ 1.6494e-01,  2.7204e-02,  4.4869e-03,  7.4005e-04,  1.2206e-04,
          2.0132e-05],
        [ 5.5548e-02,  3.0856e-03,  1.7140e-04,  9.5209e-06,  5.2887e-07,
          2.9378e-08],
        [-1.2363e+00,  1.5285e+00, -1.8898e+00,  2.3364e+00, -2.8885e+00,
          3.5712e+00],
        [ 6.2126e-01,  3.8596e-01,  2.3978e-01,  1.4896e-01,  9.2544e-02,
          5.7494e-02]])
tensor([[16.0656],
        [1[...]<jupyter_text>Reset the gradients to zero, perform a forward pass to get prediction, and compute the loss<jupyter_code>fc.zero_grad()
output = F.smooth_l1_loss(fc(sample_x), sample_y)
loss1 = output.item()
print(loss1)<jupyter_output>18.689815521240234<jupyter_text>Random did not give a good prediction. Let us do a backpropagation and update model parameters with gradients.<jupyter_code>output.backward() 
for param in fc.parameters():  
    param.data.add_(-0.1 * param.grad.data)<jupyter_output><empty_output><jupyter_text>Check the updated weights and respective loss<jupyter_code>print(fc.weight)
output = F.smooth_l1_loss(fc(sample_x), sample_y)
loss2 = output.item()
print(loss2)
print(loss1-loss2)<jupyter_output>0.8722133636474609
{'fraction_non_alphanumeric': 0.12896725440806045, 'fraction_numerical': 0.09689336691855584, 'mean_word_length': 3.757188498402556, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '539578', 'license': 'MIT', 'n_tokens_mistral': 2415, 'n_tokens_neox': 2108, 'n_words': 653, 'path': 'quiz/Mock Quiz 2 Solution.ipynb', 'repo_name': 'SilvesterHsu/COM6509', 'size': '5955'}