<jupyter_start><jupyter_text>Linear Regression - Only using Math, Numpy, Scipy packages which are already imported in the file. Python 3.5 or 3.6 (for full support for typing annotations)  Implementation Implementations in linear_regression/lr.pyFor linear regression with l2 loss (a.k.a. Ridge loss), here are two useful links: - [a tutorial blog](https://goo.gl/iTX39z)- [a lecture slide](http://www.stat.cmu.edu/~ryantibs/datamining/lectures/16-modr1.pdf).<jupyter_code># for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2
from linear_regression.lr import LinearRegression, LinearRegressionWithL2Loss
from linear_regression.utils import mean_squared_error

import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Sanity testDo the following steps, as a simple test to check your model works correctly.- Load data (features and values) from function `generate_data_part_1`.- Create a LinearRegression model.- Train the model using the loaded data.- Calculate the MSE metric by your implementation of the mean squared error function.<jupyter_code>from linear_regression.data import generate_data_part_1
features, values = generate_data_part_1()
model = LinearRegression(nb_features=1)
model.train(features, values)

mse = mean_squared_error(values, model.predict(features))
print('[Linear Regression]\tmse: {mse:.5f}'.format(mse=mse))
plt.scatter([x[0] for x in features], values, label='origin');
plt.plot([x[0] for x in features], model.predict(features), label='predicted');
plt.legend()<jupyter_output><empty_output><jupyter_text>Feature EngineeringIn this part, we are following the same procedure as that in part a). First we will try the same process as above.We will try two cases : - No extra features- Adding polynomial features  Part 1.3.1 No extra features<jupyter_code>from linear_regression.data import generate_data_part_2
features, values = generate_data_part_2()
model = LinearRegression(nb_features=1)
model.train(features, values)

mse = mean_squared_error(values, model.predict(features))
print('[Linear Regression]\tmse: {mse:.5f}'.format(mse=mse))
plt.scatter([x[0] for x in features], values, label='origin');
plt.plot([x[0] for x in features], model.predict(features), label='predicted');
plt.legend()<jupyter_output><empty_output><jupyter_text>Adding polynomial featuresNote that in that synthetic dataset, the value is actually not linear with the feature (it is exponential). Inspired by Taylor expansion, we are going to add some polynomial features based on existing features.More specifically, support $${\vec {x}} = [x_1, ..., x_n]$$is the feature vector of one sample (whose value is $y$). Instead of modeling the relationship between $\vec{x}$ and $y$, we modeling the relationship of $y$ and $\vec{x'}$, where (suppose we are adding up to $k$-th degree polynomials)$${\vec {x'}}_k = [x_1 , x_2 , ..., x_n , x_{21} , ..., x_{2n}, ..., x_{k1}, ..., x_{kn} ]. $$Repeat the 5 steps, then report the MSE value on training set and model weights for the following three cases: $k = 2, k = 4, k = 8$ (for all numbers, keep 6 digits after the decimal point).<jupyter_code>from linear_regression.utils import polynomial_features
features, values = generate_data_part_2()
plt.scatter([x[0] for x in features], values, label='origin');

for k in [2, 4, 8]:
    features_extended = polynomial_features(features, k)
    model = LinearRegression(nb_features=k)
    model.train(features_extended, values)
    mse = mean_squared_error(values, model.predict(features_extended))
    print('[Polynomial Features]\tk: {k:d}\tmse: {mse:.5f}'.format(k=k, mse=mse))
    plt.plot([x[0] for x in features_extended], model.predict(features_extended), label='k={k}'.format(k=k));
plt.legend()<jupyter_output>[Polynomial Features]	k: 2	mse: 0.02315
[Polynomial Features]	k: 4	mse: 0.00002
[Polynomial Features]	k: 8	mse: 0.00000<jupyter_text>Train, validation, test Data processing Do the following steps:- Load data (features and values) from function generate data part 3. It’s a classification dataset, but we just use it as a regression dataset in this assignment.- Check that there are 150 data samples and each sample have a feature vector of length 4.- Split the whole data set into three parts:    - the train set contains first 100 samples (0th - 99th samples),    - the validation set contains the next 20 samples (100th - 119th samples),    - the test set contains the rest 30 samples (120th - 149th samples).<jupyter_code>from linear_regression.data import generate_data_part_3
features, values = generate_data_part_3()

train_features, train_values = features[:100], values[:100]
valid_features, valid_values = features[100:120], values[100:120]
test_features, test_values = features[120:], values[120:]

assert len(train_features) == len(train_values) == 100
assert len(valid_features) == len(valid_values) == 20
assert len(test_features) == len(test_values) == 30<jupyter_output><empty_output><jupyter_text>LinearRegression  Hyper-parameter and model selection For linear regression model with extra polynomial features, $k$ is a hyper-parameter. To choose the best one, we have to     - train a model with that hyper-parameter based on the train set,     - calculate its performance on the validation set    - select the best hyper-parameter (the trained model has the best performance on validation set).     In this task, we only search $k$ among the set {1, 3, 10}.<jupyter_code>best_mse, best_k = 1e10, -1
for k in [1, 3, 10]:
#     train_features_extended = polynomial_features(train_features, k)
    train_features_extended = train_features
    model = LinearRegression(nb_features=k)
    model.train(train_features_extended, train_values)
    train_mse = mean_squared_error(train_values, model.predict(train_features_extended))

#     valid_features_extended = polynomial_features(valid_features, k)
    valid_features_extended = valid_features
    valid_mse = mean_squared_error(valid_values, model.predict(valid_features_extended))
    print('[Model Selection]\tk: {k:d}\t'.format(k=k) + 
          'train mse: {train_mse:.5f}\tvalid mse: {valid_mse:.5f}'.format(
              train_mse=train_mse, valid_mse=train_mse))

    if valid_mse < best_mse:
        best_mse, best_k = valid_mse, k
# combined_features_extended = polynomial_features(train_features + test_features, best_k)
combined_features_extended = train_features + test_features
model = LinearRegression(nb_features=best_k)
model.train(combined_features_extended, train_values + test_values)

# test_features_extended = polynomial_features(test_features, best_k)
test_features_extended = test_features
test_mse = mean_squared_error(test_values, model.predict(test_features_extended))
print('[Linear Regression]\tbest_k: {best_k:d}\ttest mse: {test_mse:.5f}'.format(
    best_k=best_k, test_mse=test_mse))<jupyter_output>[Linear Regression]	best_k: 1	test mse: 0.08778<jupyter_text>Linear Regression With L2 Loss (Ridge Regression) For Ridge (linear regression with l2 loss) Regression, we still need to search $k$ among {1, 3, 10}.$\alpha$ is also a hyper-parameter, in this task, we search $\alpha$ among the set {0.01, 0.1, 1, 10}.<jupyter_code>from itertools import product

best_mse, best_k, best_alpha = 1e10, -1, -1
for k, alpha in product([1, 3, 10], [0.01, 0.1, 1, 10]):
    train_features_extended = polynomial_features(train_features, k)
#     train_features_extended = train_features
    model = LinearRegressionWithL2Loss(nb_features=k, alpha=alpha)
    model.train(train_features_extended, train_values)
    train_mse = mean_squared_error(train_values, model.predict(train_features_extended))

    valid_features_extended = polynomial_features(valid_features, k)
#     valid_features_extended = valid_features
    valid_mse = mean_squared_error(valid_values, model.predict(valid_features_extended))
    print('[Ridge Regression]\tk: {k:d}\talpha: {alpha}\t'.format(k=k, alpha=alpha) +
          'train mse: {train_mse:.5f}\tvalid mse: {valid_mse:.5f}'.format(
              train_mse=train_mse, valid_mse=valid_mse))

    if valid_mse < best_mse:
        best_mse, best_k, best_alpha = valid_mse, k, alpha
# combined_features_extended = polynomial_features(train_features + test_features, best_k)
combined_features_extended = train_features + test_features
model = LinearRegressionWithL2Loss(nb_features=best_k, alpha=best_alpha)
model.train(combined_features_extended, train_values + test_values)

# test_features_extended = polynomial_features(test_features, best_k)
test_features_extended = test_features
test_mse = mean_squared_error(test_values, model.predict(test_features_extended))
print('[best k]\tbest_k: {best_k:d}\tbest_alpha: {best_alpha:f}\t'.format(
    best_k=best_k, best_alpha=best_alpha) +
      'test mse: {test_mse:.5f}'.format(test_mse=test_mse))<jupyter_output>[best k]	best_k: 1	best_alpha: 0.010000	test mse: 0.08777
{'fraction_non_alphanumeric': 0.1062548779128108, 'fraction_numerical': 0.023079496041922175, 'mean_word_length': 5.795454545454546, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '167547', 'license': 'MIT', 'n_tokens_mistral': 3059, 'n_tokens_neox': 2818, 'n_words': 809, 'path': 'MyKit-Learn/Linear Regression.ipynb', 'repo_name': 'theFong/MyKit-Learn', 'size': '8969'}