<jupyter_start><jupyter_text>Simple Linear Regression y = mx + c, where m is the slope and c the intercept. In this case -- regression task, m represents the feature and c the bias<jupyter_code>import matplotlib.pyplot as plt
import numpy as np<jupyter_output><empty_output><jupyter_text>Initialize data that follows a linear assumption  Generate dataGenerate training and test data where y which is the output based on the equation y = mx + c<jupyter_code># create a function that creates data
# training data range is set smaller than test data range to demonstrate extrapolation 
# i.e the model will predict the future and we can observe performance of the model as seen in the last section
def create_data(n_sample=10, train_start=-20, train_end=20, test_start=-30, test_end=30, noise_sigma=0.1):
    x = np.linspace(train_start, train_end, n_sample)
    x_test = np.linspace(test_start, test_end, n_sample).reshape((-1, 1))

    intercept = np.random.randn()
    linear_coff = np.random.randn()

    y = linear_coff * x + intercept
    y += noise_sigma * np.random.randn(*y.shape)
    y_ground_truth = linear_coff * x_test + intercept

    return x.reshape((n_sample, -1)), y.reshape((n_sample, 1)), y_ground_truth, x_test, (intercept, linear_coff)<jupyter_output><empty_output><jupyter_text>`intercept` and `linear_coff` are the weights in this model, `y` is the predicted output, `x` is the input, `y_ground_truth` is the ground truth<jupyter_code>x, y, y_ground_truth, x_test, weights = create_data(n_sample=100, noise_sigma=1.1)
# Plot x and y as points in space, and the test data and ground truth
plt.plot(x_test, y_ground_truth, 'r-')
plt.scatter(x, y)
plt.show()<jupyter_output><empty_output><jupyter_text>Parameter EstimationWe need to optimize the weights so the models fits to the data.The estimated parameters (weights, w in this case) is defined as$$ w = X^+y $$where $X^+ = (X^TX)^{-1}X^T$ i.e the pseudo-inverse of X np.pinv(X). For a simple linear regression task like this, its a simple and effective solution.First1. Add one to `x` to represent to the intercept/bias to $X$2. Then $w = X^+y$<jupyter_code>def get_X(x):
    return np.hstack([x, np.ones_like(x)])
def get_weights(x, y):
    X = get_X(x)
    return np.dot(np.linalg.pinv(X), y)<jupyter_output><empty_output><jupyter_text>Now use the new weights to fit the model<jupyter_code>def get_y_pred(x, weights):
    return np.dot(get_X(x), weights)
weights = get_weights(x, y)
# pred = get_y_pred(x, weights)
# Plot the test data and the predicted data using the learned weights
plt.plot(x_test, get_y_pred(x_test, weights), 'k--')

# Plot the training data and the predicted data using the learned weights
plt.plot(x, get_y_pred(x, weights), 'bo')

# plot training data
plt.plot(x, y, 'rx')

# plot the ground truth
plt.plot(x_test, y_ground_truth, 'c-')<jupyter_output><empty_output><jupyter_text>Loss functionUsing mean squared error<jupyter_code>def get_loss(y, y_pred):
    return np.mean(np.square(y - y_pred))
# get loss for the learned weights i.e comparing the before learning and 
# y predicted with learned weights
loss_learned = get_loss(y, get_y_pred(x, weights))

# if weights were to 0
loss_no_weights = get_loss(y, 0)

# print the loss
print({"loss_learned": loss_learned, "loss_no_weights": loss_no_weights})<jupyter_output>{'loss_learned': 1.2077052241825554, 'loss_no_weights': 414.8765805912659}
{'fraction_non_alphanumeric': 0.10724808485562758, 'fraction_numerical': 0.0173836181496759, 'mean_word_length': 5.369606003752345, 'pattern_counts': {'":': 2, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '469131', 'license': 'MIT', 'n_tokens_mistral': 1228, 'n_tokens_neox': 1171, 'n_words': 375, 'path': 'linear_regression/simple_linear_regression.ipynb', 'repo_name': 'JohnOlushola/machine-learning', 'size': '3394'}