<jupyter_start><jupyter_text>Compute slope * $r = \frac{\sum_i(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_i(x_i-\bar{x})}\sqrt{\sum_i(y_i-\bar{y})}}$* $a = \frac{\sum_i(x_i-\bar{x})(y_i-\bar{y})}{\sum_i(x_i-\bar{x})^2}=r\frac{\sigma_y}{\sigma_x}$<jupyter_code>mu_x = x.mean()
mu_y = y.mean()
# notice the denominator and numerator are both corrected by (n - 1), which cancels out
cov = sum([(x_ - mu_x) * (y_ - mu_y) for x_, y_ in zip(x, y)]) / (len(df) - 1)

r = cov / (x.std(ddof=1) * y.std(ddof=1))
r
# r^2: coefficient of determination
r ** 2
a = r * y.std(ddof=1) / x.std(ddof=1)
a<jupyter_output><empty_output><jupyter_text>Compute interceptBest fit line must pass through the average point.<jupyter_code>b = mu_y - a * mu_x
b<jupyter_output><empty_output><jupyter_text>Best fit line$y = 0.2x + 161.4$<jupyter_code>from matplotlib import pyplot as plt
x_anchor = np.array([min(x) - 100, max(x) + 100])
y_anchor = a * x_anchor + b
plt.scatter(x, y)
plt.plot(x_anchor, y_anchor)<jupyter_output><empty_output><jupyter_text>Error measure<jupyter_code>y_pred = a * x + b
# sum of sqared error (SSE)
se = [(y_p - y_) ** 2 for y_p, y_ in zip(y_pred, y)]
sse = sum(se)
sse
# standard error of estimate (SE)
# average deviation of y from the regression line
dof = len(df) - 2
np.sqrt(sse / dof)
# root mean squared error (RMSE)
np.sqrt(np.mean(np.array(se)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.186231884057971, 'fraction_numerical': 0.016666666666666666, 'mean_word_length': 4.901709401709402, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '16269', 'license': 'MIT', 'n_tokens_mistral': 639, 'n_tokens_neox': 612, 'n_words': 133, 'path': 'concepts/linear_regression.ipynb', 'repo_name': 'shawlu95/Data_Science_Toolbox', 'size': '1380'}