<jupyter_start><jupyter_text>In-class activity 2: Numpy, Pandas, Matplotlib  1. Numpy<jupyter_code># import numpy as np
import numpy as np
# the following code generates 1000 data points with 4 features each, and you are trying to predict one outcome y
# this makes the results reproducible
np.random.seed(10)
# generate X and Y
X = np.random.random(size=(1000, 4))
y = np.random.random(size=(1000, 1))<jupyter_output><empty_output><jupyter_text>1.1 Find optimal OLSYou can find the optimal parameters for the regression$$ \hat{y} = \hat{b}_1 x_1 + \hat{b}_2 x_2 + \hat{b}_3 x_3 + \hat{b}_4 x_4$$using the following expression$$\hat{b}=(X^T X)^{-1} X^T y$$To inverse a matrix, use the function `np.linalg.inv`. You can multiple matrices with the `@` operator.<jupyter_code># use numpy to find such expression and assign it to variable b
b = np.linalg.inv(X.T @ X) @ X.T @ y
print(b)<jupyter_output>[[0.24618073]
 [0.25870506]
 [0.14816792]
 [0.24958788]]<jupyter_text>1.2 Find sum of squared errors Compute the sum of squares errors using Numpy$$ SS=(Xb - y)^T (Xb - y)$$<jupyter_code># print SS
print((X @ b - y).T @ (X @ b - y))<jupyter_output>[[97.53639536]]<jupyter_text>2. Pandas  2.1 Load an import data<jupyter_code># load Pandas as alias it as "pd"
# your code here
import pandas as pd
# load remote TSV file "http://www4.stat.ncsu.edu/~boos/var.select/diabetes.tab.txt" into variable data
# you need to load it with function read_table because it is a tab separated file
data = pd.read_table('http://www4.stat.ncsu.edu/~boos/var.select/diabetes.tab.txt')
data<jupyter_output><empty_output><jupyter_text>2.2 Plot<jupyter_code># plot AGE vs BMI as an scatter plot (set x, y, and kind in the plot method)
data.plot(x="AGE", y="BMI", kind="scatter")<jupyter_output><empty_output><jupyter_text>2.3 Visualize by groups using seaborn It is known that men tend to have higher BMI than women. The variable SEX = 2 is man. Use the `seaborn` function `boxplot` to show that graphically. https://seaborn.pydata.org/generated/seaborn.boxplot.html<jupyter_code># import seaborn as sns
import seaborn as sns
# plot BMI by SEX
sns.boxplot(x='SEX', y="BMI", data=data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1278538812785388, 'fraction_numerical': 0.04018264840182648, 'mean_word_length': 5.086111111111111, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '196094', 'license': 'BSD-4-Clause-UC', 'n_tokens_mistral': 861, 'n_tokens_neox': 797, 'n_words': 253, 'path': 'activities/week2-activities-sol.ipynb', 'repo_name': 'daniel-acuna/ist718', 'size': '2190'}