<jupyter_start><jupyter_text>In this assignment, you'll practice some data manipulation in pandas, and perform a simple regression task using the Abalone data set from the UCI repositoryAbalone are a type of mollusk (you may have eaten one before).  Each row of this data set is an individual abalone, with a variety of measurements.  You can learn about the age of an abalone by counting rings in its shell (kind of like counting tree rings).  https://archive.ics.uci.edu/ml/datasets/AbaloneThis link describes the data.  Take a look at the attribute information so you know what each column is referring to.Please put all your code into a single script `hw2.py`, which will generate the image `regression.png`.  Submit both files to canvas.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

pd.set_option('display.max_rows', 10)<jupyter_output><empty_output><jupyter_text>1. Read the data into a DataFrameuse pandas to read the data at this link into a DataFrame:https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.dataHints: * decide if `pd.read_table` or `pd.read_csv` is more appropriate.* you can pass the url provided as a string (like in the Lecture 5 exercise)* You'll want to pass in at least 2 keyword arguments:    - names - a list of column names ("sex", "length", "diam", "height", wt_whole", "wt_shucked", "wt_viscera", "wt_shell", "rings")    - index_col=False - indicates that pandas should just create an index for each entry (we don't have ids for the data)<jupyter_code>fname = "https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data"
fname = "../abalone.data"
df = pd.read_csv(fname, 
                   names=["sex", "length", "diam", "height", "wt_whole", "wt_shucked", "wt_viscera", "wt_shell", "rings"],
                   index_col=False)
df
df.describe()<jupyter_output><empty_output><jupyter_text>2. Set up the data for a regression problemThe problem we're going to try to solve is to predict the number of rings in an abalone shell (the "rings" column) from the other features. This is a proxy for the age of the animal.This means we want a response vector `y` that contains the data in the `"rings"` column, and a design matrix `X` that contains all the data we'll use to predict the response (the other columns).Use the pasty library's `dmatrices` function to form your data and response matrices (see lecture 6 for an example).  Hints:* Since `"sex"` is categorical you'll want to use `C(sex)` in your model specification.<jupyter_code>from patsy import dmatrices
y, X = dmatrices("rings ~ C(sex) + length + diam + height + wt_whole + wt_shucked + wt_viscera + wt_shell", 
                 df, return_type="dataframe")<jupyter_output><empty_output><jupyter_text>you'll see that X has a column called `"Intercept"`.  We will not need this column, so remove it from the dataframe.```pythonX.drop(X[["Intercept"]], axis=1, inplace=True)```<jupyter_code>X.drop(X[['Intercept']], axis=1, inplace=True)
X<jupyter_output><empty_output><jupyter_text>3. Split the data into train and test setsNow, we're going to start using Scikit learn.Split your data into train and test sets (See lecture 6 for an example)Set your test size to be 30% of the data<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
X_train<jupyter_output><empty_output><jupyter_text>4. Fit a Linear Regression model to the data1. Use Scikit learn's linear regression class to fit the model: `from sklearn.linear_model import LinearRegression`2. Use your training data to fit the model3. Predict the number of rings in your test data using the `predict` method.4. create a scatter plot of your prediction vs. the true number of rings.  Save this figure as `regression.png` and sumbit it with your homework<jupyter_code>from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
model.score(X_test, y_test)
import matplotlib.pyplot as plt
predicted = model.predict(X_test)
fig, ax = plt.subplots(1, 1, figsize=(10, 8))

ax.scatter(predicted, y_test)

ax.set_xlabel(r'predicted')
ax.set_ylabel(r'true')
ax.set_title("Number of Rings in Abalone")


plt.savefig('regression.png')
plt.show()<jupyter_output><empty_output><jupyter_text>5. (Bonus) Try another regression classifierPick another regression classifier (e.g. try ridge, lasso, decision trees, nearest neighbors, ...) and repeat parts 3 and 4.  If you do this, name your image after the classifier you used e.g. `lasso.png`<jupyter_code>from sklearn.neighbors import KNeighborsRegressor

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

model = KNeighborsRegressor()

model.fit(X_train, y_train)

predicted = model.predict(X_test)

fig, ax = plt.subplots(1, 1, figsize=(10, 8))

ax.scatter(predicted, y_test)

ax.set_xlabel(r'predicted')
ax.set_ylabel(r'true')
ax.set_title("Number of Rings in Abalone")


plt.savefig('knn.png')
plt.show()
model.score(X_test, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09893784421715185, 'fraction_numerical': 0.007081038552321007, 'mean_word_length': 5.17861482381531, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '626285', 'license': 'MIT', 'n_tokens_mistral': 1688, 'n_tokens_neox': 1599, 'n_words': 593, 'path': 'hw/2018_fall/hw2/ipynb/hw2.ipynb', 'repo_name': 'samuelcheang0419/cme193', 'size': '5084'}