<jupyter_start><jupyter_text>Simple linear regression   Import the data [pandas](https://pandas.pydata.org/) provides excellent data reading and querying module,[dataframe](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html), which allows you to import structured data and perform SQL-like queries. Here we imported some house price records from [Trulia](https://www.trulia.com/?cid=sem|google|tbw_br_nat_x_x_nat!53f9be4f|Trulia-Exact_352364665_22475209465_aud-278383240986:kwd-1967776155_260498918114_). For more about extracting data from Trulia, please check [my previous tutorial](https://www.youtube.com/watch?v=qB418v3k2vk).<jupyter_code>import pandas
df = pandas.read_excel('house_price.xlsx')
df[:10]<jupyter_output><empty_output><jupyter_text>Prepare the data We want to use the price as the dependent variable and the area as the independent variable, i.e., use the house areas to predict the house prices<jupyter_code>X = df['area']
print (X[:10])
X_reshape = X.values.reshape(-1,1) # reshape the X to a 2D array
print (X_reshape[:10])
y = df['price']<jupyter_output>0    1541
1    1810
2    1456
3    2903
4    2616
5    3850
6    1000
7     920
8    2705
9    1440
Name: area, dtype: int64
[[1541]
 [1810]
 [1456]
 [2903]
 [2616]
 [3850]
 [1000]
 [ 920]
 [2705]
 [1440]]<jupyter_text>[sklearn](http://scikit-learn.org/stable/) provides a [split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function that can split the data into training data and testing data.<jupyter_code>import sklearn
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_reshape,y, test_size = 0.3) # put 30% data as the testing data

print ('number of training data:',len(X_train),len(y_train))
print ('number of testing data:',len(X_test),len(y_test))<jupyter_output>number of training data: 28 28
number of testing data: 13 13<jupyter_text>Train the model Use the [Linear Regression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) to estimate parameters from the training data.<jupyter_code>from sklearn import linear_model
slr = linear_model.LinearRegression() #create an linear regression model objective 

slr.fit(X_train,y_train) # estimate the patameters
print('beta',slr.coef_)
print('alpha',slr.intercept_)<jupyter_output>beta [99.0653637]
alpha 103007.2821439009<jupyter_text>Evaluate the model Let's calculate the [mean squared error](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.htmlsklearn.metrics.mean_squared_error) and the [r square](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.htmlsklearn.metrics.r2_score) of the model based on the testing data.<jupyter_code>from sklearn.metrics import mean_squared_error, r2_score

y_predict = slr.predict(X_test) # predict the Y based on the model
mean_squared_error = mean_squared_error(y_test,y_predict) # calculate mean square error
r2_score = r2_score(y_test,y_predict) #calculate r square

print ('mean square error:',mean_squared_error )
print ('r square:',r2_score )<jupyter_output>mean square error: 68539924787.35116
r square: -0.043685817412512984<jupyter_text>Visualize the model We use the [matplotlib](https://matplotlib.org/) to visualize our data.<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

plt.scatter(X_test, y_test,  color='black') # create a scatterplot to visualize the test data
plt.plot(X_test, y_predict, color='blue', linewidth=3) # add a line chart to visualize the model

plt.xlabel('area')
plt.ylabel('price')

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10827679782903664, 'fraction_numerical': 0.06540027137042062, 'mean_word_length': 6.284584980237154, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '137332', 'license': 'MIT', 'n_tokens_mistral': 1435, 'n_tokens_neox': 1271, 'n_words': 315, 'path': 'lab3.ipynb', 'repo_name': 'owainscholbe/ia480-real-', 'size': '3685'}