<jupyter_start><jupyter_text>Regression | | <NAME> | Wednesday March 28th 2018<jupyter_code># Import the necessary libraries

import numpy as np
import pandas as pd
import datetime as date
import matplotlib.pyplot as plt
%matplotlib inline
import numpy.polynomial.polynomial as poly
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import matplotlib
import seaborn as sns
matplotlib.rcParams['figure.figsize'] = [12.0, 8.0]
df = pd.read_csv('./ecommerce-customers.csv', error_bad_lines=False).drop_duplicates()
pd.DataFrame(df.head(100))
df.info()
df.describe()<jupyter_output><empty_output><jupyter_text>Use seaborn jointplot function to see which fields correlate well with the "Yearly Amount Spent" column. Write your findings. From our analysis it seems Length of Membership correalates very well with a larger average per year spend.<jupyter_code>sns.jointplot(df['Avg. Session Length'], df['Yearly Amount Spent'])
sns.jointplot(df['Time on App'], df['Yearly Amount Spent'])
sns.jointplot(df['Time on Website'], df['Yearly Amount Spent'])
sns.jointplot(df['Length of Membership'], df['Yearly Amount Spent'])
sns.pairplot(df)
# sns.lmplot(x='Length of Membership', y='Yearly Amount Spent')<jupyter_output><empty_output><jupyter_text>Create two dataframes: one for the target variable ("Yearly Amount Spent"), the other - containing all the rest of numerical features<jupyter_code>YAS_df = df['Yearly Amount Spent']
all_but_YAS_df = df[['Avg. Session Length', 'Time on App', 'Time on Website', 'Length of Membership']]<jupyter_output><empty_output><jupyter_text>Split the data into a training and test sets. Make a test set size 0.3 and random seed 123<jupyter_code># Generate the full data and split it into a training set and test set.

X_full = all_but_YAS_df['Length of Membership']
Y_full = YAS_df


X_train, X_test, Y_train, Y_test = train_test_split(X_full, Y_full, test_size=0.3, random_state=123)
plt.plot(X_train, Y_train, ".", markersize=20)
plt.title("Test Set")<jupyter_output><empty_output><jupyter_text>Fit a regression model on the training set<jupyter_code>coefs_lin = poly.polyfit(X_train, Y_train, 1)

X_line = np.linspace(0, 10, 10000)
ffit_lin = poly.polyval(X_line, coefs_lin)
plt.ylim(min(Y_train - 100), max(Y_train + 100))
plt.plot(X_train, Y_train, "r.", markersize=5)
plt.plot(X_line, ffit_lin, "g")
plt.show()<jupyter_output><empty_output><jupyter_text>Print out the coefficients of the model<jupyter_code>print(coefs_lin)<jupyter_output>[271.2757493   64.69085272]<jupyter_text>Make a prediction of the target variable from features dataframe<jupyter_code># Length of Membership would make a good target variable<jupyter_output><empty_output><jupyter_text>Calculate the Mean Squared Error (using sklearn.metrics module)<jupyter_code>mse_lin = mean_squared_error(Y_train, poly.polyval(X_train, coefs_lin))
print("MSE for a linear regressor: ", mse_lin)<jupyter_output>MSE for a linear regressor:  2167.1366186816913<jupyter_text>Using Seaborn distplot show the histogram of the residuals - differences between the target variable and predicted target variable<jupyter_code>sns.distplot(df['Length of Membership'])<jupyter_output>/Users/TBone/code_fellows/401/data-science/ENV/lib/python3.6/site-packages/matplotlib/axes/_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "
{'fraction_non_alphanumeric': 0.10532374100719424, 'fraction_numerical': 0.025323741007194246, 'mean_word_length': 7.140515222482436, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '512248', 'license': 'MIT', 'n_tokens_mistral': 1286, 'n_tokens_neox': 1196, 'n_words': 323, 'path': 'regression.ipynb', 'repo_name': 'tyler-fishbone/data-science', 'size': '3483'}