<jupyter_start><jupyter_text>Step 1: Model  Regression with Statsmodel  Correlation<jupyter_code>finalmerged_clean.corr()<jupyter_output><empty_output><jupyter_text>Correlation map<jupyter_code>sns.heatmap(finalmerged_clean.corr(), cmap = 'spring' , annot = True)<jupyter_output><empty_output><jupyter_text>Correlations plotted<jupyter_code>sns.pairplot(finalmerged_clean, size = 1.2, aspect=1.5)<jupyter_output><empty_output><jupyter_text>Least sqaures regressionLeast square regression works by making the total of the square of the errors as small as possible (that is why it is called "least squares").<jupyter_code># Create your feature matrix (X) and target vector (y)
y, X = patsy.dmatrices('Final_Time_Hund ~ Date + PercentWin + Thirds + Seconds + Firsts + Starts + ThreeF + FourF + FiveF', data=finalmerged_clean, return_type="dataframe") 

# Create your model
model = sm.OLS(y,X)
# Fitting the model to the training set
fit = model.fit()
# Print summary statistics of the model's performance
fit.summary()<jupyter_output><empty_output><jupyter_text>Let's try doing this again, removing the inputs with the largest p values (Seconds, Thirds and Starts)<jupyter_code>y, X = patsy.dmatrices('Final_Time_Hund ~ Date + Firsts + Starts + ThreeF + FourF + FiveF', data=finalmerged_clean, return_type="dataframe") 
model = sm.OLS(y,X)
fit = model.fit()
fit.summary()<jupyter_output><empty_output><jupyter_text>Plotting ResidualsLet's take a quick look at a plot of our residuals (actual value - predicted value) with this model. We do this because in a good model we essentially want our errors to be random. If our residuals look systematic (e.g. missing high for one range and low for another) then we probably are missing the actual functional dependency underlying the data (perhaps it's not really linear).<jupyter_code># Using statsmodels to plot the residuals
##This is pretty common for plotting how well your model is doing!
fit.resid.plot(style='o', figsize=(12,8));<jupyter_output><empty_output><jupyter_text>Regression with Sklearn  Getting to R^2<jupyter_code># Create an empty model
lr = LinearRegression()

# Choose the predictor variables, here all but the first which is the response variable
# This model is analogous to the Y ~ X1 + X2 + X3 + X4 + X5 + X6 model
X = finalmerged_clean.iloc[:,1:]
#Give me all the columns, and rows 1+ (so we don't hit the first row)
# Choose the response variable(s)
y = finalmerged_clean.iloc[:,0]
# Fit the model to the full dataset
lr.fit(X,y)
# Print out the R^2 for the model against the full dataset
lr.score(X,y)<jupyter_output><empty_output><jupyter_text>Running the slightly better model than before<jupyter_code>lr1 = LinearRegression()

# Choose the predictor variables, here all but the first which is the response variable
# This model is analogous to the Y ~ X1 + X3 + X6 model
X = finalmerged_clean.iloc[:,[1,3,7,8,9]]
# Choose the response variable(s)
y = finalmerged_clean.iloc[:,0]
# Fit the model to the full dataset
lr1.fit(X,y)
# Print out the R^2 for the model against the full dataset
lr1.score(X,y)<jupyter_output><empty_output><jupyter_text>Maybe we're interesting the the coefficient or the y-intercept. Sk learn makes us work for the raw values (not as clean and obvious like statsmodels).<jupyter_code>print(lr1.intercept_)
# print out other coefficients
lr1.coef_
#Coeficients for each variable
#Pickleing pandas
finalmerged_clean.to_pickle('practice.pkl')
### WHY DO MY PICKELS SHOW AN ERROR??
#Pickleing stats model
# Pickle fit2 to a file 
fit.save('practice2.pkl')
#Pickleing sklearn
from sklearn.externals import joblib
joblib.dump(lr, 'practice3.pkl')<jupyter_output><empty_output><jupyter_text>Polynomial Regression<jupyter_code># import PolynomialFeatures and make_pipeline for Polynomial Regression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline<jupyter_output><empty_output><jupyter_text>Step 2 : Train Test  Option 1 :Train test split<jupyter_code>from sklearn.linear_model import LinearRegression

lr = LinearRegression()
# Split the data into training and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
# Fit the model against the training data
#Training data X and y gathered from above. 

lr.fit(X_train, y_train)
# Evaluate the model against the testing data

lr.score(X_test, y_test)
#R^2
lr3 = LinearRegression()
lr3.fit(X, y)
lr3.score(X, y)

#Not totally sure what the outputs for these last two mean...?<jupyter_output><empty_output><jupyter_text>Option 2: K-test Split (cross validation)<jupyter_code># 10-fold cross-validation with our fake data
reg = LinearRegression()
meansquared = cross_val_score(reg, X, y, cv=3, scoring='mean_squared_error')
#rsquare = 


# scores output is negative, a sklearn quirk bc mse is used to min. optimization func.
print(-scores)

##reg = LinearRegression()
#scores = cross_val_score(reg, X_poly3, y, cv=10, scoring='mean_squared_error')
#scores = cross_val_score(reg, X_poly4, y, cv=10, scoring='mean_squared_error')
np.mean(-scores)
##We've run a controlled study, and this is how it can expect to perform on a test. 
##Mean squared error. Variance of the residuals. Take the square root for root mean squared error. 
np.std(-scores)
# >>> print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
#Accuracy: 0.98 (+/- 0.03)<jupyter_output><empty_output><jupyter_text>Model and Train Test - Regularized Regression  Lasso RegularizationWeren't sure what features to use... your model is way too complex, you want to 0 some out.<jupyter_code>X = finalmerged_clean.iloc[:,[1,3,7,8,9]]
y = finalmerged_clean.iloc[:,0]
from sklearn.linear_model import Lasso 
model = Lasso(1.0)
fit = model.fit(X,y)
fit.score(X,y)<jupyter_output><empty_output><jupyter_text>Ridge RegularizationIf you have colinearities.<jupyter_code>from sklearn.linear_model import Ridge 
model = Ridge(1.0)
fit = model.fit(X,y)
fit.score(X,y)<jupyter_output><empty_output><jupyter_text>Elastic Net<jupyter_code>from sklearn.linear_model import ElasticNet 
model = ElasticNet(1.0, l1_ratio = 0.5) ##This alpha seems high. 
fit = model.fit(X,y)
fit.score(X,y)<jupyter_output><empty_output><jupyter_text>Ridge cross validation<jupyter_code>from sklearn.linear_model import RidgeCV
rcv = RidgeCV(cv=10)
rcv.fit(X, y)
rcv.score(X_test, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10213366802635708, 'fraction_numerical': 0.013178537809852526, 'mean_word_length': 6.227891156462585, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 68, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '171355', 'license': 'Apache-2.0', 'n_tokens_mistral': 2250, 'n_tokens_neox': 2130, 'n_words': 663, 'path': 'Horse Race Data - Race time prediction/Draft Notebooks/Step3_Regressions_FirstPass.ipynb', 'repo_name': 'vszuflita/Metis', 'size': '6374'}