<jupyter_start><jupyter_text>Python statistics essential training - 05_04_crossvalidation Standard imports<jupyter_code>import math
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as pp
%matplotlib inline
import statsmodels
import statsmodels.api as sm
import statsmodels.formula.api as smf<jupyter_output><empty_output><jupyter_text>Loading gapminder data for year 1985 (Live Aid!).<jupyter_code>gapminder = pd.read_csv('gapminder.csv')
gdata = gapminder.query('year == 1985')<jupyter_output><empty_output><jupyter_text>Refactored plotting so that it works on arbitrary datasets.<jupyter_code>def sc(data):
    return 1e-6 * data.population, data.region.map({'Africa': 'skyblue', 'Europe': 'gold',
                                                   'America': 'palegreen', 'Asia': 'coral'})

def plotdata(data=gdata):
    size, colors = sc(data)
    data.plot.scatter('age5_surviving','babies_per_woman',
                      c=colors,s=size,linewidths=0.5,edgecolor='k',alpha=0.5)
                      
def plotfit(fit,data=gdata):
    plotdata(data)

    size, colors = sc(data)
    pp.scatter(data.age5_surviving,fit.predict(data),
               c=colors,s=30,linewidths=0.5,edgecolor='k',marker='D')<jupyter_output><empty_output><jupyter_text>Model formulas from the last two videos<jupyter_code>f_groupmeans = 'babies_per_woman ~ -1 + region'
f_surviving = 'babies_per_woman ~ -1 + region + age5_surviving'
f_srp = ('babies_per_woman ~ -1 + region + age5_surviving'
         '+ age5_surviving:region - age5_surviving + population')
# To divide up our data we first shuffle it with a Panda's dataframe method sample(). 
shuffled = gdata.sample(len(gdata))
# Then we use Panda's locator iloc[] which goes by row number to divide the rows in two
training, testing = shuffled.iloc[:len(gdata)//2], shuffled.iloc[len(gdata)//2:]
# We now fit the model to the training data
surviving = smf.ols(f_surviving, data=training).fit()
plotfit(surviving, training)
surviving.mse_resid
# We now try out the model on the testing data
plotfit(surviving,testing)<jupyter_output><empty_output><jupyter_text>To obtain the mean square of error, we need to do a little work1. Compute the residuals by applying modern prediction to the testing dataset. We subtract the observe data2. We compute the number of degrees of freedom which is the length of the data minus the number of patameters in the model minus 1 since the number of parameters don't count the constant, but we should.3. Finally, we square up the residuals, sum them, and divide by the number of degrees of freedom<jupyter_code>resid = surviving.predict(testing) - testing.babies_per_woman
df = len(testing) - surviving.df_model - 1
np.sum(resid**2)/ df<jupyter_output><empty_output><jupyter_text>Let's automate this procedure and use it on the three models. Since the results depend on the shuffling and are rather noisy, we'll repeat a few times and take the average.1. Let's make a function that's generic in terms of data, formula, response variable that we're looking at, and of the number of repitions2. We repeat in times, and we are going to take the code from above and modify them properly.3. We shuffle4. We divide in two5. We train6. And we compute the mean square of error7. We grab an arbitrary response viable column8. And finally we collect all these numbers in a python list and return its average.<jupyter_code>def cross_validate(data, formula, response, n=100):
    ret = []
    
    for i in range(n):
        shuffled = data.sample(len(data))
        
        training, testing = shuffled.iloc[:len(gdata)//2], shuffled.iloc[len(gdata)//2:]
        
        trained = smf.ols(f_surviving, data=training).fit()
        resid = trained.predict(testing) - testing.babies_per_woman
        df = len(testing) - trained.df_model - 1
        mse = np.sum(resid**2)/ df
        
        ret.append(mse)
        
    return np.mean(ret)
cross_validate(gdata, f_groupmeans, 'babies_per_woman')
cross_validate(gdata, f_surviving, 'babies_per_woman')
cross_validate(gdata, f_srp, 'babies_per_woman')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09653165170991997, 'fraction_numerical': 0.01309725927722532, 'mean_word_length': 4.24015247776366, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '108370', 'license': 'MIT', 'n_tokens_mistral': 1354, 'n_tokens_neox': 1283, 'n_words': 406, 'path': 'Python Statistics/Exercise Files/chapter5/05_04/05_04_crossvalidation_begin.ipynb', 'repo_name': 'atalebizadeh/DS-Career-Track', 'size': '4123'}