<jupyter_start><jupyter_text>Final Project  "Forecasting the winner of tennis matches in the Men’s ATP World Tour"  Problem Statement The goal of the project is to predict the probability that the higher-ranked player will win a tennis match. We will call that a `win`. This can be stated more formally as:For two players ${\cal P}_1$ and ${\cal P}_2$ where ${{\rm Rank}_1} > {{\rm Rank}_2}$ we calculate the probability that ${\cal P}_1$ will win the match, or that a `win` will happen.  Dataset Results for the men's ATP tour date back to January 2000 from the dateset http://www.tennis-data.co.uk/data.php (obtained from Kaggle). The original dataset is shown below.<jupyter_code>import pandas as pd
df_atp = pd.read_csv('Data.csv')
df_atp.head()<jupyter_output><empty_output><jupyter_text>Features  The features for each match that were used in the project follow:- `Date`: date of the match - `Series`: name of ATP tennis series (we kept the four main current categories namely Grand Slams, Masters 1000, ATP250, ATP500)- `Surface`: type of surface (clay, hard or grass)- `Round`: round of match (from first round to the final)- `Best of`: maximum number of sets playable in match (Best of 3 or Best of 5)- `WRank`: ATP Entry ranking of the match winner as of the start of the tournament- `LRank`: ATP Entry ranking of the match loser as of the start of the tournament The output variable is binary. The better player has higher rank by definition. We define the `win` variable by: ${\rm{win}} = \left\{ {\begin{array}{*{20}{c}}1&{{\rm{higher\,\,ranked\,\,player\,\, wins}}}\\0&{{\rm{higher\,\,ranked\,\,player\,\, loses}}}\end{array}} \right.$  Importing basic modules<jupyter_code>import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt
from sklearn import metrics
import seaborn as sns
sns.set_style("darkgrid")
import pylab as pl
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Pre-Processing of dataset After loading the dataset we proceed as following:- Keep only completed matches i.e. eliminate matches with injury withdrawals and walkovers.- For convenience we rename `Best of` to `Best_of`- Choose the features listed above- Drop `NaN` entries- Consider the two final years only (to avoid comparing different categories of tournaments which existed in the past). We note that this choice is somewhat arbitrary and can be changed if needed.- Choose only higher ranked players for better accuracy (as suggested by Corral and Prieto-Rodriguez (2010) and confirmed here)<jupyter_code>df_atp['Date'] = pd.to_datetime(df_atp['Date']) 
# Restricing dates
df_atp = df_atp.loc[(df_atp['Date'] > '2014-11-09') & (df_atp['Date'] <= '2016-11-09')]
# Keeping only completed matches
df_atp = df_atp[df_atp['Comment'] == 'Completed'].drop("Comment",axis = 1)
# Rename Best of to Best_of
df_atp.rename(columns = {'Best of':'Best_of'},inplace=True)
# Choosing features
cols_to_keep = ['Date','Series','Surface', 'Round','Best_of', 'WRank','LRank']
# Dropping NaN
df_atp = df_atp[cols_to_keep].dropna()
# Dropping errors in the dataset and unimportant entries (e.g. there are very few entries for Masters Cup)
df_atp = df_atp[(df_atp['LRank'] != 'NR') & (df_atp['WRank'] != 'NR') & (df_atp['Series'] != 'Masters Cup')]
df_atp.head()
# Transform strings into numerical values
df_atp[['Best_of','WRank','LRank']] = df_atp[['Best_of','WRank','LRank']].astype(int)
df_atp.head()<jupyter_output><empty_output><jupyter_text>We now create an extra columns for the variable `win` described above using an auxiliary function `win(x)`<jupyter_code>def win(x):
    if x > 0:
        return 0
    elif x <= 0:
        return 1  
    
df_atp['win'] = (df_atp['WRank'] - df_atp['LRank']).apply(win)
df_atp.head()<jupyter_output><empty_output><jupyter_text>Following Corral and Prieto-Rodriguez (2010) we restrict the analysis to higher ranked players (the analysis including all players has indeed less predictive power as we tested)<jupyter_code>newdf = df_atp.copy()
newdf2 = newdf[(newdf['WRank'] <= 150) & (newdf['LRank'] <= 150)]
newdf2.head()<jupyter_output><empty_output><jupyter_text>Best_of = 5We now restrict our analysis to matches of Best_of = 5. Since only Grand Slams have 5 sets we can drop the new `Series` column. The case of Best_of = 3 will be considered afterwards.<jupyter_code>df3 = newdf2.copy()
df3 = df3[df3['Best_of'] == 5]
# Drop Best_of and Series columns
df3.drop("Series",axis = 1,inplace=True)
df3.drop("Best_of",axis = 1,inplace=True)
df3.head()<jupyter_output><empty_output><jupyter_text>Now we notice that our dataset is uneven in terms of frequency of `wins`:<jupyter_code>print(df3['win'].sum(axis=0))
print(len(df3.index))
df3['win'].sum(axis=0)/float(len(df3.index))<jupyter_output>618
816<jupyter_text>To correct this problem, and create a balanced dataset, we use a stratified sampling procedure.<jupyter_code>y_0 = df3[df3.win == 0] 
y_1 = df3[df3.win == 1] 
n = min([len(y_0), len(y_1)]) 
y_0 = y_0.sample(n = n, random_state = 0) 
y_1 = y_1.sample(n = n, random_state = 0)
df_strat = pd.concat([y_0, y_1]) 
X_strat = df_strat[['Date', 'Surface', 'Round','WRank', 'LRank']]
y_strat = df_strat.win
X_strat.head()
y_strat.head()
X_strat_1=X_strat.copy()
X_strat_1['win']=y_strat
X_strat_1.head()<jupyter_output><empty_output><jupyter_text>We now define the variables ${\cal P}_1$ and ${\cal P}_2$ where ${{\rm Rank}_1} > {{\rm Rank}_2}$<jupyter_code>df = X_strat_1.copy()
df["P1"] = df[["WRank", "LRank"]].max(axis=1)
df["P2"] = df[["WRank", "LRank"]].min(axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Exploratory Analysis for Best_of = 5 We first look at percentage of wins for each surface. We find that when the `Surface` is Clay there is a higher likelihood of upsets (opposite of wins) i.e. the percentage of wins is lower. The difference is not too large tough.<jupyter_code>win_by_Surface = pd.crosstab(df.win, df.Surface).apply( lambda x: x/x.sum(), axis = 0 )
win_by_Surface
win_by_Surface = pd.DataFrame( win_by_Surface.unstack() ).reset_index()
win_by_Surface.columns = ["Surface", "win", "total" ]
fig2 = sns.barplot(win_by_Surface.Surface, win_by_Surface.total, hue = win_by_Surface.win )
fig2.figure.set_size_inches(8,5)<jupyter_output><empty_output><jupyter_text>What about the dependence on rounds? The relation is not very clear but we can clearly see that upsets are unlikely to happen on the semifinals.<jupyter_code>win_by_round = pd.crosstab(df.win, df.Round).apply( lambda x: x/x.sum(), axis = 0 )
win_by_round
win_by_round = pd.DataFrame(win_by_round.unstack() ).reset_index()
win_by_round.columns = ["Round", "win", "total" ]
fig2 = sns.barplot(win_by_round.Round, win_by_round.total, hue = win_by_round.win )
fig2.figure.set_size_inches(8,5)<jupyter_output><empty_output><jupyter_text>Let's check these results before stratification<jupyter_code>win_by_Surface_before_strat = pd.crosstab(df3.win, df3.Surface).apply( lambda x: x/x.sum(), axis = 0 )
win_by_Surface_before_strat = pd.DataFrame( win_by_Surface_before_strat.unstack() ).reset_index()
win_by_Surface_before_strat.columns = ["Surface", "win", "total" ]
fig2 = sns.barplot(win_by_Surface_before_strat.Surface, win_by_Surface_before_strat.total, hue = win_by_Surface_before_strat.win )
fig2.figure.set_size_inches(8,5)
win_by_round_before_strat = pd.crosstab(df3.win, df3.Round).apply( lambda x: x/x.sum(), axis = 0 )
win_by_round_before_strat = pd.DataFrame(win_by_round_before_strat.unstack() ).reset_index()
win_by_round_before_strat.columns = ["Round", "win", "total" ]
fig2 = sns.barplot(win_by_round_before_strat.Round, win_by_round_before_strat.total, hue = win_by_round_before_strat.win )
fig2.figure.set_size_inches(8,5)<jupyter_output><empty_output><jupyter_text>Dummy variables------ To keep the dataframe cleaner we transform the `Round` entries into numbers. We then transform rounds into dummy variables<jupyter_code>df1 = df.copy()
def round_number(x):
    if x == '1st Round':
        return 1
    elif x == '2nd Round':
        return 2
    elif x == '3rd Round':
        return 3
    elif x == '4th Round':
        return 4
    elif x == 'Quarterfinals':
        return 5
    elif x == 'Semifinals':
        return 6
    elif x == 'The Final':
        return 7
    
df1['Round'] = df1['Round'].apply(round_number)

dummy_ranks = pd.get_dummies(df1['Round'], prefix='Round')

df1 = df1.join(dummy_ranks.ix[:, 'Round_2':])
df1[['Round_2', 'Round_3',
       'Round_4', 'Round_5', 'Round_6', 'Round_7']] = df1[['Round_2', 'Round_3','Round_4', 'Round_5', 'Round_6', 'Round_7']].astype('int_')
df1.head()<jupyter_output><empty_output><jupyter_text>We repeat this for the `Surface` variable<jupyter_code>dummy_ranks = pd.get_dummies(df1['Surface'], prefix='Surface')
dummy_ranks.head()
df_2 = df1.join(dummy_ranks.ix[:, 'Surface_Grass':])
df_2.drop("Surface",axis = 1,inplace=True)
df_2[['Surface_Grass','Surface_Hard']] = df_2[['Surface_Grass','Surface_Hard']].astype('int_')
df_2.drop("Round",axis = 1,inplace=True)
df_2.head()<jupyter_output><empty_output><jupyter_text>We now take the logarithms of ${\cal P}_1$ and ${\cal P}_2$, then create a variable `D`<jupyter_code>df4 = df_2.copy()
df4['P1'] = np.log2(df4['P1'].astype('float64')) 
df4['P2'] = np.log2(df4['P2'].astype('float64')) 
df4['D'] = df4['P1'] - df4['P2']
df4['D'] = np.absolute(df4['D'])
df4.head()<jupyter_output><empty_output><jupyter_text>Model 1: Logistic Regression We now use a logistic regression to study our data.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
# Logit Function
def logit(x):
    return np.exp(x)  / (1 + np.exp(x)) 
    
x = np.linspace(-6,6,50, dtype=float)

y = logit(x)

plt.plot(x, y)
plt.ylabel("Probability")
plt.show()

df4.columns.tolist()
feature_cols = ['Round_2',
 'Round_3',
 'Round_4',
 'Round_5',
 'Round_6',
 'Round_7',
 'Surface_Grass',
 'Surface_Hard',
 'D']
dfnew = df4.copy()
dfnew[feature_cols].head()
X = dfnew[feature_cols]
y = dfnew.win
#pd.value_counts(dfnew['Best_of_5'].values, sort=False)
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred_class = logreg.predict(X_test)
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred_class))
from __future__ import print_function
print('True:', y_test.values[0:40])
print('Pred:', y_pred_class[0:40])
y_pred_prob = logreg.predict_proba(X_test)[:, 1]
auc_score = metrics.roc_auc_score(y_test, y_pred_prob)
auc_score
y_pred_prob = logreg.predict_proba(X_test)[:, 1]
auc_score = metrics.roc_auc_score(y_test, y_pred_prob)
auc_score
fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)
fig = plt.plot(fpr, tpr,label='ROC curve (area = %0.2f)' % auc_score )
plt.plot([0, 1], [0, 1], 'k--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.title('ROC curve for win classifier')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.legend(loc="lower right")
plt.grid(True)
import statsmodels.api as sm

X = dfnew[feature_cols]
X = sm.add_constant(X)
y = dfnew['win']

lm = sm.Logit(y, X)
result = lm.fit()

result.summary()
from sklearn.cross_validation import cross_val_score
cross_val_score(logreg, X, y, cv=10, scoring='roc_auc').mean()
logreg.fit(dfnew[["D"]],dfnew["win"])
pred_probs = logreg.predict_proba(dfnew[["D"]])
plt.scatter(dfnew["D"], pred_probs[:,1])
plt.title('Win Probability for 5 sets matches')
plt.xlabel('D')
plt.ylabel('Win Probability for 5 sets matches')
plt.legend(loc="lower right")
plt.grid(True)<jupyter_output><empty_output><jupyter_text>Decision Trees and Random Forests------- We now build a decision tree model to predict the upsets likelihood of a given match<jupyter_code>from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier()

X = dfnew[feature_cols].dropna()
y = dfnew['win']


model.fit(X, y)
from sklearn.cross_validation import cross_val_score

scores = cross_val_score(model, X, y, scoring='roc_auc', cv=5)
print('AUC {}, Average AUC {}'.format(scores, scores.mean()))
model = DecisionTreeClassifier(
                max_depth = 4,
                min_samples_leaf = 6)

model.fit(X, y)
scores = cross_val_score(model, X, y, scoring='roc_auc', cv=5)
print('CV AUC {}, Average AUC {}'.format(scores, scores.mean()))
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import cross_val_score

X = dfnew[feature_cols].dropna()
y = dfnew['win']

model = RandomForestClassifier(n_estimators = 200)
    
model.fit(X, y)


features = X.columns
feature_importances = model.feature_importances_

features_df = pd.DataFrame({'Features': features, 'Importance Score': feature_importances})
features_df.sort_values('Importance Score', inplace=True, ascending=False)

features_df
feature_importances = pd.Series(model.feature_importances_, index=X.columns)
feature_importances.sort()
feature_importances.plot(kind="barh", figsize=(7,6))
from sklearn.cross_validation import cross_val_score

scores = cross_val_score(model, X, y, scoring='roc_auc')
print('AUC {}, Average AUC {}'.format(scores, scores.mean()))

for n_trees in range(1, 100, 10):
    model = RandomForestClassifier(n_estimators = n_trees)
    scores = cross_val_score(model, X, y, scoring='roc_auc')
    print('n trees: {}, CV AUC {}, Average AUC {}'.format(n_trees, scores, scores.mean()))<jupyter_output>AUC [ 0.72956841  0.66758494  0.64152893], Average AUC 0.679560759106
n trees: 1, CV AUC [ 0.65151515  0.66666667  0.5632461 ], Average AUC 0.627142638506
n trees: 11, CV AUC [ 0.74437557  0.64543159  0.62775482], Average AUC 0.672520661157
n trees: 21, CV AUC [ 0.71613866  0.65381084  0.63636364], Average AUC 0.668771043771
n trees: 31, CV AUC [ 0.72623967  0.67229109  0.64577594], Average AUC 0.681435567799
n trees: 41, CV AUC [ 0.73783287  0.65048209  0.64015152], Average AUC 0.676155494337
n trees: 51, CV AUC [ 0.72463269  0.65438476  0.64290634], Average AUC 0.673974594429
n trees: 61, CV AUC [ 0.71705693  0.66069789  0.65587695], Average AUC 0.677877257423
n trees: 71, CV AUC [ 0.73519284  0.66609275  0.64623508], Average AUC 0.682506887052
n trees: 81, CV AUC [ 0.72945363  0.65943526  0.62775482], Average AUC 0.672214569942
n trees: 91, CV AUC [ 0.74368687  0.67171717  0.63429752], Average AUC 0.683233853688<jupyter_text>Best_of = 3We now restrict our analysis to matches of Best_of = 3.<jupyter_code>import pandas as pd
df_atp = pd.read_csv('Data.csv')
df_atp['Date'] = pd.to_datetime(df_atp['Date']) 
# Restricing dates
df_atp = df_atp.loc[(df_atp['Date'] > '2014-11-09') & (df_atp['Date'] <= '2016-11-09')]
# Keeping only completed matches
df_atp = df_atp[df_atp['Comment'] == 'Completed'].drop("Comment",axis = 1)
# Rename Best of to Best_of
df_atp.rename(columns = {'Best of':'Best_of'},inplace=True)
# Choosing features
cols_to_keep = ['Date','Series','Surface', 'Round','Best_of', 'WRank','LRank']
# Dropping NaN
df_atp = df_atp[cols_to_keep].dropna()
# Dropping errors in the dataset and unimportant entries (e.g. there are very few entries for Masters Cup)
df_atp = df_atp[(df_atp['LRank'] != 'NR') & (df_atp['WRank'] != 'NR') & (df_atp['Series'] != 'Masters Cup')]
df_atp[['Best_of','WRank','LRank']] = df_atp[['Best_of','WRank','LRank']].astype(int)
def win(x):
    if x > 0:
        return 0
    elif x <= 0:
        return 1  
    
df_atp['win'] = (df_atp['WRank'] - df_atp['LRank']).apply(win)
df_atp.head()
newdf = df_atp.copy()
newdf2 = newdf[(newdf['WRank'] <= 100) & (newdf['LRank'] <= 100)]
newdf2.head()
df3 = newdf2.copy()
df3 = df3[df3['Best_of'] == 3]
# Drop Best_of and Series columns
df3.drop("Series",axis = 1,inplace=True)
df3.drop("Best_of",axis = 1,inplace=True)
df3.head()
y_0 = df3[df3.win == 0] 
y_1 = df3[df3.win == 1] 
n = min([len(y_0), len(y_1)]) 
y_0 = y_0.sample(n = n, random_state = 0) 
y_1 = y_1.sample(n = n, random_state = 0)
df_strat = pd.concat([y_0, y_1]) 
X_strat = df_strat[['Date', 'Surface', 'Round','WRank', 'LRank']]
y_strat = df_strat.win
X_strat_1=X_strat.copy()
X_strat_1['win']=y_strat
X_strat_1.head()
df = X_strat_1.copy()
df["P1"] = df[["WRank", "LRank"]].max(axis=1)
df["P2"] = df[["WRank", "LRank"]].min(axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Exploratory Analysis for Best_of = 3<jupyter_code>#Without stratification
win_by_Surface = pd.crosstab(df3.win, df3.Surface).apply( lambda x: x/x.sum(), axis = 0 )
win_by_Surface
win_by_Surface = pd.DataFrame( win_by_Surface.unstack() ).reset_index()
win_by_Surface.columns = ["Surface", "win", "total" ]
fig2 = sns.barplot(win_by_Surface.Surface, win_by_Surface.total, hue = win_by_Surface.win )
fig2.figure.set_size_inches(8,5)
#without stratification
win_by_round = pd.crosstab(df.win, df.Round).apply( lambda x: x/x.sum(), axis = 0 )
win_by_round
win_by_round = pd.DataFrame(win_by_round.unstack() ).reset_index()
win_by_round.columns = ["Round", "win", "total" ]
fig2 = sns.barplot(win_by_round.Round, win_by_round.total, hue = win_by_round.win )
fig2.figure.set_size_inches(8,5)<jupyter_output><empty_output><jupyter_text>Dummy variables------<jupyter_code>df1 = df.copy()
def round_number(x):
    if x == '1st Round':
        return 1
    elif x == '2nd Round':
        return 2
    elif x == '3rd Round':
        return 3
    elif x == '4th Round':
        return 4
    elif x == 'Quarterfinals':
        return 5
    elif x == 'Semifinals':
        return 6
    elif x == 'The Final':
        return 7
    
df1['Round'] = df1['Round'].apply(round_number)

dummy_ranks = pd.get_dummies(df1['Round'], prefix='Round')

df1 = df1.join(dummy_ranks.ix[:, 'Round_2':])
df1[['Round_2', 'Round_3',
       'Round_4', 'Round_5', 'Round_6', 'Round_7']] = df1[['Round_2', 'Round_3','Round_4', 'Round_5', 'Round_6', 'Round_7']].astype('int_')
df1.head()
dummy_ranks = pd.get_dummies(df1['Surface'], prefix='Surface')
df_2 = df1.join(dummy_ranks.ix[:, 'Surface_Grass':])
df_2.drop("Surface",axis = 1,inplace=True)
df_2[['Surface_Grass','Surface_Hard']] = df_2[['Surface_Grass','Surface_Hard']].astype('int_')
df_2.drop("Round",axis = 1,inplace=True)
df_2.head()
df4 = df_2.copy()
df4['P1'] = np.log2(df4['P1'].astype('float64')) 
df4['P2'] = np.log2(df4['P2'].astype('float64')) 
df4['D'] = df4['P1'] - df4['P2']
df4['D'] = np.absolute(df4['D'])
df4.head()<jupyter_output><empty_output><jupyter_text>Model 1: Logistic Regression<jupyter_code>df4.columns.tolist()
#feature_cols = ['P1','P2','Round_2','Round_3','Round_4','Round_5','Round_6','Round_7','Surface_Grass','Surface_Hard','D']
feature_cols = ['D','Surface_Hard','Surface_Grass','Round_6','Round_5','Round_3']
dfnew = df4.copy()
X = dfnew[feature_cols]
y = dfnew.win
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred_class = logreg.predict(X_test)
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred_class))
y_pred_prob = logreg.predict_proba(X_test)[:, 1]
auc_score = metrics.roc_auc_score(y_test, y_pred_prob)
print(auc_score)
y_pred_prob = logreg.predict_proba(X_test)[:, 1]
auc_score = metrics.roc_auc_score(y_test, y_pred_prob)
auc_score
fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)
fig = plt.plot(fpr, tpr,label='ROC curve (area = %0.2f)' % auc_score )
plt.plot([0, 1], [0, 1], 'k--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.title('ROC curve for win classifier')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.legend(loc="lower right")
plt.grid(True)
from sklearn.cross_validation import cross_val_score
cross_val_score(logreg, X, y, cv=10, scoring='roc_auc').mean()
logreg.fit(dfnew[["D"]],dfnew["win"])
pred_probs = logreg.predict_proba(dfnew[["D"]])
plt.scatter(dfnew["D"], pred_probs[:,1])
plt.title('Win Probability for 3 sets matches')
plt.xlabel('D')
plt.ylabel('Win Probability for 3 sets matches')
plt.legend(loc="lower right")
plt.grid(True)<jupyter_output><empty_output><jupyter_text>Decision Trees and Random Forests<jupyter_code>from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier()

X = dfnew[feature_cols].dropna()
y = dfnew['win']


model.fit(X, y)
from sklearn.cross_validation import cross_val_score

scores = cross_val_score(model, X, y, scoring='roc_auc', cv=5)
print('AUC {}, Average AUC {}'.format(scores, scores.mean()))
model = DecisionTreeClassifier(
                max_depth = 4,
                min_samples_leaf = 6)

model.fit(X, y)
scores = cross_val_score(model, X, y, scoring='roc_auc', cv=5)
print('CV AUC {}, Average AUC {}'.format(scores, scores.mean()))
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import cross_val_score

X = dfnew[feature_cols].dropna()
y = dfnew['win']

model = RandomForestClassifier(n_estimators = 200)
    
model.fit(X, y)


features = X.columns
feature_importances = model.feature_importances_

features_df = pd.DataFrame({'Features': features, 'Importance Score': feature_importances})
features_df.sort_values('Importance Score', inplace=True, ascending=False)

features_df
feature_importances = pd.Series(model.feature_importances_, index=X.columns)
feature_importances.sort()
feature_importances.plot(kind="barh", figsize=(7,6))
from sklearn.cross_validation import cross_val_score

scores = cross_val_score(model, X, y, scoring='roc_auc')
print('AUC {}, Average AUC {}'.format(scores, scores.mean()))

for n_trees in range(1, 100, 10):
    model = RandomForestClassifier(n_estimators = n_trees)
    scores = cross_val_score(model, X, y, scoring='roc_auc')
    print('n trees: {}, CV AUC {}, Average AUC {}'.format(n_trees, scores, scores.mean()))<jupyter_output>AUC [ 0.56410714  0.55746329  0.54182243], Average AUC 0.554464288773
n trees: 1, CV AUC [ 0.54352679  0.52182012  0.52169808], Average AUC 0.529014994355
n trees: 11, CV AUC [ 0.57148597  0.5538084   0.55104636], Average AUC 0.558780245814
n trees: 21, CV AUC [ 0.55047194  0.54987731  0.53842448], Average AUC 0.546257911153
n trees: 31, CV AUC [ 0.57475765  0.55339731  0.54241338], Average AUC 0.556856114695
n trees: 41, CV AUC [ 0.55816327  0.5527357   0.54013952], Average AUC 0.550346161769
n trees: 51, CV AUC [ 0.5441199   0.55787438  0.53792988], Average AUC 0.546641388544
n trees: 61, CV AUC [ 0.56126913  0.55277424  0.54369805], Average AUC 0.552580476248
n trees: 71, CV AUC [ 0.56765306  0.55975643  0.53744813], Average AUC 0.554952539745
n trees: 81, CV AUC [ 0.56794643  0.55797716  0.54158477], Average AUC 0.555836118697
n trees: 91, CV AUC [ 0.5585523   0.55412315  0.53579733], Average AUC 0.549490924948
{'fraction_non_alphanumeric': 0.15051121154943176, 'fraction_numerical': 0.060774935275790955, 'mean_word_length': 5.695064629847239, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 92, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '387546', 'license': 'MIT', 'n_tokens_mistral': 9473, 'n_tokens_neox': 8558, 'n_words': 2011, 'path': 'tennis/notebooks/Final_Project_Marco_Tavora-DATNYC41_GA.ipynb', 'repo_name': 'marcotav/machine-learning-classification-projects', 'size': '22789'}