<jupyter_start><jupyter_text>check data: ATP finals in the past 10 yrs- processed data: finals extracted from ATP tournaments in the past 10 yrs (2008-2017), yr 2018 is not included because dataset is not complete- explore data, and select parameters for predictionLin Zhou, Sept 2018<jupyter_code>#### imports ####
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from scipy import stats
import os
from sklearn import linear_model


# configurations
%matplotlib inline  
#save_path = 'analysis_results/'  # folder for saving analysis results 
#### Main ####

# import csv 
file_path = '../data/processed/finals_yr08-17.csv'
df_finals = pd.read_csv(file_path)  

# create save path for figures 
save_path = '../reports/figures/ATP_finals_2008-2017/'<jupyter_output><empty_output><jupyter_text>Explore ages of players<jupyter_code># plot distribution of winner age
final_age_plt = df_finals.boxplot(column = ['winner_age'], by = 'tourney_year', showmeans = True)
plt.ylabel('Age (yrs)')

# one way ANOVA test on ages of players in finals, 
# p_value < 0.05: ages are significantly different; p_value > 0.05: ages are not significantly different
f_value, p_value = stats.f_oneway(df_finals['winner_age'][df_finals['tourney_year'] == 2008],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2009],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2010],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2011],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2012],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2013],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2014],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2015],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2016],
                                  df_finals['winner_age'][df_finals['tourney_year'] == 2017])

print(f_value, p_value)
if p_value <= 0.05:
    print('ages in finals are significantly different from each year')
else:
    print('ages in finals are not significantly different from each year')<jupyter_output>17.3316868113 7.21674678312e-26
ages in finals are significantly different from each year<jupyter_text>so there seems to be an upward trend for ages of winners in tournaments, which is consistent with the fact that in recent years, top players remain amost always the same ones let's confirm with linear regression?<jupyter_code># linear regression to see if there's a linear increase of player age over the years
X = df_finals['tourney_year'].values
y = df_finals['winner_age'].values

lm = linear_model.LinearRegression()
model = lm.fit(X.reshape(-1, 1) ,y)

# see how good the model is performing: 
print('coefficient of determination R^2 of the prediction = '+ str(model.score(X.reshape(-1, 1) ,y)))
# plot the trend of age increase
plt.scatter(X, y,  color='black')
plt.plot(X.reshape(-1, 1), model.predict(X.reshape(-1, 1)), color='blue', linewidth=3)
plt.xticks(np.arange(min(X), max(X)+1, 1.0))
plt.legend(['Prediction', 'Data'])
plt.title('Tournament winner ages \n Linear regression')
plt.ylabel('Age (yrs)')
plt.xlabel('Tournament year')
plt.xticks(rotation=45)

# save figure
plt.savefig(save_path + 'winner_age_year.png', bbox_inches='tight', dpi=150)<jupyter_output><empty_output><jupyter_text>the model has a low score 0.176 (the best model would have 1.0), from the figure we can see there's a upward trend, but data points are very dispersed  Explore players who are winning the most 1. Find ways to describe "most-winning": number of tournaments per player over the years? By tourney levels?2. Is number of wins per player consistent at different tournament levels?<jupyter_code># plot player wins by trourney levels

index = df_finals.winner_name.unique()  # list player names 
GS_count = df_finals.groupby('winner_name', sort=False)['tourney_level'].size()
fig = GS_count.plot.bar()
# there are too many players, select first 10，sorted by number of tournament they won
win_count = df_finals.groupby('winner_name', sort=False)['winner_name'].size().sort_values(ascending=False).head(10)
fig = win_count.sort_values().plot.barh()
plt.title('Number of ATP Tournaments Won by Players - Top 10 \n 2008-2017')
plt.xlabel('Tournament count')

# save figure
plt.savefig(save_path + 'winner_tournament_count.png', bbox_inches='tight', dpi=150)<jupyter_output><empty_output><jupyter_text>Pivot Table: winners by tourney levelsgroup win counts by player names and tournament type. plot bars for each player with tournament types* see if there's a correlation between number of grand-slam wins, and number of regular tournament wins<jupyter_code># make a pivot table for top 10 players who won the most grand slams, sorted by tournament levels 
df_top_grand_slam = []
df_by_level = df_finals[['winner_name', 'tourney_level']]
sum_by_level = df_by_level.pivot_table(index='winner_name',columns='tourney_level',aggfunc=len, fill_value=0)
sum_by_level.head(10)


# make a pivot table for top 10 players who won the most regular tournaments, sorted by tournament levels<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1080623942867882, 'fraction_numerical': 0.02950573200526217, 'mean_word_length': 4.419551934826884, 'pattern_counts': {'":': 1, '<': 22, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '552953', 'license': 'MIT', 'n_tokens_mistral': 1836, 'n_tokens_neox': 1608, 'n_words': 499, 'path': 'notebooks/ATP_finals_yr08-17.ipynb', 'repo_name': 'Linn39/Tennis_Analysis', 'size': '5321'}