<jupyter_start><jupyter_text>Pandas & Seaborn[Pandas](https://pandas.pydata.org/) is a Python library for data analysis with easy-to-use data structures, mostly commonly tables of data called _data frames_. [Seaborn](https://seaborn.pydata.org/) is a Python library that provides a high-level interface for producing statistical graphics. Here 'high-level' means we can produce graphics with only a few lines of code that specify the type of graphic, instead of building it ourselves from with the more basic plotting tools provided by matplotlib.Used together Pandas and Seaborn allow us to quickly explore, visualise and understand a dataset. The key to this workflow is to wrangle the data into a dataframe in [Tidy Format](https://seaborn.pydata.org/introduction.htmlorganizing-datasets). Put simply Tidy Format is a table where:1. Each variable is represented by a column - e.g. "passes per 90".2. Each observation/record forms a row - e.g. rows are different players.For this notebook we provide an example dataframe for player passing data stored as a .csv file in Tidy Format. This data was prepared using the [Wyscout](https://wyscout.com/) API for a subset of leagues and resampled with replacement. This means that it does not represent real match data but maintains a similar distribution. Player and team names are anonymised.  Tidy data in Pandas First let's load the data with pandas and look at the first few rows.<jupyter_code>df = pd.read_csv('../data/pass_stats.csv', index_col=0)
df.head()<jupyter_output><empty_output><jupyter_text>This is data in Tidy Format - each row represents a player (indexed by a playerId) and the different columns contain information such as position or statistics about proficiency in different kinds of pass. Many Pandas functions are designed to work on data in this format which means we can now quickly explore the dataset.For example, we can derive **summary statistics** such as the mean long pass accuracy:<jupyter_code>#Computing the mean long pass accuracy for all players
df.longPassesAccPC.mean()<jupyter_output><empty_output><jupyter_text>We can also **filter** the dataset based on values in specific columns, for example by position. Note that `~` here is used as negation.<jupyter_code>#Filter for goalkeepers
df_gk = df[df.position=='gk']
#Filter for outfield players
df_of = df[~(df.position=='gk')]

#Compute the mean long pass accuracy for goalkeepers
gk_lpa = df_gk.longPassesAccPC.mean()
#Compute the mean long pass accuracy for outfield players
of_lpa = df_of.longPassesAccPC.mean()

print(gk_lpa, of_lpa)<jupyter_output>76.84345331567046 49.17923399988484<jupyter_text>So the average long pass accuracy for a goalkeeper is better than for an outfield player. This may not be an exciting result but it's not bad for a few lines of code. We can also **sort the data** by the values in the different columns. Note that this does not result in `df` being sorted after it is called, instead it returns a new dataframe that is a sorted version of `df`.<jupyter_code>df.sort_values(by='longPassesP90', ascending=False).head(5)<jupyter_output><empty_output><jupyter_text>Player 588 at team 89 tops the ranking for long passes p90, but his accuracy (39%) is well below average. Finally, in Tidy Format we can **combine columns** quickly to compute new statistics. For example, in this dataset every pass is characterised as either short or long so we can compute the total passes p90 by adding the two columns together.<jupyter_code>#Create a new column as the sum of two others
df['passesP90'] = df.shortPassesP90 + df.longPassesP90
#View team, position and passesP90 for the first 5 rows of the table
df[['teamId', 'position', 'passesP90']].head(5)<jupyter_output><empty_output><jupyter_text>Tidy data in SeabornSeaborn can quickly create graphics such as scatterplots, histograms and density plots by providing a dataframe and the associated column names for the values to plot. For example, we can check the relationship between short passes p90 and long passes p90 with a single line of code.<jupyter_code>#Produces a scatter plot of the 'shortPassesP90' columns against the 'longPassesP90' in df
sns.scatterplot(data=df, x='shortPassesP90', y='longPassesP90');<jupyter_output><empty_output><jupyter_text>Additional properties for plot can be associated with different columns of the dataframe. For example, we can highlight goalkeepers in the above scatterplot by setting a different marker.<jupyter_code>#Create a new Boolean column which is 1 for goalkeepers and 0 for outfield players
df['is_gk'] = (df.position == 'gk')
#Create the scatter plot highlighting the goalkeepers
sns.scatterplot(data=df, x='shortPassesP90', y='longPassesP90', hue='is_gk');<jupyter_output><empty_output><jupyter_text>[Many plots](https://seaborn.pydata.org/examples/index.html) can be produced using this basic pattern, for example swarmplots to compare the long passing accuracy of goalkeepers to outfield players<jupyter_code>sns.swarmplot(data=df, x='is_gk', y='longPassesAccPC');<jupyter_output><empty_output><jupyter_text>This works well for the goalkeepers but it looks like there's too many outfield players to really get a sense for the distribution as they bunch up and overlap along the left and right. In this case, it might be better to use a [violinplot](https://seaborn.pydata.org/generated/seaborn.violinplot.html) which gives a density estimate instead of plotting all of the points. Fortunately, changing between swamplots, violinplots and boxplots is as easy as changing one line of code (see below).  Customising plots with matplotlibThese seaborn graphics provide a quick way to explore the data but aren't polished enough for publication. Because seaborn is built using matplotlib we can tweak the seaborn figures easily by using matplotlib methods. For example, we can specify the size, alter the axis labels or add custom elements to the plot. Below we build a more complex graphic showing how a given player's pass accuracy compares to others by combining violinplots, swamplots and matplotlib plotting. (Note that here we specify colours such as `xkcd:pale red`. Matplotlib recognises this format for any colour in the [xkcd colour survery](https://xkcd.com/color/rgb/). This is a useful tool for finding good plot colours and remembering them by name).<jupyter_code>#First we extract the data on a given player of interest 
p_interest = df.loc[2085] #Data for the player with id 2085
spa, fpa, lpa = p_interest[['shortPassesAccPC', 'longPassesAccPC', 'forwardPassesAccPC']]
#initialise a 15x4 figure 
plt.figure(figsize=(15,4))

#Add a subplot to the figure as the first in a 3x1 layout
ax = plt.subplot(131)

#Use seaborn to add a violin plot to the ax object by provinding it to the seaborn method (ax=ax)
sns.violinplot(ax=ax, data=df, y='shortPassesAccPC', color='xkcd:grey')
# On put black markers at 10% opacity on top of the violin
sns.swarmplot(ax=ax, data=df, y='shortPassesAccPC', color='k', alpha=0.1)
#Highlight the player of interest with a red hexagon outlined in black.
plt.plot([0], [spa], marker='H', ms=12, c='xkcd:pale red',  markeredgewidth=1.5, markeredgecolor='k')

#Customise the matplotlib style
ax.set_ylim(0,)
ax.set_title('Short passing', size=16)
ax.set_ylabel('Pass accuracy %', size=14)

#The same for forward passes
ax = plt.subplot(132)
sns.violinplot(ax=ax, data=df, y='forwardPassesAccPC', color='xkcd:grey')
sns.swarmplot(ax=ax, data=df, y='forwardPassesAccPC', color='k', alpha=0.1)
plt.plot([0], [fpa], marker='H', ms=12, c='xkcd:pale red',  markeredgewidth=1.5, markeredgecolor='k')

ax.set_ylim(0,)
ax.set_title('Forward passing', size=16)
ax.set_ylabel('')

#The same for long passes
ax = plt.subplot(133)
sns.violinplot(ax=ax, data=df, y='longPassesAccPC', color='xkcd:grey')
sns.swarmplot(ax=ax, data=df, y='longPassesAccPC', color='k', alpha=0.1)
plt.plot([0], [lpa], marker='H', ms=12, c='xkcd:pale red',  markeredgewidth=1.5, markeredgecolor='k')
ax.set_ylim(0,)
ax.set_title('Long passing', size=16)
ax.set_ylabel('');

#Add a title for the full figure (use the y parameter to move this title up)
plt.suptitle('Player 2085', size=30, y=1.1);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08762509153038808, 'fraction_numerical': 0.01806199658286551, 'mean_word_length': 5.744855967078189, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '422992', 'license': 'MIT', 'n_tokens_mistral': 2589, 'n_tokens_neox': 2452, 'n_words': 1026, 'path': 'notebooks/TidyData_PandasSeaborn.ipynb', 'repo_name': 'ProformAnalytics/tutorial_nbs', 'size': '8194'}