<jupyter_start><jupyter_text>EDA & ProfilingThis EDA is based on the European Soccer Database with more than 25,000 matches and more than 10,000 players for European professional soccer seasons from 2008 to 2016.  Import Librariescustomplot: contains functions written for this notebook<jupyter_code>import sqlite3
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn.preprocessing import scale
!find . | grep customplot
%%!
mkdir customplot && touch ./customplot/__init__.py
cp ../_pycode/customplot.py ./customplot/customplot.py
from customplot import *<jupyter_output><empty_output><jupyter_text>Data Download the data from: https://www.kaggle.com/hugomathien/soccer  Ingest Data<jupyter_code>%find.. 'database.sqlite'
# Create your connection.
cnx = sqlite3.connect('../_data/database.sqlite')
df = pd.read_sql_query("SELECT * FROM Player_Attributes", cnx)<jupyter_output><empty_output><jupyter_text>Exploring DataFrame<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Feature stats<jupyter_code>df.describe().T<jupyter_output><empty_output><jupyter_text>Check nulls, NaN's, etc.<jupyter_code>df.isnull().any().any()
'percentage null: '; df.isnull().any().sum() / df.shape[0] * 100<jupyter_output><empty_output><jupyter_text>Percentage nulls<jupyter_code>%precision 2
df.isnull().sum(axis=0).describe()
df.isnull().sum(axis=0).max() * 100 / df.shape[0], 'max % NaN'<jupyter_output><empty_output><jupyter_text>Drop nulls<jupyter_code>df = df.dropna()<jupyter_output><empty_output><jupyter_text>Sanity check<jupyter_code>df.isnull().sum(axis=0).max() * 100 / df.shape[0], 'max % NaN'
df.info()<jupyter_output><empty_output><jupyter_text>Shuffle df<jupyter_code>df = df.reindex(np.random.permutation(df.index))<jupyter_output><empty_output><jupyter_text>Predicting: 'overall_rating' of a player<jupyter_code>df.sample(5)<jupyter_output><empty_output><jupyter_text>Feature Correlation Analysis Next, we will check if 'penalties' is correlated to 'overall_rating'. We are using a similar selection operation, bu this time for all the rows and within the correlation function.<jupyter_code>df[:10][['penalties', 'overall_rating']]
df['overall_rating'].corr(df['penalties'])<jupyter_output><empty_output><jupyter_text>Create a list of potentially correlated features<jupyter_code>potentialFeatures = ['acceleration', 'curve', 'free_kick_accuracy', 'ball_control', 'shot_power', 'stamina']<jupyter_output><empty_output><jupyter_text>Check correlation coefficient of "overall_rating" of a player with each feature we added to the list as potential.<jupyter_code>for f in potentialFeatures:
    related = df['overall_rating'].corr(df[f])
    print("%s: %f" % (f,related))
df.columns.values.shape
cols = ['potential',  'crossing', 'finishing', 'heading_accuracy',
       'short_passing', 'volleys', 'dribbling', 'curve', 'free_kick_accuracy',
       'long_passing', 'ball_control', 'acceleration', 'sprint_speed',
       'agility', 'reactions', 'balance', 'shot_power', 'jumping', 'stamina',
       'strength', 'long_shots', 'aggression', 'interceptions', 'positioning',
       'vision', 'penalties', 'marking', 'standing_tackle', 'sliding_tackle']
corr_list = [(f, df['overall_rating'].corr(df[f])) for f in cols]
df2 = pd.DataFrame(corr_list, columns=['attributes', 'correlation'])
df2.sample(5)
p25 = df2.describe().loc['25%',][0]
p50 = df2.describe().loc['50%',][0]
p75 = df2.describe().loc['75%',][0]
p25, p50, p75<jupyter_output><empty_output><jupyter_text>Visualisation of correlations<jupyter_code>def plot_dataframe(df, y_label):  
    global p25, p50, p75
    color='coral'
    fig = plt.gcf()
    fig.set_size_inches(20, 6)
    plt.title(y_label)

    ax = df['correlation'].plot(linewidth=3.3, color=color)
    ax.axhline(p25, c='gray')
    ax.axhline(p50, c='k')
    ax.axhline(p75, c='gray')
    ax.xaxis.grid()
    ax.set_xticks(df.index)
    ax.set_xticklabels(df.attributes, rotation=75); #Notice the ; (remove it and see what happens !)
    plt.show()
plot_dataframe(df2, 'Player\'s Overall Rating')<jupyter_output><empty_output><jupyter_text>Correlation heatmapThe features with highest correlation coefficients are indicative for high Overall Rating. However we are never sure if the top features are independent!<jupyter_code>import seaborn as sns

plt.figure(figsize=(20, 12))
sns.set(style="white")
cmap = sns.diverging_palette(220, 10, as_cmap=True)

cor = df.loc[:, cols].corr()
cor.shape
mask = np.zeros_like(cor)
mask[np.triu_indices_from(mask)] = True

ax = sns.heatmap(cor, mask=mask, cmap=cmap, vmax=.85);<jupyter_output><empty_output><jupyter_text>Clustering Players into similar groupsWe can group similar players based on certain features.Note: Generally, someone with domain knowledge needs to define important features. We could have also selected some of the features with highest correlation with overall_rating. However, it does not guarantee best outcome always as we are not sure if the top five features are independent. For example, if 4 of the 5 features depend on the remaining 1 feature, taking all 5 does not give new information.  Select features for clustering - looking for youg mid-field player<jupyter_code>sel_features = ['reactions', 'short_passing', 'long_passing', 'vision', 'interceptions', 'standing_tackle', 'potential']
df_select = df[sel_features].copy(deep=True)
df_select.head()<jupyter_output><empty_output><jupyter_text>Perform K-Means ClusteringWe use K-Means to cluster the selected features in K clusters.<jupyter_code># Perform scaling on the dataframe containing the features
data = scale(df_select)

# Define number of clusters
k = 4

# Train a model
model = KMeans(init='k-means++', n_clusters=k, n_init=20).fit(data)<jupyter_output><empty_output><jupyter_text>DataFrame with feature coords for each cluster center<jupyter_code>df = pd.DataFrame(model.cluster_centers_)
df.columns = sel_features
df['players'] = pd.value_counts(model.labels_, sort=False)
df['cluster'] = df.index.astype(int)
df<jupyter_output><empty_output><jupyter_text>Cluster profilesWe have K clusters based on the selected features and visualise them as profiles for similar groups of players. Each point is the average value of the cluster for that feature.<jupyter_code>repeat = len(data)//5 + len(data) % 5
my_colors = list('brgyk' * repeat)[:len(data)]
from pandas.plotting import parallel_coordinates

plt.figure(figsize=(15,8)).gca().axes.set_ylim([-2.5, +2.5])
df.pop('players')
parallel_coordinates(df, 'cluster', color = my_colors, marker='o');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1305467211873391, 'fraction_numerical': 0.01711343328789944, 'mean_word_length': 6.899521531100478, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '307375', 'license': 'MIT', 'n_tokens_mistral': 2341, 'n_tokens_neox': 2220, 'n_words': 527, 'path': '_EDA/EDA_soccer.ipynb', 'repo_name': 'aixpact/data-science', 'size': '6603'}