<jupyter_start><jupyter_text>Exploring Soccer players in FIFA - 20<jupyter_code># Import Packages
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pandas import DataFrame

from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error, median_absolute_error
from sklearn.ensemble import RandomForestRegressor
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Business Understanding I am a huge fan of Soccer. So, In this notebook I am using data science to answer three questions involving players in fifa 20 gameThe FIFA game mirrors the real world attributes of Soccer players. Their real wages are recorded here and also an approximate value of a player is also recorded. The skill points given to a player almost match how they play in real lifeSo from a team management point of view, I would like to see the following three questions answered by data science* Which type of players costs more money for a team (Value) ? and which type of players earn the most (Wage)?* What are the top 5 features that influences the value and wage of a player ?* How are skills distributed among different category of players (Forward, Midfield and Defense) ?  Understanding Data<jupyter_code>#import data and setting visibility to maximum
players_df = pd.read_csv('Downloads/players_20.csv')
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
players_df.head(10)
# Displaying the number of rows and columns
players_df.shape
# Displaying the data type of each column
players_df.dtypes<jupyter_output><empty_output><jupyter_text>Data Preparation We are going to remove the goal keepers from the dataset since their values are usually lower and they have negligible numbers in most pof the other skills. Removing them will remove null data in most columns that are applicable for outfield players. Also adding them will skew the data science we are about to do Data preparation steps we are about to do here are* Remove Goal keepers from Dataset* Remove columns with more null data* Remove other unnecessary columns like player face, id, long name and url* Replace NaNs with 0's* There are some columns in which there is multiple data, split them into separate columns* Remove columns with goalkeeping attributes and skills (Since they are null for outfield players)<jupyter_code># Remove Goal Keepers from the dataset
remove_gk = players_df[players_df["player_positions"]=='GK'].index
outfield_df = players_df.drop(remove_gk)
# Remove goalkeeping attributes (since they are null for outfield players)
outfield_df = outfield_df.loc[:, ~outfield_df.columns.str.startswith('goalkeeping')]<jupyter_output><empty_output><jupyter_text>position * Most of the players are capable of playing multipl positions. But they play in one position regularly for their club and their attributes are most suited for that role. * For example <NAME>'s player positions are RW,CF,ST. But he plays in RW almost everytime so his attributes correlate for a right winger* So we split the column and drop the second and third playing position and keep only the primary position for analysis workrate* Similarly workrate of a player is divided into two categories - attack and defense. In the dataset it is presented in one column. We split and keep the work rate data in two different columns<jupyter_code>outfield_df[['position','p_2','p_3']] = outfield_df['player_positions'].str.split(',',expand=True)
outfield_df['attack_workrate'] = outfield_df['work_rate'].map(lambda x: x.split('/')[0])
outfield_df['defense_workrate'] = outfield_df['work_rate'].map(lambda x: x.split('/')[1])
#Handle Columns with more null values
columns_with_null = outfield_df.isnull().sum()
columns_with_null = columns_with_null[columns_with_null > 0]/(outfield_df.shape[0]) * 100
columns_with_null.sort_values(inplace=True)
columns_with_null.plot.bar(figsize=(15,10))
plt.xlabel('Columns with null values')
plt.ylabel('Percentage of null rows')
plt.show()
# Remove Columns that has more than 50% of the rows with null values
more_than_50_null = [col for col in outfield_df.columns if (outfield_df[col].isnull().sum()/outfield_df.shape[0]) * 100 > 50]
for col in more_than_50_null:
    outfield_df.drop(col, axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Remove other Unnecessary rows* The rows of object data type at the end  like cb,rb,rs,ls shows how one player is capable in all possible positions. This is not a necessary data since we only want to know the player's overall ability and his specific skill sets, not his ability to be versatile. (No need to know how good a forward will fare as a defender)* attributes like team_position (Duplicate of player position), real_face, dob, long_name and url* Also remove second and third position along with work rate column since we have new relevant columns for those data needed for the analysis<jupyter_code>other_col_dropped = ['ls','rs','lf','rf','cf','st','lw','rw','lam','cam','ram','lm','cm','rm','lcm','rcm','ldm','rdm','cdm','lwb','rwb','lb','rb','lcb','rcb','cb','dob','long_name','sofifa_id','player_url','team_position','player_positions','p_2','real_face','work_rate','joined','contract_valid_until','release_clause_eur','team_jersey_number']
for col in other_col_dropped:
    outfield_df.drop(col, axis=1, inplace=True)
# Replace NaNs with 0's
outfield_df.fillna(0, inplace=True)
processed_outfield_df = outfield_df.reset_index(drop=True)
processed_outfield_df.shape
processed_outfield_df.head(10)<jupyter_output><empty_output><jupyter_text>Now, We have processed the data for the analysis. Although there will be some more minor preparations to the data depending on the machine learning task.  Question 1 Which type of players costs more money for a team (Value) ? and which type of players earn the most (Wage)?<jupyter_code># Group the dataset by Position and create two such datasets for sum and count
position_sum = processed_outfield_df.groupby('position').sum()
position_count = processed_outfield_df.groupby('position').count()
# Formulae for value per position and wage per position
position_sum['number_of_players'] = position_count['short_name']
position_sum['value_per_position'] = position_sum['value_eur'] / position_sum['number_of_players']
position_sum['wage_per_position'] = position_sum['wage_eur'] / position_sum['number_of_players']
position_sum.head(15)
# Function to visualize in descensing order
def visualize_by_position (xyz):
    position_sum.sort_values(xyz +'_per_position', ascending=False, inplace=True)
    position_sum[xyz +'_per_position'].plot(kind='bar',color=np.random.rand(4))
    fig = matplotlib.pyplot.gcf()
    fig.set_size_inches(18.5, 10.5)
    plt.title("{} of a player by position.".format(xyz))
    print("{} of a player by position.".format(xyz))
# Assign v for value and w for wage
v = 'value'
w = 'wage'
# Value of a player by position
visualize_by_position(v)
#wage of a player by position 
visualize_by_position(w)<jupyter_output>wage of a player by position.<jupyter_text>Observations* Value of a player from a specific position is proportional to the wages they earn except center backs earn slightly better than how much is spent on them* Center Forwards are the most expensive players to acquire (value) and maintain(wage). They almost cost twice as the second most expensive position and earn almost twice as the next best earner* In general forwards earn better than midfielders and defenders* Defenders, especially full backs earn less than all other positions and also cost very less for the teams Question 2 What are the top 5 features that decides value and wage of a player ?<jupyter_code>players_feature_df = outfield_df.reset_index(drop=True)
# Removing some more columns that are not necessary for the analysis
unnecessary_columns = ['short_name', 'age', 'club','nationality','potential','overall']
for col in unnecessary_columns:
    players_feature_df.drop(col, axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Reason for removing Overall, age and club is because* Overall - Players with good overall have good wages and value so it will skew the analysis* Age - Players who are very old or very young don't have good value* Club - The players playing for big clubs get hefty wages regardless of ability. This will again skew the analysis<jupyter_code># One Hot encoding these 5 categorical columns
one_hot_columns = ['position', 'attack_workrate', 'defense_workrate','preferred_foot','body_type']
players_feature_df = pd.get_dummies(players_feature_df, columns=one_hot_columns, prefix = one_hot_columns)
players_feature_df.shape
# Setting two different target variables - Value and Wage
y1 = players_feature_df['wage_eur']
y2 = players_feature_df['value_eur']
X = players_feature_df.drop(['value_eur', 'wage_eur'], axis=1)
# Traininng using a Random Forest Regressor , Showing the accuracy and displaying the top 5 features
def model_train(y) :
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2, random_state=42)
    RFR = RandomForestRegressor(n_estimators=500)
    RFR.fit(X_train, y_train)
    y_test_preds = RFR.predict(X_test)
    print(r2_score(y_test, y_test_preds))
    coefficients_df = pd.DataFrame()
    coefficients_df['Features'] = X_train.columns
    coefficients_df['Coefficients'] = RFR.feature_importances_
    coefficients_df.sort_values('Coefficients', ascending=False).head(5)
    coefficients_df.set_index('Features', inplace=True)
    coefficients_df.sort_values('Coefficients', ascending=False).head(5).plot(kind='bar', color= 'blue' )
model_train(y1)<jupyter_output>0.7636865024895105<jupyter_text>Observation - Important features for wage   For a player to get good wages he must have * Good reactions * Good ball control * High International reputation  * Has good sliding tackle capability * Has good overall defending skills<jupyter_code>model_train(y2)<jupyter_output>0.8731659940410103<jupyter_text>Observation - Important features for value   For a player to be valuable he must have * Good reactions * Good ball control * Has good overall defending skills * Has good sliding tackle capability * Has good sprint speed  Question_3 How are skills distributed among different category of players (Forward, Midfield and Defense) ?<jupyter_code>players_attributes_df = processed_outfield_df.reset_index(drop=True)
# Preparing Data for the Seaborn Heatmap
# Drop the column 'skill_moves' since it does not reflect the skill attributes of a player
players_attributes_df.drop('skill_moves', axis=1, inplace=True)

#Group the players into more generalized category of positions
dict_pos = {
    'ST': 'Forward - Center',
    'CF': 'Forward - Center',
    'LW': 'Forward - Wide',
    'RW': 'Forward - Wide',
    'CM': 'Midfielder - Central',
    'CAM': 'Midfielder - Attacking ',
    'CDM': 'Midfielder - Defensive',
    'LM': 'Midfielder - Wide',
    'RM': 'Midfielder - Wide',
    'CB': 'Defense - Center',
    'LB': 'Defense - Full Back',
    'RB': 'Defense - Full Back',
    'LWB': 'Defense - Wing Back',
    'RWB': 'Defense - Wing Back',
}

players_attributes_df['position_category'] = players_attributes_df['position'].map(dict_pos)

#Function to plot all the attributes in a seaborn heatmap
def attribute_importance(x):
    if x == 'basic' :
        cols = ['pace', 'shooting', 'passing', 'dribbling', 'defending', 'physic']
    else:
            cols = [col for col in players_attributes_df if col.startswith(x)]
            
    for col in cols:
        players_attributes_df.loc[:,'n_' + col] = players_attributes_df[col]/ players_attributes_df['overall']
    req_col_att = [ 'n_' + col for col in cols]
    skills_analysis_df = players_attributes_df[req_col_att + ['position_category']]
    tmp = skills_analysis_df.groupby('position_category').mean()
    sns.set(rc={'figure.figsize':(14,10)})
    sns.heatmap(tmp,cmap = 'viridis_r' ,annot=True)
# Importance of all basic skills by category
attribute_importance('basic')<jupyter_output><empty_output><jupyter_text>Basic Skills* Pace is an important skill for all players except Center backs and defensive midfielders. Even the full backs and wing backs rely more on pace than the strikers. Wide players are the fastest players overall* shooting skills are important forwards, followed by wide and attacking midfielders. The center backs have a very poor shooting skill* Almost everyone's passing skills are more or less the same with center backs not as good as the others* Again for dribbling center backs are the poorest in this. forwards and wide players have good dribbling abilities* As expected for defending the center backs are the best followed by the full backs. the defensive midfielders are more adept in this than the wing backs* physical attributes are amost same for all categories but the center backs are better than others and wide mids and forwards along with the attacking mid are lacking in this area<jupyter_code># Importance of all basic skills by category
attribute_importance('skills')<jupyter_output><empty_output><jupyter_text>Mentality* Defensive players have good aggression and it redices as it goes to the top. The center forwards have slightly better aggression than the rest of the attack minded players* Again the defensive players have good interception skills. Forwards and wide midfielders are lacking in this area* The center forward is best when it comes to positioning followed by other attacking players. The center backs are very poor in this* It's the midfielders who are masters in this followed by the forwards and then the defenders* The center forwards are good in penalties mostly* The midfielders have the best composure but even the others are not far behind<jupyter_code># Importance of all basic skills by category
attribute_importance('skill')<jupyter_output><empty_output><jupyter_text>Skill* Wide Forwards and midfielders are good dribblers while center backs are poorest in this* Forwards have the skill-curve better than others * Usually the attacking midfielders have the best free kick accuracy* Midfielders have the best long passing capacity* Except the center backs almost everyone have similar and excellent ball control<jupyter_code># Importance of all basic skills by category
attribute_importance('power')<jupyter_output><empty_output><jupyter_text>Power* The forwards have the best shot power followed by mids and then the defenders* The center backs are best in jumping with others akmost having the same except wide forwards, wide mids and attacking mids who are a little behind others* Central midfielders and wide defenders are the players with best stamina. Attacking midfielders are behind the rest* Center backs are the strongest players and the attacking mids are the weakest ones* Forwards have best long shooting capabilities while defenders are bad in this<jupyter_code># Importance of all basic skills by category
attribute_importance('movement')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07883153261304522, 'fraction_numerical': 0.007469654528478058, 'mean_word_length': 5.853290676416819, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '232061', 'license': 'FTL', 'n_tokens_mistral': 4345, 'n_tokens_neox': 4158, 'n_words': 1700, 'path': 'Soccer_Stats.ipynb', 'repo_name': 'akshaykarthick/Data-Science-Blog-Post-Udacity', 'size': '15000'}