<jupyter_start><jupyter_text>SIT742: Modern Data Science **(2020 Assessment Task 02: Exploration FIFA2019 Data)**---- Materials in this module include resources collected from various open-source online repositories.- You are free to use, change and distribute this package.Prepared by **SIT742 Teaching Team**---Enter your information in the following cell. Please make sure you fill your Student ID and Name correctly.  **Student Group Information:** Please fill your information below- Names with Student IDs:  - Group ID:  - Emails:   ---  OverviewRecently, [Kaggle](https://www.kaggle.com) (a data science community and competition platform) released one data set '[FIFA19](https://www.kaggle.com/karangadiya/fifa19)‘, which consists of 18K+ FIFA 19 player with around 90 attributes extracted from FIFA database. In this assessment task, we make it available as the data set:- [2020T2Data.csv](https://github.com/tulip-lab/sit742/raw/master/Assessment/2020/data/2020T2Data.csv): The data set for this assignment task.In this task, you will need to **use Spark packages**- **Part 1**: Exploratory Data Analysis- **Part 2**: Clustering Analysis, and identify the position profiles of each cluster- **Part 3**: Classification Analysis, and evaluate the performance of different algorithms using cross validation;You will be required to complete the coding to answer the questions with a neat and clean format, and your commenting and adherence to code standards will be considered when marking. Further information on Python standards, please visit https://www.python.org/dev/peps/pep-0008/ **Note**: You are not necessary to stick to the partial code provided, and you can write your own complete code to implement the required tasks. But you should provide sufficient comments.  Part 1 - What we could know about FIFA 2019 Players?  1.0. Libraries and data files***Import the necessary Spark environment, and load the data set [2020T2Data.csv](https://github.com/tulip-lab/sit742/raw/master/Assessment/2020/data/2020T2Data.csv).<jupyter_code>!pip install wget
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
!tar xf spark-2.4.0-bin-hadoop2.7.tgz
!pip install -q findspark
import os,wget
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-2.4.0-bin-hadoop2.7"


link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Assessment/2020/data/2020T2Data.csv'
DataSet = wget.download(link_to_data)
import findspark
findspark.init()
from pyspark.sql import SparkSession<jupyter_output><empty_output><jupyter_text>1.1 Data Exploration*Remind: You must only use **PySpark** to complete the following data processing and model building. Otherwise, you lose all marks.******Code**:     import the csv file as a Spark dataframe and name it as df<jupyter_code># Import the '2020T2Data.csv' as a Spark dataframe and name it as df
spark = SparkSession.builder.appName('SIT742T2').getOrCreate()

# Your Code<jupyter_output><empty_output><jupyter_text>******Code**:     Check statistics (min, mean and max) for features on Age, Overall. Then find out the Avg Overall on Position, Avg Overall on Nationality (Sort by avg Overall on Nationality)    **Report**:     **1.1.A** Please answer questions with proper section title  '1.1.A':                  Which are the (min, mean and max) for Age          Which are the (min, mean and max) for Overall          Which position the talented player (based on Avg Overall) are playing?          Which are the top 3 countres that most likely have the genies player (based on sort of Avg Overall) and<jupyter_code>from pyspark.sql import functions as F

#Your code for statistics on Age


#Your Code for statistics on Overall


#Your Code for which position 



#Your Code to output top 3 countries most likely having good players<jupyter_output><empty_output><jupyter_text>******Code**:     You are required to find out the Avg Potentials on Country by Positions, and show the top 10 by ordering your results on Country with alphabet? (Hint: you may need to use pivot)    **Report**:     **1.1.B** Please answer questions with proper section title  '1.1.B':                  Show the results on Avg Potentials on Country by Position with ordering the results on country by alphabet         Which position the young talented player (by Avg Potential) are playing in Australia?<jupyter_code>#Your code to identify top 10 countryes by alphabet



#Your Code to identify which position has the most talend young players in Australia<jupyter_output><empty_output><jupyter_text>Interesting, is there a age that the player is fully relased his potential? If there is, could you find out?******Code**:     You are required to plot the finding visually by having Age as the x-axis and avg potential on age with avg overall on age at y-axis. (Hint: you may need to select and aggreate on data first and then turn it into pandas for plotting)    **Report**:     **1.1.C** Please include the following plot into your report with proper section title  '1.1.C':                  Plot the finding visually by having Age as the x-axis and avg potential on age with avg overall on age at y-axis.          What you can find for the age at which the player is fully released his potential?<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

# Your Code for plot<jupyter_output><empty_output><jupyter_text>Part 2 - Unsupervised Learning: Kmeans***In this part, you are required to build a Kmeans clustering for explore the player segmentation.  2.1 Data Preparation*Remind: You must only use **pyspark** to complete the following data processing and model building. Otherwise, you lose all marks.* ******Code**:     You will need to remove the Goal Keepers (Position = 'GK') and only use the skillset attributes (Height(CM),Weight(KG),Crossing,Finishing,HeadingAccuracy,ShortPassing,Volleys,Dribbling,Curve,FKAccuracy,LongPassing,BallControl,Acceleration,SprintSpeed,Agility,Reactions,Balance,ShotPower,Jumping,Stamina,Strength,LongShots,Aggression,Interceptions,Positioning,Vision,Penalties,Composure,Marking,StandingTackle,SlidingTackle)<jupyter_code># Your code to select relevent features and filtering by leaving out the GK<jupyter_output><empty_output><jupyter_text>To make the later stage easier, we define the position group by using the position feature.- DEF = [LB,LWB,RB,LCB,RCB,CB,RWB] ,- FWD = [RF,LF,LW,RS,RW,LS,CF,ST] ,- MID = [LCM,LM,RDM,CAM,RAM,RCM,CM,CDM,RM,LAM,LDM]******Code**:     You are required to create a new column called Position_Group with only DEF/FWD/MID in the dataframe you created in previous question<jupyter_code>from pyspark.sql.functions import when,col

# Your code to complete
DEF= 
FWD=  
MID=  

# Your code here to create a new variable df_kmeans_new with a new column Position_Group, ..
#<jupyter_output><empty_output><jupyter_text>Now, we remove the Position_Group and Position to create the feature for Kmeans<jupyter_code>from pyspark.ml.clustering import KMeans
from pyspark.ml.feature import VectorAssembler

FEATURES_COL = ['Height(CM)', 'Weight(KG)', 
                      'Crossing', 'Finishing', 'HeadingAccuracy', 
                      'ShortPassing', 'Volleys', 'Dribbling', 'Curve',
                      'FKAccuracy', 'LongPassing', 'BallControl', 
                      'Acceleration', 'SprintSpeed', 'Agility', 
                      'Reactions', 'Balance', 'ShotPower', 'Jumping', 
                      'Stamina', 'Strength', 'LongShots', 'Aggression', 
                      'Interceptions', 'Positioning', 'Vision', 'Penalties', 
                      'Composure', 'Marking', 'StandingTackle', 'SlidingTackle']

vecAssembler = VectorAssembler(inputCols=FEATURES_COL, outputCol="features")
df_kmeans_ = vecAssembler.transform(df_kmeans_new).select('ID','features')
df_kmeans_.show(3)<jupyter_output><empty_output><jupyter_text>Now in order to evaluate your Kmeans Model, please plot the elbow plot**Code**:     You are required to plot the elbow plot, with a varying K from 2 to 20.    **Report**:     **2.1** In your report with proper section title  '2.1', Please include the elbow plot.<jupyter_code>cost = np.zeros(20)

for k in range(2,20):
# Your code

fig, ax = plt.subplots(1,1, figsize =(8,6))
ax.set_xlabel('k')
ax.set_ylabel('cost')

# Your Code<jupyter_output><empty_output><jupyter_text>2.2 K-MeansCould you tell out the optimized K value? ******Code**:     You are required to choose a K value as 8 and then summarize each cluster with the count on Position_Group.    **Report**:     **2.2** In your report with proper section title  '2.2', Please summarize the following:                  What you could find from those clusters (by cluster central point)?          Is there any Position_Group belonging to particular Cluster?<jupyter_code>k = 8

# Your code
from pyspark.sql import SQLContext

# Your code<jupyter_output><empty_output><jupyter_text>Part 3 - Supervised Learning: Classification on Position_Group***In last part, you use the player's skillset values to segment the players into 8 clusters. Now we want to see whether we could accuratly predict the position_group of the player.*Remind: You must only use **PySpark** to complete the following data processing and model building. Otherwise, you lose all marks.* 3.1 Data PreparationWe remove the feature of position and use all other skillset features and the cluster prediction as the input for the model. Your target for classification is "Position_Group".<jupyter_code>FEATURES_COL_ = ['Height(CM)', 'Weight(KG)', 
                      'Crossing', 'Finishing', 'HeadingAccuracy', 
                      'ShortPassing', 'Volleys', 'Dribbling', 'Curve',
                      'FKAccuracy', 'LongPassing', 'BallControl', 
                      'Acceleration', 'SprintSpeed', 'Agility', 
                      'Reactions', 'Balance', 'ShotPower', 'Jumping', 
                      'Stamina', 'Strength', 'LongShots', 'Aggression', 
                      'Interceptions', 'Positioning', 'Vision', 'Penalties', 
                      'Composure', 'Marking', 'StandingTackle', 'SlidingTackle','Cluster']


vecAssembler_ = VectorAssembler(inputCols=FEATURES_COL_, outputCol="features")
df_class_ = vecAssembler_.transform(df_kmeans_pred_).select('features','Position_Group')
df_class_.show(3)<jupyter_output><empty_output><jupyter_text>In many data science modeling work, feature scaling is very important.In here, we use standard scaling on the fetaures.<jupyter_code>from pyspark.ml.feature import StandardScaler

standardscaler=StandardScaler().setInputCol("features").setOutputCol("Scaled_features")
raw_data=standardscaler.fit(df_class_).transform(df_class_)
raw_data.select("features","Scaled_features",'Position_Group').show(5)<jupyter_output><empty_output><jupyter_text>In Spark, you could not use string as Target data type, Please encode the Position_Group column by using following encoding: FWD = 0DEF = 1MID = 2*Hint: Data type after encoding should be numeric.*<jupyter_code>raw_data_ = raw_data.withColumn('Target',when(col("Position_Group") == "DEF", 1)
      .when(col("Position_Group")== "FWD", 0)
      .otherwise(2))<jupyter_output><empty_output><jupyter_text>3.2 Training Test EvaluationWe remove the feature of position and use all other skillset features and the cluster prediction as the input for the model. The target for classification is "Position_Group".Now, we split your data into train/Test, and evaluate one model's performance.<jupyter_code>train, test = raw_data_.randomSplit([0.7, 0.3], seed=12)
from pyspark.ml.classification import LogisticRegression
lr = LogisticRegression(labelCol="Target", featuresCol="Scaled_features",maxIter=10)
model=lr.fit(train)

predict_train=model.transform(train)
predict_test=model.transform(test)
predict_test.select("Target","prediction").show(10)<jupyter_output><empty_output><jupyter_text>******Code**:     You are required to evaluate the model by using confusion matrix. Please also print out your model's Precision, Recall and F1 score.    **Report**:     **3.2** In your report with proper section title  '3.2', Please summarize the following:                  The confusion matrix          The Precision, Recall and F1 score<jupyter_code>from sklearn.metrics import confusion_matrix


# Your Code

from sklearn.metrics import classification_report

# Your Code<jupyter_output><empty_output><jupyter_text>3.3 K-fold Cross-ValidationWe surely missed something during the modeling work -- Hyperparameter tuning! We can use K-fold cross validation to find out the best hyperparameter set.******Code**:     Please implement K-fold cross validation for three (any three) classification models.    **Report**:     **3.3** In your report with proper section title  '3.3', Please summarize the following:                  Your code design and running results          Your findings on hyperparameters based on this cross-validation results         Any difficuties you had          Any other possible tasks we could do on this dataset     Remind: fail to list the reference will lead a penalty and possibly plagiarism<jupyter_code>from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.ml.tuning import ParamGridBuilder, CrossValidator

# Your code 

# Your code<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09425168306576903, 'fraction_numerical': 0.014870163497817564, 'mean_word_length': 4.497356649044327, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 66, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '225620', 'license': 'MIT', 'n_tokens_mistral': 4032, 'n_tokens_neox': 3779, 'n_words': 1418, 'path': 'Assessment/2020/SIT742Task2.ipynb', 'repo_name': 'hltr/sit742', 'size': '13517'}