<jupyter_start><jupyter_text>Karan VombatkereCSC 440: Data Mining - Final ProjectBelow is an implementation of Python code on a dataset with detailed tennis match information from the ATP World Tour.This notebook contains scripts that import, clean and process the match data to extract the required features to be used in the ML models. The pandas library was used extensively to store, clean and process the data. First unique tennis player information was extracted into a Pandas dataframe. The following attributes were extracted from the match data - 'PlayerID', 'PName', 'Age', 'Height', 'MaxRank', 'Hand', 'Country'Then a function computePlayerStats() was written to calculate the following cumulative (average) statistics of a players: 'PlayerID', 'PName', 'Height(cm)', 'Matches Played', 'Overall Win%', 'Top 100 Win %', 'Top 30 Win %', 'First Serve %', 'First Serve Win %', 'BPSave %', 'BPConv %', 'Aces/Match', 'DF/Match', 'Future Top 30'The required features were extracted for valid players (having played ~50 matches under the age of 26) and these tuples were returned as a new dataframe. Finally, this dataframe was exported as a .csv file, to be used as training and test data for the Machine Learning models (Logistic Regression and Neural Network).<jupyter_code>#<NAME>
#CSC 440: Data Mining Final Project
#November - Decemeber 2017

#Imports needed
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import time

import itertools as itools
from statistics import *

#Load ATPTennis.csv Dataset into a dataframe

ATPData = pd.read_csv("atp_matches_combined.csv", error_bad_lines=False)
ATPData.name = "ATP Match Data"
#View the data
ATPData.head(10)
#Create dataframe for Player Data and label columns
PlayerColumns = ['PlayerID', 'PName', 'Age', 'Height', 'MaxRank', 'Hand', 'Country']

PlayerData = pd.DataFrame(columns = PlayerColumns)
PlayerData.set_index('PlayerID')


#Add data to the PlayerData dataframe
for index, row in ATPData.head(50180).iterrows():
    #If player was the winner
    p_id = int(row.winner_id)
    
    #Check if value exists or not
    if(p_id not in PlayerData.index):
        PlayerData.loc[p_id] = [p_id, row.winner_name, row.winner_age, row.winner_ht, row.winner_rank, row.winner_hand, row.winner_ioc]

    #Update rank with best rank value
    if(row.winner_rank < PlayerData.loc[p_id].MaxRank and not np.isnan(PlayerData.loc[p_id].MaxRank)):
        PlayerData.loc[p_id] = [p_id, row.winner_name, row.winner_age, row.winner_ht, row.winner_rank, row.winner_hand, row.winner_ioc]
    
    else:
        PlayerData.loc[p_id] = [p_id, row.winner_name, row.winner_age, row.winner_ht, PlayerData.loc[p_id].MaxRank, row.winner_hand, row.winner_ioc]
   

    #If the player lost
    p_id2 = int(row.loser_id)
    
    #Check if value exists or not
    if(p_id2 not in PlayerData.index):
        PlayerData.loc[p_id2] = [p_id2, row.loser_name, row.loser_age, row.loser_ht, row.loser_rank, row.loser_hand, row.loser_ioc]

    if(row.loser_rank < PlayerData.loc[p_id2].MaxRank and not np.isnan(PlayerData.loc[p_id2].MaxRank)):
        PlayerData.loc[p_id2] = [p_id2, row.loser_name, row.loser_age, row.loser_ht, row.loser_rank, row.loser_hand, row.loser_ioc]

    else:
        PlayerData.loc[p_id2] = [p_id2, row.loser_name, row.loser_age, row.loser_ht, PlayerData.loc[p_id2].MaxRank, row.loser_hand, row.loser_ioc]

#View the Player Data
PlayerData.head(100)
PlayerData.loc[103819]
ATPData.columns
#Create Data Frame for calculating Player Statistics
#Specify Column Names
PlayerStatsColumns = ['PlayerID', 'PName', 'Height(cm)', 'Matches Played', 'Overall Win%', 'Top 100 Win %', 'Top 30 Win %', 'First Serve %', 'First Serve Win %', 'BPSave %', 'BPConv %', 'Aces/Match', 'DF/Match', 'Future Top 30']

PlayerStats = pd.DataFrame(index = PlayerData.index.copy(), columns = PlayerStatsColumns)
#Function to compute statistics and Populate the PlayerStats Table
#Note that we reject all tuples that have invalid/incomplete statistics data

#Inputs: 
#playerCount = number of players to calculate data for
#matchRows = number of rows of match data to consider
#initMatches = initial number of matches played to calculate features

#Output: Updates the Player Stats table dynamicall using loc pandas function

def computePlayerStats(playerCount, matchRows, initMatches):
    playerAge = 27 #Age of player to be considered for initial matches
    
    start = time.time() #compute the start time
    validPlayers = 0 #valid player/data count
    
    #Iterate over each player
    for p_indx, player in PlayerData.head(playerCount).iterrows(): #Search stats for each player in PlayerData Table

        #Initialize cumulative stat data for each player for a particular year
        FirstServe = 0
        FirstServeWin = 0
        FirstServeCounter = 0

        BPSaved = 0
        BPFaced = 0

        BPChances = 0
        BPWon = 0

        Aces = 0
        DF = 0
        numMatches = 0
        Wins = 0

        top30 = False #Boolean to keep track if the player ever breaks into the top 30
        #This is computationally expensive since we must check every match
        
        #Keep track of win record against top 30 opponents
        top30matches = 0
        top30wins = 0
        
        #Keep track of win record against top 100 opponents
        top100matches = 0
        top100wins = 0

        #Iterate over the match data for each player
        for index, row in ATPData.head(matchRows).iterrows():

            #If player won the match and is under age playerAge
            if(p_indx == row.winner_id and row.winner_age < playerAge):
                #Only compute statistics for first 50 matches of the player
                if(numMatches < initMatches):
                    
                    #if(p_indx == 103819): #Federer matches check!
                        #print(player.PName, row.score, row.loser_name)
                        #print(row.WAces, row.WDF, FirstServe, BPSaved, BPFaced)
                        #print(index)
                        
                    #Check whether or not the match was a walkover and has valid stats
                    if(not np.isnan(row.w_ace) and not np.isnan(row.w_df) and not np.isnan(row.w_svpt) and not np.isnan(row.w_bpFaced) and not np.isnan(row.w_bpSaved) and row.w_svpt != 0):
                        Aces += row.w_ace
                        DF += row.w_df
                        FirstServe += (row.w_1stIn / row.w_svpt) #calculate first serve percentage
                        FirstServeWin += (row.w_1stWon/ row.w_1stIn) #calculate first serve win rate
                        FirstServeCounter += 1

                        BPSaved += row.w_bpSaved
                        BPFaced += row.w_bpFaced

                        BPChances += row.l_bpFaced
                        BPWon += (row.l_bpFaced - row.l_bpSaved)
                        numMatches += 1
                        Wins += 1
                        
                        #Increment top30 wins if player beat a top 30 opponent
                        if(row.loser_rank <= 30):
                            top30wins += 1
                            top30matches += 1
                            
                        #Increment top100 wins if player beat a top 100 opponent
                        if(row.loser_rank <= 100):
                            top100wins += 1
                            top100matches += 1
                            
                #Break out of loop once first 50 matches have been calculated           
                else:
                    break

            #If player lost the match and is under age 26
            if(p_indx == row.loser_id and row.loser_age < playerAge):
                #Only compute statistics for first 50 matches of the player
                if(numMatches < initMatches):
                    
                    #if(p_indx == 103819):
                        #print(row.winner_name, row.score, player.PName)
                        #print(Aces, DF, FirstServe, BPSaved, BPFaced)

                        
                    #Check whether or not the match was a walkover and has valid stats
                    if(not np.isnan(row.l_ace) and not np.isnan(row.l_df) and not np.isnan(row.l_svpt) and not np.isnan(row.l_bpFaced) and not np.isnan(row.l_bpSaved) and row.l_svpt != 0):
                        Aces += row.l_ace
                        DF += row.l_df
                        FirstServe += (row.l_1stIn / row.l_svpt) #calculate first serve percentage
                        FirstServeWin += (row.l_1stWon/ row.l_1stIn) #calculate first serve win rate
                        FirstServeCounter += 1

                        BPSaved += row.l_bpSaved
                        BPFaced += row.l_bpFaced

                        BPChances += row.w_bpFaced
                        BPWon += (row.w_bpFaced - row.w_bpSaved)
                        numMatches += 1
                        
                        #Increment top30 matches if player lost to a top 30 opponent
                        if(row.winner_rank <= 30):
                            top30matches += 1
                        
                        #Increment top100 matches if player lost to a top 100 opponent
                        if(row.winner_rank <= 100):
                            top100matches += 1
                    
                #Break out of loop once first 50 matches have been calculated           
                else:
                    break

        
        #Update the cumulative statistics for the Player
        winPercent = 0
        servePercentage = 0
        firstservewinRate = 0
        BPsaveRate = 0
        BPconvRate = 0
        
        top30winPercent = 0
        top100winPercent = 0
        
        #Mark true if the player ever broke into the top 30 in the dataset
        if(player.MaxRank <= 30):
            top30 = True

        #Calculate first serve percentage, win percentage and BP percentages
        if(numMatches != 0 and FirstServeCounter != 0):
            winPercent = Wins/numMatches
            #compute aces and double faults per match
            Aces = Aces/numMatches
            DF = DF/numMatches
            
            servePercentage = FirstServe/FirstServeCounter #average first serve percentage
            firstservewinRate = FirstServeWin/FirstServeCounter #average first serve win rate

            if(BPFaced != 0 and BPChances != 0):
                BPsaveRate = BPSaved/BPFaced
                BPconvRate = BPWon/BPChances
            
            if(top30matches != 0):
                top30winPercent = top30wins/top30matches
                top100winPercent = top100wins/top100matches
                
            
            #Check if this is a valid entry that the player has played initMatches matches below age 26
            if(numMatches == initMatches):
                validPlayers += 1
                
                #Add the cumulative statistics to the database
                print("Updating Cumulative Player Statistics for Player:", player.PName)
                PlayerStats.loc[p_indx] = [p_indx, player.PName, player.Height, numMatches, winPercent, top100winPercent, top30winPercent, servePercentage, firstservewinRate, BPsaveRate, BPconvRate, Aces, DF, top30]
    
    
    #compute runtime
    end = time.time()
    runTime = end - start
    
    print("Player Statistics Generation Runtime =", runTime, 'seconds')
    print("Number of valid players with", initMatches, "matches of initial match data =", validPlayers)
    
    #Drop all rows that have invalid statistics (insufficient matches, stats etc) and return final feature set
    #Copy the valid data with final extracted features to dataframe to be returned
    extractedFeatures = PlayerStats.dropna() 
    
    #Return the cleaned and processed player information
    return extractedFeatures
#Run the statistic generator function
#Specify inputs (total players = 1981, totalRows = 50180) -> Takes about 2.5 hours to run

numPlayers = 1981
numRows = 50180
initMatches = 40

#Call the statistic generator with inputs specified
ATPplayerFeatures = computePlayerStats(numPlayers, numRows, initMatches)
#Add Heights(cm) for the missing players
PlayerStats.loc[105311, 'Height(cm)'] = 185
PlayerStats.loc[105138, 'Height(cm)'] = 183
PlayerStats.loc[105807, 'Height(cm)'] = 188
PlayerStats.loc[106233, 'Height(cm)'] = 185
PlayerStats.loc[105526, 'Height(cm)'] = 196
PlayerStats.loc[106210, 'Height(cm)'] = 198
PlayerStats.loc[106401, 'Height(cm)'] = 193
PlayerStats.loc[106432, 'Height(cm)'] = 185
#Use query function to return smaller subsets
q1 = PlayerStats[(PlayerStats['Overall Win%'] >= 0) & (PlayerStats['Height(cm)'] != 200)]

print(len(q1.index))
print(len(PlayerStats.index))
q1
ATPplayerFeatures = PlayerStats.dropna()
print(len(ATPplayerFeatures.index))
ATPplayerFeatures.head(20)
#Export the extracted feature set to a csv file
ATPplayerFeatures.to_csv("ATPPlayerFeatures40.csv", index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07904200780472875, 'fraction_numerical': 0.025480143851863188, 'mean_word_length': 2.2287549407114624, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '315652', 'license': 'MIT', 'n_tokens_mistral': 4054, 'n_tokens_neox': 3639, 'n_words': 1206, 'path': 'Feature Extraction.ipynb', 'repo_name': 'kvombatkere/Tennis-Performance-Prediction', 'size': '13079'}