<jupyter_start><jupyter_text>**Tracking Data** Three matches from Swedish first division from 2019 season1. 2019/07/22: Hammarby Vs IFElfsborg2. 2019/10/20: Hammarby Vs MalmöFF3. 2019/09/30: Hammarby Vs ÖrebröThings to think about:* Set pieces will cause issues;* There's an entity resolution problem - I wonder whether you could computationall adjust  for this;* Sanity check: is a player faster than Usain Bolt?* Could maybe do some kind of replacement metric?**The assignment**The data analyst at you club has just received tracking data for three matches of her club (Hammarby, playing in Sweden). She wants to know what she can reliably infer from this data. She first asks you to calculate speed and direction of all players in the data using the techniques outlined in Laurie's videos. So, for any given frame you can plot the direction and speed of the players.  Give examples of 4 such frames from the matches. (3 points)Write a program which plots distance from goal, speed and acceleration of all Hammarby players as a short time series, i.e plot of these variables over time. The analyst is aware there are limitations and errors in the data. Provide her with a visualisation of the three variables for a few different examples of a 15 second sequence of play. At least one of these should show where the tracking data is relatively accurate and one should show a case whee there is an error. Write a short report using these figures on the possibilities and limitations of the data. (2 points)Now the analyst now asks you to analyse Hammarby's goals. She would like you to look at distance to nearest opposition player for each of the Hammarby players during the possession (you will have to find the start of possession time frame by hand) leading up to an open play goal.  Measure the distance to the nearest defending player and nearest teammate during  this buildup.  Use this to analyse one Hammarby player in particular. (3 points)Implement a further metric of you own choice that you think will help understand how Hammarby score goals. Use this to analyse your chosen Hammarby player.  (2 points)---Output:1. a non-technical description of how your method for measuring distance, speed, acceleration and relative positions of other players works.the examples of frames with speed and direction and time series of these variables for short sequences.an explanation of the strengths and weaknesses of the data as asked for by the data analyst.an analysis of the attacking runs of one of the Hammarby players in particular.2. A runnable, commented code  as a (preferably) Python or R script that generates all the plots from the report and explains the method you have used. Important: this code should be a single file run immediately if placed with in the same directory as the Signality folder.  **Data Preparation Approach*** Let's take advantage of the excellent work done already by <NAME>* Laurie has written functions to calculate various metrics on the Metrica dataset, which comes in a different format to the Signality data* Seems like a good idea to load the Signality data into a similar format to the Metrica dataset<jupyter_code>import pprint as pp
import pandas as pd
import numpy as np
import datetime

# imports required by data prep functions
import json
import math
import pickle
from copy import deepcopy

import re
import os
from collections import Counter
import pprint as pp

# plotting
import matplotlib.pyplot as plt

pd.options.display.max_rows = 100
pd.options.display.max_columns = 20

signalityRepo = r'2019/Tracking Data/'<jupyter_output><empty_output><jupyter_text>**Helper / Data Preparation Functions**> Main timeseries data (@25Hz) is in **tracks** and metadata is in **info_live**<jupyter_code>def initialise_dic_tracks(df_homePlayers, df_awayPlayers):
    """
    Initialises dictionaries for both home and away player locations
    """

    dic_home_tracks = {}
    dic_away_tracks = {}

    for homePlayer in df_homePlayers.playerIndex:
        for xy in ['x','y']:
            dic_home_tracks[f'Home_{homePlayer}_{xy}'] = []

    for awayPlayer in df_awayPlayers.playerIndex:
        for xy in ['x','y']:
            dic_away_tracks[f'Away_{awayPlayer}_{xy}'] = []
            
    return dic_home_tracks, dic_away_tracks
def populate_df_tracks(homeAway, homeAway_tracks, playersJerseyMapping, dic_tracks, df_players):
    """
    For a given team (home OR away), will transform the JSON track data to produce a dataframe just like Laurie's
    """
    
    lst_playerJerseys = df_players.jersey_number.values
    
    # iterating through frames for home/away team
    for n, frame in enumerate(homeAway_tracks):

        lst_playerJerseysPerFrame = []

        for player in frame:
            jersey_number = player.get('jersey_number')
            playerIndex = playersJerseyMapping[jersey_number]
            x,y = player.get('position', [np.nan, np.nan])

            # keeping track of jerseys that have a position for that frame
            lst_playerJerseysPerFrame.append(jersey_number)

            dic_tracks[f'{homeAway}_{playerIndex}_x'].append(x)
            dic_tracks[f'{homeAway}_{playerIndex}_y'].append(y)

        # list of jerseys that aren't in the frame
        lst_playerJerseysNotInFrame = list(set(lst_playerJerseys) - set(lst_playerJerseysPerFrame))

        # adding the jerseys that aren't in frame and providing an x,y position of nan, nan
        for jersey_number in lst_playerJerseysNotInFrame:
            playerIndex = playersJerseyMapping[jersey_number]
            x,y = [np.nan, np.nan]

            dic_tracks[f'{homeAway}_{playerIndex}_x'].append(x)
            dic_tracks[f'{homeAway}_{playerIndex}_y'].append(y)
    
    # transforming tracking dic to a tracking dataframe
    df_tracks = pd.DataFrame(dic_tracks)
    
    return df_tracks
rootFileNames = ['20190930.Hammarby-Örebrö'
                ,'20191020.Hammarby-MalmöFF'
                ,'20190722.Hammarby-IFElfsborg']

halfModifiers = ['.1','.2']
%%time

# loops through the combinations of matches and halves
for rootFileName in rootFileNames[:1]:
    for halfModifier in halfModifiers[:1]:
        print (f'Processing {rootFileName}-{halfModifier}...')
        
        # producing filename prefix (just need to add either "-info_live.json" or "-tracks.json")
        fileNamePrefix = rootFileName + halfModifier
        
        # load info
        ## looks like the info JSON is duplicated between the two halves
        with open(os.path.join(signalityRepo, f'{fileNamePrefix}-info_live.json')) as f:
            info = json.load(f)
            
        # load tracks
        #with open(os.path.join(signalityRepo, f'{fileNamePrefix}-tracks.json')) as f:
            #tracks = json.load(f)
            
        # unpacking info
        ## looks like .1 and .2 files are duplicated, so just looking at the .1 (first half file)
        if halfModifier == '.1':
            matchId = info.get('id')
            venueId = info.get('venueId')
            timeStart = info.get('time_start')
            pitchLength, pitchWidth = info.get('calibration').get('pitch_size')
            homeTeam = info.get('team_home_name')
            awayTeam = info.get('team_away_name')

            # unpacking players
            homePlayers = info.get('team_home_players')
            awayPlayers = info.get('team_away_players')
            homeLineup = info.get('team_home_lineup')
            awayLineup = info.get('team_away_lineup')
            homeLineupSwitch = {homeLineup[i]:i for i in homeLineup}
            awayLineupSwitch = {awayLineup[i]:i for i in awayLineup}

            # putting player metadata in dataframe
            df_homePlayers = pd.DataFrame(homePlayers)
            df_awayPlayers = pd.DataFrame(awayPlayers)
            df_homePlayers['teamName'] = homeTeam
            df_awayPlayers['teamName'] = awayTeam

            # adding matchId to the player dataframes
            df_homePlayers['matchId'] = matchId
            df_awayPlayers['matchId'] = matchId
            df_homePlayers['matchName'] = rootFileName
            df_awayPlayers['matchName'] = rootFileName

            # adding 1-11 + sub player indices (will probably use these for the final column names like Laurie in the tracks df)
            df_homePlayers['playerIndex'] = [int(homeLineupSwitch[i]) if i in homeLineupSwitch else np.nan for i in df_homePlayers.jersey_number.values]
            df_awayPlayers['playerIndex'] = [int(awayLineupSwitch[i]) if i in awayLineupSwitch else np.nan for i in df_awayPlayers.jersey_number.values]
            df_homePlayers.loc[pd.isna(df_homePlayers['playerIndex']) == True, 'playerIndex'] = np.arange(int(np.nanmax(df_homePlayers.playerIndex))+1, len(df_homePlayers)+1)
            df_awayPlayers.loc[pd.isna(df_awayPlayers['playerIndex']) == True, 'playerIndex'] = np.arange(int(np.nanmax(df_awayPlayers.playerIndex))+1, len(df_awayPlayers)+1)
            df_homePlayers['playerIndex'] = df_homePlayers.playerIndex.apply(lambda x: int(x))
            df_awayPlayers['playerIndex'] = df_awayPlayers.playerIndex.apply(lambda x: int(x))

            # re-jigging cols and re-ordering rows
            df_homePlayers = df_homePlayers[['matchId','matchName','teamName','playerIndex','jersey_number','name']].sort_values('playerIndex')
            df_awayPlayers = df_awayPlayers[['matchId','matchName','teamName','playerIndex','jersey_number','name']].sort_values('playerIndex')
            
            homePlayersJerseyMapping = {i:j for i, j in zip(df_homePlayers.jersey_number, df_homePlayers.playerIndex)}
            awayPlayersJerseyMapping = {i:j for i, j in zip(df_awayPlayers.jersey_number, df_awayPlayers.playerIndex)}
        
        ## parsing the track data
        phase = int(halfModifier[-1])
        
        # extracting home and away tracks
        home_tracks = [i.get('home_team') for i in tracks]
        away_tracks = [i.get('away_team') for i in tracks]

        # ball tracks
        ball_tracks = [i.get('ball') for i in tracks]
        ball_tracks_position = [(i.get('position')[0],i.get('position')[1],i.get('position')[2]) if i.get('position') != None else (np.nan, np.nan, np.nan) for i in ball_tracks]
        ball_x = [i[0] for i in ball_tracks_position]
        ball_y = [i[1] for i in ball_tracks_position]
        ball_z = [i[2] for i in ball_tracks_position]
        
        # match timestamps
        match_time = [i.get('match_time') for i in tracks]
        period = [i.get('phase') for i in tracks]
        timeStamp = pd.to_datetime([datetime.datetime.utcfromtimestamp(i.get('utc_time')/1000) for i in tracks])
        
        # unpacking tracks
        ## 1) initialising dictionaries for home and away teams
        dic_home_tracks, dic_away_tracks = initialise_dic_tracks(df_homePlayers, df_awayPlayers)
        ## 2) producing home tracking dataframe
        df_home_tracks = populate_dic_tracks('Home', home_tracks, homePlayersJerseyMapping, dic_home_tracks, df_homePlayers)
        ## 3) producing away tracking dataframe
        df_away_tracks = populate_dic_tracks('Away', away_tracks, awayPlayersJerseyMapping, dic_away_tracks, df_awayPlayers)
        
        # putting things together
        ## 1) home
        df_home_tracks['index'] = df_home_tracks.index
        df_home_tracks['matchId'] = matchId
        df_home_tracks['matchName'] = rootFileName
        df_home_tracks['Period'] = period
        df_home_tracks['Time [s]'] = np.array(match_time) / 1000
        df_home_tracks['TimeStamp'] = timeStamp
        df_home_tracks['ball_x'] = ball_x
        df_home_tracks['ball_y'] = ball_y
        df_home_tracks['ball_z'] = ball_z
        
        ## 2) away
        df_away_tracks['index'] = df_away_tracks.index
        df_away_tracks['matchId'] = matchId
        df_away_tracks['matchName'] = rootFileName
        df_away_tracks['Period'] = period
        df_away_tracks['Time [s]'] = np.array(match_time) / 1000
        df_away_tracks['TimeStamp'] = timeStamp
        df_away_tracks['ball_x'] = ball_x
        df_away_tracks['ball_y'] = ball_y
        df_away_tracks['ball_z'] = ball_z
        
            
        
        
def parse_raw_to_df(signalityRepo, rootFileName):
    """
    Takes raw root of a match string e.g. 20190930.Hammarby-Örebrö and transforms it into 4 dataframes
    """
    
    lst_df_home = []
    lst_df_away = []
    
    for half in ['.1','.2']:
        
        # producing filename prefix (just need to add either "-info_live.json" or "-tracks.json")
        fileNamePrefix = rootFileName + half

        # load info
        ## looks like the info JSON is duplicated between the two halves
        with open(os.path.join(signalityRepo, f'{fileNamePrefix}-info_live.json')) as f:
            info = json.load(f)

        # load tracks
        with open(os.path.join(signalityRepo, f'{fileNamePrefix}-tracks.json')) as f:
            tracks = json.load(f)

        # unpacking info
        ## looks like .1 and .2 files are duplicated, so just looking at the .1 (first half file)
        if half == '.1':
            matchId = info.get('id')
            venueId = info.get('venueId')
            timeStart = info.get('time_start')
            pitchLength, pitchWidth = info.get('calibration').get('pitch_size')
            homeTeam = info.get('team_home_name')
            awayTeam = info.get('team_away_name')

            # unpacking players
            homePlayers = info.get('team_home_players')
            awayPlayers = info.get('team_away_players')
            homeLineup = info.get('team_home_lineup')
            awayLineup = info.get('team_away_lineup')
            homeLineupSwitch = {homeLineup[i]:i for i in homeLineup}
            awayLineupSwitch = {awayLineup[i]:i for i in awayLineup}

            # putting player metadata in dataframe
            df_homePlayers = pd.DataFrame(homePlayers)
            df_awayPlayers = pd.DataFrame(awayPlayers)
            df_homePlayers['teamName'] = homeTeam
            df_awayPlayers['teamName'] = awayTeam

            # adding matchId to the player dataframes
            df_homePlayers['matchId'] = matchId
            df_awayPlayers['matchId'] = matchId
            df_homePlayers['matchName'] = rootFileName
            df_awayPlayers['matchName'] = rootFileName

            # adding 1-11 + sub player indices (will probably use these for the final column names like Laurie in the tracks df)
            df_homePlayers['playerIndex'] = [int(homeLineupSwitch[i]) if i in homeLineupSwitch else np.nan for i in df_homePlayers.jersey_number.values]
            df_awayPlayers['playerIndex'] = [int(awayLineupSwitch[i]) if i in awayLineupSwitch else np.nan for i in df_awayPlayers.jersey_number.values]
            df_homePlayers.loc[pd.isna(df_homePlayers['playerIndex']) == True, 'playerIndex'] = np.arange(int(np.nanmax(df_homePlayers.playerIndex))+1, len(df_homePlayers)+1)
            df_awayPlayers.loc[pd.isna(df_awayPlayers['playerIndex']) == True, 'playerIndex'] = np.arange(int(np.nanmax(df_awayPlayers.playerIndex))+1, len(df_awayPlayers)+1)
            df_homePlayers['playerIndex'] = df_homePlayers.playerIndex.apply(lambda x: int(x))
            df_awayPlayers['playerIndex'] = df_awayPlayers.playerIndex.apply(lambda x: int(x))

            # re-jigging cols and re-ordering rows
            df_homePlayers = df_homePlayers[['matchId','matchName','teamName','playerIndex','jersey_number','name']].sort_values('playerIndex')
            df_awayPlayers = df_awayPlayers[['matchId','matchName','teamName','playerIndex','jersey_number','name']].sort_values('playerIndex')

            homePlayersJerseyMapping = {i:j for i, j in zip(df_homePlayers.jersey_number, df_homePlayers.playerIndex)}
            awayPlayersJerseyMapping = {i:j for i, j in zip(df_awayPlayers.jersey_number, df_awayPlayers.playerIndex)}

        ## parsing the track data
        phase = int(half[-1])

        # extracting home and away tracks
        home_tracks = [i.get('home_team') for i in tracks]
        away_tracks = [i.get('away_team') for i in tracks]

        # ball tracks
        ball_tracks = [i.get('ball') for i in tracks]
        ball_tracks_position = [(i.get('position')[0],i.get('position')[1],i.get('position')[2]) if i.get('position') != None else (np.nan, np.nan, np.nan) for i in ball_tracks]
        ball_x = [i[0] for i in ball_tracks_position]
        ball_y = [i[1] for i in ball_tracks_position]
        ball_z = [i[2] for i in ball_tracks_position]

        # match timestamps
        match_time = [i.get('match_time') for i in tracks]
        period = [i.get('phase') for i in tracks]
        timeStamp = pd.to_datetime([datetime.datetime.utcfromtimestamp(i.get('utc_time')/1000) for i in tracks])

        # unpacking tracks
        ## 1) initialising dictionaries for home and away teams
        dic_home_tracks, dic_away_tracks = initialise_dic_tracks(df_homePlayers, df_awayPlayers)
        ## 2) producing home tracking dataframe
        df_home_tracks = populate_dic_tracks('Home', home_tracks, homePlayersJerseyMapping, dic_home_tracks, df_homePlayers)
        ## 3) producing away tracking dataframe
        df_away_tracks = populate_dic_tracks('Away', away_tracks, awayPlayersJerseyMapping, dic_away_tracks, df_awayPlayers)

        # putting things together
        ## 1) home
        df_home_tracks['index'] = df_home_tracks.index
        df_home_tracks['matchId'] = matchId
        df_home_tracks['matchName'] = rootFileName
        df_home_tracks['Period'] = period
        df_home_tracks['Time [s]'] = np.array(match_time) / 1000
        df_home_tracks['TimeStamp'] = timeStamp
        df_home_tracks['ball_x'] = ball_x
        df_home_tracks['ball_y'] = ball_y
        df_home_tracks['ball_z'] = ball_z

        ## 2) away
        df_away_tracks['index'] = df_away_tracks.index
        df_away_tracks['matchId'] = matchId
        df_away_tracks['matchName'] = rootFileName
        df_away_tracks['Period'] = period
        df_away_tracks['Time [s]'] = np.array(match_time) / 1000
        df_away_tracks['TimeStamp'] = timeStamp
        df_away_tracks['ball_x'] = ball_x
        df_away_tracks['ball_y'] = ball_y
        df_away_tracks['ball_z'] = ball_z
        
        lst_df_home.append(df_home_tracks)
        lst_df_away.append(df_away_tracks)
        
    df_homeTracks = pd.concat(lst_df_home, ignore_index=True)
    df_awayTracks = pd.concat(lst_df_away, ignore_index=True)
    
    return df_homePlayers, df_awayPlayers, df_homeTracks, df_awayTracks

%%time

for rootFileName in rootFileNames[:1]:
    df_homePlayers, df_awayPlayers, df_homeTracks, df_awayTracks = parse_raw_to_df(rootFileName)
df_homePlayers
df_awayPlayers
df_homeTracks
df_awayTracks<jupyter_output><empty_output><jupyter_text>Parsing the tracks<jupyter_code>y_home = [len(i.get('home_team')) for i in tracks]
x_home = np.arange(len(y_home))

plt.figure(figsize=(16,8))
plt.plot(x_home,y_home)
plt.show()
y_away = [len(i.get('away_team')) for i in tracks]
x_away = np.arange(len(y_away))

plt.figure(figsize=(16,8))
plt.plot(x_away,y_away)
plt.show()
def LoadDataHammarbyNewStructure2020(data_file_name,raw_dir):
    """
    This function spits out 7 lists:
    
    1) 
    2)
    3)
    4)
    5)
    6)
    7)
    
    """
    print()
    print('Loading data, this might take some seconds...')
    
    info_dir = raw_dir+data_file_name+'-info_live.json'
    tracks_dir = raw_dir+data_file_name+'-tracks.json'
    events_dir = raw_dir+data_file_name+'-events.json'

    with open(info_dir) as json_data:
        info = json.load(json_data)

    with open(tracks_dir) as json_data:
        data = json.load(json_data)

    with open(events_dir) as json_data:
        events = json.load(json_data)
        
    # Add to info, so that is the same as the non-live
    info['home_team'] = {}
    info['home_team']['name'] = info['team_home_name']
    info['away_team'] = {}
    info['away_team']['name'] = info['team_away_name']

    #Get the track ids of all the players to stablish the order in the rest of arrays
    order_list = []
    players_jersey = []
    players_names = []
    players_team_id = []
    players_team_id_list = []

    
    track_id = 0
    for player in info['team_home_players']:
        order_list.append(track_id)
        players_jersey.append(player['jersey_number'])
        players_names.append(player['name'])
        players_team_id.append(2)
        num_players_home = len(order_list)
        track_id += 1
    for player in info['team_away_players']:
        order_list.append(track_id)
        players_jersey.append(player['jersey_number'])
        players_names.append(player['name'])
        players_team_id.append(7)
        num_players_away = len(order_list)-num_players_home
        track_id += 1
        
    num_players_total = num_players_home+num_players_away
    
    
    #Create the lists of arrays
    players_position = []
    ball_position = []
    
    pitch_length = info['calibration']['pitch_size'][0]
    pitch_width = info['calibration']['pitch_size'][1]

    for frame in range(len(data)):
        #Get position of ball for each frame
        if data[frame]['ball'].get('position',np.asarray([np.inf,np.inf,np.inf])) is None:
            bp = np.asarray([np.inf,np.inf,np.inf])
        else:
            bp = np.asarray(data[frame]['ball'].get('position',np.asarray([np.inf,np.inf,np.inf])))
        bp = np.delete(bp,-1)
        bp[0] = (bp[0]+pitch_length/2)/pitch_length
        bp[1] = (bp[1]+pitch_width/2)/pitch_width
        ball_position.append(bp)

        #Append arrays of positions
        players_position.append(np.full((num_players_total,2),-1.0))
            
        # Get players
        # Home players
        for player in range(len(data[frame]['home_team'])):
            try:
                jersey_player = int(data[frame]['home_team'][player].get('jersey_number',-1))
                position_player = np.asarray(data[frame]['home_team'][player].get('position',np.asarray([-1,-1])))
            except:
                jersey_player = -1
                position_player = np.asarray([-1,-1])
            
            try:
                idx = players_jersey[:num_players_home].index(jersey_player)
                players_position[frame][idx][0] = (position_player[0]+pitch_length/2)/pitch_length
                players_position[frame][idx][1] = (position_player[1]+pitch_width/2)/pitch_width
            except:
                pass

                
        for player in range(len(data[frame]['away_team'])):
            try:
                jersey_player = int(data[frame]['away_team'][player].get('jersey_number',-1))
                position_player = np.asarray(data[frame]['away_team'][player].get('position',np.asarray([-1,-1])))
            except:
                jersey_player = -1
                position_player = np.asarray([-1,-1])
            
            try:
                idx = players_jersey[num_players_home:].index(jersey_player)+num_players_home
                players_position[frame][idx][0] = (position_player[0]+pitch_length/2)/pitch_length
                players_position[frame][idx][1] = (position_player[1]+pitch_width/2)/pitch_width
            except:
                pass
        
        players_team_id_list.append(np.array(players_team_id))

    print('Data has been loaded')
    print()
    
    return ball_position,players_position,players_team_id_list,events,np.array(players_jersey),info,players_names<jupyter_output><empty_output><jupyter_text>Loading in Events Data for the 3 Games<jupyter_code>df_events = pd.read_csv('df_subevents_Swedish_1920.csv')
len(df_events)
df_events<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10155392281057343, 'fraction_numerical': 0.009500886749429947, 'mean_word_length': 3.103083853083853, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '518345', 'license': 'MIT', 'n_tokens_mistral': 7543, 'n_tokens_neox': 7082, 'n_words': 1810, 'path': '3 Tracking Data/Signality/.ipynb_checkpoints/Tracking Assignment-Backup-checkpoint.ipynb', 'repo_name': 'christiangilson/Mathematical-Modelling-of-Football-Assignments', 'size': '23687'}