<jupyter_start><jupyter_text>Working With Play by PlayWorking with play by play can be interesting work in that there's a lot of unknown types of data as well as parsing of strings. In addition, there a ton of cool things that can be done with play be play like sending the feed into a pub\sub model so other systems can interact with it, build your own UI, or a whole host of other ideas.The goal of this notebook is to walk through the play by play feed examining data such as 1. `EVENTMSGTYPE` which provides the play type (e.g. FIELD_GOAL_MADE, FIELD_GOAL_MISSED, TIMEOUT, PERIOD_BEGIN, etc.)2. `EVENTMSGACTIONTYPE`which provides a subcatagorization of `EVENTMSGTYPE` (e.g. REVERSE_LAYUP, 3PT_JUMP_SHOT, HOOK_SHOT, etc.)This notebook builds on top of the following notebooks: [Finding Games](notebook2.ipynb), [Basics Notebook](Basics.ipynb), and of course, dives into `PlayByPlay` endpoint. Note that the `PlayByPlayV2` endpoint is an extension of `PlayByPlay`.So with that...let's get started!The goals are1. Get the last game the Pacers played (maybe we'll get lucky and get a current game)2. Examine the feed and the fields that are returned3. See how Regex can be applied to the play by play3. Dynamically build a unique list of NBA Player Actions Events using EVENTMSGACTIONTYPE4. See what's hiding in the feed...need to get those BLOCKS from the shot blockers!<jupyter_code># Game IDs needed
list_game_ids = ['0041900165', '0041900105', '0041900401', '0041900214', '0041900163', '0041900313', '0041900174', '0041900134', '0041900112', '0041900237', '0041900205', '0041900145', '0041900305', '0041900144', '0041900406', '0041900175', '0041900162', '0041900151', '0041900164', '0041900405', '0041900171', '0041900312', '0041900404', '0041900153', '0041900104', '0041900177', '0041900217', '0041900176', '0041900131', '0041900306', '0041900402', '0041900123', '0041900202', '0041900216', '0041900154', '0041900235', '0041900231', '0041900311', '0041900161', '0041900304', '0041900143', '0041900232', '0041900111', '0041900167', '0041900152', '0041900301', '0041900102', '0041900303', '0041900203', '0041900103', '0041900155', '0041900172', '0041900114', '0041900173', '0041900213', '0041900121', '0041900314', '0041900315', '0041900223', '0041900204', '0041900224', '0041900133', '0041900156', '0041900221', '0041900132', '0041900142', '0041900222', '0041900212', '0041900211', '0041900236', '0041900302', '0041900166', '0041900124', '0041900141', '0041900225', '0041900101', '0041900403', '0041900215', '0041900122', '0041900233', '0041900234', '0041900113', '0041900201']<jupyter_output><empty_output><jupyter_text>Retrieving the play by play dataNow that we've got a game_id, let's pull some play by play data<jupyter_code># Query for the play by play of that most recent regular season game
from nba_api.stats.endpoints import playbyplay

for x in list_game_ids:
    df = playbyplay.PlayByPlay(x).get_data_frames()[0]
    df.to_csv(
    f"../../data/interim/games/nba-stats-play-by-play-{x}-2019-20.csv", index=False
    )

df = playbyplay.PlayByPlay('0041900406').get_data_frames()[0]
df.head() #just looking at the head of the data<jupyter_output><empty_output><jupyter_text>Optional: Dataframes can become large. In pandas you can set some options to make it more visible if needed<jupyter_code>#Since the datset is fairly large you'll see plenty of elipses(...). 
#If that's the case, you can set the following options to expand the data 
#You can adjust these as you'd like
import pandas
pandas.set_option('display.max_colwidth',250)
pandas.set_option('display.max_rows',250)<jupyter_output><empty_output><jupyter_text>Some of the most valuable fields of `PlayByPlay`are the following:`EVENTMSGTYPE``EVENTMSGACTIONTYPE``HOMEDESCRIPTION`and `VISITORDESCRIPTION`.`EVENTMSGTYPE` gives us the type of event that has occurred. This can vary per game. This is why finding these and placing them into an Enum or other type structure is a good idea.<jupyter_code>#List unique values in the df['EVENTMSGTYPE'] colum
print(f'EVENTMSGTYPE: {sorted(df.EVENTMSGTYPE.unique())}')
#For quick refernce, here's an Enum for `EVENTMSGTYPE`
#This list may be incomplete as a thourogh play by play scan is necessary

from enum import Enum

class EventMsgType(Enum):
    FIELD_GOAL_MADE = 1
    FIELD_GOAL_MISSED = 2
    FREE_THROWfree_throw_attempt = 3
    REBOUND = 4
    TURNOVER = 5
    FOUL = 6
    VIOLATION = 7
    SUBSTITUTION = 8
    TIMEOUT = 9
    JUMP_BALL = 10
    EJECTION = 11
    PERIOD_BEGIN = 12
    PERIOD_END = 13<jupyter_output><empty_output><jupyter_text>Using the `EVENTMSGTYPE` field we can begin to examine the event types to see what typical values will be in the `EVENTMSGACTIONTYPE` `HOMEDESCRIPTION` and `VISITORDESCRIPTION` fields.<jupyter_code>#### pull the data for a specfic EVENTMSGTYPE
df.loc[df['EVENTMSGTYPE'] == 1].head() #hint: use the EVENTMSGTYPE values above to see different data<jupyter_output><empty_output><jupyter_text>Now that we've seen what the output of `EVENTMSGTYPE` is, let's dig into `EVENTMSGACTIONTYPE`.For this next exercise, let's pull all unique `EVENTMSGACTIONTYPE` values for `EVENTMSGTYPE = 1`_Note: `EVENTMSGACTIONTYPE` ids have a very loose correlation to `EVENTMSGTYPE` ids. This means that `EVENTMSGTYPE` ids share some of the same `EVENTMSGACTIONTYPE` ids. This allows the NBA to have a 'Missed Field Goal' share the same '3PT Jump Shot' with a 'Made Field Goal'. Now, that being said, they are not always unique. We'll see this towards the end._<jupyter_code>#List unique values in the df['EVENTMSGTYPE'] column
emt_df = df.loc[df['EVENTMSGTYPE'] == 1]
print(f'EVENTMSGACTIONTYPE: {sorted(emt_df.EVENTMSGACTIONTYPE.unique())}')<jupyter_output>EVENTMSGACTIONTYPE: [1, 3, 5, 6, 7, 41, 44, 47, 50, 52, 58, 66, 71, 72, 73, 75, 76, 78, 79, 80, 86, 97, 98, 99, 108]<jupyter_text>So how do we know what each `EVENTMSGACTIONTYPE` is?Let the fun begin.Apply some regular expressions, that are `EVENTMSGTYPE` specific, against `HOMEDECSRIPTION` and `VISITORDESCRIPTION` while keeping track of the `EVENTMSGACTIONTYPE`. To see the regular expressions in action, take the example listed in the comments, along with the regex, and head on over to https://regex101.com/ or your favorite regex interative tool.  `EVENTMSGTYPE == 1`The following regex expression `'(\s{2}|\' )([\w+ ]*)` will look for the type of basket within the `VISITORDESCRIPTION` or `HOMEDESCRIPTION` and tie that to the `EVENTMSGACTIONTYPE`.Example: Given a `VISITORDESCRIPTION == 'Young Cutting Layup Shot (2 PTS) (Collison 1 AST)'` and a `EVENTMSGACTIONTYPE = 98`, the code will produce an output of `CUTTING_LAYUP_SHOT = 99`Let's see it in action..._Note: The regex may need to be adjusted over time to account for changes in the data_<jupyter_code>#Mapping out all of the EventMsgActionTypes for EventMsgType 1
import re
import operator

#the following expression is specific to EventMsgType 1
p = re.compile('(\s{2}|\' )([\w+ ]*)')

#get the PlayByPlay data from the Pacers game_id
plays = playbyplay.PlayByPlay(game_id).get_normalized_dict()['PlayByPlay']

#declare a few variables
description = ''
event_msg_action_types = {}

#loop over the play by play data
for play in plays:
    if play['EVENTMSGTYPE'] == 1:
        description = play['HOMEDESCRIPTION'] if play['HOMEDESCRIPTION'] is not None else play['VISITORDESCRIPTION']
        if description is not None:
            #do a bit of searching(regex) and a little character magic: underscores and upper case
            event_msg_action = re.sub(' ', '_', p.search(description).groups()[1].rstrip()).upper()
            #Add it to our dictionary
            event_msg_action_types[event_msg_action] = play['EVENTMSGACTIONTYPE']
            
#sort it all
event_msg_action_types = sorted(event_msg_action_types.items(), key=operator.itemgetter(0))

#output a class that we could plug into our code base
for action in event_msg_action_types:
    print(f'\t{action[0]} = {action[1]}')<jupyter_output>3PT_JUMP_SHOT = 1
	3PT_PULLUP_JUMP_SHOT = 79
	3PT_STEP_BACK_JUMP_SHOT = 80
	ALLEY_OOP_DUNK = 52
	CUTTING_DUNK_SHOT = 108
	CUTTING_FINGER_ROLL_LAYUP_SHOT = 99
	CUTTING_LAYUP_SHOT = 98
	DRIVING_FINGER_ROLL_LAYUP = 75
	DRIVING_LAYUP = 6
	DRIVING_REVERSE_LAYUP = 73
	DUNK = 7
	FINGER_ROLL_LAYUP = 71
	FLOATING_JUMP_SHOT = 78
	HOOK_SHOT = 3
	JUMP_BANK_SHOT = 66
	JUMP_SHOT = 1
	LAYUP = 5
	PULLUP_JUMP_SHOT = 79
	PUTBACK_LAYUP = 72
	REVERSE_LAYUP = 44
	RUNNING_DUNK = 50
	RUNNING_FINGER_ROLL_LAYUP = 76
	RUNNING_LAYUP = 41
	STEP_BACK_JUMP_SHOT = 80
	TIP_LAYUP_SHOT = 97
	TURNAROUND_FADEAWAY = 86
	TURNAROUND_HOOK_SHOT = 58
	TURNAROUND_JUMP_SHOT = 47<jupyter_text>`EVENTMSGTYPE == 2`We'll reuse the regex expression `(\s{2}|' )([\w+ ]*)` from `EVENTMSGTYPE == 1` for `EVENTMSGTYPE == 2`. EventMsgType 2 are missed field goals. Again, it'll look for the type of basket within the `VISITORDESCRIPTION` or `HOMEDESCRIPTION` and tie that to the `EVENTMSGACTIONTYPE`.Example: Given a `HOMEDESCRIPTION == 'MISS Collison 24' 3PT Jump Shot'` and a `EVENTMSGACTIONTYPE = 2`, the code will produce an output of `3PT_JUMP_SHOT = 1`Let's see it in action...<jupyter_code>#Mapping out all of the EventMsgActionTypes for EventMsgType 2
import re
import operator

#the following expression is specific to EventMsgType 1
p = re.compile('(\s{2}|\' )([\w+ ]*)')

#get the PlayByPlay data from the Pacers game_id
plays = playbyplay.PlayByPlay(game_id).get_normalized_dict()['PlayByPlay']

#declare a few variables
description = ''
event_msg_action_types = {}

#loop over the play by play data
#do a bit of findall(regex) and a little character magic: underscores and upper case
#we're using a findall here as we have to deal with the extra word MISS at the beginning of the text.
#that extra text means we'll have multiple matches for our regex.
for play in plays:
    if play['EVENTMSGTYPE'] == 2:
        match = list()
        if play['HOMEDESCRIPTION'] is not None: 
            match = p.findall(play['HOMEDESCRIPTION'])
        
        if not match:
            match = p.findall(play['VISITORDESCRIPTION'])

        event_msg_action = re.sub(' ', '_', match[0][1]).upper()
        event_msg_action_types[event_msg_action] = play['EVENTMSGACTIONTYPE']
        
       # if play['EVENTMSGACTIONTYPE']
        
event_msg_action_types = sorted(event_msg_action_types.items(), key=operator.itemgetter(0))

for action in event_msg_action_types:
    print(f'\t{action[0]} = {action[1]}')<jupyter_output>3PT_JUMP_SHOT = 1
	3PT_PULLUP_JUMP_SHOT = 79
	3PT_STEP_BACK_JUMP_SHOT = 80
	ALLEY_OOP_DUNK = 52
	ALLEY_OOP_LAYUP = 43
	CUTTING_LAYUP_SHOT = 98
	DRIVING_FINGER_ROLL_LAYUP = 75
	DRIVING_LAYUP = 6
	DRIVING_REVERSE_LAYUP = 73
	FADEAWAY_JUMPER = 63
	FINGER_ROLL_LAYUP = 71
	FLOATING_JUMP_SHOT = 78
	HOOK_SHOT = 3
	JUMP_BANK_SHOT = 66
	JUMP_SHOT = 1
	LAYUP = 5
	PULLUP_JUMP_SHOT = 79
	REVERSE_LAYUP = 44
	RUNNING_FINGER_ROLL_LAYUP = 76
	RUNNING_JUMP_SHOT = 2
	RUNNING_LAYUP = 41
	STEP_BACK_JUMP_SHOT = 80
	TIP_DUNK_SHOT = 107
	TIP_LAYUP_SHOT = 97
	TURNAROUND_FADEAWAY_BANK_JUMP_SHOT = 105
	TURNAROUND_JUMP_SHOT = 47<jupyter_text>What About Blocks?So if you've taken a close look at the data, especially that where `EVENTMSGTYPE == 2` you may have noticed that a few of the missed field goals were due to some incredible shot blocking players. By adding a few lines of code, we can find these shot blockers. Dealing with this data is a bit beyond the scope of this notebook, but it's worth pointing out that the data is in there. One idea is to play it into it's own play by play block (just a thought).<jupyter_code>#Blocks are not included in the event feed but are a part of the EVENTMSGTYPE 2
import re
import operator

print('------------------')

#the following expression is specific to EventMsgType 1
p = re.compile('(\s{2}|\' )([\w+ ]*)')

#get the PlayByPlay data from the Pacers game_id
plays = playbyplay.PlayByPlay(game_id).get_normalized_dict()['PlayByPlay']

#declare a few variables
description = ''
event_msg_action_types = {}

#loop over the play by play data
#do a bit of findall(regex) and a little character magic: underscores and upper case
#we're using a findall here as we have to deal with the extra word MISS at the beginning of the text.
#that extra text means we'll have multiple matches for our regex.
for play in plays:
    if play['EVENTMSGTYPE'] == 2:
        match = list()
        if play['HOMEDESCRIPTION'] is not None: 
            match = p.findall(play['HOMEDESCRIPTION'])

            #looking for blocks
            if len(match) & (play['VISITORDESCRIPTION'] is not None):
                print(play['VISITORDESCRIPTION'])

        if not match:
            match = p.findall(play['VISITORDESCRIPTION'])
            
            #looking for blocks
            if len(match) & (play['HOMEDESCRIPTION'] is not None):
                print(play['HOMEDESCRIPTION'])


        event_msg_action = re.sub(' ', '_', match[0][1]).upper()
        event_msg_action_types[event_msg_action] = play['EVENTMSGACTIONTYPE']
            
event_msg_action_types = sorted(event_msg_action_types.items(), key=operator.itemgetter(0))

print('------------------')<jupyter_output>------------------
Antetokounmpo BLOCK (1 BLK)
Joseph BLOCK (1 BLK)
Turner BLOCK (1 BLK)
Young BLOCK (1 BLK)
Turner BLOCK (2 BLK)
Snell BLOCK (1 BLK)
Lopez BLOCK (1 BLK)
------------------
{'fraction_non_alphanumeric': 0.10331384015594541, 'fraction_numerical': 0.0817213975108712, 'mean_word_length': 4.668933276668083, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '40524', 'license': 'MIT', 'n_tokens_mistral': 5298, 'n_tokens_neox': 4466, 'n_words': 1479, 'path': 'gathering/nba-api-py-play/PlayByPlay2019Playoffs.ipynb', 'repo_name': 'JovaniPink/nba-data', 'size': '13338'}