<jupyter_start><jupyter_text>Data importAssumes data is in `DATA_DIR` and includes the directory and file structure of data as downloaded from [retrosheet.org](retrosheet.org).<jupyter_code>DATA_DIR = '../data'
import glob

event_files = glob.glob("{}/*seve/*.EV*".format(DATA_DIR))<jupyter_output><empty_output><jupyter_text>Constrain to years of interest<jupyter_code>years = list(range(2010, 2016))
year_files = [f for f in event_files if int(f.split('/')[-1][:4]) in years]<jupyter_output><empty_output><jupyter_text>Parse game information, including plays and lineup changes.<jupyter_code>from io import StringIO

def parse_event_file(filename):
    game_info_io, game_play_io, lineup_io = str_io_list = [StringIO() for _ in range(3)]
    with open(filename) as f:
        game_id = ''
        lineup_id = 0
        new_game = True
        for line in f:
            line = line.rstrip('\n')
            if line.startswith('id'):
                game_id = line.split(',')[-1]
            elif line.startswith('start'):
                if not new_game:
                    new_game = True
                    lineup_id = 0
                lineup_io.write(','.join([game_id, str(lineup_id)] 
                                         + line.split(',')[1:]) + '\n')
            elif line.startswith('info'):
                game_info_io.write(','.join([game_id] + line.split(',')[1:]) + '\n')
            elif line.startswith('play'):
                game_play_io.write(','.join([game_id, str(lineup_id)] 
                                            + line.split(',')[1:]) + '\n')
            elif line.startswith('sub'):
                new_game = False
                lineup_id += 1
                lineup_io.write(','.join([game_id, str(lineup_id)] 
                                         + line.split(',')[1:]) + '\n')
                
    
    # "rewind" to the beginning of the StringIO object
    for str_io in str_io_list:
        str_io.seek(0)
    
    return (pd.read_csv(game_info_io, header=None, 
                    names=['Game_ID','Var','Value']).pivot('Game_ID','Var','Value'),
            pd.read_csv(game_play_io, header=None, index_col=False,
                    names=['Game_ID', 'Lineup_ID', 'Inning', 'Home', 'Retrosheet_ID', 
                           'Count', 'Pitches', 'Play']),
           pd.read_csv(lineup_io, header=None, index_col=False,
                      names=['Game_ID', 'Lineup_ID', 'Retrosheet_ID', 'Name', 'Home', 
                             'Order', 'Position']))<jupyter_output><empty_output><jupyter_text>Parse all files<jupyter_code>parsed_files = [parse_event_file(f) for f in year_files]<jupyter_output><empty_output><jupyter_text>Concatenate game info, plays and lineup data<jupyter_code>games = pd.concat([df[0] for df in parsed_files], ignore_index=True)
games.shape
plays = pd.concat([df[1] for df in parsed_files], ignore_index=True)
plays.shape
plays.loc[301:320]
lineups = pd.concat([df[2] for df in parsed_files], ignore_index=True)
lineups.shape<jupyter_output><empty_output><jupyter_text>Create hierarchical index for lineups<jupyter_code>lineups_hi = lineups.set_index(['Game_ID', 'Lineup_ID', 'Home', 'Order'])
lineups_hi.head(25)<jupyter_output><empty_output><jupyter_text>Function for constructing a given lineup from lineup changes<jupyter_code>def get_lineup(game_id, lineup_id, data=lineups_hi):
    game_data = data.loc[game_id]
    current_lineup = game_data.loc[0].copy()
    try:
        for l in range(lineup_id):
            lineup_change = game_data.loc[l+1]
            current_lineup.loc[lineup_change.index] = lineup_change
    except IndexError:
        print('Invalid lineup number', lineup_id)
        return None
    return current_lineup<jupyter_output><empty_output><jupyter_text>For example, pick arbitrary play and reconstruct the lineup at the time:<jupyter_code>plays.loc[24776]
get_lineup('BAL201006220', 4)<jupyter_output><empty_output><jupyter_text>Chadwick events tableUse `cwevent` to parse events files<jupyter_code>fields = pd.read_csv('../data/fields.csv', index_col=0)
def cwevent(year):
    try:
        year_dir = '/'.join(glob.glob("{}/*seve/{}*.EV*".format(DATA_DIR, year))[0].split('/')[:-1])
    except:
        print('No data for', year)
        return
    !cd $year_dir; cwevent -y $year -q -f 0-96 $year*.EV* > tmp.csv
    df = pd.read_csv(year_dir+'/tmp.csv', header=None, names=fields.Header)
    !rm $year_dir/tmp.csv
    return df
events1920s = pd.concat([cwevent(y) for y in range(1921, 1930)])
events1920s.shape<jupyter_output><empty_output><jupyter_text>RostersParse roster files and create single table<jupyter_code>roster_files = glob.glob("{}/*seve/*.ROS".format(DATA_DIR))
roster_files[-1].split('/')[-1][3:7]
def parse_roster_file(filename):
    
    df = pd.read_csv(filename, header=None, 
                     names=("Retrosheet_ID", "Last_Name", "First_Name",
                            "Bats", "Pitches", "Team", "Position"), 
                     na_values=['X'])
    df['Year'] = int(filename.split('/')[-1][3:7])
    return df
rosters = pd.concat([parse_roster_file(f) for f in roster_files])
rosters.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14357585139318885, 'fraction_numerical': 0.01489938080495356, 'mean_word_length': 3.03828125, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '280484', 'license': 'MIT', 'n_tokens_mistral': 1818, 'n_tokens_neox': 1729, 'n_words': 323, 'path': 'notebooks/Retrosheet.ipynb', 'repo_name': 'fonnesbeck/baseball', 'size': '5168'}