<jupyter_start><jupyter_text>Load M19 datasetThis notebook describes how to load the standardized M19 dataset.<jupyter_code>import os
import pickle

import pandas as pd

import draftsimtools as ds<jupyter_output><empty_output><jupyter_text>Show standardized dataset<jupyter_code>data_folder = "./standardized_m19/"
display(os.listdir(data_folder))<jupyter_output><empty_output><jupyter_text>1. Load draft ratings<jupyter_code>cur_set = pd.read_csv(data_folder + 'standardized_m19_rating.tsv', delimiter="\t")
display(cur_set)<jupyter_output><empty_output><jupyter_text>2. Load drafts with cardnames<jupyter_code>def load_data(path):
    """
    Load a pickle file from disk. 
    """
    with open(path, "rb") as f:
        return pickle.load(f)
drafts_train = load_data(data_folder + 'drafts_train.pkl')
drafts_test = load_data(data_folder + 'drafts_test.pkl')<jupyter_output><empty_output><jupyter_text>Show the train/test split<jupyter_code>print(len(drafts_train), len(drafts_test))<jupyter_output>86359 21590<jupyter_text>Show the first 2 picks of the first draft<jupyter_code>print(drafts_train[0][:2])<jupyter_output>[['Volcanic_Dragon', 'Lena_Selfless_Champion', 'Exclusion_Mage', 'Gallant_Cavalry', 'Dragon_Egg', 'Manalith', 'Goblin_Instigator', 'Salvager_of_Secrets', 'Gearsmith_Guardian', 'Bogstomper', 'Disperse', 'Trumpet_Blast', 'Duress', 'Walking_Corpse', 'Island'], ['Departed_Deckhand', 'Star-Crowned_Stag', 'Meteor_Golem', 'Aviation_Pioneer', 'Oreskos_Swiftclaw', 'Abnormal_Endurance', 'Gearsmith_Guardian', 'Divination', 'Centaur_Courser', 'Mind_Rot', 'Blanchwood_Armor', 'Talons_of_Wildwood', 'Root_Snare', 'Mountain']]<jupyter_text>3. Load drafts with indices The drafts_tensor format replaces cardnames with indices.This was previously referred to as the "Intermediate Draft Representation"<jupyter_code>drafts_tensor_train = load_data(data_folder + 'drafts_tensor_train.pkl')
drafts_tensor_test = load_data(data_folder + 'drafts_tensor_test.pkl')<jupyter_output><empty_output><jupyter_text>Show size of draft tensors<jupyter_code>print(drafts_tensor_train.shape, drafts_tensor_test.shape)<jupyter_output>(86359, 45, 15) (21590, 45, 15)<jupyter_text>Show the first 2 picks of the first draft<jupyter_code>drafts_tensor_train[0, :2, :]<jupyter_output><empty_output><jupyter_text>4. Creating One Hot Encoded DatasetThe one hot encoded representation is useful for training most ML models. Currently, this representation is dynamically generated from the intermediate representation. In the future, the data may be serialized in this format to improve training performance.  First, create a cardname -> index mapping<jupyter_code>le = ds.create_le(cur_set["Name"].values)
print(le.classes_[:5])<jupyter_output>['Abnormal_Endurance' 'Act_of_Treason' 'Aegis_of_the_Heavens'
 'Aerial_Engineer' 'Aether_Tunnel']<jupyter_text>Then, define dynamically generated datasets<jupyter_code>train_dataset = ds.DraftDataset(drafts_tensor_train, le)
test_dataset = ds.DraftDataset(drafts_tensor_test, le)<jupyter_output><empty_output><jupyter_text>5. Using the One Hot Encoded Dataset<jupyter_code>x, y = train_dataset[10]<jupyter_output><empty_output><jupyter_text>Input Representation  x is a vector of length 2n, where n is the number of cards in the set.  x[:n] represent the counts of cards already picked by the user.  x[n:2n] represent the cards in the current pack (1 if card is present in pack).<jupyter_code>print(x)<jupyter_output>tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 2., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., [...]<jupyter_text>Output Representationy is a vector of length ny[i]=1, where i is the index of the card picked by the user.<jupyter_code>print(y)<jupyter_output>tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0])
{'fraction_non_alphanumeric': 0.2009090909090909, 'fraction_numerical': 0.09872727272727273, 'mean_word_length': 3.3555027711797307, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 61, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '510930', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2810, 'n_tokens_neox': 2287, 'n_words': 765, 'path': 'bots/load_standardized_set.ipynb', 'repo_name': 'khakhalin/mtg', 'size': '5500'}