<jupyter_start><jupyter_text>Mess around with reading in the json files and figure out a schema to use for the data<jupyter_code>import re
import json
from pathlib import Path
import pandas as pd
json_dir = Path(".").resolve().parent / "data" / "raw"<jupyter_output><empty_output><jupyter_text>Just load a random file and poke around at it a bit to get a sense of the data<jupyter_code>eg_json = json_dir / "2014-01.json"
with open(eg_json) as f:
    data = json.load(f)
len(data)<jupyter_output><empty_output><jupyter_text>From the schema, each item in the list is supposed to be a game so it looks like we have 393 games for this month Let's poke around a bit more<jupyter_code>data[0].keys()
data[0]["game"]
for i in range(20,30):
    print(data[i]["game"])
data[0]["last_update"]
data[0]["player_count"]
data[0]["factions"]
data[0]["base_map"]
event_eg = data[0]["events"]
event_eg.keys()<jupyter_output><empty_output><jupyter_text>Location wasn't listed in the schema, let's take a look at it<jupyter_code>event_eg["location"]<jupyter_output><empty_output><jupyter_text>OK, I don't think I need that for now, but I'll update the schema text so it's noted<jupyter_code>event_eg["faction"]["alchemists"]["vp"]["round"]["all"]<jupyter_output><empty_output><jupyter_text>Well that's a pretty deeply nested place to get victory points, should be fun<jupyter_code>event_eg["global"]<jupyter_output><empty_output><jupyter_text>Ok, so to know what scoring tiles were used in the game I have to check for them in the keys of "global". I don't get the nested schema below that, all seems like it's always going to be one, and then the other key just tells me what round it was actually in. So in this example I read it as  Round 1: Scoring tile 2  Round 2: Scoring tile 7  Round 3: Scoring tile 5  Round 4: Scoring tile 1  Round 5: Scoring tile 3  Round 6: Scoring tile 8<jupyter_code>data[0]["factions"]<jupyter_output><empty_output><jupyter_text>I think I should have x + 3 = 6 passing tiles for this game, let's figure out what they are<jupyter_code>event_eg["faction"]["all"].keys()<jupyter_output><empty_output><jupyter_text>If I'm reading this right to get the passing tiles for the game I'd have to go through here and for each faction find the keys that have "pass"/"act":BON let's see if that makes sense<jupyter_code>fact_eg = event_eg["faction"]
rgx = r":(BON\d+)"
# I could do this in python 3.8 but trying to get everything else working there sent me to dependency hell. Oh well
# set(match.groups(0)[0] for x in fact_eg["all"].keys() if (match := re.compile(rgx).search(x)))
set(re.compile(rgx).search(x).groups(0)[0] for x in fact_eg["all"].keys() if re.compile(rgx).search(x))<jupyter_output><empty_output><jupyter_text>That looks like it's it then. I guess the risk is that if a bonus tile is available but is never played in the game I won't be able to identify it from this. Something to validate for when I'm building the dataset Day two of exploring<jupyter_code>eg_game = data[0]<jupyter_output><empty_output><jupyter_text>I'd like some way to consistently record which player is playing what, but there's no particular order so I'll have to figure that out some other way<jupyter_code>eg_game["factions"]
eg_game["events"]["faction"]["alchemists"]["order:1"]<jupyter_output><empty_output><jupyter_text>I was kind of hoping there'd be an easy way to figure out first player but this looks gross to parse and might not be worth it<jupyter_code>eg_game["events"]["global"]["faction-count"]
eg_game["events"]["global"].keys()
for eg_game in data:
    if "drop-faction" in eg_game["events"]["global"].keys():
        print(eg_game["game"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11352525801195003, 'fraction_numerical': 0.011678435632808256, 'mean_word_length': 6.193359375, 'pattern_counts': {'":': 2, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '302939', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1247, 'n_tokens_neox': 1221, 'n_words': 437, 'path': 'notebooks/01_ip_explore-json.ipynb', 'repo_name': 'ianepreston/terra-mystica-models', 'size': '3682'}