<jupyter_start><jupyter_text>Data Acquisition and ProcessingThis notebook contains the instructions for downloading and processing the data for training the Hidden Markov Models for BeatMapSynth. I do not suggest using this code because it can take over 24 hours to run, but rather download the zipped folder I have made available [here](https://drive.google.com/open?id=18l7p0K_gwRQ4NSVnuRGlqcM95TFsdbi6) and unzip the contents in the `data/processed_data/` folder of the repo (if `/processed_data/` doesn't exist please make create it).  _If you insist though..._First, import the metadata from beatsaver.com that contain the download keys needed for making API calls:<jupyter_code>with open('../data/metadata.pkl', 'rb') as f:
    metadata_total = pickle.load(f)<jupyter_output><empty_output><jupyter_text>Second, I trained my Hidden Markov Models on two sets of data.1. Custom maps with > 90% rating on beatsaver.com2. Custom maps with > 70% rating on beatsaver.comFor the first set, subset the metadata with this code:<jupyter_code>first_set = list(filter(lambda x: x['stats']['rating'] >= .9, metadata_total))<jupyter_output><empty_output><jupyter_text>Then, if you wanted to download and process the data:<jupyter_code>download_and_process(first_set)<jupyter_output><empty_output><jupyter_text>I trained my first model before downloading the second set, so you could go over to the modeling notebook and do that before proceeding to the next step. (In theory you could do all of the downloading at once and then train on the subset that is rated 90% or higher first, but that's just not the order I did things).After you've trained the first HMM, download the second dataset. This one will take a really long time. AGAIN, I DO NOT SUGGEST THIS!<jupyter_code>second_set = list(filter(lambda x: x['stats']['rating'] < .9 and x['stats']['rating'] >= .7, metadata_total))
download_and_process(second_set)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08311822405782138, 'fraction_numerical': 0.010841507485802787, 'mean_word_length': 5.946236559139785, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '229146', 'license': 'MIT', 'n_tokens_mistral': 592, 'n_tokens_neox': 552, 'n_words': 233, 'path': 'notebooks/Data_acquisition_and_processing.ipynb', 'repo_name': 'wvsharber/BeatMapSynthesizer', 'size': '1937'}