<jupyter_start><jupyter_text>Aggergate osu random dumps into MongoDB**Contributors:** <NAME>**Achievement:** Migrate osu_user_stats, osu_scores_high, osu_beatmaps, osu_beatmap_difficulty_attribs tables from several random data dumps into mongodb. Prioritize migrating more recent dumps to avoid duplicates. Created basic indecies on osu_scores_high & a unique compound index on osu_beatmap_attribs.**Requirements:**- External    - MySQL DB has imported osu random dumps from https://data.ppy.sh/<jupyter_code>import sys
sys.path.append('../..')
from exploration.config import sql_inst, mongo_inst
from mlpp.data_collection.sql_migration import SqlDumpMigrator<jupyter_output><empty_output><jupyter_text>Declare SQL dumps to migrate**Note:** Order does matter in the declaration. More recent dumps should be declared first in order to prioritize more recent user information. Duplicate users found in older dumps will be skipped.<jupyter_code># Order of insertion matters

SQL_DUMPS = [
    "osu_random_2021_02",
    "osu_random_2020_12",
    "osu_random_2020_11",
    "osu_random_2020_10",
    "osu_random_2020_09",
    "osu_random_2020_08"
]

migrator = SqlDumpMigrator(sql_inst)
osu_random_db = mongo_inst["osu_random_db"]<jupyter_output><empty_output><jupyter_text>**TODO** : Code to create a compound index on (user_id, beatmap_id, and enabled_mods) would ensure score uniqueness. However, there are currently duplicates scores in some dumps that violates the uniqueness. Commented until further clarification from osu! devs.<jupyter_code># mongo_db['osu_scores_high'].create_index(
#     [('user_id', pymongo.ASCENDING), ('beatmap_id', pymongo.ASCENDING) , ('enabled_mods', pymongo.ASCENDING)],
#     unique=True
# )<jupyter_output><empty_output><jupyter_text>Migrating Tables: Users, Scores, Beatmaps, Attribs<jupyter_code>osu_random_db['osu_scores_high'].create_index('user_id')
osu_random_db['osu_scores_high'].create_index('beatmap_id')

migrator.migrate_users_and_scores(SQL_DUMPS, osu_random_db)
migrator.migrate_beatmaps(SQL_DUMPS, osu_random_db)
migrator.migrate_beatmap_attribs(SQL_DUMPS, osu_random_db)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09361900326036329, 'fraction_numerical': 0.016767582673497903, 'mean_word_length': 7.592, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '332286', 'license': 'MIT', 'n_tokens_mistral': 788, 'n_tokens_neox': 743, 'n_words': 165, 'path': 'exploration/1_sql_migration/random_dump_migration.ipynb', 'repo_name': 'Cozieee/dsu-mlpp', 'size': '2151'}