<jupyter_start><jupyter_text>Estimate user pp at time of play**Contributors:** <NAME>**Achievement:** Player pp at the time a score was set can be estimated by computing pp on all scores the player set before the play.**Requirements:**1. *sql_migration/sql_to_mongo_migration.ipynb*  IntroductionExcept for very recent plays, a player's pp does not reflect their skill at the time a score was set. Therefore, the amount of useable data would increase significantly if an accurate estimate of the user's pp at the time of play can be calculated<jupyter_code>import sys
sys.path.append('../..')
from pymongo import UpdateOne
from tqdm import tqdm
import numpy as np
from exploration.config import mongo_inst
from mlpp.data_collection.pp_calculation import fast_pp_hist
osu_db = mongo_inst['val_random_db']
user_ids = list(map(lambda c: c['_id'], osu_db['osu_user_stats'].find({}, {})))
cnt_bonus = []
curr = 1
for i in range(100000):
    curr *= .9994
    cnt_bonus.append(416.6667 * (1 - curr))
cnt_bonus = np.array(cnt_bonus)<jupyter_output><empty_output><jupyter_text>Estimating scores for each Playerfast_pp_history enables fast calculation of the player's estimated pp for a date-sorted list of scores. Score estimation calculations grouped by player.<jupyter_code>for user_id in tqdm(user_ids):
    scores = list(osu_db['osu_scores_high'].find({'user_id': user_id}, {'pp': 1, 'date': 1}).sort('date'))
    score_hist = list(map(lambda x: x['pp'] if x['pp'] else 0, scores))
    pp_hist = np.array(fast_pp_hist(score_hist))
    bonus_pp_hist = pp_hist + cnt_bonus[:len(pp_hist)]

    score_updates = []
    for i in range(len(scores)):
        query = {'_id': scores[i]['_id']}
        update = {
            '$set': {
                'mlpp.est_user_raw_pp': pp_hist[i],
                'mlpp.est_user_pp': bonus_pp_hist[i]
            }
        }
        score_updates.append(UpdateOne(query, update))
    
    if len(score_updates) > 0:
        osu_db['osu_scores_high'].bulk_write(score_updates)

        osu_db['osu_user_stats'].update_one({'_id': user_id}, {
            '$set': {
                'mlpp': {
                    'est_current_raw_pp': pp_hist[-1],
                    'est_current_pp': bonus_pp_hist[-1]
                }
    if len(score_updates) > 0:
        osu_random_db['osu_scores_high'].bulk_write(score_updates)

    osu_random_db['osu_user_stats'].update_one({'_id': user_id}, {
        '$set': {
            'mlpp': {
                'est_current_raw_pp': pp_hist[-1],
                'est_current_pp': bonus_pp_hist[-1]
        osu_db['osu_user_stats'].update_one({'_id': user_id}, {
            '$set': {
                'mlpp': {
                    'est_current_raw_pp': pp_hist[-1],
                    'est_current_pp': bonus_pp_hist[-1]
                }
            }
        })
    else:
        osu_db['osu_user_stats'].remove({'_id': user_id})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1262002743484225, 'fraction_numerical': 0.010631001371742112, 'mean_word_length': 3.023448275862069, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '45765', 'license': 'MIT', 'n_tokens_mistral': 1012, 'n_tokens_neox': 969, 'n_words': 220, 'path': 'exploration/2_estimate_real_user_pp/1_est_user_pp_historical.ipynb', 'repo_name': 'Cozieee/dsu-mlpp', 'size': '2920'}