<jupyter_start><jupyter_text>Make popular subsetCreating a subset of ```titlemeta``` that is selected to maximize the ```copiesin25yrs``` column.This may not be "popularity" in an absolute sense; please don't take the name of the file in a literal-minded fashion.<jupyter_code>import pandas as pd
import numpy as np
from scipy.stats import pearsonr
title = pd.read_csv('../titlemeta.tsv', sep = '\t', index_col = 'docid', low_memory = False)
frames = []

for floor in range(1800, 2010, 10):
    decade = title.loc[(title.latestcomp >= floor) & (title.latestcomp < (floor + 10)), : ]
    sample = decade.nlargest(100, 'copiesin25yrs')  
    frames.append(sample)

subset = pd.concat(frames)
subset.shape
subset.head()
subset.to_csv('most_popular_subset.tsv', sep = '\t', index_label = 'docid')<jupyter_output><empty_output><jupyter_text>validate the logic of ```copiesin25yrs```This version of ```most_popular_subset``` was created after experimentally recalculating our "number of copies" statistics. The earlier version of that calculation privileged multi-volume works. I want to confirm that our new calculation works better.<jupyter_code>oldsub = pd.read_csv('old_popular_subset.tsv', sep = '\t', index_col = 'docid')
lengths = []
copies = []

for recordid, df in oldsub.groupby('recordid'):
    lengths.append(len(df))
    copies.append(np.mean(df.copiesin25yrs))

print(np.mean(lengths))
pearsonr(lengths, copies)
    
lengths = []
copies = []

for recordid, df in subset.groupby('recordid'):
    lengths.append(len(df))
    copies.append(np.mean(df.copiesin25yrs))

print(np.mean(lengths))
pearsonr(lengths, copies)<jupyter_output>1.1751538892
{'fraction_non_alphanumeric': 0.1321212121212121, 'fraction_numerical': 0.02181818181818182, 'mean_word_length': 5.474509803921569, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '180377', 'license': 'MIT', 'n_tokens_mistral': 584, 'n_tokens_neox': 544, 'n_words': 145, 'path': 'manuallists/make_frequently_reprinted_subset.ipynb', 'repo_name': 'tedunderwood/noveltmmeta', 'size': '1650'}