<jupyter_start><jupyter_text>Merge non-US canonWe have created a list of 52 volumes not by American authors that are reprinted by <NAME>. To extend this up to 1980, we have in a few cases selected novels by authors whose short fiction is included in the *Norton Anthology of English Literature*.However, not all of these volumes are already present in our sample. So we probably need to add them and run another model.*This notebook is going to be the notebook that creates the third — and let's hope final — version of project metadata.*<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>the non-US canon<jupyter_code>nonus = pd.read_csv('canon/nonusnorton.tsv', sep = '\t', index_col = 'docid')
nonus.head()<jupyter_output><empty_output><jupyter_text>metadata documenting hypotheses<jupyter_code>hypmeta = pd.read_csv('../supplement2/second_supplement_deduped.tsv', sep = '\t', index_col = 'docid')
hypmeta.head()
allmeta = pd.read_csv('../supplement2/supp2nationalitymeta.tsv', sep = '\t', index_col = 'docid')
allmeta.head()<jupyter_output><empty_output><jupyter_text>Initial analysis of the situationHow many of the non-us vols are already in our hypothesis list? How many are in the larger list?<jupyter_code>print('Total number of non-us vols: ', len(set(nonus.index)))
hypoverlap = set(nonus.index).intersection(set(hypmeta.index))
print("In the hypothesis list: ", len(hypoverlap))
alloverlap = set(nonus.index).intersection(set(allmeta.index))
print("In the larger list: ", len(alloverlap))
print(len(alloverlap.intersection(hypoverlap)))<jupyter_output>13<jupyter_text>what about problems in hypmeta itselfsome vols in it aren't in allmeta<jupyter_code>print(hypmeta.shape)
largeoverlap = set(allmeta.index).intersection(set(hypmeta.index))
print(len(largeoverlap))
abouttolose = set(hypmeta.index) - set(allmeta.index)
hypmeta.loc[abouttolose, : ]<jupyter_output><empty_output><jupyter_text>That requires some special thinking. Basically, I happen to know the ```uc1.b``` volumes are Hathi-level problems. But the other three are things I've been able to download. Don't know why not in allmeta. Let's try a second time.<jupyter_code>keepthese = ['mdp.39015031600409', 'nyp.33433081852711', 'mdp.39015004063601']
print('Already in allmeta: ', len(set(keepthese).intersection(set(allmeta.index))))<jupyter_output>Already in allmeta:  0<jupyter_text>Let's build an addition to allmeta<jupyter_code>workmeta = pd.read_csv('../../noveltmmeta/workmeta.tsv', sep = '\t', index_col = 'docid', low_memory = False)
intersectcolumns = set(workmeta.columns).intersection(set(allmeta.columns))
intersectcolumns
set(allmeta.columns) - set(workmeta.columns)
missingvols = set(nonus.index) - set(allmeta.index)
missingvols = list(missingvols) + keepthese
print(len(missingvols))
newallmeta = workmeta.loc[missingvols, intersectcolumns]
newallmeta.head()
newallmeta = newallmeta.assign(nationality = 'nonus')
for k in keepthese:
    newallmeta.loc[k, 'nationality'] = 'us'
def get_last_name(aname):
    if pd.isnull(aname) or len(aname) < 2:
        return('anonymous')
    else:
        names = aname.split()
        lastname = names[0].strip('()[],.')
        return lastname

newallmeta = newallmeta.assign(lastname = newallmeta.author.map(get_last_name))
newallmeta = pd.concat([allmeta, newallmeta])
newallmeta.shape<jupyter_output>/Users/tunder/miniconda3/lib/python3.5/site-packages/ipykernel/__main__.py:1: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=False'.

To retain the current behavior and silence the warning, pass 'sort=True'.

  if __name__ == '__main__':<jupyter_text>fixing a significant bug in the second supplementA rather consequential detail here: I notice that the anonymization routine last time around failed to distinguish different anonymous authors!<jupyter_code>print(len(set(newallmeta.lastname)))
ctr = 0
for idx in newallmeta.index:
    if newallmeta.loc[idx, 'lastname'].startswith('anonymou'):
        newallmeta.loc[idx, 'lastname'] = 'anonymous' + str(ctr)
        ctr += 1
print(len(set(newallmeta.lastname)))<jupyter_output>10298
13228<jupyter_text>add birthdate<jupyter_code>print(newallmeta.shape)
birthmeta = newallmeta.join(workmeta.authordate, how = 'left')
birthmeta.shape
def get_birth(authdate):
    if pd.isnull(authdate):
        return np.nan
    elif len(authdate) < 5:
        return np.nan
    elif len(authdate) == 5 and authdate.endswith('-'):
        try:
            birth = int(authdate[0:4])
            return birth
        except:
            return np.nan
    elif len(authdate) > 8:
        try:
            birth = int(authdate[0:4])
            return birth
        except:
            return np.nan
    elif authdate.startswith('b.') and authdate[-4:].isnumeric():
        return int(authdate[-4 : ])
    else:
        return np.nan

birthmeta = birthmeta.assign(birth = birthmeta.authordate.map(get_birth))
birthmeta.shape
sum(~pd.isnull(birthmeta.birth))
birthmeta.to_csv('../supplement3/third_all_meta.tsv', sep = '\t', index_label = 'docid')<jupyter_output><empty_output><jupyter_text>Let's build an addition to hypmeta<jupyter_code>intersectcolumns = set(workmeta.columns).intersection(set(hypmeta.columns))
intersectcolumns
set(hypmeta.columns) - set(workmeta.columns)
abouttolose = set(hypmeta.index) - set(allmeta.index)
print(abouttolose)
for k in keepthese:
    abouttolose.remove(k)
abouttolose
keep = set(hypmeta.index) - abouttolose
hypmeta = hypmeta.loc[keep, : ]
hypmeta.shape
print('Total number of non-us vols: ', len(set(nonus.index)))
hypoverlap = set(nonus.index).intersection(set(hypmeta.index))
print("In the hypothesis list: ", len(hypoverlap))
hypmeta = hypmeta.assign(nonusnorton = False)
for h in hypoverlap:
    hypmeta.loc[h, 'nonusnorton'] = True
needed2add = set(nonus.index) - set(hypmeta.index)
print(len(needed2add))
addition = workmeta.loc[needed2add, intersectcolumns]
addition.head()
booleans = {'bestseller', 'contrast4reviewed', 'heath', 'mostdiscussed', 'norton', 'nortonshort', 
            'preregistered', 'reviewed'}
for b in booleans:
    addition[b] = False
addition['nonusnorton'] = True
addition.head()
set(hypmeta.columns) - set(addition.columns)
addition['firstpub'] = np.nan
for idx in addition.index:
    addition.loc[idx, 'firstpub'] = nonus.loc[idx, 'firstpub']
addition.head()
addition['gender'] = np.nan
addition['nationality'] = 'non-us'
addition = addition.assign(lastname = addition.author.map(get_last_name))
addition.shape
hypmeta.shape
newhypmeta = pd.concat([hypmeta, addition])
newhypmeta.shape
newhypmeta.to_csv('../supplement3/third_hypothesis_meta.tsv', sep = '\t', index_label = 'docid')
toget = set(birthmeta.index) - set(allmeta.index)
print(len(toget))
with open('../gettexts/thirdsupplementIDs.txt', mode = 'w', encoding = 'utf-8') as f:
    for tg in toget:
        f.write(tg + '\n')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12670842824601367, 'fraction_numerical': 0.012670842824601366, 'mean_word_length': 5.238898756660746, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '430518', 'license': 'MIT', 'n_tokens_mistral': 2480, 'n_tokens_neox': 2380, 'n_words': 555, 'path': 'meta/merge_nonus_canon.ipynb', 'repo_name': 'tedunderwood/asymmetry', 'size': '7030'}