<jupyter_start><jupyter_text>Make weighted manual subsetTo reduce the amount of manual coding labor, we didn't create the weighted subset through a completely separate sampling operation. Instead Underwood calculated how a weighted sample would be distributed across the range of reprinting-frequencies, [and randomly sampled a *supplement*](https://github.com/tedunderwood/noveltmmeta/blob/master/manuallists/make_weighted_supplement.ipynb) to make up the difference between a sample representing distinct titles and a sample representing volumes.Now we have to load that supplement and merge it with the title sample, after randomly removing volumes from the title sample. All the volumes we remove will be those only appearing once in Hathi; the volumes we add will all have more than one appearance.<jupyter_code>import random
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from collections import Counter
from difflib import SequenceMatcher
from sklearn.metrics import cohen_kappa_score
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load the data, standardize data dictionary<jupyter_code>j = pd.read_csv('weighteddata/jessicamanycopies.csv', index_col = 'docid')
p = pd.read_csv('weighteddata/patrickmanycopies.csv', index_col = 'docid')
t = pd.read_csv('weighteddata/tedmanycopies.csv', index_col = 'docid')
j['gender'] = j['gender'].fillna(value = 'u')
p['gender'] = p['gender'].fillna(value = 'u')
t['gender'] = t['gender'].fillna(value = 'u')
def dominant_category(astring):
    ''' Accepts a category string that may contain multiple
    values, and reduces it to a single category.
    '''
    astring = astring.replace(' ', '')
    cats = astring.split('|')
    if 'nonfic' in cats:
        return 'nonfic'
    if 'poetry' in cats:
        return 'poetry'
    if 'drama' in cats:
        return 'drama'
    if 'reprint' in cats:
        return 'reprint'
    if 'juvenile' in cats:
        return 'juvenile'
    if 'shortstories' in cats:
        return 'shortstories'
    if 'novel' in cats:
        return 'novel'
    else:
        return 'error'

j = j.assign(category = j.category.map(dominant_category))
p = p.assign(category = p.category.map(dominant_category))
t = t.assign(category = t.category.map(dominant_category))<jupyter_output><empty_output><jupyter_text>How much do human readers agree?The three readers in this portion of the project worked on overlapping sets of books, allowing us to roughly estimate average levels of agreement.Note that "agree" may be the wrong verb here, if it suggests as the other alternative a settled and irreconcilable *dis*agreement. Many cases of "disagreement" below are just data entry errors. In other cases, we probably could have come to consensus given a bit more time to discuss the meanings of categories. But in this project, we weren't working to stabilize a local consensus for the purposes of a particular experiment. We were rather producing a dataset that we expect to be borrowed by other people, who may or may not share our consensus. So it seemed appropriate to report a first (low) estimate of agreement; that will be more likely to reflect the intersubjective conditions operative for users of this project who know nothing but the names of the categories we used.<jupyter_code>def measure_agreement(category):
    agreement = Counter()
    differences = []
    
    firstlist = []
    secondlist = []
    
    frames = [j, p, t]
    for index1 in range(0, 3):
        for index2 in range(index1 + 1, 3):
            print(index1, ' - ', index2)
            frame1 = frames[index1]
            frame2 = frames[index2]
            overlap = set(frame1.index).intersection(set(frame2.index))
            for docid in overlap:
                value1 = frame1.loc[docid, category]
                value2 = frame2.loc[docid, category]
                firstlist.append(value1)
                secondlist.append(value2)
                
                if value1 == value2:
                    agreement[True] += 1
                elif pd.isnull(value1) and pd.isnull(value2):
                    agreement[True] += 1
                else:
                    agreement[False] += 1
                    differences.append((value1, value2, docid))
    
    k = cohen_kappa_score(firstlist, secondlist)

    return agreement, differences, k
genagreement, genderdiffs, k = measure_agreement('gender')
genagreement[True] / (genagreement[False] + genagreement[True])
genderdiffs
k
catagreement, catdiffs, kappa = measure_agreement('category')
catagreement[True] / (catagreement[False] + catagreement[True])
catdiffs
kappa
nationagreement, nationdiffs, kappa = measure_agreement('nationality')
nationagreement[True] / (nationagreement[False] + nationagreement[True])
nationdiffs
kappa<jupyter_output><empty_output><jupyter_text>what fraction of the errors are caused by blanks?<jupyter_code>ctr = 0
for d1, d2, doc in nationdiffs:
    if pd.isnull(d1) or pd.isnull(d2):
        ctr += 1
ctr
13 / len(nationdiffs)<jupyter_output><empty_output><jupyter_text>Combining the dataframesThis is a two-stage process, because we have to first reconcile the disagreements detailed above. make list of contested volumes<jupyter_code>cat = 0
reasons = dict()
allcontested = set()

for diffs in [genderdiffs, catdiffs, nationdiffs]:
    cat += 1
    for a, b, docid in diffs:
        if docid not in reasons:
            reasons[docid] = ""
        if cat == 1:
            reasons[docid] += "gender|"
        elif cat == 2:
            reasons[docid] += "category|"
        else:
            reasons[docid] += "nation|"
            
        allcontested.add(docid)

subset = list(allcontested)

tsub = t.loc[list(set(subset).intersection(set(t.index))), : ]
psub = p.loc[list(set(subset).intersection(set(p.index))), : ]
jsub = j.loc[list(set(subset).intersection(set(j.index))), : ]
tsub = tsub.assign(reader = 't')
psub = psub.assign(reader = 'p')
jsub = jsub.assign(reader = 'j')
subsets = pd.concat([tsub, psub, jsub])
subsets.shape

# drop duplicate rows

subsets = subsets[~subsets.index.duplicated(keep='first')]
subsets.shape
subsets['debate'] = subsets.index.to_series().map(reasons)<jupyter_output><empty_output><jupyter_text>resolve disagreements<jupyter_code># subsets.to_csv('weighteddata/contested_volumes.csv', index_label = 'docid')

# Note that I'm commenting that out, because I don't want it to re-run 
# every time this notebook is run. But on the first pass, this is where I
# wrote to file, and manually edited, a list of contested volumes.<jupyter_output><empty_output><jupyter_text>make list of uncontested volumes, and combine with contested<jupyter_code>allvols = pd.concat([j, p, t])
allvols.drop_duplicates(inplace = True, keep = False)
allvols.shape
contested = pd.read_csv('weighteddata/contested_volumes.csv', index_col = 'docid')
contested.drop(labels = ['debate', 'reader'], axis = 1, inplace = True)
contested.shape
allvols = pd.concat([allvols, contested])
allvols.shape<jupyter_output><empty_output><jupyter_text>reconsider ```reprint``` category<jupyter_code>reprints = allvols.loc[allvols.category == 'reprint', : ]
reprints.shape
# commenting out because I don't actually want this step to re-run

# reprints.to_csv('weighteddata/reprints.csv', index_label = 'docid')
reprints = pd.read_csv('weighteddata/reprints.csv', index_col = 'docid')
noreprints = allvols.loc[allvols.category != 'reprint', : ]
noreprints = noreprints.assign(hathiadvent = 'contemporary')
print("reprints: ", reprints.shape)
print("noreprints: ", noreprints.shape)
allvols = pd.concat([noreprints, reprints], sort = False)
allvols.shape<jupyter_output><empty_output><jupyter_text>Merging the frequently-reprinted supplement with the title_subsetTo do this, we first have to remove an equal number of rarely-reprinted volumes, distributed similarly across the timeline.<jupyter_code>title = pd.read_csv('manual_title_subset.tsv', sep = '\t', index_col = 'docid')
toremove = []

for floor in range(1800, 2010, 10):
    originals = title.loc[(title.inferreddate >= floor) & 
                          (title.inferreddate < (floor + 10)) &
                         (title.copiesin25yrs.astype(int) < 2), : ]
    new = allvols.loc[(allvols.inferreddate >= floor) & 
                          (allvols.inferreddate < (floor + 10)), : ]
    k = new.shape[0]
    sample = random.sample(originals.index.tolist(), k)
    toremove.extend(sample)  
len(toremove)
title.shape
title.drop(labels = toremove, inplace = True)
title.shape
all = pd.concat([title, allvols], sort = False)
all.shape<jupyter_output><empty_output><jupyter_text>Redefining categories to avoid misunderstandingIn manual coding we used the terms "novel" and "shortstories." But these phrases are in reality often misleading. Folktales or anecdotes are not really short stories, and some older or experimental fiction might not be quite "a novel."Let's use looser terms.<jupyter_code>def remap_categories(cat):
    accepted = {'longfiction', 'shortfiction', 'notfiction', 'poetry', 'juvenile', 'drama'}
    
    if cat == 'novel':
        return 'longfiction'
    elif cat == 'shortstories':
        return 'shortfiction'
    elif cat == 'nonfic':
        return 'notfiction'
    elif cat in accepted:
        return cat
    else:
        print(cat)
        return cat

all = all.assign(category = all.category.map(remap_categories))
all.to_csv('weighted_subset.tsv', sep = '\t', index_label = 'docid')<jupyter_output><empty_output><jupyter_text>EDA on gender<jupyter_code>def isfiction(astring):
    ''' Note that this doesn't count juvenile fiction.
    '''
    
    if pd.isnull(astring):
        return 'not'

    if 'longfiction' in astring or 'shortfiction' in astring:
            return 'fic'
    else:
        return 'not'

all = all.assign(isfic = all.category.map(isfiction))
def bootstrap_ratio(numtrue, numfalse):
    population = [True] * numtrue + [False] * numfalse
    results = []
    
    for i in range(1000):
        sample = np.random.choice(population, size = len(population), replace = True)
        ratio = sum(sample) / len(sample)
        results.append(ratio)
    
    results.sort()
    return results[49], results[950]
timeaxis = []
percentages = []
minima = []
maxima = []

for decade in range(1800, 2010, 15):
    in_decade = all.loc[(all.firstpub >= decade) & (all.firstpub < (decade + 15)) & ((all.isfic == 'fic')), : ]
    masculine = sum(in_decade.gender == 'm') 
    feminine = sum(in_decade.gender == 'f')
    pct = feminine / (masculine + feminine)
    timeaxis.append(np.mean(in_decade.firstpub))
    percentages.append(pct)
    minimum, maximum = bootstrap_ratio(feminine, masculine)
    maxima.append(maximum)
    minima.append(minimum)

plt.figure(figsize = (9, 6))
plt.xlim(1800,2010)
plt.ylim(0.0, 0.6)
plt.title('Percentage of fiction by women')
downward = np.array(percentages) - np.array(minima)
upward = np.array(maxima) - np.array(percentages)
plt.errorbar(timeaxis, percentages, yerr = [downward, upward], fmt='--o', ecolor = 'k', color = 'b')
plt.savefig('figures/weightedsubset/pctwomen.png')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11226685796269728, 'fraction_numerical': 0.00860832137733142, 'mean_word_length': 3.870305676855895, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 48, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '617277', 'license': 'MIT', 'n_tokens_mistral': 3596, 'n_tokens_neox': 3431, 'n_words': 990, 'path': 'manuallists/make_weighted_subset.ipynb', 'repo_name': 'tedunderwood/noveltmmeta', 'size': '11152'}