<jupyter_start><jupyter_text>Make gender-balanced subsetFor most of the two centuries this project covers, more works of fiction were published by men than by women. This dataset experimentally levels that imbalance, using the simple technique of discarding works by men. In a few periods at the beginning and end of the timeline, more works were published by women, and we have to reverse the strategy.This approach means that coverage will be uneven across time; we'll have fewer works overall in periods where gender imbalance was severe.Given unlimited time, we could go back and select a supplement to fill in those dips. But I don't know for sure that this dataset will be used by anyone outside the current project; it seems unwise to be a perfectionist before knowing the size of the audience.The concept of "gender balance" implies a binaristic simplification of a more complex reality, but I want to avoid simplifying more than necessary. So I'm not going to discard works where gender is marked as 'u' or 'o.' Instead I'll downsample those works to keep them in proportion to the smaller subset.Mostly this is going to affect collections, anonymous works, and works with multiple authors.<jupyter_code>import pandas as pd
import random
title = pd.read_csv('manual_title_subset.tsv', sep = '\t', index_col = 'docid')
numwomen = sum(title.gender == 'f')
nummen = sum(title.gender == 'm')
numother = len(title) - (numwomen + nummen)
print("START: ", numwomen, nummen, numother)
print()

frames = []

for floor in range (1800, 2010, 5):
    fiveyrs = title.loc[(title.firstpub >= floor) & 
                        (title.firstpub < floor + 5) &
                        (title.category.isin({'longfiction', 'shortfiction'}))
                        , :]
    
    numwomen = sum(fiveyrs.gender == 'f')
    nummen = sum(fiveyrs.gender == 'm')
    numother = len(fiveyrs) - (numwomen + nummen)
    if numwomen > nummen:
        asterisk = "*"
    else:
        asterisk = ''
        
    print(floor, numwomen, nummen, numother, asterisk)
    
    women = fiveyrs.loc[fiveyrs.gender == 'f', :]
    men = fiveyrs.loc[fiveyrs.gender == 'm', :]
    others = fiveyrs.loc[(fiveyrs.gender == 'u') | (fiveyrs.gender == 'o'), :]
    
    if numwomen < nummen:
        men = men.sample(n = numwomen)
    elif nummen < numwomen:
        women = women.sample(n = nummen)
    
    reduction = (len(men) + len(women)) / (nummen + numwomen)
    numothers = len(others)
    reduced = int(numothers * reduction)
    
    if reduced > 0:
        others = others.sample(n = reduced)
        newframe = pd.concat([women, men, others], sort = False)
    else:
        newframe = pd.concat([women, men], sort = False)
        
    frames.append(newframe)
    
print()
newtitle = pd.concat(frames)
print(newtitle.shape)
numwomen = sum(newtitle.gender == 'f')
nummen = sum(newtitle.gender == 'm')
numother = len(newtitle) - (numwomen + nummen)
print("END: ", numwomen, nummen, numother)
newtitle.to_csv('gender_balanced_subset.tsv', sep = '\t', index_label = 'docid')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10692232694182645, 'fraction_numerical': 0.0035749106272343193, 'mean_word_length': 3.3659574468085105, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '198330', 'license': 'MIT', 'n_tokens_mistral': 945, 'n_tokens_neox': 883, 'n_words': 332, 'path': 'manuallists/make_gender_balanced_subset.ipynb', 'repo_name': 'tedunderwood/noveltmmeta', 'size': '3077'}