<jupyter_start><jupyter_text>Exploratory analysis of corpus metadataThe corpus for our experiment has been selected to achieve several different goals. We need a substantial subset of books where we are confident both about a) the book's first publication date and b) the author's date of birth. An author's date of birth is pretty commonly recorded in library metadata, but it is hard to be confident that the publication date of any particular copy is also the *first* publication date for the title.To get more confidence about publication date, we aligned HathiTrust metadata to metadata drawn from three sources: manual (title-by-title) [research at the University of Illinois,](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/B4D4H3) metadata manually recorded for the Chicago Novel Corpus, and [the NYPL digitized copy of the US copyright registry.](https://github.com/NYPL/catalog_of_copyright_entries_project) The copyright registry covers more volumes, but is restricted to the period 1923-1969.This evidence is still not infallible.E.g., although we took a book's first appearance in the copyright registry--if earlier than recorded publication date of our library copy--it is still possible that some books we have as post-1923 were first registered before 1923. The copyright registry won't give us the true earliest date for those. However, it did correct library metadata by 6 years or more in about 4% of cases.Another source of error: we've used metadata where only a single author is recorded--but some books are coauthored, or were strongly influenced by an editor, whose identity may or may not be known. We've removed coauthored books that we can identify, but the role of editors will remain murkier. We plan to run two different experiments: 1) a regression experiment that asks whether publication date or DoB is more predictive for particular textual variables, and 2) structural equation modeling on sequences of books written by the same author. For the first (regression) experiment we want to hold nationality constant across time, to avoid creating confounds. For the second experiment, comparisons will take place only within an author's oeuvre, so authors of different nationalities can be used.This means our corpus will contain two overlapping subsets:1. Books by authors of known US nationality.2. Books by authors who have more than three volumes in our corpus, of any nationality.In order to create a corpus evenly distributed across time for topic modeling, we also added3. Books to balance out the timeline, and extend it 20 years in both directions beyond the century (1890-1989) we plan to analyze. (There can otherwise be edge effects near the end of a timeline.)Right now "balance" means "the same number of volumes in each year." But we might reconsider that to mean "the same number of words in each year," and remove some volumes from category (3) to even things out. (And also shrink the corpus to speed modeling a little.)The cells below explore all three sets, visualizing the distribution of publication date and DoB.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
from matplotlib import pyplot as plt
from collections import Counter
corpus = pd.read_csv('metadata/modelcorpus.tsv', sep = '\t', low_memory = False, 
                     dtype = {'docid': object, 'hathi_author': object, 'hathi_title': object,
                            'authordate': object, 'birthyear': float, 'deathyear': float,
                            'chi_date': float, 'ukw_date': float, 'copyright_date': float,
                             'firstpub': int, 'datewrong': float, 'nationality': object,
                             'nonficprob': float, 'distance': float, 'copyright_corpus': bool,
                             'us_national': bool, 'pubdate_known': bool, 'authof3ormore': bool})               
corpus.head()<jupyter_output><empty_output><jupyter_text>1. Meaning of columns:Not all of these columns are likely to be used in our analysis. Some are left over from the process of data cleaning, where we used various kinds of modeling to discover outliers in need of a manual check.|field | meaning||------|---------|| **docid**  | HathiTrust volume ID || **hathi_author** | Author name from HathiTrust, trailing punct trimmed||**hathi_title** | Short title from HathiTrust || **authordate** | string with bio dates, variety of nonstandard formats|| **birthyear**   | year of author's birth, or NaN || **deathyear**  | year of author's death, or NaN || **chi_date**   | if present, we matched vol to Chicago Novel Corpus || **ukw_date**   | if present, we matched vol to Underwood, Kimutis, Witte 2020 || **copyright_date** | if present, we matched vol to US copyright registry || **firstpub** | best estimate for date of first publication || **datewrong** | not error in firstpub, but error in a predictive model used in cleaning || **nationality** | author's, if known || **nonficprob**  | used in cleaning || **distances** | from corpus centroid, used in cleaning || **copyright_corpus** | Boolean, whether firstpub confirmed against registry || **manual_corpus** | Boolean, whether firstpub confirmed by Chicago or UKW || **us_national** | Boolean, whether nationality == 'us' or 'US' || **pubdate_known** | Boolean, whether firstpub has been checked against any source but Hathi || **authof3ormore** | Boolean, whether hathi_author has 3 or more vols in this corpus |  2. Distribution of corpora across timeFirst, the whole corpus.<jupyter_code>plt.figure(figsize=(8, 6), dpi=80)
sns.histplot(corpus.firstpub, binwidth = 5)
plt.show()<jupyter_output><empty_output><jupyter_text>This is unexciting. We have 42,000 volumes distributed evenly between 1870 and 2009, or 300 vols/year for 140 years. US national corpus<jupyter_code>usnational = corpus.loc[corpus.us_national, ['firstpub', 'birthyear']]
usnational.shape<jupyter_output><empty_output><jupyter_text>We have 6,642 volumes where US nationality is confirmed. But note, a smaller number of those have confirmed DoB.<jupyter_code>usnational = usnational.loc[(~pd.isnull(usnational.birthyear)) &
                            (usnational.firstpub >= 1890) &
                            (usnational.firstpub <= 1989), : ]
usnational.shape
sns.histplot(usnational.firstpub, binwidth = 5)<jupyter_output><empty_output><jupyter_text>The curve is a bit bell-shaped, because it was easier to confirm publication dates in the middle period covered by the copyright registry.Date of birth is even more bell-shaped, because of the above distribution and the Central Limit Theorem.<jupyter_code>sns.histplot(usnational.birthyear, binwidth = 5)<jupyter_output><empty_output><jupyter_text>The sequence corpusThis is the subset of books where we have a sequence 3 or more by the same author.<jupyter_code>sequencecorpus = corpus.loc[corpus.authof3ormore, :]
sequencecorpus.shape
sns.histplot(sequencecorpus.firstpub, binwidth = 5)<jupyter_output><empty_output><jupyter_text>We see there that the author-diversity of the corpus tends to increase after 1970. There are fewer books by multiple authors.But don't get too excited about that mass of 22,812 books. The subset where we feel confident about publication year is smaller, and getting the *sequence* of books right is pretty important for this analysis. Also, we would ideally like to have author's DoB for this analysis too.<jupyter_code>sum((sequencecorpus.pubdate_known) &  (~pd.isnull(sequencecorpus.birthyear)))<jupyter_output><empty_output><jupyter_text>How is that subset distributed?<jupyter_code>sns.histplot(sequencecorpus,
    x = "firstpub", hue="pubdate_known",
    multiple="stack",
    palette="Paired",
    edgecolor=".3",
    linewidth=.5)<jupyter_output><empty_output><jupyter_text>As you can see, the corpus with known publication dates is concentrated in the mid-20c. This could be remediated a little by manual confirmation of pubdates on the "shoulders" of that curve. We do have books by multiple authors where DoB is known, or can be known. We just haven't confirmed pub dates.What about DoB distribution?<jupyter_code>knownsubset = sequencecorpus.loc[(sequencecorpus.pubdate_known == True) &
                                (~pd.isnull(sequencecorpus.birthyear)), : ]
sns.histplot(knownsubset.birthyear, binwidth = 5)
age = knownsubset.firstpub - knownsubset.birthyear
knownsubset = knownsubset.assign(age = age)
sns.histplot(knownsubset.age, binwidth = 1)
sum(age <= 30)<jupyter_output><empty_output><jupyter_text>We only have 795 books published in a year when the author was 30 or younger. This will limit our ability to check for a "plasticity period."  What about the distribution of words?I selected an even number of volumes across time. But how even is the distribution of words?<jupyter_code>wordspervol = Counter()

with open('/Users/tunder/work/tomallet/cohort1.txt', encoding = 'utf-8') as f:
    for line in f:
        words = line.split()
        docid = words[0].split('_')[0]
        numwords = len(words) - 2
        wordspervol[docid] += numwords
wordsdf = pd.DataFrame.from_dict(wordspervol, orient = 'index', columns = ['wordcount'])
wordsdf.reset_index(inplace = True)
wordsdf.rename(columns = {'index': 'docid'}, inplace = True)
wordsdf.sample(10)
sum(wordspervol.values())<jupyter_output><empty_output><jupyter_text>We've got a total of two billion words in this corpus.<jupyter_code>wordsdf = wordsdf.merge(corpus.loc[ : , ['docid', 'firstpub']], on = 'docid')
yearsum = wordsdf.groupby('firstpub').sum()
yearsum = yearsum.reset_index()
yearsum
sns.lineplot(data = yearsum, x = 'firstpub', y = 'wordcount')
plt.ylim(0, None)
plt.show()<jupyter_output><empty_output><jupyter_text>That's probably flat enough, but if we wanted to trim the size of the corpus we could do it by removing selected books to flatten the curve a bit.  Preparing texts for LIWC<jupyter_code>textstouse = set(corpus.loc[corpus.pubdate_known, 'docid'].tolist())
thisdoc = []
lastdocid = 'null'

with open('/Users/tunder/work/tomallet/cohort1.txt', encoding = 'utf-8') as f:
    for line in f:
        words = line.split()
        docid = words[0].split('_')[0]
        
        if docid != lastdocid and lastdocid != 'null':
            if lastdocid in textstouse:
                with open('/Users/tunder/work/tomallet/corpustexts/' + lastdocid + '.txt', mode = 'w', encoding = 'utf-8') as f1:
                    f1.write(' '.join(thisdoc))
            thisdoc = []
            
        thisdoc.extend(words[2: ])
        lastdocid = docid<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09035062848502032, 'fraction_numerical': 0.012380682355164919, 'mean_word_length': 4.3907284768211925, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 50, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '328601', 'license': 'MIT', 'n_tokens_mistral': 3093, 'n_tokens_neox': 2867, 'n_words': 1199, 'path': 'metadata/InitialMetadataAnalysis.ipynb', 'repo_name': 'tedunderwood/period-cohort', 'size': '10581'}