<jupyter_start><jupyter_text>Within a single author's life, is there any pattern to resonance?Because we idly speculated that it might decline ...<jupyter_code>import pandas as pd
import numpy as np
from scipy.stats import pearsonr, zscore
import random
from matplotlib import pyplot as plt
%matplotlib inline
zdata = pd.read_csv('../supp2results/zdata.tsv', sep = '\t', index_col= 'docid')
meta = pd.read_csv('../supplement2/supp2nationalitymeta.tsv', sep = '\t', index_col = 'docid')<jupyter_output><empty_output><jupyter_text>get some extra metadata for author birth and death dates<jupyter_code>work = pd.read_csv('../../noveltmmeta/workmeta.tsv', sep = '\t', index_col = 'docid', low_memory = False)<jupyter_output><empty_output><jupyter_text>I'm going to actually add ```author``` to zdata, but keep ```authordate``` in work, because there can be different authordates for the same author, and it complicates matters to do a join.<jupyter_code>print(zdata.shape)
zdata = zdata.join(meta.author, how = 'inner')
print(zdata.shape)
zdata.columns<jupyter_output><empty_output><jupyter_text>test the average correlation of publication date with resonance across an author's life<jupyter_code>correlations = []
ages = []
resonances = []
errors = 0

grouped = zdata.groupby('author')
for auth, df in grouped:
    rows = df.shape[0]
    if rows > 3:
        r, p = pearsonr(df['resonance_0.05_25'], df['latestcomp'])
        correlations.append(r)
        
        if not np.isnan(r):
            docid = df.index[0]
            if docid not in work.index:
                errors += 1
                continue
                
            authordate = work.loc[docid, 'authordate']
            if pd.isnull(authordate):
                continue
            if len(authordate) > 5 and '-' in authordate:
                try:
                    birth = int(authordate[0 : 4])
                except:
                    continue
            
                idx = random.sample([df.index.tolist()], 1)[0]
                date = df.loc[idx, 'latestcomp'].values[0]
                age = date - birth
                if age > 90:
                    continue
                else:
                    ages.append(age)
                    resonances.append(df.loc[idx, 'resonance_0.05_25'].values[0])        

print("There are ", sum(np.isnan(correlations)), " NaN vals.")       
correlations = np.array(correlations)
correlations = correlations[~np.isnan(correlations)]
print('And ', len(correlations), ' non-NaN.')

print('Raw mean: ', np.nanmean(correlations))
print()
fisher_transformed = np.arctanh(correlations)
themean = np.mean(fisher_transformed)
print('Better estimate: ', np.tanh(themean))<jupyter_output>There are  394  NaN vals.
And  1280  non-NaN.
Raw mean:  -0.503245459878

Better estimate:  -0.656071653109<jupyter_text>An average correlation of -0.65 is bloody strong.It suggests that resonance goes down as you get older.But hey, it's sort of weird to average *r* across a lot of datasets with small *n*. So we also randomly pulled out some works, and recorded author age and resonance in each case.<jupyter_code>plt.scatter(ages, resonances)
plt.show()
pearsonr(ages, resonances)<jupyter_output><empty_output><jupyter_text>InterpretationWe need to be a little cautious, because we don't know how close books are to date of first publication on average. Maybe this is just telling us that reprints lack resonance, because (obvs) they're dated too late.So let's take some sets where we know they're dated to firstpub.<jupyter_code>hypothesis_meta = pd.read_csv('../supplement2/hypothesis_wnationality_meta.tsv', sep = '\t', index_col = 'docid')
reviewset = hypothesis_meta[hypothesis_meta.reviewed].index.tolist() + hypothesis_meta[hypothesis_meta.contrast4reviewed].index.tolist()
len(reviewset)
puredata = zdata.loc[reviewset, :]
correlations = []
ages = []
resonances = []
errors = 0

grouped = puredata.groupby('author')
for auth, df in grouped:
    rows = df.shape[0]
    if rows > 2:
        r, p = pearsonr(df['resonance_0.05_25'], df['latestcomp'])
        correlations.append(r)
        
        if not np.isnan(r):
            docid = df.index[0]
            if docid not in work.index:
                errors += 1
                continue
                
            authordate = work.loc[docid, 'authordate']
            if pd.isnull(authordate):
                continue
            if len(authordate) > 5 and '-' in authordate:
                try:
                    birth = int(authordate[0 : 4])
                except:
                    continue
            
                for idx in df.index:
                    date = df.loc[idx, 'latestcomp']
                    age = date - birth
                    if age > 90:
                        continue
                    else:
                        ages.append(age)
                        resonances.append(df.loc[idx, 'resonance_0.05_25'])       

print("There are ", sum(np.isnan(correlations)), " NaN vals.")       
correlations = np.array(correlations)
correlations = correlations[~np.isnan(correlations)]
print('And ', len(correlations), ' non-NaN.')

print('Raw mean: ', np.nanmean(correlations))
print()
fisher_transformed = np.arctanh(correlations)
themean = np.mean(fisher_transformed)
print('Better estimate: ', np.tanh(themean))
plt.scatter(ages, resonances)
plt.show()
pearsonr(ages, resonances)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11276948590381426, 'fraction_numerical': 0.015662428597751982, 'mean_word_length': 2.6331994645247656, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '404503', 'license': 'MIT', 'n_tokens_mistral': 1781, 'n_tokens_neox': 1638, 'n_words': 433, 'path': 'analysis/author_age.ipynb', 'repo_name': 'tedunderwood/asymmetry', 'size': '5427'}