<jupyter_start><jupyter_text>Initial Analysis of the Fan Fiction DatasetThe dataset can be downloaded here: https://pan.webis.de/clef21/pan21-web/author-identification.htmlThe description of the dataset, also found at the previous link is: ```The train (calibration) and test datasets consists of pairs of (snippets from) two different fanfics, that were obtained drawn from fanfiction.net. Each pair was assigned a unique identifier and we distinguish between same-author pairs and different-authors pairs. Additionally, we offer metadata on the fandom (i.e. thematic category) for each text in the pair (note that fanfic "crossovers" were not included and only single-fandom texts were considered). The fandom distribution in these datasets maximally approximates the (long-tail) distribution of the fandoms in the original dataset. The test dataset is structured in the exact same way, but participants should expect a significant shift in the relation between authors and fandoms.The training dataset comes in two variants: a smaller dataset, particularly suited for symbolic machine learning methods and a large, dataset, suitable for applying data-hungry deep learning algorithms. Participants have to specify which of the two datasets was used to train their model. Models using the small set will be evaluated separately from models using the large set. We encourage participants to try the small dataset as a challenge, though participants can submit separate submissions for either one or both.Both the small and the large dataset come with two newline delimited JSON files each (*.jsonl). The first file contains pairs of texts (each pair has a unique ID) and their fandom labels:{"id": "6cced668-6e51-5212-873c-717f2bc91ce6", "fandoms": ["Fandom 1", "Fandom 2"], "pair": ["Text 1...", "Text 2..."]}{"id": "ae9297e9-2ae5-5e3f-a2ab-ef7c322f2647", "fandoms": ["Fandom 3", "Fandom 4"], "pair": ["Text 3...", "Text 4..."]}...The second file, ending in *_truth.jsonl, contains the ground truth for all pairs. The ground truth is composed of a boolean flag indicating if texts in a pair are from the same author and the numeric author IDs:{"id": "6cced668-6e51-5212-873c-717f2bc91ce6", "same": true, "authors": ["1446633", "1446633"]}{"id": "ae9297e9-2ae5-5e3f-a2ab-ef7c322f2647", "same": false, "authors": ["1535385", "1998978"]}...Data and ground truth are in the same order and can be ingested line-wise in parallel without the need for a reshuffle based on the pair ID. The fandom labels will be given in both the training and testing datasets. The ground truth file will only be available for the training data.``` Key Notes 1. Each data point contains snippets of text from two different fanfics.2. Fanfic "crossovers" were not included (so texts originate from only one fandom)3. "The fandom distribution in these datasets maximally approximates the (long-tail) distributuion of the fandoms in the original dataset" (I'm not sure what they mean here. 4. Participants should expect a significant shift in the relation between authors and fandoms between the train and test set.  What questions do we have? 1. How many fandoms?2. How many authors? 3. What is the distribution of texts/author? fandoms/author? author/fandom?4. In the given pairs, what is the percentage of same author pairs vs different author pairs? 5. Are pairs constructed only with texts from the same (or different) fandoms?6. Is the training/testing set balanced (both the one I created and what should we expect)?7. What is the accuracy on the dataset if we just predict the most common author/score?<jupyter_code>import json 
import pandas as pd
from tqdm.notebook import tqdm
import matplotlib.pyplot as plt
from multiprocessing import Pool
import time
import numpy as np
# First, read in the data 
# Start by getting the truth file
truth = {}
first_id = None
with open('/home/jtyo/Repos/AuthorshipAttribution/data/PAN20/pan20-authorship-verification-training-large/pan20-authorship-verification-training-large-truth.jsonl', 'r') as f:

    lines = f.readlines()

    for line in lines:
        
        line = json.loads(line)
        
        if line['id'] not in truth.keys():
            truth[line['id']] = {}
            
        if first_id is None:
            first_id = line['id']
            
        truth[line['id']]['authors'] = line['authors']
        truth[line['id']]['same'] = line['same']

# Now read the training data - I think all we need is the fandoms
data = None
with open('/home/jtyo/Repos/AuthorshipAttribution/data/PAN20/pan20-authorship-verification-training-large/pan20-authorship-verification-training-large.jsonl', 'r') as f:
    
    for line in tqdm(f):
        
        line = json.loads(line)
        
        if data is None:
            data = {'id': [line['id']],
                    'same': [truth[line['id']]['same']], 
                    'fandom1': [line['fandoms'][0]], 
                    'fandom2': [line['fandoms'][1]],
                    'auth1': [truth[line['id']]['authors'][0]],
                    'auth2': [truth[line['id']]['authors'][1]],
                    'text_len1': [len(line['pair'][0])],
                    'text_len2': [len(line['pair'][1])]
                   }
        else:
            data['id'].append(line['id'])
            data['same'].append(truth[line['id']]['same'])
            data['fandom1'].append(line['fandoms'][0])
            data['fandom2'].append(line['fandoms'][1])
            data['auth1'].append(truth[line['id']]['authors'][0])
            data['auth2'].append(truth[line['id']]['authors'][1])
            data['text_len1'].append(len(line['pair'][0]))
            data['text_len2'].append(len(line['pair'][1]))
        
data = pd.DataFrame(data=data)
# now get all of the data from the dataframe.
data.describe()
data.head()
# how many are same author pairs?
num_same = len(data[data['same'] == True])
print(f'There are {len(data)} pairs, {num_same} of them are same author, which is {num_same/len(data)} percent.')

# how many are same fandom?
num_same_fandom = len(data[data['fandom1'] == data['fandom2']])
print(f'{num_same_fandom} are same fandom, or {num_same_fandom/len(data)} percent.')

# what percent of the same author, (and diff auth pairs) are same fandom?
tmp = data[data['same'] == True]
same_auth_same_fandom = len(tmp[tmp['fandom1'] == tmp['fandom2']])
print(f'Of the same author pairs, {same_auth_same_fandom} of them are same fandom, or {same_auth_same_fandom/num_same} percent.')

tmp = data[data['same'] == False]
diff_auth_same_fandom = len(tmp[tmp['fandom1'] == tmp['fandom2']])
print(f'Of the different author pairs, {diff_auth_same_fandom} of them are same fandom, or {diff_auth_same_fandom/num_same} percent.')

# how many fandoms?
num_fandom1 = len(pd.unique(data['fandom1']))
num_fandom2 = len(pd.unique(data['fandom2']))
print(f'there were {num_fandom1} and {num_fandom2} fandoms.')

# how many authors?
num_auth1 = len(pd.unique(data['auth1']))
num_auth2 = len(pd.unique(data['auth2']))
print(f'there were {num_auth1} and {num_auth2} authors.')

# how many samples per author? (multiprocess for speed)

def get_texts_for_auth(auth):
    return len(data[data['auth1'] == auth])

print('getting all auths')

all_auths = pd.unique(data['auth1'])
start_time = time.time()
texts_per_auth = [] 

with Pool(8) as p:
    texts_per_auth.append(p.map(get_texts_for_auth, all_auths))

print(f'took {time.time() - start_time} seconds')


print(f'avg samples per author: {np.mean(texts_per_auth)}')
print(f'median samples per author: {np.median(texts_per_auth)}')
print(f'std of samples per author: {np.std(texts_per_auth)}')
print(f'min samples per author: {np.min(texts_per_auth)}')
print(f'max samples per author: {np.max(texts_per_auth)}')
# plot a histogram of texts per author
plt.hist(texts_per_auth, bins=10)
# plot a histogram of fandoms per author
# how many samples per author? 

def get_auths_per_fndm(fndm):
    return len(data[data['fandom1'] == fndm])

auth_per_fndm = []
all_fndms = pd.unique(data['fandom1'])
with Pool(8) as p:
    auth_per_fndm.append(p.map(get_auths_per_fndm, all_fndms))

print(f'avg samples per author: {np.mean(auth_per_fndm)}')
print(f'median samples per author: {np.median(auth_per_fndm)}')
print(f'std of samples per author: {np.std(auth_per_fndm)}')
print(f'min samples per author: {np.min(auth_per_fndm)}')
print(f'max samples per author: {np.max(auth_per_fndm)}')
plt.hist(auth_per_fndm, bins=50)
# plot a histogram of authors per fandom
# plot a histogram of fandoms per author
# how many samples per author? 
def get_fndms_per_auth(auth):
    return len(pd.unique(data[data['auth1'] == auth]['fandom1']))

fndm_per_auth = []
with Pool(8) as p:
    fndm_per_auth.append(p.map(get_fndms_per_auth, all_auths))

print(f'avg samples per author: {np.mean(fndm_per_auth)}')
print(f'median samples per author: {np.median(fndm_per_auth)}')
print(f'std of samples per author: {np.std(fndm_per_auth)}')
print(f'min samples per author: {np.min(fndm_per_auth)}')
print(f'max samples per author: {np.max(fndm_per_auth)}')
plt.hist(fndm_per_auth, bins=5)
# What is the accuracy on the dataset if we just predict the most common author/score? - Low AF, not even worth doing<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12483745123537061, 'fraction_numerical': 0.021348071087993064, 'mean_word_length': 4.378205128205129, 'pattern_counts': {'":': 12, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '226082', 'license': 'MIT', 'n_tokens_mistral': 3065, 'n_tokens_neox': 2938, 'n_words': 968, 'path': 'src/AuthorshipAttribution/notebooks/fanfiction_dataset_analysis.ipynb', 'repo_name': 'JacobTyo/PAN21_SAV', 'size': '9228'}