<jupyter_start><jupyter_text>Explore forum postsHow many words per post, how many posts per author, that sort of thing.  Intended to help inform the design of our authorship attribution models.  Note that we are including only posts with at least some text (_selftext_) and ignoring posts that are just an image/url.<jupyter_code>import json
lines = open('../data/RS_2011-01', "r").readlines()
posts = []

i = 0
while len(posts) < 1000000 and i < len(lines):
    try:
        i = i + 1
        p = json.loads(lines[i-1].rstrip())
        
        # Keep only posts with at least some text
        if len(p['selftext']) == 0:
            continue
            
        posts.append(p)
    except:
        continue
        
print("Loaded " + str(len(posts)) + " posts.")
print("Sample post:")
posts[9]['selftext']
posts[9].keys()<jupyter_output><empty_output><jupyter_text>Words per post?This might affect our opinions about the size of the sequences we need to work with.<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd

counts = []
for p in posts:
    counts.append(len(p['selftext'].split(" ")))
    
df = pd.DataFrame({"counts": counts})
df['counts'][df['counts'] < 1000].hist()<jupyter_output><empty_output><jupyter_text>How prolific?How many posts from the typical author?<jupyter_code>author_counts = dict()
for p in posts:
    if 'author' not in p:
        continue
        
    if p['author'] == '[deleted]':
        continue
        
    if p['author'] not in author_counts:
        author_counts[p['author']] = 1
    else:
        author_counts[p['author']] = author_counts[p['author']] + 1
        
df_authors = pd.DataFrame({'author': list(author_counts.keys()), 'posts': list(author_counts.values())})
df_authors.posts.hist()<jupyter_output><empty_output><jupyter_text>How many authors?<jupyter_code>print(len(df_authors.author.unique()))<jupyter_output>67811<jupyter_text>How many posts from our top/best-case authors?<jupyter_code>df_authors = df_authors.sort_values(by=['posts'])
df_authors.tail()<jupyter_output><empty_output><jupyter_text>And what do their posts look like?Let's take a per-author look...<jupyter_code>target_author = "SwirlStick"
samples = []
for p in posts:
    if 'author' not in p:
        continue
    
    if p['author'] == target_author:
        samples.append(p['selftext'])
    
samples<jupyter_output><empty_output><jupyter_text>Prepare for trainingStrip down to the essentials - text of the post, author name<jupyter_code>post_author = []
post_content = []
for p in posts:
    if p['author'] != '[deleted]' and author_counts[p['author']] >= 10:
        post_author.append(p['author'])
        post_content.append(p['selftext'])

df_post_and_author = pd.DataFrame({"author":post_author, "text":post_content})
df_post_and_author.to_csv("../data/sample_author_and_post.csv", index=False)
df_post_and_author.head()<jupyter_output><empty_output><jupyter_text>How many samples do we have?<jupyter_code>print(df_post_and_author.shape[0])<jupyter_output>11104<jupyter_text>How many authors in our sample?<jupyter_code>print(len(set(df_post_and_author.author)))<jupyter_output>740<jupyter_text>Words per author?<jupyter_code>import matplotlib.mlab as mlab

post_word_count = df_post_and_author.text.apply(lambda x: len(x.split(" ")))
df_post_and_author['word_count'] = post_word_count
words_per_author = df_post_and_author.groupby(['author'])['word_count'].sum()

num_bins = 20

fig, ax = plt.subplots()

# the histogram of the data
n, bins, patches = ax.hist(words_per_author, num_bins)

# add a 'best fit' line
#y = mlab.normpdf(bins)
#ax.plot(bins, y, '--')
ax.set_xlabel('Word count')
ax.set_ylabel('Authors')
ax.set_title(r'Histogram of word counts')

# Tweak spacing to prevent clipping of ylabel
fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Now, we prepare our data for easy ingestion by FastAI. Labels should be in the first column, but first we'll replace our author names with a numeric code.<jupyter_code>keys = list(set(df_post_and_author.author))
values = list(range(0, len(keys)))
author_to_label = dict(zip(keys, values))
df_post_and_author['label'] = df_post_and_author['author'].apply(lambda x: author_to_label[x])
df_post_and_author.head()                                                               
df_fastaiformat = pd.DataFrame({'label':df_post_and_author.label, 'text':df_post_and_author.text})
df_fastaiformat.head()
from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(df_fastaiformat.text,df_fastaiformat.label,stratify=df_fastaiformat.label)
df_fai_train = pd.DataFrame({'label' : y_train, 'text' : x_train})
df_fai_test = pd.DataFrame({'label' : y_test, 'text' : x_test})
df_fai_train.head()
df_fai_train.to_csv('../data/df_authorship_fastai_train.csv')
df_fai_test.to_csv('../data/df_authorship_fastai_test.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13078797725426483, 'fraction_numerical': 0.008935824532900082, 'mean_word_length': 4.919471153846154, 'pattern_counts': {'":': 3, '<': 45, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '293996', 'license': 'Apache-2.0', 'n_tokens_mistral': 1775, 'n_tokens_neox': 1721, 'n_words': 361, 'path': 'src/explore_the_data.ipynb', 'repo_name': 'EdwardDixon/authorship', 'size': '4924'}