<jupyter_start><jupyter_text>How Random is Random? So the question may arise how random the corpora are that we use to validate our results. We have several ways of generating random corpora, such as the permutation (which we previously used to validate fitting results). This permutation was set up in such a way to control for word length and unigram frequencies, but not bigram frequencies. Here we try to quantify unigram and bigram frequency distribution differences.<jupyter_code>%matplotlib inline

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import random

import compare_bigrams<jupyter_output><empty_output><jupyter_text>Comparing unordered categorical distributions How do we quantify how close two unigram distributions are? Or how close to bigram distributions are?Really what we are comparing is how one group is distributed over a number of unordered categorical (discrete) variables, relative to a second group.So for example, here are the unigram distributions of the original corpus, together with those in a bootstrapped version of the corpus. In bootstrapping, we generate a new corpus by randomly choosing (with replacement) a sentence from the corpus, and add it to our bootstrap corpus. We keep going until our bootstrap corpus has the same number of sentences as the original corpus. Since we work with replacement, we will end up with a slightly different distribution of unigrams and bigrams.<jupyter_code># Read the input file and obtain a list of list of strings, i.e. the list of sentences
INPUT_FILE = "../corpus/cath8.txt"
f = open(INPUT_FILE,'r')
lines = f.readlines()
f.close()
cath8 = [ l.strip().split(" ") for l in lines ]

original_unigrams = compare_bigrams.unigram_counts( cath8 )

# Take a bootstrap sample
bootstrap_corpus = []
for j in range(len(cath8)):
    bootstrap_corpus.append( random.choice(cath8) )
    
bootstrap_unigrams = compare_bigrams.unigram_counts( bootstrap_corpus )
# All we have to do is generate a common set of bigrams
all_ngrams = list({**original_unigrams,**bootstrap_unigrams}.keys())
# ... and then evaluate each according to this common set,
# substituting zero if the bigram in question does not occur
orig_counts    = list(map(lambda x: original_unigrams.get(x,0),all_ngrams))
bootstr_counts = list(map(lambda x: bootstrap_unigrams.get(x,0),all_ngrams))

unigrams = pd.DataFrame({'ngram':all_ngrams,'orig.count':orig_counts,'bootstr.counts':bootstr_counts})
unigrams["total.counts"]=unigrams["orig.count"]+unigrams["bootstr.counts"]
unigrams = unigrams.sort('total.counts',ascending=False)
plt.figure(figsize=(16,6))
x = np.arange(len(all_ngrams))
plt.bar(x+.3,unigrams["orig.count"], width=.3,edgecolor="darkred",color="darkred")
plt.bar(x,unigrams["bootstr.counts"],width=.3,edgecolor="darkblue",color="darkblue")
plt.ylabel("Frequency")
plt.xlabel("Unigram (ordered by # of occurrence")<jupyter_output><empty_output><jupyter_text>Eyeballing it, the distributions seem fairly similar. There are some unigrams where differences are somewhat bigger, but that is to be expected by chance. Similarly, in some of the low-frequency unigrams, the bootstrap corpus simply didn't capture the one or two sentences that contained that unigram, and therefore would miss it altogether. A nice metric to capture similarity of distributions is chi-squared. It looks at the two corpora as columns in a contingency table where the rows are the unigrams. Kind of like this, for illustration, for the highest-frequency unigrams:<jupyter_code>unigrams.iloc[:10][["ngram","orig.count","bootstr.counts"]]<jupyter_output><empty_output><jupyter_text>Now if the two corpora are identically distributed, the two columns should be independent. Note that the columns may sum to different numbers of items, but that will not affect whether they are independent and should not affect our metric of similarity of distribution. The chi-square metric has exactly that property. It computes the marginal sums (the sums of the rows and columns) and computes the expected number of items in each cell. Then, it computes how far the actual number in each cell deviates from the expected, and that results in the chi square measure.In our case we get the following chi-squared:<jupyter_code>comp = compare_bigrams.compare_Ngrams( original_unigrams, bootstrap_unigrams )
print (comp["chisq"])<jupyter_output><empty_output><jupyter_text>However, the chi squared measure is unbounded, it takes arbitrary large values if the number of items in the table increases. It is therefore more elegant to use a "normalised" version, which is Cramer's V.For details: http://en.wikipedia.org/wiki/Cram%C3%A9r%27s_VThe neat property of this quantity is that it is in the range from 0 (meaning identical distributions, in our case) to 1.0 (meaning completely different distributions).<jupyter_code>print (comp["cramer.V"])<jupyter_output><empty_output><jupyter_text>Permuted Corpora Bigram Distributions Ok, we now have the tools to investigate similarity of distributions. In our previous analysis, we used a clever randomisation procedure in which we permuted the words in the corpus, keeping sentence boundaries intact. In this way, we have an identical distribution of unigrams and sentence lengths, but not bigrams. So how far off are the bigrams?Here are some 10k permutations and the corresponding Cramer V values.<jupyter_code>bootstraps = pd.DataFrame.from_csv('interim/bootstrap_corpora_stats.csv')
permutations = pd.DataFrame.from_csv('interim/permutations_stats.csv')
bins = np.linspace(0,.9,500)
plt.figure(figsize=(13,5))
plt.hist(list(permutations["bigrams.cramer.V"]),bins,color="darkblue",edgecolor="darkblue",alpha=.9,label="permuted")
plt.hist(list(bootstraps["bigrams.cramer.V"]),bins,color="darkred",edgecolor="darkred",alpha=.9,label="bootstraps")
plt.xlabel("Bigram distributions dissimilarity (Cramer V)")
plt.legend(loc='upper left')<jupyter_output><empty_output><jupyter_text>As you can see, the permuted corpora have very different bigram distributions, relative to bootstrap samples of similar size. I had this idea we could "cherry-pick" some permutations that have reasonably good bigram distributions, but as you can see there is really no hope we'll ever get even close to the bootstrap samples.  Bigram-generated corpora So here is another try to get randomised corpora for our testing, trying to approach the bigram distributions more closely. The idea is as follows. We first generate a transition matrix from the bigrams observed in the corpus. We also count word boundaries in these bigrams, e.g. ".aiw" is a bigram, indicating the aiw occurs initially with a particular probability.Then, we use this transition matrix to start generating a humongous set of sentences of all sorts of lengths. In this way, we hope to approximate the distribution of bigrams as observed in the real corpus.Then, we generate corpora from this set by choosing for each sentence in the real corpus a sentence from this randomised supercorpus with the same length. In this way, we make sure we have a randomised corpus with the same number of (1) words, (2) sentences, (3) and sentence lengths, but not necessarily the same unigram or bigram distributions.In order to check whether these come close to the real corpus (again, taking the bootstrap samples as our "ground truth"), here are the Cramer V's for the unigram and bigram frequencies.<jupyter_code>bootstraps = pd.DataFrame.from_csv('interim/bootstrap_corpora_stats.csv')
bigramgens = pd.DataFrame.from_csv('interim/bigramgen_corpus_stats.csv')
bins = np.linspace(0,.3,100)
plt.figure()
plt.hist(list(bigramgens["bigrams.cramer.V"]),bins,color="darkblue",alpha=.7,label="bigram-generated")
plt.hist(list(bootstraps["bigrams.cramer.V"]),bins,color="darkred",alpha=.7,label="bootstraps")
plt.xlabel("Bigram distributions dissimilarity (Cramer V)")
plt.legend()
plt.figure()
plt.hist(list(bigramgens["unigrams.cramer.V"]),bins,color="darkblue",alpha=.7,label="bigram-generated")
plt.hist(list(bootstraps["unigrams.cramer.V"]),bins,color="darkred",alpha=.7,label="bootstraps")
plt.xlabel("Unigram distributions dissimilarity (Cramer V)")
plt.legend()<jupyter_output><empty_output><jupyter_text>So this is encouraging. The two distributions still don't completely overlap, but that could be taken care ofby setting appropriate cutoffs.  Number of unique bigrams Is there a difference in number of bigrams?<jupyter_code>original_bigrams = compare_bigrams.bigram_counts( cath8 )
n_cath8_bigrams = len(original_bigrams.keys())

bins = np.linspace(0,1900,500)
plt.figure(figsize=(13,5))
plt.hist(list(permutations["n.unique.bigrams"]),bins,color="darkblue",edgecolor="darkblue",alpha=.7,label="permuted")
plt.hist(list(bootstraps["n.unique.bigrams"]),bins,color="darkred",edgecolor="darkred",alpha=.7,label="bootstraps")
plt.hist(list(bigramgens["n.unique.bigrams"]),bins,color="darkgreen",edgecolor="darkgreen",alpha=.7,label="bigram-generated (length signatures)")
plt.axvline(x=n_cath8_bigrams,lw=3,color="purple",label="Original CATH8")
plt.xlabel("Number of unique bigrams")
plt.legend(loc='upper right')
plt.title("Bigram counts")<jupyter_output><empty_output><jupyter_text>Unigrams and Bigrams Serial Distributions in CATH8 Meaghan observed that the unigrams and bigrams show markedly different distributions for different points in the corpus. Let's see if we can back that observation up with more data.  Unigrams<jupyter_code>counts_per_sentence = [] # unigram counts per sentence

## First build a list of unigrams, in the order in which they first occur in the corpus
unigrams = []
for sentence in cath8:
    for word in sentence:
        if word not in unigrams:
            unigrams.append(word)
    
## Then make the counts
counts = np.zeros( (len(cath8),len(unigrams)) )
for i,sentence in enumerate(cath8):
    for word in sentence:
        if word in unigrams:
            counts[i][unigrams.index(word)]+=1
plt.figure(figsize=(8,6))
plt.imshow(counts.T,aspect='auto',interpolation='none')
plt.xlabel("Sentence # in CATH8")
plt.ylabel("Unigram (numbered by first occurrence)")
plt.colorbar(label="Number of occurrences")
plt.title("Unigram frequencies")<jupyter_output><empty_output><jupyter_text>Bigrams<jupyter_code>counts_per_sentence = [] # bigram counts per sentence

cath8_concat = ["#"]
for sentence in cath8:
    cath8_concat+=sentence+["#"]

## First build a list of bigrams, in the order in which they first occur in the corpus
bigrams = []
for i in range(len(cath8_concat)-1):
    bigr = "%s.%s"%(cath8_concat[i],cath8_concat[i+1])
    if bigr not in bigrams:
        bigrams.append(bigr)

## Then make the counts
counts = np.zeros( (len(cath8),len(bigrams)) )
for i,sentence in enumerate(cath8):
    pad_sentence = ["#"]+sentence+["#"]
    for j in range(len(pad_sentence)-1):
        bigr = "%s.%s"%(pad_sentence[j],pad_sentence[j+1])
        counts[i][bigrams.index(bigr)]+=1
plt.figure(figsize=(8,6))
plt.imshow(counts.T,aspect='auto',interpolation='none')
plt.xlabel("Sentence # in CATH8")
plt.ylabel("Bigram (numbered by first occurrence)")
plt.colorbar(label="Number of occurrences")
plt.title("Bigram frequencies")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0984895879882027, 'fraction_numerical': 0.00795424077218697, 'mean_word_length': 6.228682170542636, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '574575', 'license': 'BSD-3-Clause-Clear', 'n_tokens_mistral': 3576, 'n_tokens_neox': 3288, 'n_words': 1159, 'path': 'surface/How_Random_Is_Random.ipynb', 'repo_name': 'megodoonch/birdsong', 'size': '11189'}