<commit_before>#!/usr/bin/env python


'''


vbox.py: contains all infrastructure for storing and processing the


CLASSES

Word: this construct replaces the dictionary entries from the previous script.
you initialize a word by passing in a string

    attributes:

    name: the word itself

    freq: the raw frequency of that word within its dictionary

    sub1: dictionary of words that follow this word

    sub2: dicionary of words that come two after this word

    norm: normalized frequency of the word (chance of occurring)

    sig: the significance score of the word, which is how much the frequency in this dictionary
    exceeds baseline frequency

    methods:

    There's a print method and there are methods to set each of the attributes

Voice: represents data from a given corpus. initialize it by passing in a dictionary

    attributes:

    D: the dictionary constructed from the source on the first pass

    activeD: a pruned version of D containing only words that occur a certain number of times

    VoiceSettings: determines the pruning criterion for this voice and the way in which its words get ranked
    (i.e., by normalized frequency, raw frequency or significance score)

    methods:

    restrict(dictionary,list): given a list of words, takes a dictionary and returns a dictionary with only
    matching words

    atLeast(dictionary,int): finds all the words in a dictionary that pass a frequency criterion, passes this
    list to restrict()


Voicebox: a collection of Voice objects and their associated weights

    attributes:
    voices: a dictionary of the various Voice objects in Voicebox

    methods:
    addVoice: adds a single Voice
    addTranscript: given a transcript where speakers are assigned to lines by SPEAKER: [line], breaks that script
    down into constituent speakers and adds a Voice for each
    makeDict: makes a dict and sets raw frequencies
    normalizeFreqs: processes a dict's frequencies into rates of occurrence
    sigScores: processes a dict's rates of occurrence into significance scores

Context: constructed using a dictionary and a list of words (the context in which we're typing),
this represents all the influences on the what list of words the program will suggest


NOTE ON SIGNIFICANCE SCORE:
I introduced this to avoid a problem with large dictionaries: the top suggestions tend to be extremely common words, even given
specific contexts. If the previous two words are 'jack the' then one continuation you'd like to have is 'ripper.'
but if you go strictly by rate of occurrence, then phrases like 'jack, the thing' [is] and [i gave] 'jack the keys'
are going to overwhelm 'jack the ripper'

To solve this, significance score tries to index how much the context in question boosts a word's frequency
over baseline. This may be what you would call a Bayesian approach:

sig = P(w|context)/P(w)

So if 'ripper' occurs 1 percent of the time after 'jack the' but only .0000001 percent of the time overall,
it gets a huge sigscore. meanwhile, 'thing' might occur 2 percent of the time after 'jack the' but .05 percent
of the time overall, giving it a more modest sigscore

To avoid giving huge sigscores to sequences that occur just once or twice, I've mulitplied the sigscore by
the log of the frequency as a way of discounting those little blips we should probably make the base of that
log function into a parameter so we can adjust how much the program privileges sequences with high frequencies.


'''


__author__ = 'jamiebrew'


###############################

# first one is weight to baseline dict, then to 1-back, then 2-back, then conjunction
default_wts = [0,0,1]
default_VSets = {'criterion':3,'vision':2,'ranktype':'norm','weights':default_wts,'loveOfSize':10}
default_VBSets = {'num_opts': 20,'direction':'forward','vision':2,
                 'showvalue':True,'showsource':True,'proportional_src':False}


import string
import operator
import os<commit_msg>updated header<commit_after>#!/usr/bin/env python


'''


vbox.py: contains all infrastructure for storing and processing the

4/3/16: Some of the documentation, especially in the header, might be out-of-date. Please let me know if you find something glaring!

CLASSES

Word: this construct replaces the dictionary entries from the previous script.
you initialize a word by passing in a string

    attributes:

    name: the word itself

    freq: the raw frequency of that word within its dictionary

    sub1: dictionary of words that follow this word

    sub2: dicionary of words that come two after this word

    norm: normalized frequency of the word (chance of occurring)

    sig: the significance score of the word, which is how much the frequency in this dictionary
    exceeds baseline frequency

    methods:

    There's a print method and there are methods to set each of the attributes
    
    NOTE ON SIGNIFICANCE SCORE:
    I introduced this to avoid a problem with large dictionaries: the top suggestions tend to be extremely common words, even given
    specific contexts. If the previous two words are 'jack the' then one continuation you'd like to have is 'ripper.'
    but if you go strictly by rate of occurrence, then phrases like 'jack, the thing' [is] and [i gave] 'jack the keys'
    are going to overwhelm 'jack the ripper'
    
    To solve this, significance score tries to index how much the context in question boosts a word's frequency
    over baseline. This may be what you would call a Bayesian approach:
    
    sig = P(w|context)/P(w)
    
    So if 'ripper' occurs 1 percent of the time after 'jack the' but only .0000001 percent of the time overall,
    it gets a huge sigscore. meanwhile, 'thing' might occur 2 percent of the time after 'jack the' but .05 percent
    of the time overall, giving it a more modest sigscore
    
    To avoid giving huge sigscores to sequences that occur just once or twice, I've mulitplied the sigscore by
    the log of the frequency as a way of discounting those little blips we should probably make the base of that
    log function into a parameter so we can adjust how much the program privileges sequences with high frequencies.

Voice: represents data from a given corpus. initialize it by passing in a dictionary

    attributes:

    D: the dictionary constructed from the source on the first pass

    activeD: a pruned version of D containing only words that occur a certain number of times

    VoiceSettings: determines the pruning criterion for this voice and the way in which its words get ranked
    (i.e., by normalized frequency, raw frequency or significance score)

    methods:

    restrict(dictionary,list): given a list of words, takes a dictionary and returns a dictionary with only
    matching words

    atLeast(dictionary,int): finds all the words in a dictionary that pass a frequency criterion, passes this
    list to restrict()


Voicebox: a collection of Voice objects and their associated weights

    attributes:
    voices: a dictionary of the various Voice objects in Voicebox

    methods:
    addVoice: adds a single Voice
    addTranscript: given a transcript where speakers are assigned to lines by SPEAKER: [line], breaks that script
    down into constituent speakers and adds a Voice for each
    makeDict: makes a dict and sets raw frequencies
    normalizeFreqs: processes a dict's frequencies into rates of occurrence
    sigScores: processes a dict's rates of occurrence into significance scores

Context: constructed using a dictionary and a list of words (the context in which we're typing),
this represents all the influences on the what list of words the program will suggest

TODO: Document the functions in this object.





'''


__author__ = 'jamiebrew'


###############################

# first one is weight to baseline dict, then to 1-back, then 2-back, then conjunction
default_wts = [0,0,1]
default_VSets = {'criterion':3,'vision':2,'ranktype':'norm','weights':default_wts,'loveOfSize':10}
default_VBSets = {'num_opts': 20,'direction':'forward','vision':2,
                 'showvalue':True,'showsource':True,'proportional_src':False}


import string
import operator
import os
{'fraction_non_alphanumeric': 0.05566430144360166, 'fraction_numerical': 0.0066063126988010765, 'mean_word_length': 3.951544518473652, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6968450', 'n_tokens_mistral': 2129, 'n_tokens_neox': 1972, 'n_words': 1158}