<jupyter_start><jupyter_text>Sample generator for all Root Q dataThis notebook is for creating the random sample .txt files for the experiment.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib as mpl<jupyter_output><empty_output><jupyter_text>Contents 1. [Look at the distribution of factors](Look-at-the-distribution-of-factors) 2. [Constrain dataset to stimuli set](Constrain-dataset-to-stimuli-set) 3. [Figuring out the distribution of factors per list](Figuring-out-the-distribution-of-factors-per-list) 4. [Add in the paraphrases](Add-in-the-paraphrases)<jupyter_code># import the database file from the TGrep2 searching
df = pd.read_csv("../results/swbd.tab", sep='\t', engine='python')
# This makes the display show more info
pd.set_option('display.max_rows', None)
pd.set_option('display.max_colwidth', None)<jupyter_output><empty_output><jupyter_text>Look at the distribution of factors<jupyter_code>df.groupby(["QuestionType"])["QuestionType"].count()
df.pivot_table(index=['QuestionType'], values="Question", aggfunc=len).groupby(["QuestionType"]).Question.transform(lambda x: x/len(df)*100).reset_index()
# combine adj + embadj + subject together as "adjunct"
(.075988 + 0.237082 + 0.137857)*100
df[df['QuestionType'] == 'root'].groupby(["Wh"])["Wh"].count()<jupyter_output><empty_output><jupyter_text>Constrain dataset to stimuli setfor experimental mock-upFirst we have to remove the questions that we don;t want to include:1. non-embedded or root questions2. no degree questions3. no identity questions4. generally only monomorphemic wh-phrases5. only what-, when-, and why-questions  Note, the original step was below, but the DB has changed during fine-tuining the EQ DB, so instead of taking the step below, skip to the column<jupyter_code># critical = df[(df['QuestionType'] == 'root') # only root questions
#               & 
#               (df['DegreeQ'] == 'no' ) # no degree questions
#               &
#               (df['IdentityQ'] == "no") # no identity questions
#               &
#               (df['Wh'].isin(['how','How','where','Where','who','Who','what','What','why','Why','when','When'])) # just these wh-words
#               &
#               (df['WhPhaseType'] == 'monomorphemic') # monomorphic wh only (might get anything not go by degQ)
#              ]
# this was 995 originally, but is now 901
# len(critical)<jupyter_output><empty_output><jupyter_text>Read in the dataset from the experiment, and use the TGrepIDs there to fiter the corpus DB here (3/9/21)<jupyter_code>exp = pd.read_csv("../../analysis/03_experiment/data/exp03_main-merged.csv")
# get the ids from the experimental files
exp_ids = exp.tgrep_id
critical = df[df["Item_ID"].isin(set(exp_ids))]
len(critical)<jupyter_output><empty_output><jupyter_text>Figuring out the distribution of factors per list<jupyter_code># Number of lists (30 items per list)
len(critical)/30
995 - 33*30<jupyter_output><empty_output><jupyter_text>33 lists of 30, 1 list of 35<jupyter_code>critical.pivot_table(index=['ModalPresent'], values="Question", aggfunc=len).groupby(["ModalPresent"]).Question.transform(lambda x: x/len(critical)).reset_index()
critical.pivot_table(index=['Wh'], values="Question", aggfunc=len).groupby(["Wh"]).Question.transform(lambda x: x/len(critical)).reset_index()
critical.pivot_table(index=['Wh','ModalPresent'], values="Question", aggfunc=len).groupby(["Wh"]).Question.transform(lambda x: x/len(critical)*100).reset_index()<jupyter_output><empty_output><jupyter_text>Add in the paraphrasesthis should take as input the entire constrained dataframe from the above section, and then generate the paraphrasesFor Who questions: Who is a person...? / Who is some person...? / "Who is every person..." / "Who is the person..."<jupyter_code># read in df with contexts
cntxts = pd.read_csv("swbd_contexts.csv")
cntxts = cntxts.drop(columns="FollowingContext")
# get the indixes from critical
crit_index = critical.Item_ID<jupyter_output><empty_output><jupyter_text>Merge back in Wh and ModalPresent colums<jupyter_code>df_WhMod = critical[["Item_ID","Wh","ModalPresent","Question"]].rename(columns={"Item_ID": "TGrepID"})
# subset to the items that are just the ones filtered in the previos section

# otherwise, if using the database file with contexts directly in there, then this step
# is not necessary
df_valid = cntxts[cntxts["TGrepID"].isin(set(crit_index))]
len(df_valid)
df_valid.columns
df_valid = df_valid.merge(df_WhMod, how = 'inner', indicator=False)
len(df_valid)
df_valid.pivot_table(index=['Wh'], values="EntireSentence", aggfunc=len).groupby(["Wh"]).EntireSentence.transform(lambda x: x/len(df_valid)).reset_index()
who = df_valid[df_valid["Wh"] == "who"]
where = df_valid[df_valid["Wh"] == "where"]
how = df_valid[df_valid["Wh"] == "how"]
when = df_valid[df_valid["Wh"] == "when"]
why = df_valid[df_valid["Wh"] == "why"]
what = df_valid[df_valid["Wh"] == "what"]
who["AResponse"] = "Who is a person...?"
# who["SomeResponse"] = "Who is some person...?"
who["AllResponse"] = "Who is every person...?"
who["TheResponse"] = "Who is the person...?"


where["AResponse"] = "What is a place...?"
# where["SomeResponse"] = "What is some place...?"
where["AllResponse"] = "What is every place...?"
where["TheResponse"] = "What is the place...?"


how["AResponse"] = "What is a way...?"
# how["SomeResponse"] = "What is some way...?"
how["AllResponse"] = "What is every way...?"
how["TheResponse"] = "What is the way...?"

when["AResponse"] = "What is a time...?"
# when["SomeResponse"] = "What is some time...?"
when["AllResponse"] = "What is every time...?"
when["TheResponse"] = "What is the time...?"


why["AResponse"] = "What is a reason...?"
# why["SomeResponse"] = "What is some reason...?"
why["AllResponse"] = "What is every reason...?"
why["TheResponse"] = "What is the reason...?"


what["AResponse"] = "What is a thing...?"
# what["SomeResponse"] = "What is some thing...?"
what["AllResponse"] = "What is every thing...?"
what["TheResponse"] = "What is the thing...?"
df_final = pd.concat([who,where,how,why,when,what])
len(df_final)
df_final.pivot_table(index=['Wh'], values="EntireSentence", aggfunc=len).groupby(["Wh"]).EntireSentence.transform(lambda x: x/len(df_final)).reset_index()
df_final.pivot_table(index=['ModalPresent'], values="EntireSentence", aggfunc=len).groupby(["ModalPresent"]).EntireSentence.transform(lambda x: x/len(df_final)).reset_index()<jupyter_output><empty_output><jupyter_text>Split EntireSentence on Question<jupyter_code># split EntireSentence
df_final["Matrix"] = df_final.apply(lambda x: x['EntireSentence'].replace(x['Question'],"").strip(),axis=1)
# add it to preceeding context
df_final["PreceedingContext"] = [' '.join([str(x), str(y)]) for x, y in zip(df_final['PreceedingContext'], df_final['Matrix'])]
df_final = df_final.rename(columns={"TGrepID": "tgrep_id"})
df_final.columns
lm = exp[["workerid","tgrep_id","paraphrase","rating"]].merge(df_final, how = 'inner', indicator=False)
lm = lm[["workerid","tgrep_id","paraphrase","rating","Question","PreceedingContext","Wh","ModalPresent"]]
lm.to_csv("../../analysis/03_experiment/data/lm_data.csv")<jupyter_output><empty_output><jupyter_text>Controls<jupyter_code>controls = pd.read_csv("../../experiments/clean_corpus/controls.csv")
controls["Wh"] = "none"
controls["ModalPresent"] = "no"
controls = controls[["TGrepID","EntireSentence","PreceedingContext","Wh","ModalPresent","AResponse","AllResponse","TheResponse"]]<jupyter_output><empty_output><jupyter_text>Create randomly sampled files  TEST SAMPLE<jupyter_code>len(df_final)/32
# 32 lists of 30, 1 list of 35
30*32 +35<jupyter_output><empty_output><jupyter_text>For Modal Balancing:- Lists 1-29: 4    - List 1-15: 4    - List 16-29: 4- List 30-32: 1- List 33: 2<jupyter_code>df_final.groupby(["ModalPresent"])["ModalPresent"].count()<jupyter_output><empty_output><jupyter_text>Modals<jupyter_code>121/33
# modal leftovers
121 - 4*29<jupyter_output><empty_output><jupyter_text>For Wh Balancing<jupyter_code>df_final.groupby(["Wh"])["Wh"].count()<jupyter_output><empty_output><jupyter_text>When:- List 1-15: 1  Who:- List 1-15: 2- List 15-21: 2- List 22-33: 1<jupyter_code>54/33
21*2 + 12<jupyter_output><empty_output><jupyter_text>Why:- List 1-15: 2- List 16-21: 2- List 22-27: 2- List 28-33: 1<jupyter_code>60/33
27*2 + 6<jupyter_output><empty_output><jupyter_text>Where:- List 1-15: 3- List 16-31: 3- List 32-33: 2<jupyter_code>97/33
31*3 + 2*2<jupyter_output><empty_output><jupyter_text>How:- List 1-15: 6- List 16-27: 6- List 28-31: 4- List 32-33: 3<jupyter_code>27*6 + 4*4 + 2*3<jupyter_output><empty_output><jupyter_text>What: - Lists 1-15: 16- List 16-29: 18**- Lists 30-32: 16**- List 33: 15**<jupyter_code>29*18+3*16+1*15
15*16 + 6*17 + 6*18 + 2*21 + 2*21 + 1*23 + 1*28
585-557<jupyter_output><empty_output><jupyter_text>Lists 1-15 (30):    1 when    2 who    2 why    3 where    6 how    16 whatList 16-21 (30):    0 when    2 who    2 why    3 where    6 how    17 whatList 22-27 (30):    0 when    1 who    2 why    3 where    6 how    18 whatList 28-29 (30):    0 when    1 who    1 why    3 where    4 how    21 whatList 30-31 (30):    0 when    1 who    1 why    3 where    4 how    21 whatList 32 (30):    0 when    1 who    1 why    2 where    3 how    23 whatList 33 (35):    1 who    1 why    2 where    3 how    28 what<jupyter_code>df_final = pd.concat([who,where,how,why,when,what])
for n in range(1,16):
    mod_sample = df_final[df_final["ModalPresent"] == "yes"].sample(4)

    i = len(mod_sample[mod_sample["Wh"] == "why"])
    j = len(mod_sample[mod_sample["Wh"] == "when"])
    k = len(mod_sample[mod_sample["Wh"] == "what"])
    l = len(mod_sample[mod_sample["Wh"] == "how"])
    m = len(mod_sample[mod_sample["Wh"] == "who"])
    o = len(mod_sample[mod_sample["Wh"] == "where"])
    df_final = df_final.drop(mod_sample.index)

    when_sample = df_final[df_final["Wh"] == "when"].sample(1-j)
    df_final = df_final.drop(when_sample.index)  

    who_sample = df_final[
        (df_final["Wh"] == "who") &
        (df_final["ModalPresent"] == "no")
    ].sample(2-m)
    df_final = df_final.drop(who_sample.index)    
    
    why_sample = df_final[
        (df_final["Wh"] == "why") &
        (df_final["ModalPresent"] == "no")
    ].sample(2-i)
    df_final = df_final.drop(why_sample.index)
    
    where_sample = df_final[
        (df_final["Wh"] == "where") &
        (df_final["ModalPresent"] == "no")
    ].sample(3-o)
    df_final = df_final.drop(where_sample.index)

    how_sample = df_final[
        (df_final["Wh"] == "how") &
        (df_final["ModalPresent"] == "no")
    ].sample(6-l)
    df_final = df_final.drop(how_sample.index)    
    
    what_sample = df_final[
        (df_final["Wh"] == "what") &
        (df_final["ModalPresent"] == "no")
    ].sample(16-k)
    df_final = df_final.drop(what_sample.index)
    
    total = pd.concat([mod_sample,why_sample,when_sample,what_sample,how_sample,who_sample,where_sample,controls])

    # save to file
    filename = f"../../experiments/clean_corpus/03_experiment/corpus_{n}.txt".format(n=n)
    total.to_csv(filename,header=True,sep="\t",index=False)
len(df_final)
df_final.groupby(["Wh"])["Wh"].count()<jupyter_output><empty_output><jupyter_text>who9/2 = 4.5who9/2 = 3.5how49/2 = 24.5<jupyter_code>df_final.groupby(["ModalPresent"])["ModalPresent"].count()
13/2<jupyter_output><empty_output><jupyter_text>Second iteration<jupyter_code>for n in range(16,22):
    mod_sample = df_final[df_final["ModalPresent"] == "yes"].sample(4)

    i = len(mod_sample[mod_sample["Wh"] == "why"])
    k = len(mod_sample[mod_sample["Wh"] == "what"])
    l = len(mod_sample[mod_sample["Wh"] == "how"])
    m = len(mod_sample[mod_sample["Wh"] == "who"])
    o = len(mod_sample[mod_sample["Wh"] == "where"])
    df_final = df_final.drop(mod_sample.index)

    who_sample = df_final[
        (df_final["Wh"] == "who") &
        (df_final["ModalPresent"] == "no")
    ].sample(2-m)
    df_final = df_final.drop(who_sample.index)    
    
    why_sample = df_final[
        (df_final["Wh"] == "why") &
        (df_final["ModalPresent"] == "no")
    ].sample(2-i)
    df_final = df_final.drop(why_sample.index)
    
    where_sample = df_final[
        (df_final["Wh"] == "where") &
        (df_final["ModalPresent"] == "no")
    ].sample(3-o)
    df_final = df_final.drop(where_sample.index)

    how_sample = df_final[
        (df_final["Wh"] == "how") &
        (df_final["ModalPresent"] == "no")
    ].sample(6-l)
    df_final = df_final.drop(how_sample.index)    
    
    what_sample = df_final[
        (df_final["Wh"] == "what") &
        (df_final["ModalPresent"] == "no")
    ].sample(17-k)
    df_final = df_final.drop(what_sample.index)
    
    total = pd.concat([mod_sample,why_sample,what_sample,how_sample,who_sample,where_sample,controls])

    # save to file
    filename = f"../../experiments/clean_corpus/03_experiment/corpus_{n}.txt".format(n=n)
    total.to_csv(filename,header=True,sep="\t",index=False)
23+4+3
len(df_final)
df_final.groupby(["ModalPresent"])["ModalPresent"].count()
995/30
df_final.groupby(["Wh"])["Wh"].count()
for n in range(22,28):
    mod_sample = df_final[df_final["ModalPresent"] == "yes"].sample(4)

    i = len(mod_sample[mod_sample["Wh"] == "why"])
    k = len(mod_sample[mod_sample["Wh"] == "what"])
    l = len(mod_sample[mod_sample["Wh"] == "how"])
    m = len(mod_sample[mod_sample["Wh"] == "who"])
    o = len(mod_sample[mod_sample["Wh"] == "where"])
    df_final = df_final.drop(mod_sample.index)

    who_sample = df_final[
        (df_final["Wh"] == "who") &
        (df_final["ModalPresent"] == "no")
    ].sample(1-m)
    df_final = df_final.drop(who_sample.index)    
    
    why_sample = df_final[
        (df_final["Wh"] == "why") &
        (df_final["ModalPresent"] == "no")
    ].sample(2-i)
    df_final = df_final.drop(why_sample.index)
    
    where_sample = df_final[
        (df_final["Wh"] == "where") &
        (df_final["ModalPresent"] == "no")
    ].sample(3-o)
    df_final = df_final.drop(where_sample.index)

    how_sample = df_final[
        (df_final["Wh"] == "how") &
        (df_final["ModalPresent"] == "no")
    ].sample(6-l)
    df_final = df_final.drop(how_sample.index)    
    
    what_sample = df_final[
        (df_final["Wh"] == "what") &
        (df_final["ModalPresent"] == "no")
    ].sample(18-k)
    df_final = df_final.drop(what_sample.index)
    
    total = pd.concat([mod_sample,why_sample,what_sample,how_sample,who_sample,where_sample,controls])

    # save to file
    filename = f"../../experiments/clean_corpus/03_experiment/corpus_{n}.txt".format(n=n)
    total.to_csv(filename,header=True,sep="\t",index=False)
len(d_final)
for n in range(28,30):
    mod_sample = df_final[df_final["ModalPresent"] == "yes"].sample(4)

    i = len(mod_sample[mod_sample["Wh"] == "why"])
    k = len(mod_sample[mod_sample["Wh"] == "what"])
    l = len(mod_sample[mod_sample["Wh"] == "how"])
    m = len(mod_sample[mod_sample["Wh"] == "who"])
    o = len(mod_sample[mod_sample["Wh"] == "where"])
    df_final = df_final.drop(mod_sample.index)

    who_sample = df_final[
        (df_final["Wh"] == "who") &
        (df_final["ModalPresent"] == "no")
    ].sample(1-m)
    df_final = df_final.drop(who_sample.index)    
    
    why_sample = df_final[
        (df_final["Wh"] == "why") &
        (df_final["ModalPresent"] == "no")
    ].sample(1-i)
    df_final = df_final.drop(why_sample.index)
    
    where_sample = df_final[
        (df_final["Wh"] == "where") &
        (df_final["ModalPresent"] == "no")
    ].sample(3-o)
    df_final = df_final.drop(where_sample.index)

    how_sample = df_final[
        (df_final["Wh"] == "how") &
        (df_final["ModalPresent"] == "no")
    ].sample(4-l)
    df_final = df_final.drop(how_sample.index)    
    
    what_sample = df_final[
        (df_final["Wh"] == "what") &
        (df_final["ModalPresent"] == "no")
    ].sample(21-k)
    df_final = df_final.drop(what_sample.index)
    
    total = pd.concat([mod_sample,why_sample,what_sample,how_sample,who_sample,where_sample,controls])

    # save to file
    filename = f"../../experiments/clean_corpus/03_experiment/corpus_{n}.txt".format(n=n)
    total.to_csv(filename,header=True,sep="\t",index=False)
len(d_final)
for n in range(30,32):
    mod_sample = df_final[df_final["ModalPresent"] == "yes"].sample(1)

    i = len(mod_sample[mod_sample["Wh"] == "why"])
    k = len(mod_sample[mod_sample["Wh"] == "what"])
    l = len(mod_sample[mod_sample["Wh"] == "how"])
    m = len(mod_sample[mod_sample["Wh"] == "who"])
    o = len(mod_sample[mod_sample["Wh"] == "where"])
    df_final = df_final.drop(mod_sample.index)

    who_sample = df_final[
        (df_final["Wh"] == "who") &
        (df_final["ModalPresent"] == "no")
    ].sample(1-m)
    df_final = df_final.drop(who_sample.index)    
    
    why_sample = df_final[
        (df_final["Wh"] == "why") &
        (df_final["ModalPresent"] == "no")
    ].sample(1-i)
    df_final = df_final.drop(why_sample.index)
    
    where_sample = df_final[
        (df_final["Wh"] == "where") &
        (df_final["ModalPresent"] == "no")
    ].sample(3-o)
    df_final = df_final.drop(where_sample.index)

    how_sample = df_final[
        (df_final["Wh"] == "how") &
        (df_final["ModalPresent"] == "no")
    ].sample(4-l)
    df_final = df_final.drop(how_sample.index)    
    
    what_sample = df_final[
        (df_final["Wh"] == "what") &
        (df_final["ModalPresent"] == "no")
    ].sample(21-k)
    df_final = df_final.drop(what_sample.index)
    
    total = pd.concat([mod_sample,why_sample,what_sample,how_sample,who_sample,where_sample,controls])

    # save to file
    filename = f"../../experiments/clean_corpus/03_experiment/corpus_{n}.txt".format(n=n)
    total.to_csv(filename,header=True,sep="\t",index=False)
len(df_final)
for n in range(32,33):
    mod_sample = df_final[df_final["ModalPresent"] == "yes"].sample(1)

    i = len(mod_sample[mod_sample["Wh"] == "why"])
    k = len(mod_sample[mod_sample["Wh"] == "what"])
    l = len(mod_sample[mod_sample["Wh"] == "how"])
    m = len(mod_sample[mod_sample["Wh"] == "who"])
    o = len(mod_sample[mod_sample["Wh"] == "where"])
    df_final = df_final.drop(mod_sample.index)

    who_sample = df_final[
        (df_final["Wh"] == "who") &
        (df_final["ModalPresent"] == "no")
    ].sample(1-m)
    df_final = df_final.drop(who_sample.index)    
    
    why_sample = df_final[
        (df_final["Wh"] == "why") &
        (df_final["ModalPresent"] == "no")
    ].sample(1-i)
    df_final = df_final.drop(why_sample.index)
    
    where_sample = df_final[
        (df_final["Wh"] == "where") &
        (df_final["ModalPresent"] == "no")
    ].sample(2-o)
    df_final = df_final.drop(where_sample.index)

    how_sample = df_final[
        (df_final["Wh"] == "how") &
        (df_final["ModalPresent"] == "no")
    ].sample(3-l)
    df_final = df_final.drop(how_sample.index)    
    
    what_sample = df_final[
        (df_final["Wh"] == "what") &
        (df_final["ModalPresent"] == "no")
    ].sample(23-k)
    df_final = df_final.drop(what_sample.index)
    
    total = pd.concat([mod_sample,why_sample,what_sample,how_sample,who_sample,where_sample,controls])

    # save to file
    filename = f"../../experiments/clean_corpus/03_experiment/corpus_{n}.txt".format(n=n)
    total.to_csv(filename,header=True,sep="\t",index=False)
len(df_final)
last = pd.concat([df_final,controls])
last.to_csv("../../experiments/clean_corpus/03_experiment/corpus_33.txt",header=True,sep="\t",index=False)<jupyter_output><empty_output><jupyter_text>Pilot Samples  Pilot1<jupyter_code>df_final.pivot_table(index=['Wh'], values="EntireSentence", aggfunc=len).groupby(["Wh"]).EntireSentence.transform(lambda x: x/99).reset_index()
1+5+1+1+1+1
995/10
mod_sample = df_final[df_final["ModalPresent"] == "yes"].sample(1)

i = len(mod_sample[mod_sample["Wh"] == "why"])
j = len(mod_sample[mod_sample["Wh"] == "when"])
k = len(mod_sample[mod_sample["Wh"] == "what"])
l = len(mod_sample[mod_sample["Wh"] == "how"])
m = len(mod_sample[mod_sample["Wh"] == "who"])
n = len(mod_sample[mod_sample["Wh"] == "where"])
df_final = df_final.drop(mod_sample.index)

why_sample = df_final[
    (df_final["Wh"] == "why") &
    (df_final["ModalPresent"] == "no")
                     ].sample(1-i)
df_final = df_final.drop(why_sample.index)

when_sample = df_final[
    (df_final["Wh"] == "when") &
    (df_final["ModalPresent"] == "no")].sample(1-j)
df_final = df_final.drop(when_sample.index)

what_sample = df_final[
    (df_final["Wh"] == "what") &
    (df_final["ModalPresent"] == "no")].sample(5-k)
df_final = df_final.drop(what_sample.index)

how_sample = df_final[
    (df_final["Wh"] == "how") &
    (df_final["ModalPresent"] == "no")
                     ].sample(1-l)
df_final = df_final.drop(how_sample.index)

who_sample = df_final[
    (df_final["Wh"] == "who") &
    (df_final["ModalPresent"] == "no")].sample(1-m)
df_final = df_final.drop(who_sample.index)

where_sample = df_final[
    (df_final["Wh"] == "where") &
    (df_final["ModalPresent"] == "no")].sample(1-n)
df_final = df_final.drop(where_sample.index)

total = pd.concat([mod_sample,why_sample,when_sample,what_sample,how_sample,who_sample,where_sample,controls])

# save to file

total
total.to_csv("../../experiments/clean_corpus/03_experiment/pilot1.txt",header=True,sep="\t",index=False)
# since df_final changed, it's a different sample set now.
# pilot = df_final.sample(10, random_state=333)
pilot = df_final.sample(10, random_state=666)
cont = controls.drop(columns=["Wh","ModalPresent"])
pilott = pd.concat([pilot,cont])
len(pilott)
pilott
pilott.to_csv("../../experiments/clean_corpus/03_experiment/pilot1.txt",header=True,sep="\t",index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1797917047323223, 'fraction_numerical': 0.02343321761374018, 'mean_word_length': 4.299685306221254, 'pattern_counts': {'":': 2, '<': 97, '<?xml version=': 0, '>': 97, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '294105', 'license': 'Apache-2.0', 'n_tokens_mistral': 8314, 'n_tokens_neox': 7905, 'n_words': 1392, 'path': 'corpus/analysis/sample_generator-root.ipynb', 'repo_name': 'mcmoyer11/whquestions-tgrep2', 'size': '21892'}