<jupyter_start><jupyter_text>Using a small sample of the 2020 corpus, without preprocessing<jupyter_code>f = open("test.txt", "r", encoding="utf8")
test = f.read()
test_dict = lexicon.analyze(test, normalize=True)
test_dict.update([('source', 'suicidewatch_2020')])
import pandas
df = pandas.DataFrame(test_dict, index=["suicidewatch_2020"])
df
dict_2 = lexicon.analyze(test)
df2 = pandas.DataFrame(dict_2, index=["sw_2020_non_normalized"])
df = pandas.concat([df, df2], axis=0)
df
lexicon.analyze(test, categories=["negative_emotion"], normalize=True)
#Note to self: you can find categories and their associated words in categories.tsv. These were generated from training over contemporary fiction; and were validated by humans. Category labels are leftmost words, their constituent words follow them in each row.<jupyter_output><empty_output><jupyter_text>Using a small sample of the 2020 corpus with preprocessing  Cleaning doesn't work, so I will try just Gensim stopwords. Empath already tokenizes! It doesn't seem such a great idea too...<jupyter_code>from gensim.parsing.preprocessing import remove_stopwords

test_filtered = (remove_stopwords(test))
lexicon.analyze(test_filtered, normalize=True)<jupyter_output><empty_output><jupyter_text>Adding "first-person pronouns" as a category  Categories (bit hard to get the categories to render right)<jupyter_code>lexicon.create_category("First_person_plural_pronoun", ["we", "our"])
lexicon.analyze(test, categories=["First_person_singular_pronoun"])
lexicon.analyze(test_filtered, categories=["First_person_singular_pronoun"])<jupyter_output><empty_output><jupyter_text>Using the entire 2020 Lexicon, without preprocessing<jupyter_code>f = open("a_lot.txt", "r", encoding="utf8")
a_lot = f.read()
lexicon.analyze(a_lot)
lexicon.analyze(a_lot, normalize=True)<jupyter_output><empty_output><jupyter_text>Checking corpora after txt dump<jupyter_code>f = open("ready_data/sw_2013.txt", "r", encoding="utf8")
sw = f.read()
##old
lexicon.analyze(sw_2010, normalize="True")
lexicon.analyze(sw, normalize="True")
print(sw)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11815150071462602, 'fraction_numerical': 0.019056693663649357, 'mean_word_length': 8.859154929577464, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '85806', 'license': 'MIT', 'n_tokens_mistral': 738, 'n_tokens_neox': 680, 'n_words': 165, 'path': 'empath_preliminary.ipynb', 'repo_name': 'kraftedcheese/SW-Study', 'size': '2099'}