<commit_before>REDMASH
==========

Don't waste your time posting this to /r/shittyprogramming. It's bad. I know. 

This scrapes reddit for posts with keywords in the titles, keydomains as the linked url, or keynames as the submitter. It compiles these into multiple TXT files which are sorted by various criteria. 

At the end, it packs the information into a pickle file. This allows you to generate more txt files without having to re-run the program. This is especially important if you're scraping several thousand submissions.

**To use the pickle file:**

Open Python interpreter

`import pickle`

`filec = open('C:\\path\\file.p', 'rb')`

`loaded = pickle.load(filec)`

'loaded' now contains all of the posts. This is in a dictionary form rather than an object form. To get the author of the sixth post in the list:

`item = loaded[5]`

`print(item['author'])

Contact me if this doesn't make any sense. I wouldn't be surprised.<commit_msg>reddit 46<commit_after>REDMASH
==========

Don't waste your time posting this to /r/shittyprogramming. It's bad. I know. 

This scrapes reddit for posts with keywords in the titles, keydomains as the linked url, or keynames as the submitter. It compiles these into multiple TXT files which are sorted by various criteria. 

At the end, it packs the information into a pickle file. This allows you to generate more txt files without having to re-run the program. This is especially important if you're scraping several thousand submissions.

**To use the pickle file:**

Open Python interpreter

`import pickle`

`filec = open('C:\\path\\file.p', 'rb')`

`loaded = pickle.load(filec)`

'loaded' now contains all of the posts. This is in a dictionary form rather than an object form. To get the author of the sixth post in the list:

`item = loaded[5]`

`print(item['author'])`

Contact me if this doesn't make any sense. I wouldn't be surprised.
{'fraction_non_alphanumeric': 0.08514013749338974, 'fraction_numerical': 0.0021152829190904283, 'mean_word_length': 4.284916201117318, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2176464', 'n_tokens_mistral': 548, 'n_tokens_neox': 520, 'n_words': 280}