<commit_before>Name:           SmeegeScrape
Version:        0.6
Date:           01/27/2014
Author:         Smeege
Contact:        <EMAIL>

Description:    SmeegeScrape.py is a simple python script to scrape text from various sources including local files and 
                web pages, and turn the text into a custom word list.  A customized word list has many uses, from web 
                application testing to password cracking, having a specific set of words to use against a target can 
                increase efficiency and effectiveness during a penetration test.  I realize there are other text scrapers 
                publicly available however I feel this script is simple, efficient, and specific enough to warrant its own 
                release. This script is able to read almost any file which has cleartext in it that python can open.  I 
                have also included support for file formats such as pdf, html, docx, and pptx. <commit_msg>Change readme<commit_after>Name:           SmeegeScrape
Version:        0.6
Date:           01/27/2014
Author:         Smeege
Contact:        <EMAIL>

Description:    SmeegeScrape.py is a simple python script to scrape text from various sources including local files and web pages, and turn the text into a custom word list.  A customized word list has many uses, from web application testing to password cracking, having a specific set of words to use against a target can increase efficiency and effectiveness during a penetration test.  I realize there are other text scrapers publicly available however I feel this script is simple, efficient, and specific enough to warrant its own release. This script is able to read almost any file which has cleartext in it that python can open.  I have also included support for file formats such as pdf, html, docx, and pptx. 
{'fraction_non_alphanumeric': 0.03040173724212812, 'fraction_numerical': 0.010857763300760043, 'mean_word_length': 2.839583333333333, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1810879', 'n_tokens_mistral': 446, 'n_tokens_neox': 421, 'n_words': 261}