<commit_before># DAWG generator #


Creates highly compressed directed acyclic word graphs from word lists.
See http://en.wikipedia.org/wiki/Trie and http://en.wikipedia.org/wiki/Directed_acyclic_word_graph for an overview of the data structure.


### Requirements


Python 2.7.


### Acknowledgement


https://github.com/chalup/dawggenerator for piquing my interest and the basic idea of using hashes for sub-trie comparison.


### Usage


Launch from console:

    $ dawg_gen.py [word list file]

Note that the word list must be sorted, delimited by space or newline and have uppercase words. The program will prompt you for 

- The output location.
- Whether you choose 3 or 4 byte nodes.

Searching algorithms are not included; you are free to write those, armed with an understanding of the data format described below. <commit_msg>readme update<commit_after># DAWG generator #


Creates highly compressed directed acyclic word graphs from word lists.
See http://en.wikipedia.org/wiki/Trie and http://en.wikipedia.org/wiki/Directed_acyclic_word_graph for an overview of the data structure.


### Requirements


Python 2.7.


### Acknowledgement


https://github.com/chalup/dawggenerator for piquing my interest and the basic idea of using hashes for sub-trie comparison.


### Usage


Launch from console:

    $ dawg_gen.py [word list file]

Note that the word list must be sorted, delimited by space or newline and have uppercase words. The program will prompt you for 

- The output location.
- Whether you choose 3- or 4-byte nodes (if available).

Searching algorithms are not included; you are free to write those, armed with an understanding of the data format described below. 
{'fraction_non_alphanumeric': 0.06627218934911243, 'fraction_numerical': 0.004733727810650888, 'mean_word_length': 4.017804154302671, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '157797', 'n_tokens_mistral': 505, 'n_tokens_neox': 462, 'n_words': 214}