<commit_before># Waltz

A tool for low-level information retrieval research.

## Non-Goals

- To be a useful search engine that you can drag and drop into another project.
- To be the fastest implementation
- To have a web frontend
- To provide tokenization (of any kind)
- To provide parsing (of any formats)
- To provide stemming
- To provide stopping or stopword lists
- To provide a query language
- To implement all the algorithms

## Goals

- To be as minimally useful as possible.
- Enable changing of everything.
- Enable storing and querying of your weird, wacky data in your weird, wacky format.

<commit_msg>Add Galago goal<commit_after># Waltz

A tool for low-level information retrieval research.

## Non-Goals

- To be a useful search engine that you can drag and drop into another project.
- To be the fastest implementation
- To have a web frontend
- To provide tokenization (of any kind)
- To provide parsing (of any formats)
- To provide stemming
- To provide stopping or stopword lists
- To provide a query language
- To implement all the algorithms

## Goals

- To be as minimally useful as possible.
- To be slightly compatible with [Galago](http://lemurproject.org/galago.php)
- Enable changing of everything.
- Enable storing and querying of your weird, wacky data in your weird, wacky format.


{'fraction_non_alphanumeric': 0.05842185128983308, 'fraction_numerical': 0.0, 'mean_word_length': 3.7446043165467624, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2571847', 'n_tokens_mistral': 363, 'n_tokens_neox': 350, 'n_words': 192}