<commit_before>JesterJ 
=======
[![License](https://img.shields.io/badge/license-Apache%202.0-B70E23.svg?style=plastic)](http://www.opensource.org/licenses/Apache-2.0) [![Build Status](https://travis-ci.org/nsoft/jesterj.svg?branch=master)](https://travis-ci.org/nsoft/jesterj)

A new highly flexible, highly scaleable document ingestion system. 

See the [web site](http://www.jesterj.org) and the [documentation](https://github.com/nsoft/jesterj/wiki/Documentation) for more info

# Getting Started

Please see the [documentation in the wiki](https://github.com/nsoft/jesterj/wiki/Documentation)

# Project Status

Current release version: 0.1. (But current HEAD revision is *much* better, please build from source until 0.2)

# Slack Chanel
If you want an invite to our slack channel just send mail to:
```
                 ____   _        _           _                
  __ _ _  _ ___ / __ \ (_)___ __| |_ ___ _ _(_)  ___ _ _ __ _ 
 / _` | || (_-</ / _` || / -_|_-<  _/ -_) '_| |_/ _ \ '_/ _` |
 \__, |\_,_/__/\ \__,_|/ \___/__/\__\___|_|_/ (_)___/_| \__, |
 |___/          \____/__/                 |__/          |___/ 
```
## Features:

In this release we have the following features

 * Embedded Cassandra server
 * Cassandra config and data location configurable, defaults to ~/.jj/cassandra
 * Initial support for fault tolerance via logging statuses to the embedded cassandra server (WIP)
 * Log4j appender to write to Cassandra where desired
 * Initial API/process for user written steps. (see [documentation](https://github.com/nsoft/jesterj/wiki/Documentation))
 * 40% test coverage (jacoco)
 * Simple filesystem scanner
 * Copy Field processor
 * Date Reformat processor
 * Human Readable File Size processor 
 * Tika processor to extract content<commit_msg>Slightly more legible<commit_after>JesterJ 
=======
[![License](https://img.shields.io/badge/license-Apache%202.0-B70E23.svg?style=plastic)](http://www.opensource.org/licenses/Apache-2.0) [![Build Status](https://travis-ci.org/nsoft/jesterj.svg?branch=master)](https://travis-ci.org/nsoft/jesterj)

A new highly flexible, highly scaleable document ingestion system. 

See the [web site](http://www.jesterj.org) and the [documentation](https://github.com/nsoft/jesterj/wiki/Documentation) for more info

# Getting Started

Please see the [documentation in the wiki](https://github.com/nsoft/jesterj/wiki/Documentation)

# Project Status

Current release version: 0.1. (But current HEAD revision is *much* better, please build from source until 0.2)

# Slack Chanel
If you want an invite to our slack channel just send mail to:
```
                     ______    _           __            _                  
   ____ ___  _______/ ____ \  (_)__  _____/ /____  _____(_)____  _________ _
  / __ `/ / / / ___/ / __ `/ / / _ \/ ___/ __/ _ \/ ___/ // __ \/ ___/ __ `/
 / /_/ / /_/ (__  ) / /_/ / / /  __(__  ) /_/  __/ /  / // /_/ / /  / /_/ / 
 \__, /\__,_/____/\ \__,_/_/ /\___/____/\__/\___/_/__/ (_)____/_/   \__, /  
/____/             \____/___/                     /___/            /____/   

```
## Features:

In this release we have the following features

 * Embedded Cassandra server
 * Cassandra config and data location configurable, defaults to ~/.jj/cassandra
 * Initial support for fault tolerance via logging statuses to the embedded cassandra server (WIP)
 * Log4j appender to write to Cassandra where desired
 * Initial API/process for user written steps. (see [documentation](https://github.com/nsoft/jesterj/wiki/Documentation))
 * 40% test coverage (jacoco)
 * Simple filesystem scanner
 * Copy Field processor
 * Date Reformat processor
 * Human Readable File Size processor 
 * Tika processor to extract content
{'fraction_non_alphanumeric': 0.1354054054054054, 'fraction_numerical': 0.009189189189189189, 'mean_word_length': 2.811534500514933, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 12, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2668100', 'n_tokens_mistral': 1227, 'n_tokens_neox': 1154, 'n_words': 281}