<commit_before>Pipeline
========
[![Build Status](https://travis-ci.org/smoketurner/pipeline.svg?branch=master)](https://travis-ci.org/smoketurner/pipeline)
[![Coverage Status](https://coveralls.io/repos/smoketurner/pipeline/badge.svg?branch=master&service=github)](https://coveralls.io/github/smoketurner/pipeline?branch=master)
[![Maven Central](https://img.shields.io/maven-central/v/com.smoketurner.pipeline/notification-parent.svg?style=flat-square)](https://maven-badges.herokuapp.com/maven-central/com.smoketurner.pipeline/pipeline-parent/)
[![GitHub license](https://img.shields.io/github/license/smoketurner/pipeline.svg?style=flat-square)](https://github.com/smoketurner/pipeline/tree/master)

Pipeline is an implementation of an event pipeline using [Amazon Web Services](https://aws.amazon.com) (AWS) [Simple Storage Service](http://aws.amazon.com/s3/) (S3) and [Simple Queue Service](http://aws.amazon.com/sqs/) (SQS). S3 can be configured to publish a notification to the [Simple Notification Service](http://aws.amazon.com/sns/) (SNS) whenever a new object is created in a bucket. SNS can then publish the notification into an SQS queue which Pipeline will consume from. Pipeline will consume from the SQS queue, then download the object from S3, optionally decompress it, then emit each line of data over a [server-sent events](https://en.wikipedia.org/wiki/Server-sent_events) (SSE) HTTP endpoint.

Installation
------------
To build this code locally, clone the repository then use [Maven](https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html) to build the jar:
```
git clone https://github.com/smoketurner/pipeline.git
cd pipeline<commit_msg>Fix badge<commit_after>Pipeline
========
[![Build Status](https://travis-ci.org/smoketurner/pipeline.svg?branch=master)](https://travis-ci.org/smoketurner/pipeline)
[![Coverage Status](https://coveralls.io/repos/smoketurner/pipeline/badge.svg?branch=master&service=github)](https://coveralls.io/github/smoketurner/pipeline?branch=master)
[![Maven Central](https://img.shields.io/maven-central/v/com.smoketurner.pipeline/pipeline-parent.svg?style=flat-square)](https://maven-badges.herokuapp.com/maven-central/com.smoketurner.pipeline/pipeline-parent/)
[![GitHub license](https://img.shields.io/github/license/smoketurner/pipeline.svg?style=flat-square)](https://github.com/smoketurner/pipeline/tree/master)

Pipeline is an implementation of an event pipeline using [Amazon Web Services](https://aws.amazon.com) (AWS) [Simple Storage Service](http://aws.amazon.com/s3/) (S3) and [Simple Queue Service](http://aws.amazon.com/sqs/) (SQS). S3 can be configured to publish a notification to the [Simple Notification Service](http://aws.amazon.com/sns/) (SNS) whenever a new object is created in a bucket. SNS can then publish the notification into an SQS queue which Pipeline will consume from. Pipeline will consume from the SQS queue, then download the object from S3, optionally decompress it, then emit each line of data over a [server-sent events](https://en.wikipedia.org/wiki/Server-sent_events) (SSE) HTTP endpoint.

Installation
------------
To build this code locally, clone the repository then use [Maven](https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html) to build the jar:
```
git clone https://github.com/smoketurner/pipeline.git
cd pipeline
{'fraction_non_alphanumeric': 0.14949374627754616, 'fraction_numerical': 0.0023823704586063135, 'mean_word_length': 5.484555984555985, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 24, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5138175', 'n_tokens_mistral': 1055, 'n_tokens_neox': 1044, 'n_words': 238}