<commit_before>mongo-λ
--------

***EXPERIMENTAL***

A mongo [lambda architecture](http://www.manning.com/marz/) implementation with simple API for providing mongo's aggregation pipepline reports. Written in javascript designed as an npm module.

### Model Implementation

```js
{
	reports: [
	{
		name: "trending",
		agg: $aggPipeline,
		frequency: 60*60*1000
	}],
	batchLayer: {
		collection: "master",
		dataRetention: 24*60*60*1000
	},
	speedLayer: {
		collection: "delta"
	},
	servingLayer: {
		combine: function(batchReports, liveDeltaReport) {
		}
	}
}
```

### Responsibilities of Module

Report Runner
 - Generating recurring reports in batch layer. Will run using `$aggPipeline`.
 - Combine report method.


Data Management
 - Scrub data from batch layer's master collection after it expires past `dataRentention`.
 - Scrub data from delta when batch view is produced.

### API

### `.insertData(data)`

Will insert data point into batch and speed layer's mongo collection.

### `.getReport("trendingTop100")`

Fetches a **"trendingTop100"** report this will use the supplied `combine` function.

#### NOTE

* *Live* means that the data is near realtime, I hesitate to use the word realtime because nothing is completely realtime.
<commit_msg>added badge to readme<commit_after>mongo-λ
--------

[![Build Status](https://travis-ci.org/smaxwellstewart/mongo-lambda.svg?branch=master)](https://travis-ci.org/smaxwellstewart/mongo-lambda)

***WARNING! EXPERIMENTAL WORK IN PROGRESS***

A mongo [lambda architecture](http://www.manning.com/marz/) implementation with simple API for providing mongo's aggregation pipepline reports. Written in javascript designed as an npm module.

### Model Implementation

```js
{
	reports: [
	{
		name: "trending",
		agg: $aggPipeline,
		frequency: 60*60*1000
	}],
	batchLayer: {
		collection: "master",
		dataRetention: 24*60*60*1000
	},
	speedLayer: {
		collection: "delta"
	},
	servingLayer: {
		combine: function(batchReports, liveDeltaReport) {
		}
	}
}
```

### Responsibilities of Module

Report Runner
 - Generating recurring reports in batch layer. Will run using `$aggPipeline`.
 - Combine report method.


Data Management
 - Scrub data from batch layer's master collection after it expires past `dataRentention`.
 - Scrub data from delta when batch view is produced.

### API

### `.insertData(data)`

Will insert data point into batch and speed layer's mongo collection.

### `.getReport("trendingTop100")`

Fetches a **"trendingTop100"** report this will use the supplied `combine` function.

#### NOTE

* *Live* means that the data is near realtime, I hesitate to use the word realtime because nothing is completely realtime.

{'fraction_non_alphanumeric': 0.11973144349123462, 'fraction_numerical': 0.017903767251025736, 'mean_word_length': 4.42914979757085, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2508461', 'n_tokens_mistral': 908, 'n_tokens_neox': 820, 'n_words': 275}