<commit_before># [mainevent](http://codeactual.github.com/mainevent)

mainevent provides a suite of tools to gain insight into your log files.

* Collect, parse and store log updates from local and SSH-accessible log files.
* Easily write new parser modules for any file format.
* Searchable timeline with optional real-time updates.
* Dashboard analytics generated by MongoDB MapReduce, Redis and jqPlot.
* Write custom Pub/Sub listeners to real-time updates.
* [more ...](http://codeactual.github.com/mainevent)

<a href="http://codeactual.github.com/mainevent"><img alt="3-screenshot set" src="http://codeactual.github.com/mainevent/img/screenshot-set.png" /></a>

## Use Cases

* Search development/production environment logs, whether they're syslog, JSON, nginx or proprietary.
* Produce dashboard data and graphs specific to your insight needs.
* Filter and replicate real-time updates into additional systems for specialized processing or alerting.

## Components

### Frontend

#### [mainevent_server.js](https://github.com/codeactual/mainevent/blob/master/bin/mainevent_server.js)

`$ bin/mainevent_server.js`

Has three responsibilities:

1. &nbsp;`/` serves the backbone.js MVC app.
1. &nbsp;`/api` serves JSON data including graph points, event objects and timeline pages.
1. &nbsp;`/socket.io` serves real-time timelime updates.

Triggers `public/build.js` on startup to build the `static/` directory.

#### public/build.js

`$ public/build.js`

* Combines and compresses JS/CSS files located in `public/`.
* Relies on `public/js/app.build.js` for RequireJS configuration.
* Triggers `public/js/templates.build.js`.

Outputs all files into `static/`.

#### public/js/templates.build.js

Compiles dust.js templates in `app/views` and `app/parsers/*/templates`.

#### public/js/app.build.js

RequireJS configuration for client-side dependencies.

### Backend

### Background

#### [tail.js](https://github.com/codeactual/mainevent/blob/master/bin/tail.js)

`$ bin/tail.js`

Spawns `tail` instances for each source described in `[config/app.js](https://github.com/codeactual/mainevent/blob/master/config/app.js.dist)`.

#### [import.js](https://github.com/codeactual/mainevent/blob/master/bin/import.js)

`$ bin/import.js --parser json --path /var/log/myApp/prod.json --tags myApp,import`

Like `tail.js` except it processes the entire file. (The file does not need to be described in `config/app.js`.)

### Testing

#### [test.js](https://github.com/codeactual/mainevent/blob/master/bin/test.js)

`$ bin/test.js`

Serves the YUI Test runner page, `app/view/test.html` and `test/browser/*.js` test cases.

## How To

### Create a new parser module

* TODO

### Create a Pub/Sub listener for log updates

* TODO

## Configuration

* TODO

## File Layout

* `app` : TODO
  * `controllers`: TODO
  * `graphs`: TODO
  * `jobs`: TODO
  * `modules`: TODO
  * `parsers`: TODO
  * `sockets`: TODO
  * `views`: TODO
* `bin`: TODO
* `config`: TODO
* `public`: TODO
  * `backbone`: TODO
  * `collections`: TODO
  * `controllers`: TODO
  * `helpers`: TODO
  * `models`: TODO
  * `observers`: TODO
  * `shared`: TODO
  * `views`: TODO
* `static`: TODO
* `test`: TODO
  * `bin`: TODO<commit_msg>Add README.md content.<commit_after># [mainevent](http://codeactual.github.com/mainevent)

mainevent provides a suite of tools to gain insight into your log files.

* Collect, parse and store log updates from local and SSH-accessible log files.
* Easily write new parser modules for any file format.
* Searchable timeline with optional real-time updates.
* Dashboard analytics generated by MongoDB MapReduce, Redis and jqPlot.
* Write custom Pub/Sub listeners to real-time updates.
* [more ...](http://codeactual.github.com/mainevent)

<a href="http://codeactual.github.com/mainevent"><img alt="3-screenshot set" src="http://codeactual.github.com/mainevent/img/screenshot-set.png" /></a>

## Use Cases

* Search development/production environment logs, whether they're syslog, JSON, nginx or proprietary.
* Produce dashboard data and graphs specific to your insight needs.
* Filter and replicate real-time updates into additional systems for specialized processing or alerting.

## Components

### Frontend

#### [mainevent_server.js](https://github.com/codeactual/mainevent/blob/master/bin/mainevent_server.js)

`$ bin/mainevent_server.js`

Has three responsibilities:

1. &nbsp;`/` serves the backbone.js MVC app.
1. &nbsp;`/api` serves JSON data including graph points, event objects and timeline pages.
1. &nbsp;`/socket.io` serves real-time timelime updates.

Triggers `public/build.js` on startup to build the `static/` directory.

#### public/build.js

`$ public/build.js`

* Combines and compresses JS/CSS files located in `public/`.
* Relies on `public/js/app.build.js` for RequireJS configuration.
* Triggers `public/js/templates.build.js`.

Outputs all files into `static/`.

#### public/js/templates.build.js

Compiles dust.js templates in `app/views` and `app/parsers/*/templates`.

#### public/js/app.build.js

RequireJS configuration for client-side dependencies.

### Backend

### Background

#### [tail.js](https://github.com/codeactual/mainevent/blob/master/bin/tail.js)

`$ bin/tail.js`

Spawns `tail` instances for each source described in `[config/app.js](https://github.com/codeactual/mainevent/blob/master/config/app.js.dist)`.

#### [import.js](https://github.com/codeactual/mainevent/blob/master/bin/import.js)

`$ bin/import.js --parser json --path /var/log/myApp/prod.json --tags myApp,import`

Like `tail.js` except it processes the entire file. (The file does not need to be described in `config/app.js`.)

### Testing

#### [test.js](https://github.com/codeactual/mainevent/blob/master/bin/test.js)

`$ bin/test.js`

Serves the YUI Test runner page, `app/view/test.html` and `test/browser/*.js` test cases.

## How To

### Create a new parser for an unsupported log format

Each parser lives in a separate directory under `[app/parsers](https://github.com/codeactual/mainevent/tree/master/app/parsers)` which holds its JS, CSS, templates and tests.

All parser classes extend a [base](https://github.com/codeactual/mainevent/blob/master/app/parsers/prototype.js) and only need to implement a small number of interfaces.

* Required
  * `parse(log)`: Accepts a log line string, returns an object of parsed fields.
* Optional
  * `buildTemplateContext(template, log)`: Modify the context object sent to dust.js based on the type of template. Currently there are two types: `preview` and `event`. See the [extension example](http://codeactual.github.com/mainevent/#extension-example) to see them in action.
  * `buildPreviewText(log)`: Skip defining a preview template and just build and return the preview string manually.
  * `extractTime(log)`: The default implementation will detect millisecond/second timestamps and `Date.parse()`-able strings in `log.time` values. For incompatible formats, define this function to extract the millisecond timestamp manually.
* Utilities
  * `namedCapture(subject, regex)`: Wrapper around [XRegExp](https://github.com/slevithan/XRegExp) named capture expression handling.
  * `candidateCapture(subject, candidates)`: Wrapper around `namedCapture` that lets you define multiple potential patterns and the first match wins.

See [app/parsers/prototype.js](https://github.com/codeactual/mainevent/blob/master/app/parsers/prototype.js) for more interface details.

Extending the base class is a simple one-call process via a backbone.js-like `extend()` function. See the [extension example](http://codeactual.github.com/mainevent/#extension-example) for a working implementation and screenshots of the output content. Or browse any of the modules under `[app/parsers](https://github.com/codeactual/mainevent/tree/master/app/parsers)`.

### Create a Pub/Sub listener for log updates

1. Create a module that exports an `on` function that receives an array of one or more log objects.
1. Perform any non-native tasks you need.
1. Find the `[config/app.js](https://github.com/codeactual/mainevent/blob/master/config/app.js.dist)` section that looks like:

```javascript
{
  // ...
  mongodb: {
    // ...
    listeners: [
      {
        // Customize this event but do not remove.
        event: 'InsertLog',
        enabled: true,
        subscribers: [
          'app/modules/redis/InsertLogPubSub.js'
        ],
      }
    ],
    // ...
  },
  // ...
}
```

1. Add the location of your listener module to the `subscribers` list.

## Configuration

* TODO

## File Layout

* `app` : TODO
  * `controllers`: TODO
  * `graphs`: TODO
  * `jobs`: TODO
  * `modules`: TODO
  * `parsers`: TODO
  * `sockets`: TODO
  * `views`: TODO
* `bin`: TODO
* `config`: TODO
* `public`: TODO
  * `backbone`: TODO
  * `collections`: TODO
  * `controllers`: TODO
  * `helpers`: TODO
  * `models`: TODO
  * `observers`: TODO
  * `shared`: TODO
  * `views`: TODO
* `static`: TODO
* `test`: TODO
  * `bin`: TODO
{'fraction_non_alphanumeric': 0.13654663386267055, 'fraction_numerical': 0.0013419816595839856, 'mean_word_length': 4.087030716723549, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '694141', 'n_tokens_mistral': 2829, 'n_tokens_neox': 2740, 'n_words': 850}