<commit_before># Precog

Precog is an advanced analytics engine for NoSQL data. It's sort of like
a traditional analytics database, but instead of working with
normalized, tabular data, it works with denormalized data that may not
have a uniform schema.

You can plop large amounts of JSON into Precog and start doing analytics
without any preprocessing, such as time series analytics, filtering,
rollups, statistics, and even some kinds of machine learning.

There's an API for developer integration, and a high-level application
called *Labcoat* for doing ad hoc and exploratory analytics.

Precog has been used by developers to build reporting features into
applications (since Precog has very comprehensive, developer-friendly APIs),
and together with *Labcoat*, Precog has been used by data scientists to perform
ad hoc analysis of semi-structured data.

This is the Community Edition of Precog. For more information about
commercial support and maintenance options, check out [SlamData,
Inc](<http://www.slamdata.com>), the official sponsor of the Precog open
source project.

# Roadmap

## Phase 1: Simplified Deployment

Precog was originally designed to be offered exclusively via the cloud in
a multi-tenant offering. As such, it has made certain tradeoffs that make it
much harder for individuals and casual users to install and maintain.

In the current roadmap, Phase 1 involves simplifying Precog to the point where
there are so few moving pieces, anyone can install and launch Precog, and keep
Precog running without anything more than an occasional restart.

The work is currently divided into the following tickets:

- Remove MongoDB dependency
- Remove Kafka dependency
- Remove Zookeeper dependency
- Simplify file system model
- Query directly from raw files
- Conversion from raw files to NihDB file format<commit_msg>add link to mongodb ticket<commit_after># Precog

Precog is an advanced analytics engine for NoSQL data. It's sort of like
a traditional analytics database, but instead of working with
normalized, tabular data, it works with denormalized data that may not
have a uniform schema.

You can plop large amounts of JSON into Precog and start doing analytics
without any preprocessing, such as time series analytics, filtering,
rollups, statistics, and even some kinds of machine learning.

There's an API for developer integration, and a high-level application
called *Labcoat* for doing ad hoc and exploratory analytics.

Precog has been used by developers to build reporting features into
applications (since Precog has very comprehensive, developer-friendly APIs),
and together with *Labcoat*, Precog has been used by data scientists to perform
ad hoc analysis of semi-structured data.

This is the Community Edition of Precog. For more information about
commercial support and maintenance options, check out [SlamData,
Inc](<http://www.slamdata.com>), the official sponsor of the Precog open
source project.

# Roadmap

## Phase 1: Simplified Deployment

Precog was originally designed to be offered exclusively via the cloud in
a multi-tenant offering. As such, it has made certain tradeoffs that make it
much harder for individuals and casual users to install and maintain.

In the current roadmap, Phase 1 involves simplifying Precog to the point where
there are so few moving pieces, anyone can install and launch Precog, and keep
Precog running without anything more than an occasional restart.

The work is currently divided into the following tickets:

- [Remove MongoDB dependency](https://github.com/precog/platform/issues/523)
- Remove Kafka dependency
- Remove Zookeeper dependency
- Simplify file system model
- Query directly from raw files
- Conversion from raw files to NihDB file format
{'fraction_non_alphanumeric': 0.039037433155080216, 'fraction_numerical': 0.0018716577540106951, 'mean_word_length': 4.919303797468355, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6753692', 'n_tokens_mistral': 928, 'n_tokens_neox': 874, 'n_words': 539}