<commit_before>[![Build Status](https://img.shields.io/travis/webstylestory/algolia-firebase-indexer.svg?style=flat-square)](https://travis-ci.org/webstylestory/algolia-firebase-indexer) [![Coverage Status](https://img.shields.io/coveralls/webstylestory/algolia-firebase-indexer.svg?style=flat-square)](https://coveralls.io/github/webstylestory/algolia-firebase-indexer) [![Dependency Status](https://img.shields.io/david/webstylestory/algolia-firebase-indexer.svg?style=flat-square)](https://david-dm.org/webstylestory/algolia-firebase-indexer) [![devDependency Status](https://img.shields.io/david/dev/webstylestory/algolia-firebase-indexer.svg?style=flat-square)](https://david-dm.org/webstylestory/algolia-firebase-indexer#info=devDependencies) [![Babel stage-2 badge](https://img.shields.io/badge/babel-stage%202-blue.svg?style=flat-square)](https://babeljs.io/docs/plugins/preset-stage-2/) [![License](http://img.shields.io/:license-mit-blue.svg?style=flat-square)](https://raw.githubusercontent.com/webstylestory/algolia-firebase-indexer/master/LICENSE) [![Made by french tech badge](https://img.shields.io/badge/made%20by-french%20%E2%9D%A4%20tech-orange.svg?style=flat-square)](http://webstylestory.com)


# algolia-firebase-indexer

⚡️ Keep your Algolia search indexes in sync with your Firebase datasets!

When using Firebase as a web or mobile app backend, Algolia search-as-a-service is a really nice addition, considering the poor search and querying capabilities of Firebase. This server is the perfect companion to make their synchronization happen in no time.

*(Codebase inspired by Scott Smith's work in this [blog post](http://scottksmith.com/blog/2014/12/09/algolia-real-time-search-with-firebase/)*

  * [Usage](#usage)
  * [Configuration](#configuration)
      * [Firebase configuration](#firebase-configuration)
      * [Reindexing, incremental indexing](#reindexing-incremental-indexing)
  * [Known Issues](#known-issues)
  * [Developers](#developers)
      * [Logging & Debugging](#logging--debugging)
      * [testing](#testing)
      * [contribute](#contribute)
  * [License](#license)


---

## WORK IN PROGRESS

IMPORTANT: This is an early release, check the issues for the remaining tasks before it can be use seriously

---

## Install

    git clone https://github.com/webstylestory/algolia-firebase-indexer.git
    cd algolia-firebase-indexer
    npm install

---

## Usage

    npm start

For production setup, I strongly encourage the use of a good process manager 
like [PM2](https://github.com/Unitech/pm2)

Important : the server needs a config file before it can runs, at least to provide your Algolia and Firebase API keys. See the [section below about configuration](#configuration).

---

## Configuration

Edit the `config.example.js` file with the data relevant to your setup, and rename it to `config.js` before running the server.


    let CONFIG = {
        // Firebase credentials
        firebase: {
            instance: 'TO_BE_CHANGED',
            secret: 'TO_BE_CHANGED',
            // Where to store server metadata
            path: 'algolia',
            // Firebase token will be generated with this uid (to write above path)
            uid: 'algolia'
        },
        // Algolia credentials
        algolia: {
            applicationId: 'TO_BE_CHANGED',
            // *Admin* API Key
            apiKey: 'TO_BE_CHANGED'
        },
        // Optional, this field will be checked against last
        // run date to see if reindexing is necessary.
        // Field type must be UNIX timestamp (example Javascript Date.now()).
        // WARNING: Without this field being corectly configured,
        // everything is re-indexed at each rerun.
        timestampField: 'modifiedAt',
        // Firebase datasets to index in Algolia (examples)
        schema: {
            todoLists: {
                // Firebase path
                path: 'app/todo',
                // Algolia index (must exist already)
                index: 'dev_todo_lists',
                // Optional, name of ID field (otherwise,
                // the Firebase object key will be used)
                key: 'id',
                // Optional, list of fields to index
                // (otherwise, every field will be indexed)
                includeFields: [
                    'name',
                    'modifiedAt'
                ],
                // Optional, list of fields to exclude from index
                // Note: if both are specified, `excludeFields` 
                // is applied *after* `includeFields`
                excludeFields: [
                    'passwdHash',
                    'private'
                ]
            },
            todoItems: {
                // Second example dataset to index, minimal config
                path: 'app/todoItems',
                index: 'dev_todo_items'
            }
        }
    };


*Note for production use: you can also uncomment production-specific settings below the main, in the same file, to deal with multi environment setup*

### Firebase configuration

In order for `algolia-firebase-indexer` to work properly, it has to store 
the last known indexing date in firebase. You can specify the path where you
want this information stored in the config ([see above](#configuration)).

This path will also be used to store data fixtures [if you run the tests](#testing).

### Reindexing, incremental indexing

This daemon supports a simple mode where every indexed object in Algolia is dropped at runtime, and then re-indexed from the current Firebase connection. **Any previously indexed data will be lost.**

For more serious usage, this is not reccomended, as it can lead to many useless operations, and bandwith waste.

I really suggest to specify `path` and `uid` in `firebase` config field, and give
write access to that uid in the Firebase Rules. Also mandatory, the last modificated 
time of each items must be set in the schema definition (for example, 
in a `modifiedAt` field). This is in your app, if you did not implement such feature 
to track the last modified time of each of your objects, you'll have to do so 
before using this server efficiently. 

To allow `algolia` user to write in your Firebase `algolia` path, 
add the following in your Firebase instance security rules :

    // Let algolia-firebase-indexer daemon keep track of what is in sync
    "algolia": {                              
        ".read": "auth.uid == 'algolia'",
        ".write": "auth.uid == 'algolia'",
    }

---

## Known issues

  * [#15](#15) If relaunching indexing after a previous one, with timestamp
    stored in Firebase, removed items since then will not be removed from index.
    Workaround is to delete timestamp in `algolia/index` backend in Firebase to 
    force a full reindex.

---
<commit_msg>[docs] How to avoid warnings running tests<commit_after>[![Build Status](https://img.shields.io/travis/webstylestory/algolia-firebase-indexer.svg?style=flat-square)](https://travis-ci.org/webstylestory/algolia-firebase-indexer) [![Coverage Status](https://img.shields.io/coveralls/webstylestory/algolia-firebase-indexer.svg?style=flat-square)](https://coveralls.io/github/webstylestory/algolia-firebase-indexer) [![Dependency Status](https://img.shields.io/david/webstylestory/algolia-firebase-indexer.svg?style=flat-square)](https://david-dm.org/webstylestory/algolia-firebase-indexer) [![devDependency Status](https://img.shields.io/david/dev/webstylestory/algolia-firebase-indexer.svg?style=flat-square)](https://david-dm.org/webstylestory/algolia-firebase-indexer#info=devDependencies) [![Babel stage-2 badge](https://img.shields.io/badge/babel-stage%202-blue.svg?style=flat-square)](https://babeljs.io/docs/plugins/preset-stage-2/) [![License](http://img.shields.io/:license-mit-blue.svg?style=flat-square)](https://raw.githubusercontent.com/webstylestory/algolia-firebase-indexer/master/LICENSE) [![Made by french tech badge](https://img.shields.io/badge/made%20by-french%20%E2%9D%A4%20tech-orange.svg?style=flat-square)](http://webstylestory.com)


# algolia-firebase-indexer

⚡️ Keep your Algolia search indexes in sync with your Firebase datasets!

When using Firebase as a web or mobile app backend, Algolia search-as-a-service is a really nice addition, considering the poor search and querying capabilities of Firebase. This server is the perfect companion to make their synchronization happen in no time.

*(Codebase inspired by <NAME>'s work in this [blog post](http://scottksmith.com/blog/2014/12/09/algolia-real-time-search-with-firebase/)*

  * [Usage](#usage)
  * [Configuration](#configuration)
      * [Firebase configuration](#firebase-configuration)
      * [Reindexing, incremental indexing](#reindexing-incremental-indexing)
  * [Known Issues](#known-issues)
  * [Developers](#developers)
      * [Logging & Debugging](#logging--debugging)
      * [testing](#testing)
      * [contribute](#contribute)
  * [License](#license)


---

## WORK IN PROGRESS

IMPORTANT: This is an early release, check the issues for the remaining tasks before it can be use seriously

---

## Install

    git clone https://github.com/webstylestory/algolia-firebase-indexer.git
    cd algolia-firebase-indexer
    npm install

---

## Usage

    npm start

For production setup, I strongly encourage the use of a good process manager 
like [PM2](https://github.com/Unitech/pm2)

Important : the server needs a config file before it can runs, at least to provide your Algolia and Firebase API keys. See the [section below about configuration](#configuration).

---

## Configuration

Edit the `config.example.js` file with the data relevant to your setup, and rename it to `config.js` before running the server.


    let CONFIG = {
        // Firebase credentials
        firebase: {
            instance: 'TO_BE_CHANGED',
            secret: 'TO_BE_CHANGED',
            // Where to store server metadata
            path: 'algolia',
            // Firebase token will be generated with this uid (to write above path)
            uid: 'algolia'
        },
        // Algolia credentials
        algolia: {
            applicationId: 'TO_BE_CHANGED',
            // *Admin* API Key
            apiKey: 'TO_BE_CHANGED'
        },
        // Optional, this field will be checked against last
        // run date to see if reindexing is necessary.
        // Field type must be UNIX timestamp (example Javascript Date.now()).
        // WARNING: Without this field being corectly configured,
        // everything is re-indexed at each rerun.
        timestampField: 'modifiedAt',
        // Firebase datasets to index in Algolia (examples)
        schema: {
            todoLists: {
                // Firebase path
                path: 'app/todo',
                // Algolia index (must exist already)
                index: 'dev_todo_lists',
                // Optional, name of ID field (otherwise,
                // the Firebase object key will be used)
                key: 'id',
                // Optional, list of fields to index
                // (otherwise, every field will be indexed)
                includeFields: [
                    'name',
                    'modifiedAt'
                ],
                // Optional, list of fields to exclude from index
                // Note: if both are specified, `excludeFields` 
                // is applied *after* `includeFields`
                excludeFields: [
                    'passwdHash',
                    'private'
                ]
            },
            todoItems: {
                // Second example dataset to index, minimal config
                path: 'app/todoItems',
                index: 'dev_todo_items'
            }
        }
    };


*Note for production use: you can also uncomment production-specific settings below the main, in the same file, to deal with multi environment setup*

### Firebase configuration

In order for `algolia-firebase-indexer` to work properly, it has to store 
the last known indexing date in firebase. You can specify the path where you
want this information stored in the config ([see above](#configuration)).

This path will also be used to store data fixtures [if you run the tests](#testing).

### Reindexing, incremental indexing

This daemon supports a simple mode where every indexed object in Algolia is dropped at runtime, and then re-indexed from the current Firebase connection. **Any previously indexed data will be lost.**

For more serious usage, this is not reccomended, as it can lead to many useless operations, and bandwith waste.

I really suggest to specify `path` and `uid` in `firebase` config field, and give
write access to that uid in the Firebase Rules. Also mandatory, the last modificated 
time of each items must be set in the schema definition (for example, 
in a `modifiedAt` field). This is in your app, if you did not implement such feature 
to track the last modified time of each of your objects, you'll have to do so 
before using this server efficiently. 

To allow `algolia` user to write in your Firebase `algolia` path, 
add the following in your Firebase instance security rules :

    // Let algolia-firebase-indexer daemon keep track of what is in sync
    "algolia": {                              
        ".read": "auth.uid == 'algolia'",
        ".write": "auth.uid == 'algolia'",
    }
    // To avoid Firebase warnings running the tests
    "tests": {
      "testData": {
        ".indexOn": "modifiedAt"
      }
    }
    
---

## Known issues

  * [#15](#15) If relaunching indexing after a previous one, with timestamp
    stored in Firebase, removed items since then will not be removed from index.
    Workaround is to delete timestamp in `algolia/index` backend in Firebase to 
    force a full reindex.

---

{'fraction_non_alphanumeric': 0.10144717800289435, 'fraction_numerical': 0.004052098408104197, 'mean_word_length': 2.7364152473641523, 'pattern_counts': {'":': 9, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 28, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5225803', 'n_tokens_mistral': 4012, 'n_tokens_neox': 3797, 'n_words': 1288}