<commit_before># clucene
=======

A lucene based cloud search engine which stores its index in a Microsoft Azure Blob Storage.

The name should be changed as clucene is already the name of the C++ port of lucene. clucene is supposed to be the contraction of cloud and Lucene. 

# Usage

## Setting up the nodes

The project is trying to be as configurable as possible from the inside to avoid recompiling for each node.
We use [JSON](http://json.org/) as a configuration format for its simplicity.

A configuration file look like:

    {
        "port": 7050,
        "azure": {
            "DefaultEndpointsProtocol":"the protocol",
            "AccountName":"account name",
            "AccountKey":"key linked to your account",
            "container": "folder to store the index",
        },
        ....
    }

A node can either be an indexer (will crawl and index document) or a searcher (will search queries sent to it). To set that just change the configuration file to right node.

For an indexer:
		"indexer": {
			"crawler": {
				"container": "where on azure the documents are stored",
				"queueSize": "The maximum number of processes that analyze a document in the thread pool",
				"nbThreads": "The number of threads allocated to parsing documents"
			},
			"indexer": {
				"commitFrequency": "how often we commit the new documents",
				"folder": "if you use a FS lucene directory where to put the directory",
				"regular": "if you don't want to use an azure directory (for stats for example)",
				"container": "the azure blob where the directory is stored",
				"queueSize": "The maximum number of processes that index document",
				"nbThreads": "The number of threads allocated to indexing documents"
			},
			"downloadDir": "Where to download the directory",
			"stats": {
				"file": "where to write the stats report",
				"frequency": "How often in ms we create a snapshot of the system status",
			}
		}

For a searcher:

    "searcher": {
        // To be completed
    }

Compulsory parameters in the JSON:

*   port
*   azure (with DefaultEndpointProtocol, AccountName, AccountKey, container)
*   For an indexer:
    *   crawler (folder, seed, domain, nbCrawler)
    *   downloadFolder
*   For a searcher:
    * To be completed

Once your configuration file is written you can either put it next to your executable jar or specify it to the executable:

    java -jar target/server-1.0-jar-with-dependencies.jar target/config.json 

## Using the indexer

## Using the searcher


## TODOs

This is a list of things that can be improved:
*		add much more unit tests
*		justify the use of mutexes in DirectoryFS
*		add compression on the nodes
*		Improve the error handling (currently most of it is just dumping the stacktrace)
------------------------

We use Travis for continuous integration! 

[![Build Status](https://secure.travis-ci.org/lahabana/clucene.png)](http://travis-ci.org/lahabana/clucene)<commit_msg>Updating and cleaning the readme<commit_after># clucene
=======

A lucene based cloud search engine which stores its index in a Windows Azure Blob Storage.

The name should be changed as clucene is already the name of the C port of lucene. clucene is supposed to be the contraction of cloud and Lucene. 

This project started (and is still) a university project the end of it is December 2012 so sorry for the mess I may leave in the repo before then (promise I'll clean ;) )
Please feel free to comment I'm pretty new to Lucene and I'm sure there's many wrong things!

# Usage

## Setting up the nodes

The project is trying to be as configurable as possible from the inside to avoid recompiling for each node.
We use [JSON](http://json.org/) as a configuration format for its simplicity.

A configuration file look like:

    {
        "port": 7050,
        "azure": {
            "DefaultEndpointsProtocol":"the protocol",
            "AccountName":"account name",
            "AccountKey":"key linked to your account",
            "container": "folder to store the index",
        },
        ....
    }

A node can either be an indexer (will crawl and index document) or a searcher (will search queries sent to it). To set that just change the configuration file to right node.

For an indexer:

    {
          "port": 7050,
          "azure": {
              "defaultEndpointsProtocol":"http",
              "accountName":"clucene2",
              "accountKey":"azure key"
          },
          "indexer": {
              "crawler": {
                  "directory": "/mnt/resource/pages",
                  "queueSize": 10,
                  "nbThreads": 1
              },
              "indexer": {
                  "commitFrequency": 5000,
                  "folder": "/mnt/resource/index",
                  "regular": false,
                  "container": "index",
                  "queueSize": 50,
                  "nbThreads": 3,
                  "bufferSize": 512
              }, 
              "stats": {
                   "file": "stats/commit10s3th512mbSearch.csv",
                   "frequency": 1000,
             }
         }
    }

For a searcher:

    {
        "port": 7051,
        "azure": {
            "defaultEndpointsProtocol":"http",
            "accountName":"clucene2",
            "accountKey":"azure key"
        },
        "searcher": {
            "folder": "/mnt/resource/index",
            "container": "index"
        }
    }

Compulsory parameters in the JSON:

*   TODO

Once your configuration file is written you can either put it next to your executable jar or specify it to the executable:

    java -jar target/server-1.0-jar-with-dependencies.jar target/config.json 

## Using the indexer

TODO

## Using the searcher

TODO

## TODOs

This is a list of things that can be improved:
*		add much more unit tests
*		refactor rename and clean code
*		add compression on the nodes
*		Improve the error handling (currently most of it is just dumping the stacktrace)
*		get some sub projects (like utils) out of clucene
*		Make abstraction of the cloud service to make usable with S3 for example

------------------------

We use Travis for continuous integration! 

[![Build Status](https://secure.travis-ci.org/lahabana/clucene.png)](http://travis-ci.org/lahabana/clucene)
{'fraction_non_alphanumeric': 0.09639323332269391, 'fraction_numerical': 0.007979572294924991, 'mean_word_length': 2.47202216066482, 'pattern_counts': {'":': 58, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2784334', 'n_tokens_mistral': 1824, 'n_tokens_neox': 1685, 'n_words': 725}