<commit_before>### System Overview:

The mongo-connector system is designed to hook up mongoDB to any backend. This allows all the
documents in mongoDB to be stored in some other system, and both mongo and the backend will remain
in sync while the connector is running.

## Usage:
---------

Since the connector does real time syncing, it is necessary to have MongoDB running, although the
connector will work with both sharded and non sharded configurations. To start the system, simply
run "python mongo_connector.py". It is likely, however, that you will need to specify some command
line options to work with your setup. They are described below:

"-m" or "--mongos" is to specify the mongos address, which is a host:port pair, or for clusters with
 one shard, the primary's address. For example, "-m localhost:27217" would be a valid argument
 to "-m".

"-b" or "--backend-url" is to specify the URL to the backend engine being used. For example, if you
were using Solr out of the box, you could use "-b http://localhost:8080/solr" with the
SolrDocManager to establish a proper connection.

"-o" or "--oplog-ts" is to specify the name of the file that stores the oplog progress timestamps.
This file is used by the system to store the last timestamp read on a specific oplog. This allows
for quick recovery from failure.

"-n" or "--namespace-set" is used to specify the namespaces we want to consider. For example, if we
wished to store all documents from the test.test and alpha.foo namespaces, we could use
"-n test.test,alpha.foo". Presently, there is no way to indicate "all namespaces" - if you want to
store every single namespace, you must specify them all in a comma delimited format as an argument
to "-n".

"-u" or "--unique-key" is used to specify the uniqueKey used by the backend. The default is "_id",
which can be noted by "-u _id"

`-a` or `--auth-file` is used to specify the path to the authentication key file. This file is used

by mongos to authenticate connections to the shards, and we'll use it in the oplog threads. If
authentication is not used, then this field can be left empty as the default is None.

An example of combining all of these is:

	python mongo_connector.py -m localhost:27217 -b http://localhost:8080/solr -o oplog_progress.txt -n alpha.foo,test.test -u _id -a auth.txt


doc_manager:

This is the only file that is engine specific. In the current version, we have provided sample<commit_msg>readme changes<commit_after>### System Overview:
---------

The mongo-connector system is designed to hook up mongoDB to any backend. This allows all the
documents in mongoDB to be stored in some other system, and both mongo and the backend will remain
in sync while the connector is running.

## Usage:
---------

Since the connector does real time syncing, it is necessary to have MongoDB running, although the
connector will work with both sharded and non sharded configurations. To start the system, simply
run "python mongo_connector.py". It is likely, however, that you will need to specify some command
line options to work with your setup. They are described below:

`-m` or `--mongos` is to specify the mongos address, which is a host:port pair, or for clusters with
 one shard, the primary's address. For example, "-m localhost:27217" would be a valid argument
 to "-m".

`-b` or `--backend-url` is to specify the URL to the backend engine being used. For example, if you
were using Solr out of the box, you could use "-b http://localhost:8080/solr" with the
SolrDocManager to establish a proper connection.

`-o` or `--oplog-ts` is to specify the name of the file that stores the oplog progress timestamps.
This file is used by the system to store the last timestamp read on a specific oplog. This allows
for quick recovery from failure.

`-n` or `--namespace-set` is used to specify the namespaces we want to consider. For example, if we
wished to store all documents from the test.test and alpha.foo namespaces, we could use
"-n test.test,alpha.foo". Presently, there is no way to indicate "all namespaces" - if you want to
store every single namespace, you must specify them all in a comma delimited format as an argument
to `-n`.

`-u` or `--unique-key` is used to specify the uniqueKey used by the backend. The default is "_id",
which can be noted by "-u _id"

`-a` or `--auth-file` is used to specify the path to the authentication key file. This file is used
by mongos to authenticate connections to the shards, and we'll use it in the oplog threads. If
authentication is not used, then this field can be left empty as the default is None.

An example of combining all of these is:

	python mongo_connector.py -m localhost:27217 -b http://localhost:8080/solr -o oplog_progress.txt -n alpha.foo,test.test -u _id -a auth.txt


doc_manager:

This is the only file that is engine specific. In the current version, we have provided sample
{'fraction_non_alphanumeric': 0.0705233033524121, 'fraction_numerical': 0.007358953393295176, 'mean_word_length': 3.8686567164179104, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2779985', 'n_tokens_mistral': 1384, 'n_tokens_neox': 1320, 'n_words': 790}