<commit_before># DIG

DIG is a visual analysis tool based on a faceted search engine that enables rapid, interactive exploration of large data sets. Users refine their queries by entering search terms or selecting values from lists of aggregated attributes. DIG can be quickly configured for a new domain through simple configuration. 

## Prerequisites
- [Node.js]
- [grunt-cli] installed globally (npm i -g grunt-cli)
- [Bower] installed globally (npm i -g bower)
- [MongoDB]
- [Elasticsearch]

### MongoDB Notes

MongoDB is used to store relevant data for users.

When running in development mode:
The necessary MongoDB collections are newly instantiated each time the application starts and auto populated with records specified in `server/config/seed.js`. 

Similarly, in a production environment a user would be supplied in the request header, but in development, the header will be auto populated with the mock user specified in the seed.js file. 

### Elasticsearch Setup

Refer to installation instructions at www.elastic.co/downloads/elasticsearch. DIG can be configured and used with any document corpus which can be loaded and indexed into Elasticsearch. If the Elasticsearch server is installed on a host machine other than the machine which dig executes, it will be necessary to either configure a reverse proxy and DIG to direct all Elasticsearch requests through the proxy first, or to enable CORS in Elasticsearch (refer to Elasticsearch reference).
A sample data set and installer can be used in initialize Elasticsearch. This is found in dig/data/ import_test_data.sh.
Depending on the version of Elasticsearch (>= 1.4.x), it may be necessary to add add **http.cors.enabled : true** to elasticsearch.yml in the config directory of Elasticsearch

# Contributing to DIG

[DIG] is built upon the [Yeoman] based [AngularJS Full-Stack Generator]. To use the yeoman angular-fullstack-generator to create new components for
this application, see the documentation at
[documentation][AngularJS Full-Stack Generator] which has useful generators to
create angular.js elements such as models, views, controllers, routes,
directives, factories, etc.

When cloning this repository for the first time, run these two commands:

  `cd dig`

  `npm install`

  `bower install`

The workflow is:

  `grunt serve # starts the server and opens up the home page in your browser`

  `grunt test # run all of the tests`

  `grunt ... # see Gruntfile.js and documentation for grunt`

Once the server is started with grunt, when you modify the
application, the browser will refresh and show those changes.

## Setting Your Environment Variables

Various application parameters can be configured via environment variables - these parameters are assigned default values in server/config/environment/index.js.  The values in the files development.js and production.js override the values in index.js when DIG is run in development and production mode, respectively.  The parameters are:

| Variable  | Default  Value | Description |
| :------------ | :--------------- | :----- |
| NODE_ENV | 'development' | Determines whether DIG runs in production or development mode.  ['development, 'test', 'production'] |
| PORT | 9000 | The port on which the application listens for browser requests |
| EUI_SERVER_URL | 'http://localhost' | The base URL at which [Elasticsearch] can be found |
| EUI_SERVER_PORT | 9200 | The port on which to connect to [Elasticsearch] |
| EUI_SEARCH_INDEX | 'dig' | The name of the [Elasticsearch] index |
| BLUR_IMAGES | 'blur' | Determines whether images linked to the [Elasticsearch] data are blurred in the user interface |
| BLUR_PERCENT | 2.5 | The amount of image blurring |
| MONGOHQ_URL | 'mongodb://localhost/dig-dev' | In development mode, this variable is ignored and the default value is used |


## Deployment

To package the application for deployment:

  `grunt build # builds the application and prepares the dist/ directory`

  `bash ./package.sh # create a makeself archive that can be used to easily deploy dig` **(please note that this requires that makeself is installed on your system)**`

The `package.sh` script creates `dig_deploy.sh`.  Running `dig_deploy.sh` will extract the dig application and download docker-compose into a directory called `dig/`.  Inside the `dig/` directory is `run.sh` which can be used to run the dig application as a daemon.

All deployment configuration for dig is done by modifying the docker-compose.yml file in the `dig/` directory.

## Configuration Files
This section of the [DIG] README will highlight [DIG]-specific configuration elements.

### Overview
[DIG] manages system and User Interface (UI) configuration variables server-side.  Both system and UI variables are defined within `server/config/environment/index.js`.  [DIG] loads this file when the system starts.  `server/app.js` uses system configuration elements as necessary or passes them along to server-side modules. `server/routes.js` defines a `/config` REST endpoint that returns all UI configuration elements within a JSON object.

The UI is built upon the [AngularJS] framework and loads configuration variables during the bootstrapping phase of [DIG]'s [AngularJS] application.  The [DIG] app pulls variabes via the `/config` REST endpoint and stores them as [AngularJS] constants or values.
<commit_msg>Edit to README file.<commit_after># DIG

DIG is a visual analysis tool based on a faceted search engine that enables rapid, interactive exploration of large data sets. Users refine their queries by entering search terms or selecting values from lists of aggregated attributes. DIG can be quickly configured for a new domain through simple configuration. 

## Prerequisites
- [Node.js]
- [grunt-cli] installed globally (npm i -g grunt-cli)
- [Bower] installed globally (npm i -g bower)
- [MongoDB]
- [Elasticsearch]

### MongoDB Notes

MongoDB is used to store relevant data for users.

When running in development mode:
The necessary MongoDB collections are newly instantiated each time the application starts and auto populated with records specified in `server/config/seed.js`. 

Similarly, in a production environment a user would be supplied in the request header, but in development, the header will be auto populated with the mock user specified in the seed.js file. 

### Elasticsearch Setup

Refer to installation instructions at www.elastic.co/downloads/elasticsearch. DIG can be configured and used with any document corpus which can be loaded and indexed into Elasticsearch. If the Elasticsearch server is installed on a host machine other than the machine which dig executes, it will be necessary to either configure a reverse proxy and DIG to direct all Elasticsearch requests through the proxy first, or to enable CORS in Elasticsearch (refer to Elasticsearch reference).
A sample data set and installer can be used in initialize Elasticsearch. This is found in dig/data/ import_test_data.sh.
Depending on the version of Elasticsearch (>= 1.4.x), it may be necessary to add add **http.cors.enabled : true** to elasticsearch.yml in the config directory of Elasticsearch

# Contributing to DIG

[DIG] is built upon the [Yeoman] based [AngularJS Full-Stack Generator]. To use the yeoman angular-fullstack-generator to create new components for
this application, see the documentation at
[documentation][AngularJS Full-Stack Generator] which has useful generators to
create angular.js elements such as models, views, controllers, routes,
directives, factories, etc.

When cloning this repository for the first time, run these two commands:

  `cd dig`

  `npm install`

  `bower install`

The workflow is:

  `grunt serve # starts the server and opens up the home page in your browser`

  `grunt test # run all of the tests`

  `grunt ... # see Gruntfile.js and documentation for grunt`

Once the server is started with grunt, when you modify the
application, the browser will refresh and show those changes.

## Setting Your Environment Variables

Various application parameters can be configured via environment variables - these parameters are assigned default values in server/config/environment/index.js.  The values in the files development.js and production.js override the values in index.js when DIG is run in development and production mode, respectively.  The parameters are:

| Variable  | Default  Value | Description |
| :------------ | :--------------- | :----- |
| NODE_ENV | 'development' | Determines whether DIG runs in production or development mode.  ['development, 'test', 'production'] |
| PORT | 9000 | The port on which the application listens for browser requests |
| EUI_SERVER_URL | 'http://localhost' | The base URL at which [Elasticsearch] can be found |
| EUI_SERVER_PORT | 9200 | The port on which to connect to [Elasticsearch] |
| EUI_SEARCH_INDEX | 'dig' | The name of the [Elasticsearch] index |
| BLUR_IMAGES | 'true' | Determines whether images linked to the [Elasticsearch] data are blurred in the user interface |
| BLUR_PERCENT | 2.5 | The amount of image blurring |
| MONGOHQ_URL | 'mongodb://localhost/dig-dev' | In development mode, this variable is ignored and the default value is used |


## Deployment

To package the application for deployment:

  `grunt build # builds the application and prepares the dist/ directory`

  `bash ./package.sh # create a makeself archive that can be used to easily deploy dig` **(please note that this requires that makeself is installed on your system)**`

The `package.sh` script creates `dig_deploy.sh`.  Running `dig_deploy.sh` will extract the dig application and download docker-compose into a directory called `dig/`.  Inside the `dig/` directory is `run.sh` which can be used to run the dig application as a daemon.

All deployment configuration for dig is done by modifying the docker-compose.yml file in the `dig/` directory.

## Configuration Files
This section of the [DIG] README will highlight [DIG]-specific configuration elements.

### Overview
[DIG] manages system and User Interface (UI) configuration variables server-side.  Both system and UI variables are defined within `server/config/environment/index.js`.  [DIG] loads this file when the system starts.  `server/app.js` uses system configuration elements as necessary or passes them along to server-side modules. `server/routes.js` defines a `/config` REST endpoint that returns all UI configuration elements within a JSON object.

The UI is built upon the [AngularJS] framework and loads configuration variables during the bootstrapping phase of [DIG]'s [AngularJS] application.  The [DIG] app pulls variabes via the `/config` REST endpoint and stores them as [AngularJS] constants or values.

{'fraction_non_alphanumeric': 0.06844710056961434, 'fraction_numerical': 0.0022411056121019704, 'mean_word_length': 4.417298937784522, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5021838', 'n_tokens_mistral': 2782, 'n_tokens_neox': 2648, 'n_words': 1445}