<commit_before># DistributedMonitoring

We propose a fault tolerant distributed monitoring system based on Nmap, and possible other tools, for scanning target hosts and networks. The system will also store the result for later retrieval, audit and statistics purposes.

## Set up

### Requirements

Install RabbitMQ server - V3.5.3 or [newer](http://www.rabbitmq.com/download.html)

Install MongoDB - V2.6.3 or [newer](http://www.mongodb.org/downloads)

Install Nmap - V6.40 or [newer](https://nmap.org/download.html) on every machine that runs a `Scanner` component.

Install JRE [1.8](http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html) on all the machines that run one of the four components (FrontEnd, Scanner, Converter, Presenter).

### Building the project manually

Install [Maven](https://maven.apache.org/download.cgi)

Install [JDK 1.8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)

For every module (FrontEnd, Scanner, Converter and Presenter), navigate to the pom.xml and run mvn `mvn -Dmaven.test.skip=true package` (this will create the jar without running unit tests)

### Configuration

Each of the four components (FrontEnd, Scanner, Converter and Presenter) has their own configuration file (conf.properties). The configuration file contains the ip and port for MongoDB and the ip, queue names and credentials for RabbitMQ.

### Using the deployment

As an alternative to manually building the project and creating the config files, the deployment directory can be used. The directory contain the latest stable jars and a directory structure that is easy to deploy and manage.
In order to use the deployment the following environment variables should be set:

* `rabbitHost` IP of the machine hosting RabbitMQ (on all machines hosting `FrontEnd`, `Scanner`, `Converter` and `Presenter` components)
* `feRabbitSendQueue` name of the queue used to send messages by the `FrontEnd` component (on machines hosting FrontEnd components)
* `scRabbitSendQueue` name of the queue used to send messages by the `Scanner` component (on machines hosting Scanner components)
* `scRabbitReceiveQueue` name of the queue used to receive messages by the `Scanner` component (on machines hosting Scanner components)
* `coRabbitSendQueue` name of the queue used to send messages by the `Converter` component (on machines hosting Converter components)
* `coRabbitReceiveQueue` name of the queue used to receive messages by the `Converter` component (on machines hosting Converter components)
* `prRabbitReceiveQueue` name of the queue used to receive messages by the `Presenter` component (on machines hosting Presenter components)
* `rabbitUser` credentials for RabbitMQ (on all machines hosting FrontEnd, Scanner, Converter and Presenter components)
* `rabbitPassword` credentials for RabbitMQ (on all machines hosting FrontEnd, Scanner, Converter and Presenter components)
* `mongoHost` IP of the machine hosting MongoDB (on all machines hosting FrontEnd, Scanner, Converter and Presenter components)
* `mongoPort` port on which MongoDB is running (on all machines hosting FrontEnd, Scanner, Converter and Presenter components)

As the relation between components is `FrontEnd`-`Scanner`-`Converter`-`Presenter` \| `Remediator`, the pairs:
```
feRabbitSendQueue - scRabbitReceiveQueue;
scRabbitSendQueue - coRabbitReceiveQueue;
coRabbitSendQueue - prRabbitReceiveQueue;
```
should have identical names.

Configuration file example for the FrontEnd component:
```
#RabbitMQ connection details
rabbitHost = 192.168.56.102
rabbitSendQueue = commands
rabbitUser = rabbitUser
rabbitPassword = <PASSWORD>

#MongoDB connection details
mongoHost = 192.168.56.101
mongoPort = 27017
```

If the environment variables are set, simply run `/bin/run.sh` for each component (this will create the configuration files and will start the component, more info can be found on the [deployment page](https://github.com/IrimieBogdan/DistributedMonitoring/wiki/Deployment).

##Test/Stress the system

Fallow the guide on the [test page](https://github.com/IrimieBogdan/DistributedMonitoring/wiki/Testing).

##Interacting with the monitoring system

Jobs can be submitted by making a HTTP POST request `http://<ip>:8080/request` with a json body. More details about the request body can be found on [request model page](https://github.com/IrimieBogdan/DistributedMonitoring/wiki/Requests).

clientResponse is a JSON with the following fields:
* `clientId` - the id the client provide, it can be used to retrieve all the jobs for a client
* `command`- the command that will be executed by the system
* `responseAddress` - is the address where the results will be sent
* `processors` - used for filtering and processing of the raw results
* `adapter` - used for adapting the results for a specific system<commit_msg>Remove old deployment description.<commit_after># DistributedMonitoring

We propose a fault tolerant distributed monitoring system based on Nmap, and possible other tools, for scanning target hosts and networks. The system will also store the result for later retrieval, audit and statistics purposes.

## Set up

### Requirements

Install RabbitMQ server - V3.5.3 or [newer](http://www.rabbitmq.com/download.html)

Install MongoDB - V2.6.3 or [newer](http://www.mongodb.org/downloads)

Install Nmap - V6.40 or [newer](https://nmap.org/download.html) on every machine that runs a `Scanner` component.

Install JRE [1.8](http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html) on all the machines that run one of the four components (FrontEnd, Scanner, Converter, Presenter).

### Building the project manually

Install [Maven](https://maven.apache.org/download.cgi)

Install [JDK 1.8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)

For every module (FrontEnd, Scanner, Converter and Presenter), navigate to the pom.xml and run mvn `mvn -Dmaven.test.skip=true package` (this will create the jar without running unit tests)

### Configuration

Each of the four components (FrontEnd, Scanner, Converter and Presenter) has their own configuration file (conf.properties). The configuration file contains the ip and port for MongoDB and the ip, queue names and credentials for RabbitMQ.

### Using the deployment

As an alternative to manually building the project and creating the config files, the deployment directory can be used. The directory contain the latest stable jars and a directory structure that is easy to deploy and manage.
In order to use the deployment the following environment variables should be set:

Configuration file example for the FrontEnd component:
```
#RabbitMQ connection details
rabbitHost = 192.168.56.102
rabbitSendQueue = commands
rabbitUser = rabbitUser
rabbitPassword = <PASSWORD>

#MongoDB connection details
mongoHost = 192.168.56.101
mongoPort = 27017
```

More info can be found on the [deployment page](https://github.com/IrimieBogdan/DistributedMonitoring/wiki/Deployment).

##Test/Stress the system

Fallow the guide on the [test page](https://github.com/IrimieBogdan/DistributedMonitoring/wiki/Testing).

##Interacting with the monitoring system

Jobs can be submitted by making a HTTP POST request `http://<ip>:8080/request` with a json body. More details about the request body can be found on [request model page](https://github.com/IrimieBogdan/DistributedMonitoring/wiki/Requests).

clientResponse is a JSON with the following fields:
* `clientId` - the id the client provide, it can be used to retrieve all the jobs for a client
* `command`- the command that will be executed by the system
* `responseAddress` - is the address where the results will be sent
* `processors` - used for filtering and processing of the raw results
* `adapter` - used for adapting the results for a specific system
{'fraction_non_alphanumeric': 0.07801778907242694, 'fraction_numerical': 0.015247776365946633, 'mean_word_length': 5.1636648394675015, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 10, 'lorem ipsum': 0, 'www.': 8, 'xml': 2}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1607575', 'n_tokens_mistral': 2241, 'n_tokens_neox': 2075, 'n_words': 932}