<commit_before>OBGS - Open Banner Grabbing System
==================================

# Installation

##Automatic installation

There is an script under the folder install.sh that will install OBGS and its dependencies with a configuration by default.

WARNING: This script is still under development so use it at your own risk!

##Manual installation

First, install dependencies. For example, with a debian based system:

```bash
$ sudo apt-get install npm nodejs-legacy mongodb-server nginx
$ sudo pip install eve
$ pip install pymongo==2.9.1
$ sudo npm install -g bower gulp
```

Then, we need to install [masscan](https://github.com/robertdavidgraham/masscan)

```
$ sudo apt-get install git gcc make libpcap-dev
$ git clone https://github.com/robertdavidgraham/masscan
$ cd masscan
$ make
$ sudo make install
```

We need to add a iptables filter to grab the banners
```
iptables -A INPUT -p tcp --dport 60000 -j DROP
```
Reference masscan's readme for further instructions.

When deploying in a server, if `systemd` is available:
```
$ sudo cp install/obgs-farmer.service /etc/systemd/system/
$ sudo cp install/obgs-api.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl start obgs-farmer.service obgs-api.service
```

### Frontend

Use `gulp` to compile the frontend, and setup apache/nginx
to serve the `public` folder (see wiki for config examples)

```bash
$ cd frontend
$ npm install
$ bower install
$ gulp
```

### API

Edit `settings.py` to match the MongoDB settings and the data schema.

Use systemd service to control the api service, or run manually `api/server.py`.

### Farmer

Use systemd service to control the farmer service, or run manually `farmer/farmer.py`.

<commit_msg>clean code<commit_after>OBGS - Open Banner Grabbing System
==================================

# Installation

##Automatic installation

There is an script under the folder install.sh that will install OBGS and its dependencies with a configuration by default.

WARNING: This script is still under development so use it at your own risk!

##Manual installation

First, install dependencies. For example, with a debian based system:

```bash
$ sudo apt-get install npm nodejs-legacy mongodb-server nginx
$ sudo pip install pymongo==2.9.1
$ sudo pip install eve
$ sudo npm install -g bower gulp
```

Then, we need to install [masscan](https://github.com/robertdavidgraham/masscan)

```
$ sudo apt-get install git gcc make libpcap-dev
$ git clone https://github.com/robertdavidgraham/masscan
$ cd masscan
$ make
$ sudo make install
```

We need to add a iptables filter to grab the banners
```
iptables -A INPUT -p tcp --dport 60000 -j DROP
```
Reference masscan's readme for further instructions.

When deploying in a server, if `systemd` is available:
```
$ sudo cp install/obgs-farmer.service /etc/systemd/system/
$ sudo cp install/obgs-api.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl start obgs-farmer.service obgs-api.service
```

### Frontend

Use `gulp` to compile the frontend, and setup apache/nginx
to serve the `public` folder (see wiki for config examples)

```bash
$ cd frontend
$ npm install
$ bower install
$ gulp
```

### API

Edit `settings.py` to match the MongoDB settings and the data schema.

Use systemd service to control the api service, or run manually `api/server.py`.

### Farmer

Use systemd service to control the farmer service, or run manually `farmer/farmer.py`.


{'fraction_non_alphanumeric': 0.10898550724637682, 'fraction_numerical': 0.00463768115942029, 'mean_word_length': 4.367029548989113, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3120412', 'n_tokens_mistral': 1102, 'n_tokens_neox': 1021, 'n_words': 428}