<commit_before>Agentless System Crawler
========================

**Disclaimer:**
---------------

```
"The strategy is definitely: first make it work, then make it right, and, finally, make it fast."
```
The current state of this project is in the middle of "make it right".

**Prereqs and Building:**
-----------
To run the crawler you'll need to do the following steps in the environment
in which you want it to run:

 * Install Python 2.7.5
 * apt-get install python-dev
 * apt-get install python-pip
 * pip install psutil
 * pip install netifaces
 * pip install bottle
 * pip install requests
 * pip install python-dateutil

If you want to run the crawler in a container then build the `crawler` image
using the provided `Dockerfile`:

`docker build -t crawler .`

or run:

`make build`

To run the test cases, run:

`make test`

**Running the Crawler:**
------------------------
To run the crawler use:
```
sudo python crawler/crawlyer.py ...
```
if running it natively on the Docker host system. 

Use:
```
sudo docker run \
    --privileged \
    --net=host \
    --pid=host \
    -v /cgroup:/cgroup:ro \
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $PWD/output:/crawler/output \
    -it crawler ...
```
if running it from within a container.

Note: this assumes a directory called `output` is present to store the output
of the crawler scans.

The following examples will use `CRAWLER` as a short-hand notation to mean
one of the above commands.

***Quick tests:***
-----------------

`$ CRAWLER --help`
Should show the help text.

`$ CRAWLER`

Should print something like this on stdout:

```
metadata "metadata" {"since_timestamp":1450194519.0,"features":"os,cpu","timestamp":"2015-12-16T20:27:25-0500","since":"BOOT","namespace":"192.168.1.3","system_type":"vm","compress":false}
os "linux" {"boottime":1450194519.0,"ipaddr":["127.0.0.1","192.168.1.3","192.168.122.1","192.168.123.1","172.17.42.1","192.168.3.11"],"osdistro":"Red Hat Enterprise Linux Workstation","osname":"Linux-2.6.32-573.8.1.el6.x86_64-x86_64-with-redhat-6.7-Santiago","osplatform":"x86_64","osrelease":"2.6.32-573.8.1.el6.x86_64","ostype":"linux","osversion":"#1 SMP Fri Sep 25 19:24:22 EDT 2015"}
cpu "cpu-0" {"cpu_idle":61.0,"cpu_nice":0.0,"cpu_user":19.5,"cpu_wait":0.0,"cpu_system":19.5,"cpu_interrupt":0.0,"cpu_steal":0.0,"cpu_used":39}
```

**Crawling Containers:**
-------------------------------
To crawl all containers running on the host use the following command:<commit_msg>Minor tweaks to clean-up README<commit_after>Agentless System Crawler
========================

**Disclaimer:**
---------------

```
"The strategy is definitely: first make it work, then make it right, and, finally, make it fast."
```
The current state of this project is in the middle of "make it right".

**Prereqs and Building:**
-----------
To run the crawler you'll need to do the following steps in the environment
in which you want it to run:

 * Install Python 2.7.5
 * apt-get install python-dev
 * apt-get install python-pip
 * pip install psutil
 * pip install netifaces
 * pip install bottle
 * pip install requests
 * pip install python-dateutil

If you want to run the crawler in a container then build the `crawler` image
using the provided `Dockerfile`:

`sudo docker build -t crawler .`

or run:

`sudo make build`

To run the test cases, run:

`sudo make test`

**Running the Crawler:**
------------------------
To run the crawler natively on the Docker host system, use:
```
sudo python crawler/crawlyer.py ...
```

To run the crawler in a container use:
```
sudo docker run \
    --privileged \
    --net=host \
    --pid=host \
    -v /cgroup:/cgroup:ro \
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $PWD/output:/crawler/output \
    -it crawler ...
```

Note: this assumes a directory called `output` is present to store the output
of the crawler scans, so you may need to create (`mkdir output`) first.

The following examples will use `CRAWLER` as a short-hand notation to mean
one of the above commands.

***Quick tests:***
-----------------

`$ CRAWLER --help`
Should show the help text.

`$ CRAWLER`

Should print something like this on stdout:

```
metadata "metadata" {"since_timestamp":1450194519.0,"features":"os,cpu","timestamp":"2015-12-16T20:27:25-0500","since":"BOOT","namespace":"192.168.1.3","system_type":"vm","compress":false}
os "linux" {"boottime":1450194519.0,"ipaddr":["127.0.0.1","192.168.1.3","192.168.122.1","192.168.123.1","172.17.42.1","192.168.3.11"],"osdistro":"Red Hat Enterprise Linux Workstation","osname":"Linux-2.6.32-573.8.1.el6.x86_64-x86_64-with-redhat-6.7-Santiago","osplatform":"x86_64","osrelease":"2.6.32-573.8.1.el6.x86_64","ostype":"linux","osversion":"#1 SMP Fri Sep 25 19:24:22 EDT 2015"}
cpu "cpu-0" {"cpu_idle":61.0,"cpu_nice":0.0,"cpu_user":19.5,"cpu_wait":0.0,"cpu_system":19.5,"cpu_interrupt":0.0,"cpu_steal":0.0,"cpu_used":39}
```

**Crawling Containers:**
-------------------------------
To crawl all containers running on the host use the following command:
{'fraction_non_alphanumeric': 0.1970260223048327, 'fraction_numerical': 0.06769712385051849, 'mean_word_length': 3.5159010600706715, 'pattern_counts': {'":': 46, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 14, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6996688', 'n_tokens_mistral': 1938, 'n_tokens_neox': 1705, 'n_words': 506}