<commit_before># gammaray

gammaray is a system implementing disk-based
[introspection](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.11.8367)
for virtual machines.  It thus far works with VMs that have disks virtualized
via [QEMU](http://www.qemu.org); however, this limitation is only due to the
scarcity of developer time.  Conceptually, and practically, gammaray can
perform introspection with any source of raw disk writes.  The instructions
below assume an [Ubuntu 12.04 LTS](http://releases.ubuntu.com/precise/) host,
although they should be similar for most distributions of Linux.

## Installation Procedure 
 
1. Follow the non-optional [dependencies](#dependencies) instructions

2. Follow the [installation procedure](#installing-gammaray) for gammaray

3. Follow the [example](#example-creation-of-gammaray-supported-disk-layout)
   virtul disk creation instructions

4. Follow the [instructions](#gammaray-pipeline) for running the gammaray
   pipeline

5. Check that things are working by using `gray-fs` and `tail` from outside
   the guest on a known text file.  Add lines within the guest and ensure that
   they eventually become visible via the `gray-fs` mounted file system.

   ```bash
   gray-fs disk.raw /mnt/disk -s -d
   ```

   Replace `/mnt/disk` with whatever folder you'd like to use as the read-only
   view into the real-time guest file system.  `gray-fs` doesn't properly parse
   command line options yet, so type the command almost exactly as it appears
   above (`-s -d` force single-threaded, debug mode).

   * Example check command outside the VM guest

      ```bash
      tail /mnt/disk/home/wolf/test.txt
      ```

   * Example check command within the VM guest

      ```bash
      while 1; do echo 'test' >> test.txt; sleep 3; done 
      ```

## License

All source code, documentation, and related artifacts associated with the
gammaray open source project are licensed under the [Apache License, Version
2.0](http://www.apache.org/licenses/LICENSE-2.0.html).

A copy of this license is reproduced in the [LICENSE](LICENSE) file, and the
licenses of dependencies and included code are enumerated in the
[NOTICE](NOTICE) file.


## Dependencies

The following libraries are needed to build and execute gammaray (most
developers can skip the Python libraries):

1. [hiredis](https://github.com/redis/hiredis) [BSD 3-clause] - the Redis
   client-side library

   ```bash
   sudo apt-get install libhiredis-dev libhiredis0.10
   ```

2. [event](http://libevent.org/) [BSD 3-clause] - the libevent event-driven
   networking library

    ```bash
    sudo apt-get install libevent-dev libevent-2.0.5 
    ```

3. [bson](https://pypi.python.org/pypi/bson/0.3.2) [Optional, BSD 3-clause] -
   Python BSON library

4. [redis-py](https://github.com/andymccurdy/redis-py) [Optional, MIT] - Python
   hiredis wrapper

The Python libraries are optional if you want to write or execute Python
monitors consuming gammaray's publish-subscribe stream of file-level updates.

In addition to `libhiredis` and `libevent`, gammaray requires a slightly
modified version of QEMU.  Clone the QEMU repository, apply a patch for
gammaray support, and then compile a gammaray-friendly QEMU.

1. Get the official QEMU source tree
 
   ```bash
   git clone git://git.qemu-project.org/qemu.git
   ```

2. Checkout a specific tagged commit to apply our patch cleanly
   <commit_msg>gammaray: adding Redis server install to the dependencies<commit_after># gammaray

gammaray is a system implementing disk-based
[introspection](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.11.8367)
for virtual machines.  It thus far works with VMs that have disks virtualized
via [QEMU](http://www.qemu.org); however, this limitation is only due to the
scarcity of developer time.  Conceptually, and practically, gammaray can
perform introspection with any source of raw disk writes.  The instructions
below assume an [Ubuntu 12.04 LTS](http://releases.ubuntu.com/precise/) host,
although they should be similar for most distributions of Linux.

## Installation Procedure 
 
1. Follow the non-optional [dependencies](#dependencies) instructions

2. Follow the [installation procedure](#installing-gammaray) for gammaray

3. Follow the [example](#example-creation-of-gammaray-supported-disk-layout)
   virtul disk creation instructions

4. Follow the [instructions](#gammaray-pipeline) for running the gammaray
   pipeline

5. Check that things are working by using `gray-fs` and `tail` from outside
   the guest on a known text file.  Add lines within the guest and ensure that
   they eventually become visible via the `gray-fs` mounted file system.

   ```bash
   gray-fs disk.raw /mnt/disk -s -d
   ```

   Replace `/mnt/disk` with whatever folder you'd like to use as the read-only
   view into the real-time guest file system.  `gray-fs` doesn't properly parse
   command line options yet, so type the command almost exactly as it appears
   above (`-s -d` force single-threaded, debug mode).

   * Example check command outside the VM guest

      ```bash
      tail /mnt/disk/home/wolf/test.txt
      ```

   * Example check command within the VM guest

      ```bash
      while 1; do echo 'test' >> test.txt; sleep 3; done 
      ```

## License

All source code, documentation, and related artifacts associated with the
gammaray open source project are licensed under the [Apache License, Version
2.0](http://www.apache.org/licenses/LICENSE-2.0.html).

A copy of this license is reproduced in the [LICENSE](LICENSE) file, and the
licenses of dependencies and included code are enumerated in the
[NOTICE](NOTICE) file.


## Dependencies

The following libraries are needed to build and execute gammaray (most
developers can skip the Python libraries):

1. [hiredis](https://github.com/redis/hiredis) [BSD 3-clause] - the Redis
   client-side library

   ```bash
   sudo apt-get install libhiredis-dev libhiredis0.10
   ```

   and install a Redis server if you don't have one already:

   ```bash
   sudo apt-get install redis-server
   ```

2. [event](http://libevent.org/) [BSD 3-clause] - the libevent event-driven
   networking library

    ```bash
    sudo apt-get install libevent-dev libevent-2.0.5 
    ```

3. [bson](https://pypi.python.org/pypi/bson/0.3.2) [Optional, BSD 3-clause] -
   Python BSON library

4. [redis-py](https://github.com/andymccurdy/redis-py) [Optional, MIT] - Python
   hiredis wrapper

The Python libraries are optional if you want to write or execute Python
monitors consuming gammaray's publish-subscribe stream of file-level updates.

In addition to `libhiredis` and `libevent`, gammaray requires a slightly
modified version of QEMU.  Clone the QEMU repository, apply a patch for
gammaray support, and then compile a gammaray-friendly QEMU.

1. Get the official QEMU source tree
 
   ```bash
   git clone git://git.qemu-project.org/qemu.git
   ```

2. Checkout a specific tagged commit to apply our patch cleanly
   
{'fraction_non_alphanumeric': 0.09183381088825215, 'fraction_numerical': 0.012320916905444125, 'mean_word_length': 3.7393075356415477, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4751843', 'n_tokens_mistral': 2196, 'n_tokens_neox': 2043, 'n_words': 819}