<commit_before>**Build Status:**
- Master [![Build Status](https://travis-ci.org/mbevilacqua/appcompatprocessor.svg?branch=master)](https://travis-ci.org/mbevilacqua/appcompatprocessor)
- Develop [![Build Status](https://travis-ci.org/mbevilacqua/appcompatprocessor.svg?branch=develop)](https://travis-ci.org/mbevilacqua/appcompatprocessor)




# AppCompatProcessor (Beta)
"_Evolving AppCompat/AmCache data analysis beyond grep_"

AppCompatProcessor has been designed to extract additional value from enterprise-wide AppCompat / AmCache data beyond the classic stacking and grepping techniques.


If you don't fancy reading check the SANS Threat Hunting 2017 presentation [here](https://www.youtube.com/watch?v=-0bYcD3_bBs)

**Note: Windows platform support has been temporarily removed (expect to see it back shortly though)** 



### Installation:

**OSX**
You need Python 2.7+, libregf and pyregf (python bindings) from https://github.com/libyal/libregf

-Option A Source ditribution package from https://github.com/libyal/libregf/releases
- ./configure --enable-python && make
- sudo make install
- python setup.py build
- python setup.py install

-Option B Direct from source
- git clone https://github.com/libyal/libregf.git
- cd libregf/
- ./synclibs.sh
- ./autogen.sh
- ./configure --enable-python && make
- sudo make install
- python setup.py build
- python setup.py install


The rest of the requirements you can handle with 'pip install -r requirements.txt'.

**Linux**
You need Python 2.7+ and 'pip -r requirements.txt' should take care of everything for you.
If you have issues with libregf or pyregf (python bindings for libregf) you can find them here: https://github.com/libyal/libregf

Note: There seems to be an issue with some libregf versions on some linux environments in which libregf apparently provides file paths encoded in UTF16 (breaking sql inserts for us) when it used to do UTF8. Still trying to pinpoint when and why that happens to handle it correctly.

**SIFT v3 users**
Sift comes pre-packaged with libregf v20141030 so 'pip -r requirements.txt' will add the few missing pieces easily.

**Windows**
Note: ACP is currently broken on Windows due to differences in implementation of multiprocessing! Will bring support back shortly.


### Ingestion Modules

The following ingestion formats are currently supported:

* AppCompat in CSV format as produced by ShimCacheParser.py
> Use flags -t -o and store as one file per host.
> File naming convention that you'll need to follow: \<HOSTNAME\>.csv.
> Note that BOM is not currently supported so avoid the '--bom' flag for the time being.
* AppCompat in Redline format
> Redline creates a folder named after the hostname in the AnalysisSession1/Audits folder.<commit_msg>Removing windows version expectations. Not seeing too many requests for it and its non-trivial.<commit_after>**Build Status:**
- Master [![Build Status](https://travis-ci.org/mbevilacqua/appcompatprocessor.svg?branch=master)](https://travis-ci.org/mbevilacqua/appcompatprocessor)
- Develop [![Build Status](https://travis-ci.org/mbevilacqua/appcompatprocessor.svg?branch=develop)](https://travis-ci.org/mbevilacqua/appcompatprocessor)




# AppCompatProcessor (Beta)
"_Evolving AppCompat/AmCache data analysis beyond grep_"

AppCompatProcessor has been designed to extract additional value from enterprise-wide AppCompat / AmCache data beyond the classic stacking and grepping techniques.


If you don't fancy reading check the SANS Threat Hunting 2017 presentation [here](https://www.youtube.com/watch?v=-0bYcD3_bBs)

**Note: Windows platform support has been temporarily removed (expect to see it back shortly though)** 



### Installation:

**OSX**
You need Python 2.7+, libregf and pyregf (python bindings) from https://github.com/libyal/libregf

-Option A Source ditribution package from https://github.com/libyal/libregf/releases
- ./configure --enable-python && make
- sudo make install
- python setup.py build
- python setup.py install

-Option B Direct from source
- git clone https://github.com/libyal/libregf.git
- cd libregf/
- ./synclibs.sh
- ./autogen.sh
- ./configure --enable-python && make
- sudo make install
- python setup.py build
- python setup.py install


The rest of the requirements you can handle with 'pip install -r requirements.txt'.

**Linux**
You need Python 2.7+ and 'pip -r requirements.txt' should take care of everything for you.
If you have issues with libregf or pyregf (python bindings for libregf) you can find them here: https://github.com/libyal/libregf

Note: There seems to be an issue with some libregf versions on some linux environments in which libregf apparently provides file paths encoded in UTF16 (breaking sql inserts for us) when it used to do UTF8. Still trying to pinpoint when and why that happens to handle it correctly.

**SIFT v3 users**
Sift comes pre-packaged with libregf v20141030 so 'pip -r requirements.txt' will add the few missing pieces easily.

**Windows**
Note: ACP is currently broken on Windows due to differences in implementation of multiprocessing!


### Ingestion Modules

The following ingestion formats are currently supported:

* AppCompat in CSV format as produced by ShimCacheParser.py
> Use flags -t -o and store as one file per host.
> File naming convention that you'll need to follow: \<HOSTNAME\>.csv.
> Note that BOM is not currently supported so avoid the '--bom' flag for the time being.
* AppCompat in Redline format
> Redline creates a folder named after the hostname in the AnalysisSession1/Audits folder.
{'fraction_non_alphanumeric': 0.08689381647737515, 'fraction_numerical': 0.00829277086713539, 'mean_word_length': 4.509433962264151, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 13, 'https://': 18, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2665617', 'n_tokens_mistral': 1644, 'n_tokens_neox': 1532, 'n_words': 657}