<commit_before># xray / data controller indicators for smartphone apps

# Pipeline 1: Dynamic analysis

In this pipeline we will donwload and install apps from the app store, set up mitmproxy, interact with the apps, and do some analysis on their traffic.

## capture logs (dynamic approach)

0. check out the repo
1. create a virtualenv, enter the virtualenv; pip install mitmproxy
2. create your own `mitm-config.json` based on mitm-config.sample.json to customise your environment
3. run the script like `mitmdump -s mitm-save.py -p 8081`

While you were interacting with an app on your device you should see traffic on your local console, and logs to be saved in the directory specified in your mitm-config.json file. 

## process the logs

1. create a copy of `config.sample.json` called `config.json`
2. you need bower, make sure you have it (or `npm install -g bower`)
3. at the top level, run bower install
4. go to the mitm directory
5. run `npm install`
6. make a 'mitm_out' directory (or whatever destination you specified in `config.json`)
7. run `node parse-output.js` to generate the data files 
8. stand at the base directory, run `http-server`
9. point a browser to to localhost:8080/prototypes/p1.html

# Pipeline 2: Static analysis

In this pipeline we will install apps from the app store, using an automated UI on an android device, pull the APKs from the device, do some static code analysis on the APKs using LibRadar, and store the results of further analysis.

## Dependencies

Ensure you have JDK version 1.8.0_25

## Downloading APKs via the app store
Requires root access to the android device.

1. Install android-developer-tools.
2. Get the device id adb devices
3. Modify `install-from-store-random-motog.py' with the device id.
4. Start ADB demon as root adb root
5. Run `python install-from-store-random-motog.py'

This should result in the APK files being written to a directory outside the repo called `apks'
<commit_msg>instructions for setting up monkey<commit_after># xray / data controller indicators for smartphone apps

# Pipeline 1: Dynamic analysis

In this pipeline we will donwload and install apps from the app store, set up mitmproxy, interact with the apps, and do some analysis on their traffic.

## capture logs (dynamic approach)

0. check out the repo
1. create a virtualenv, enter the virtualenv; pip install mitmproxy
2. create your own `mitm-config.json` based on mitm-config.sample.json to customise your environment
3. run the script like `mitmdump -s mitm-save.py -p 8081`

While you were interacting with an app on your device you should see traffic on your local console, and logs to be saved in the directory specified in your mitm-config.json file. 

## process the logs

1. create a copy of `config.sample.json` called `config.json`
2. you need bower, make sure you have it (or `npm install -g bower`)
3. at the top level, run bower install
4. go to the mitm directory
5. run `npm install`
6. make a 'mitm_out' directory (or whatever destination you specified in `config.json`)
7. run `node parse-output.js` to generate the data files 
8. stand at the base directory, run `http-server`
9. point a browser to to localhost:8080/prototypes/p1.html

## Automated MITM traffic logging

Another way is to use monkey to simulate user events. Logs traffic data using mitmproxy.

Requires Android Developer Studio tools (for monkeyrunner)

Requires rooted phone with adb root shell access, network log tool with superuser permissions.

Requires working mitmproxy setting up, as described above.

1. Edit `logger.sh` to specify packages to log
2. turn on network logging on the device
3. Activate the python virtual environment in xray directory, run `mitmdump -s mitm-save.py -p 8080`
4. Run `logger.sh` to start UI fuzzing on packages and logging mitmproxy traffic in the background
5. When the fuzzer has done its business, `Ctrl+C` the mitmproxy logging.
6. There will be a big CSV in the data directory. It can be split into smaller csvs for each app using `csplit filename.csv 'endsession_apppackagename'` (an ugly solution, will do for now!). Place in the data directory
7. export network log data from device, place in the data directory
8. auto-cleaner.py is work in progress, it uses the network log to filter out non-app traffic from the mitmproxy log csv file.


# Pipeline 2: Static analysis

In this pipeline we will install apps from the app store, using an automated UI on an android device, pull the APKs from the device, do some static code analysis on the APKs using LibRadar, and store the results of further analysis.

## Dependencies

Ensure you have JDK version 1.8.0_25

## Downloading APKs via the app store
Requires root access to the android device.

1. Install android-developer-tools.
2. Get the device id adb devices
3. Modify `install-from-store-random-motog.py` with the device id.
4. Start ADB demon as root adb root
5. Run `python install-from-store-random-motog.py`

This should result in the APK files being written to a directory outside the repo called `apks'

{'fraction_non_alphanumeric': 0.059207920792079205, 'fraction_numerical': 0.015841584158415842, 'mean_word_length': 4.107178968655207, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7601353', 'n_tokens_mistral': 1470, 'n_tokens_neox': 1403, 'n_words': 786}