<commit_before># Introduction

Most of the data analysis used in this project is conducted with
[R](http://www.r-project.org) (a free software environment for statistical computing and graphics).
Data analysis was mostly done as a classification task trying to classify
Android applications either to malware or goodware. Classifiers that
are used are:
- Support Vector Machines (ref.)
- Random Forest (ref.)
- LASSO (ref.)
- Ridge (ref.)

# Dependencies

Since we wanted to use the newest (and thus probably the best) version of R and
its packages data analysis was done with R 3.1.1. Packages used are:
- caret (version 6.0-35)
- doMC (version 1.3.3)
- e1071 (version 1.6-4)
- foreach (version 1.4.2)
- glmnet (version 1.9-8)
- Matrix (version 1.1-4)
- randomForest (version 4.6-10)
- rpart (version 4.1-8)
- rpart.plot (version 1.4-4)
- RRF (version 1.6)
- dplyr (version >= 0.2)

# Installation

R version 3.1.1 is not in the official Ubuntu 12.04 LTS repository so
it should be built from the source. To do so appropriate build tools
need to be installed. After build tools are installed and R is built
and installed appropriate R packages should be installed.

## Ubuntu packages

Packages needed to build R-3.1.1 from source can be obtained as follows:

```bash
$ sudo apt-get install build-essential fort77 libreadline-dev
```<commit_msg>data analysis readme update<commit_after># Introduction

Most of the data analysis used in this project is conducted with
[R](http://www.r-project.org) (a free software environment for statistical computing and graphics).
Data analysis was mostly done as a classification task trying to classify
Android applications either to malware or goodware. Classifiers that
are used are:
- [Support Vector Machines](http://www.support-vector-machines.org/)
- [Random Forest](https://www.stat.berkeley.edu/~breiman/RandomForests/)
- [LASSO](https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html)
- [Ridge](https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html)

# Dependencies

Since we wanted to use the newest (and thus probably the best) version of R and
its packages data analysis was done with R 3.1.1. Packages used are:
- caret (version 6.0-35)
- doMC (version 1.3.3)
- e1071 (version 1.6-4)
- foreach (version 1.4.2)
- glmnet (version 1.9-8)
- Matrix (version 1.1-4)
- randomForest (version 4.6-10)
- rpart (version 4.1-8)
- rpart.plot (version 1.4-4)
- RRF (version 1.6)
- dplyr (version >= 0.2)

# Installation

R version 3.1.1 is not in the official Ubuntu 12.04 LTS repository so
it should be built from the source. To do so appropriate build tools
need to be installed. After build tools are installed and R is built
and installed appropriate R packages should be installed.

## Ubuntu packages

Packages needed to build R-3.1.1 from source can be obtained as follows:

```bash
$ sudo apt-get install build-essential fort77 libreadline-dev
```
{'fraction_non_alphanumeric': 0.09793103448275862, 'fraction_numerical': 0.03586206896551724, 'mean_word_length': 3.942078364565588, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1743502', 'n_tokens_mistral': 967, 'n_tokens_neox': 890, 'n_words': 379}