<commit_before># coastSHARK

[![Build Status](https://travis-ci.org/smartshark/coastSHARK.svg?branch=master)](https://travis-ci.org/smartshark/coastSHARK)
[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg)(https://smartshark.github.io/coastSHARK/)]

Collect AST Information for smartSHARK.

Currently Java and Python are supported.
For Python the build in [ast](https://docs.python.org/3/library/ast.html) package is used, for java the [javalang](https://github.com/c2nes/javalang) package is used.

## Install

### via PIP
```bash
pip install https://github.com/smartshark/coastSHARK/zipball/master --process-dependency-links
```
The --process-dependency-links switch is needed because we install pycoSHARK as a requirement directly from git.

### via setup.py
```bash
python setup.py install
```

## Run Tests
```bash
python setup.py test
```

## Execution

To run coastSHARK needs an already checked out repository. It also depends on a running MongoDB and that the MongoDB is filled for this project by vcsSHARK.
```bash
python smartshark_plugin.py -U $DBUSER -P $DBPASS -DB $DBNAME -i $PATH_TO_REPOSITORY -r $REVISION_HASH -u $REPOSITORY_GIT_URI -a $AUTHENTICATION_DB
```

## Python AST extraction

Using the builtin ast package has the advantage of not requiring much code. Although, it has the drawback that it uses the python runtime the coastSHARK is executed with. It is not possible to extract newer python 3.6 nodes if coastSHARK runs on python 3.5 (and the nodes are not present in 3.6, e.g., new keywords). 
There is also the problem with python2 code, at the moment coastSHARK just runs the python code through the lib2to3 package to avoid parsing errors.
A future version of the coastSHARK may mitigate this problem by probing for the version of the python file first. Probably by incrementing the python version for each probe if it catches parsing errors until it can parse the file.

## Java AST extraction

The node types are defined by the javalang package. Other packages for parsing java code and extracting the AST may have different names for the node types.<commit_msg>add doc badge on top2<commit_after># coastSHARK

[![Build Status](https://travis-ci.org/smartshark/coastSHARK.svg?branch=master)](https://travis-ci.org/smartshark/coastSHARK)
[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://smartshark.github.io/coastSHARK/)

Collect AST Information for smartSHARK.

Currently Java and Python are supported.
For Python the build in [ast](https://docs.python.org/3/library/ast.html) package is used, for java the [javalang](https://github.com/c2nes/javalang) package is used.

## Install

### via PIP
```bash
pip install https://github.com/smartshark/coastSHARK/zipball/master --process-dependency-links
```
The --process-dependency-links switch is needed because we install pycoSHARK as a requirement directly from git.

### via setup.py
```bash
python setup.py install
```

## Run Tests
```bash
python setup.py test
```

## Execution

To run coastSHARK needs an already checked out repository. It also depends on a running MongoDB and that the MongoDB is filled for this project by vcsSHARK.
```bash
python smartshark_plugin.py -U $DBUSER -P $DBPASS -DB $DBNAME -i $PATH_TO_REPOSITORY -r $REVISION_HASH -u $REPOSITORY_GIT_URI -a $AUTHENTICATION_DB
```

## Python AST extraction

Using the builtin ast package has the advantage of not requiring much code. Although, it has the drawback that it uses the python runtime the coastSHARK is executed with. It is not possible to extract newer python 3.6 nodes if coastSHARK runs on python 3.5 (and the nodes are not present in 3.6, e.g., new keywords). 
There is also the problem with python2 code, at the moment coastSHARK just runs the python code through the lib2to3 package to avoid parsing errors.
A future version of the coastSHARK may mitigate this problem by probing for the version of the python file first. Probably by incrementing the python version for each probe if it catches parsing errors until it can parse the file.

## Java AST extraction

The node types are defined by the javalang package. Other packages for parsing java code and extracting the AST may have different names for the node types.
{'fraction_non_alphanumeric': 0.08399812294697326, 'fraction_numerical': 0.005396527451900516, 'mean_word_length': 4.616600790513834, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2385112', 'n_tokens_mistral': 1276, 'n_tokens_neox': 1201, 'n_words': 537}