<commit_before>[![Build Status](https://travis-ci.org/CSC-IT-Center-for-Science/spark-analysis.svg?branch=master)](https://travis-ci.org/CSC-IT-Center-for-Science/spark-analysis)

# Spark Analysis
Analyzing large datasets using spark 


# Installation
python setup.py sdist to make the distribution.
pip install ./<foldername> to install (run from parent directory)

#Directory Structures (Before Installation)
Place 'etc' and 'spark_jars' in the shared data folder of your cluster. Create tmp and mods directories in shared data folder itself. Edit the config.yml template in etc. Be sure to provide the read and write access to all the 4 directories.
Also make sure, the input files should be somewhere in shared data folder for all the spark workers to read

# Example Usage
```
from sparkles.modules.utils.runner import SparkRunner
sr = SparkRunner('path/to/config.yml')
sr.import_dataset(inputs='/path/to/filename.extension',userdatadir='swift://containerFiles.SparkTest',description='some',details='some')
sr.import_analysis(name='modulename', description='description', details='details', '/path/to/module.py', params='params that are used in module', inputs='type of input it reads', outputs='kind of output it generates')

params = {'tablename': 'ORDERS', 'start_time': 1349159961141, 'end_time': 1349160643981, 'interval': 60}
inputs = ['filename']
sr.run_analysis(modulename='modulename', params=params, inputs=inputs)
```
If you need to generate a feature set out of the modules
```
features = {'userdatadir': 'swift://containerFeatures.SparkTest', 'description': 'event counting', 'details': 'something', 'modulename': 'modulename', 'featureset_name': 'featuresetname'}
params = {'tablename': 'ORDERS', 'start_time': 1349159961141, 'end_time': 1349160643981, 'interval': 60}
inputs = ['filename']

sr.run_analysis(modulename='modulename', params=params, inputs=inputs, features=features)
```<commit_msg>missing params added for clarity<commit_after>[![Build Status](https://travis-ci.org/CSC-IT-Center-for-Science/spark-analysis.svg?branch=master)](https://travis-ci.org/CSC-IT-Center-for-Science/spark-analysis)

# Spark Analysis
Analyzing large datasets using spark 


# Installation
python setup.py sdist to make the distribution.
pip install ./<foldername> to install (run from parent directory)

#Directory Structures (Before Installation)
Place 'etc' and 'spark_jars' in the shared data folder of your cluster. Create tmp and mods directories in shared data folder itself. Edit the config.yml template in etc. Be sure to provide the read and write access to all the 4 directories.
Also make sure, the input files should be somewhere in shared data folder for all the spark workers to read

# Example Usage
```
from sparkles.modules.utils.runner import SparkRunner
sr = SparkRunner('path/to/config.yml')
sr.import_dataset(inputs='/path/to/filename.extension',userdatadir='swift://containerFiles.SparkTest',description='description of dataset',details='details')
sr.import_analysis(name='modulename', description='description', details='details', filepath='/path/to/module.py', params='params that are used in module', inputs='type of input it reads', outputs='kind of output it generates')

params = {'tablename': 'ORDERS', 'start_time': 1349159961141, 'end_time': 1349160643981, 'interval': 60}
inputs = ['filename']
sr.run_analysis(modulename='modulename', params=params, inputs=inputs)
```
If you need to generate a feature set out of the modules
```
features = {'userdatadir': 'swift://containerFeatures.SparkTest', 'description': 'something', 'details': 'something', 'modulename': 'module used to create this', 'featureset_name': 'featuresetname'}
params = {'tablename': 'ORDERS', 'start_time': 1349159961141, 'end_time': 1349160643981, 'interval': 60}
inputs = ['filename']

sr.run_analysis(modulename='modulename', params=params, inputs=inputs, features=features)
```
{'fraction_non_alphanumeric': 0.1302672147995889, 'fraction_numerical': 0.029290853031860225, 'mean_word_length': 6.633333333333334, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4342278', 'n_tokens_mistral': 1235, 'n_tokens_neox': 1106, 'n_words': 357}