<commit_before># Dromedar: Drill on Mesos

This is a simple wrapper/enabler for running [Apache Drill](http://drill.apache.org/) on [Apache Mesos](http://mesos.apache.org/).

Dromedar (DRill On MEsos aDAptoR) gets launched via [Marathon](https://github.com/mesosphere/marathon/) and whenever a query request comes in, it launches a number of Drillbits, depending on the dataset size under query. The query-scale-factor determines how many Drillbits are launched and defaults to 1 Drillbit per 100MB (1:100) or `qfs=100` for short.


## Dependencies

* Apache [Drill 0.8.0](http://getdrill.org/drill/download/apache-drill-0.8.0.tar.gz)
* Apache [Mesos 0.22.x](http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz)
* [Marathon 0.8.1](https://downloads.mesosphere.io/marathon/v0.8.1/marathon-0.8.1.tgz)
* [marathon-python](https://github.com/thefactory/marathon-python)

## Usage

### Install

TBD

### Run

TBD

## To Do

- [ ] Bootstrap (install Drill, launch Dromedar via Marathon)
- [ ] QSF implementation
- [ ] Playa test
- [ ] Cluster test
- [ ] Doc, examples, video<commit_msg>updates todos<commit_after># Dromedar: Drill on Mesos

This is a simple wrapper/enabler for running [Apache Drill](http://drill.apache.org/) on [Apache Mesos](http://mesos.apache.org/).

Dromedar (DRill On MEsos aDAptoR) gets launched via [Marathon](https://github.com/mesosphere/marathon/) and whenever a query request comes in, it launches a number of Drillbits, depending on the dataset size under query. The query-scale-factor determines how many Drillbits are launched and defaults to 1 Drillbit per 100MB (1:100) or `qfs=100` for short.


## Dependencies

* Apache [Drill 0.8.0](http://getdrill.org/drill/download/apache-drill-0.8.0.tar.gz)
* Apache [Mesos 0.22.x](http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz)
* [Marathon 0.8.1](https://downloads.mesosphere.io/marathon/v0.8.1/marathon-0.8.1.tgz)
* [marathon-python](https://github.com/thefactory/marathon-python)

## Usage

### Install

TBD

### Run

TBD

## To Do

- [x] Bootstrap (install Drill, launch Dromedar via Marathon)
- [ ] Worker deployment
- [ ] HAProxy?
- [ ] QSF implementation
- [ ] Playa test
- [ ] Cluster test
- [ ] Doc, examples, video
{'fraction_non_alphanumeric': 0.14799820062977959, 'fraction_numerical': 0.033288349077822764, 'mean_word_length': 4.054545454545455, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2289150', 'n_tokens_mistral': 852, 'n_tokens_neox': 797, 'n_words': 202}