<commit_before># Cook Scheduler

The Cook scheduler is a Mesos framework that is able to scheduler jobs submitted to it via its REST API from multiple users.
It's raison d'etre is to prioritize the user experience on a shared cluster.
It does so by allowing users to use as much resources as possible, preempting users with greater shares when new users show up, and allowing users to have new jobs jump to the head of their queue.
You can learn more about the algorithms used by Cook in this talk by <NAME> at MesosCon: https://www.youtube.com/watch?v=BkBMYUe76oI.

## Building

You can build the final jar by running `lein uberjar`.

## Running

You can run the scheduler using the build system by running `lein run $config`, where config is an `.edn` file.
See `dev-config.edn` for example usage.
You should be able to start a sample instance (without any persistence) using the `dev-config.edn`.
You'll need to have the environment variable `MESOS_NATIVE_JAVA_LIBRARY` set.

To deploy the scheduler, simply build a jar as described above, and then run it with `java -cp target/cook-0.1.0-SNAPSHOT-standalone.jar cook.components dev-config.edn`.
See the Asciidoc document `docs/configuration.asc` for details on the config file format and options.

## API

Cook comes with two APIs: the REST API, and the Datomic API.
To use the REST API, see the `docs/scheduler-rest-api.asc` Asciidoc document for the API description.
Most users should use the REST API, since it's compatible and will not change.

For advanced integration with JVM-based projects, you can write new jobs directly to Datomic, and Cook will automatically detect these jobs and scheduler them.
To use this API, you should already be familiar with (Datomic)[docs.datomic.com].
You can see how to create a new job and query it by looking at the implementation of the REST API in `src/cook/mesos/api.clj`.
Job creation is done in the `submit-jobs` function, and job querying is done in `fetch-job-map`.
The direct Datomic access is especially useful for computing analytics on the Cook scheduler's behavior.
Because of Datomic's architecture, you can spin up a Datomic peer and freely do heavy full-db scans and analytics on the production database without impacting the performance of the Cook server.

## Dependencies

Cook is written in Clojure.
To develop Cook, all you need is a JVM and Mesos installed and configured.
Cook will automatically start embedded copies of the rest of its dependencies.

When deploying Cook in production, you'll also want to provide a few other dependencies:<commit_msg>Explain what stdout and stderr are. Fixes #39<commit_after># Cook Scheduler

The Cook scheduler is a Mesos framework that is able to scheduler jobs submitted to it via its REST API from multiple users.
It's raison d'etre is to prioritize the user experience on a shared cluster.
It does so by allowing users to use as much resources as possible, preempting users with greater shares when new users show up, and allowing users to have new jobs jump to the head of their queue.
You can learn more about the algorithms used by Cook in this talk by <NAME> at MesosCon: https://www.youtube.com/watch?v=BkBMYUe76oI.

## Building

You can build the final jar by running `lein uberjar`.

## Running

You can run the scheduler using the build system by running `lein run $config`, where config is an `.edn` file.
See `dev-config.edn` for example usage.
You should be able to start a sample instance (without any persistence) using the `dev-config.edn`.
You'll need to have the environment variable `MESOS_NATIVE_JAVA_LIBRARY` set.

When Cook starts running, it prints out as it initializes itself.
Once logging is initialized, all further logging data will be written to the log file, not stdout or stderr.
`libmesos`, the Mesos client library, prints all of its logging to stdout and stderr.
Thus, all messages printed to stdout and stderr are about the connection to Mesos.

To deploy the scheduler, simply build a jar as described above, and then run it with `java -cp target/cook-0.1.0-SNAPSHOT-standalone.jar cook.components dev-config.edn`.
See the Asciidoc document `docs/configuration.asc` for details on the config file format and options.

## API

Cook comes with two APIs: the REST API, and the Datomic API.
To use the REST API, see the `docs/scheduler-rest-api.asc` Asciidoc document for the API description.
Most users should use the REST API, since it's compatible and will not change.

For advanced integration with JVM-based projects, you can write new jobs directly to Datomic, and Cook will automatically detect these jobs and scheduler them.
To use this API, you should already be familiar with (Datomic)[docs.datomic.com].
You can see how to create a new job and query it by looking at the implementation of the REST API in `src/cook/mesos/api.clj`.
Job creation is done in the `submit-jobs` function, and job querying is done in `fetch-job-map`.
The direct Datomic access is especially useful for computing analytics on the Cook scheduler's behavior.
Because of Datomic's architecture, you can spin up a Datomic peer and freely do heavy full-db scans and analytics on the production database without impacting the performance of the Cook server.

## Dependencies

Cook is written in Clojure.
To develop Cook, all you need is a JVM and Mesos installed and configured.
Cook will automatically start embedded copies of the rest of its dependencies.

When deploying Cook in production, you'll also want to provide a few other dependencies:
{'fraction_non_alphanumeric': 0.0511147362697118, 'fraction_numerical': 0.002175095160413268, 'mean_word_length': 4.431102362204724, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4968707', 'n_tokens_mistral': 1487, 'n_tokens_neox': 1428, 'n_words': 856}