<commit_before>GoWorker
====
A plugin based work management system implimented in pure go.

This project is still under heavy development. APIs are subject to change.

## Installation
To install GoWorker, simply run 
`go get github.com/barracudanetworks/GoWorker/...`

##### Hint
In order for this to work, you must have a valid GOPATH set
(read more about GOPATH [here](https://code.google.com/p/go-wiki/wiki/GOPATH))

## Testing
To run all tests for the project simply navigate to the root of the project folder and run
`go test ./...`

To test an individual packages tests, navigate to the root of that package and run
`go test`

To run benchmarks contained in a package's tests run
`go test --bench=.`

More on testing in go can be found [here](http://www.golang-book.com/12/index.htm)

## Usage
The main program for this project can be built by navigating to cmd/goworker/ and either running 
`go run goworker.go` for testing purproses.

or

`go build` to compile the full binary for repeated use. 

## Composition
The work manager is compried of three main componants.
* Provider
	* Provides jobs to the manager from an external source.
* Worker
	* Takes a job, exicutes it, and returns its status and statistics about the job back to the manager.
* Manager 
	* Manages the work pipline. Requests jobs from the providers and sends them to the workers.

### Pipeline
There are three main stages in the worker pipeline.

1. The manager requests a job from the provider.
2. The Manager hands the job of to a worker that can handle the job.
3. Once the job is complete, the worker returns the job back to the manager, along with information about the result of the job. If the job was unsuccessful, and the job is configured to retry, step two and three will repeat until the job succeeds or the retries have been exhuasted. Once the retries have been exhausted, the manager hands the job off to the manager's failure handlers, if they have been configured.

So graphically...
__Provider->Manager->Worker->Manager__ 
where each arrow represents a transition of ownership of the job.

## Plugins
Workers and providers are implemented as plugins. Because go compiles down to a static binary, you must recompile when you add new plugins.
To load your plugin into the binary, add a blank import to your package in the main go program inside of the cmd directory. Your package must have an init function that calls either `provider.LoadProvider(YourProviderFactory)` or `worker.LoadWorker(YourWorkerFactory)`. Once your worker or provider has been loaded, it will be accessable in the config file by its name, in all lowercase.

More on writing plugins [worker](http://godoc.org/github.com/barracudanetworks/GoWorker/worker) [provider](http://godoc.org/github.com/barracudanetworks/GoWorker/provider)<commit_msg>Update README.md for readability<commit_after>GoWorker
====
A plugin-based worker system implimented in Golang. APIs subject to change.

## Installation
1. Make sure the [GOPATH](https://code.google.com/p/go-wiki/wiki/GOPATH) environment variable is set.
2. Run `go get github.com/barracudanetworks/GoWorker/...`


## Testing
From the project root, run `go test ./...` to test all packages. To test a single package, run `go test` from within the package's directory.

To run benchmarks for a package, run `go test --bench=.`

[Read more about Golang testing](http://www.golang-book.com/12/index.htm)

## Usage
You may either run the program with `go run goworker.go` or build a binary with `go build`.

## Composition
GoWorker is comprised of three main components:

1. Provider: Provides jobs to the manager from an external source (e.g. Redis)
2. Worker: Executes a job and returns results and stats to the manager.
3. Manager: Manages the work pipline. Requests jobs from providers and routes them to the workers.

### Pipeline
__Provider -> Manager -> Worker -> Manager__ 

There are three main stages in the worker pipeline.

1. The Manager requests a job from a provider.
2. The Manager hands the job off to a worker.
3. The Worker returns the job back to the Manager after execution, with additional information including the result of the job. Jobs may optionally be set to retry in the case of failure until a maximum number of retries has been exhausted, or until the job succeeds. If all retries have been exhausted, the job is passed to the Manager's failure handlers, if any have been configured.

## Plugins
Workers and Providers are implemented as plugins. Because Go can compile down to a static binary, you must recompile GoWorker when you add new plugins.

To load your plugin into the binary, add a blank import to your package in the `main` package (`gowork.go` by default), located inside the `cmd` directory. Your package must have an `init()` function that calls either `provider.LoadProvider(YourProviderFactory)` or `worker.LoadWorker(YourWorkerFactory)`.

Once your worker or provider has been loaded, it will be accessible in the config file by its name, in all lowercase.

You may learn more by reading the GoDoc entries for [workers](http://godoc.org/github.com/barracudanetworks/GoWorker/worker) and [providers](http://godoc.org/github.com/barracudanetworks/GoWorker/provider).

{'fraction_non_alphanumeric': 0.0693752395553852, 'fraction_numerical': 0.002874664622460713, 'mean_word_length': 4.293103448275862, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '997647', 'n_tokens_mistral': 1418, 'n_tokens_neox': 1370, 'n_words': 747}