<commit_before># Ferrari Worker

## Process   programs in any language from different  sources. 

[![Build Status](https://travis-ci.org/ottogiron/ferrariworker.svg?branch=master)](https://travis-ci.org/ottogiron/ferrariworker)
## Example
This example shows jobs processing from rabbitmq using a Node.js script as job processor.

```
ferrariworker process rabbit \
    --uri=amqp://guest:guest@localhost:5672 \
    --queue-name=hello \
    --consumer-auto-ack=true \
    --command="node hello.js" \
    --command-run-path="/Users/ogiron" \
    --max-concurrency=8 
```

### Example job script for Node

***hello.js***

```javascript
console.time("hello task");

const payloadbase64 = process.argv[2];`
var buf = new Buffer(payloadbase64, 'base64');
console.log(buf.toString());
console.timeEnd("hello task")
process.exit(0);
/* Task was successful */
```

### Example output

```
...
hello 96
hello task: 3ms
2016/05/26 10:48:07 Thread 2 from 8 end processing took 169.387777ms
2016/05/26 10:48:07 Thread 7 from 8 end processing took 197.641664ms
2016/05/26 10:48:07 Thread 8 from 8 end processing took 198.734157ms
hello 99
hello task: 2ms
2016/05/26 10:48:07 Thread 3 from 8 end processing took 136.934854ms

```
## Processors
Processors are the sources from which you can process jobs e.g rabbit.

```
ferrariworker process <processor_name> <[flags]>
```

### Available Processors

* [RabbitMQ](processor/rabbit)

### Processing

## Job Payload

The program will receive the payload of a job as an argument encoded in base64. 

### Example in Node

```
//You can get a job payload using arguments array from the language you are using. 
const payloadbase64 = process.argv[2];
var buf = new Buffer(payloadbase64, 'base64');
console.log(buf.toString());
```

## Global Flags
This flags apply to all the available processors

<table>
    <tr>
        <th>Variable</th>
        <th>Default</th>
        <th>Description</th>
    </tr>
      <tr>
        <td>command</td>
        <td>echo "You should  set your own command :)"</td>
        <td>Command to be run when a new job arrives to the queue.</td>
    </tr>
      <tr>
        <td>command-path</td>
        <td>.</td>
        <td>Context path of the command to be run.</td>
    </tr>
      <tr>
        <td>max-concurrency</td>
        <td>1</td>
        <td>Max number of jobs proccessed concurrently.</td>
    </tr>
      <tr>
        <td>wait-timeout</td>
        <td>200</td>
        <td>Time to wait in milliseconds for existing jobs to be processed. </td>
    </tr>
</table>

## Development

### Processor Adapters
An adapter is an interface that defines the functionallity for processing jobs from any source.

```
//Adapter defines an processor source
type Adapter interface {
	Open() error
	Close() error
	Messages(chan<- *Message) error
	ResultHandler(jobResult *JobResult, message *Message) error
}
```

#### Processor Factory
An processor factory is interface which defines a "New" method that the ferrari core will use to create an instance of an specific adapter.
"New" receives   a "Config" object which contains all the configuration values provided by the user.

```
//Factory defines a actory from stream adapters
type Factory interface {
	New(config *Config) Adapter
}
```
#### Processor Factory Registration
By convention every adapter must implement a package level function called Register, which will be in charge of registering the adapter factory.

Example: 

```
//Register registers a processor adapter to  be used by the process command
func Register() {
	configSchema := &processor.ConfigurationSchema{
		Name:             "rabbit",
		ShortDescription: "Executes jobs coming from rabbit",
		LongDescription:  "Executes jobs coming from rabbit",
		Properties: []processor.ConfigurationProperty{
			processor.ConfigurationProperty{
				Name:        hostPropertyKey,
				Type:        processor.PropertyTypeString,
				Description: "Rabbit host url e.g. amqp://guest:guest@localhost:5672/",
				Default:     "amqp://guest:guest@localhost:5672/",
				Optional:    false,
			},
		},
	}
	processor.RegisterProcessordapterFactory(&factory{}, configSchema)
}
```

This function needs to be called explictly in cmd/modules.go


```
package cmd

import "github.com/ottogiron/ferrariworker/processor/rabbit"

func init() {
	rabbit.Register()
  // Other adapters registration
}
``` 

#### Processor Configuration Schema
Every adapters has to define their configuration metadata, that means the adapter name/identifier and all the related configuration fields.
This information is necessary for the adapter to be registered as processor command, and to be able to parse the configuration values that will be provided to the factory.


Please check the [RabbitMQ](processor/rabbit/rabbit.go) adapter for an example of a working processor adapter.


### Prerequisites

Go 1.6 +
<commit_msg>Update REAME<commit_after># Ferrari Worker

## Process   programs in any language from different  sources. 

[![Build Status](https://travis-ci.org/ottogiron/ferrariworker.svg?branch=master)](https://travis-ci.org/ottogiron/ferrariworker)
## Example
This example shows jobs processing from rabbitmq using a Node.js script as job processor.

```
ferrariworker process rabbit \
    --uri=amqp://guest:guest@localhost:5672 \
    --queue-name=hello \
    --consumer-auto-ack=true \
    --command="node hello.js" \
    --command-run-path="/Users/ogiron" \
    --max-concurrency=8 
```

### Example job script for Node

***hello.js***

```javascript
console.time("hello task");

const payloadbase64 = process.argv[2];`
var buf = new Buffer(payloadbase64, 'base64');
console.log(buf.toString());
console.timeEnd("hello task")
process.exit(0);
/* Task was successful */
```

### Example output

```
...
hello 96
hello task: 3ms
2016/05/26 10:48:07 Thread 2 from 8 end processing took 169.387777ms
2016/05/26 10:48:07 Thread 7 from 8 end processing took 197.641664ms
2016/05/26 10:48:07 Thread 8 from 8 end processing took 198.734157ms
hello 99
hello task: 2ms
2016/05/26 10:48:07 Thread 3 from 8 end processing took 136.934854ms

```
## Processor Adapters

Processors  Adapters are in charge of consuming jobs from different sources.

```
ferrariworker process <processor_adapter_name> <[flags]>
```

### Available Processor Adapters

* [RabbitMQ](processor/rabbit)

### Processing

## Job Payload

The program will receive the payload of a job as an argument encoded in base64. 

### Example in Node

```
//You can get a job payload using arguments array from the language you are using. 
const payloadbase64 = process.argv[2];
var buf = new Buffer(payloadbase64, 'base64');
console.log(buf.toString());
```

## Global Flags
This flags apply to all the available processors

<table>
    <tr>
        <th>Variable</th>
        <th>Default</th>
        <th>Description</th>
    </tr>
      <tr>
        <td>command</td>
        <td>echo "You should  set your own command :)"</td>
        <td>Command to be run when a new job arrives to the queue.</td>
    </tr>
      <tr>
        <td>command-path</td>
        <td>.</td>
        <td>Context path of the command to be run.</td>
    </tr>
      <tr>
        <td>max-concurrency</td>
        <td>1</td>
        <td>Max number of jobs proccessed concurrently.</td>
    </tr>
      <tr>
        <td>wait-timeout</td>
        <td>200</td>
        <td>Time to wait in milliseconds for existing jobs to be processed. </td>
    </tr>
</table>

## Development

### Processor Adapters
An adapter is an interface that defines the functionallity for processing jobs from any source.

```
//Adapter defines an processor source
type Adapter interface {
	Open() error
	Close() error
	Messages(chan<- *Message) error
	ResultHandler(jobResult *JobResult, message *Message) error
}
```

#### Processor Adapter Factory
An processor adapter factory is interface which defines a "New" method that the ferrari core will use to create an instance of an specific adapter.
"New" receives   a "Config" object which contains all the configuration values provided by the user.

```
//Factory defines a actory from stream adapters
type Factory interface {
	New(config *AdapterConfig) Adapter
}
```
#### Processor Factory Registration
By convention every adapter must implement a package level function called Register, which will be in charge of registering the adapter factory.

Example: 

```
//Register registers a processor adapter to  be used by the process command
func Register() {
	configSchema := &processor.ConfigurationSchema{
		Name:             "rabbit",
		ShortDescription: "Executes jobs coming from rabbit",
		LongDescription:  "Executes jobs coming from rabbit",
		Properties: []processor.ConfigurationProperty{
			processor.ConfigurationProperty{
				Name:        hostPropertyKey,
				Type:        processor.PropertyTypeString,
				Description: "Rabbit host url e.g. amqp://guest:guest@localhost:5672/",
				Default:     "amqp://guest:guest@localhost:5672/",
				Optional:    false,
			},
		},
	}
	processor.RegisterProcessordapterFactory(&factory{}, configSchema)
}
```

This function needs to be called explictly in cmd/modules.go


```
package cmd

import "github.com/ottogiron/ferrariworker/processor/rabbit"

func init() {
	rabbit.Register()
  // Other adapters registration
}
``` 

#### Processor Adapters Configuration Schema
Every adapters has to define their configuration metadata, that means the adapter name/identifier and all the related configuration fields.
This information is necessary for the adapter to be registered as processor command, and to be able to parse the configuration values that will be provided to the factory.


Please check the [RabbitMQ](processor/rabbit/rabbit.go) adapter for an example of a working processor adapter.


### Prerequisites

Go 1.6 +

{'fraction_non_alphanumeric': 0.11100879885410272, 'fraction_numerical': 0.02905668099038265, 'mean_word_length': 3.6614210777300906, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 91, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '391518', 'n_tokens_mistral': 3147, 'n_tokens_neox': 2792, 'n_words': 1025}