<commit_before>Vertigo
=======

**Need support? Check out the [Vertigo Google Group][google-group]**

**[Getting Started](#getting-started) | [Java User Manual](#java-user-manual) | [Javadoc](http://vertigo.kuujo.net/java)**

[Javascript][vertigo-js] | [Python][vertigo-python]

Vertigo is a durable polyglot event processing framework built on the
[Vert.x](http://vertx.io/) application platform. Combining concepts of cutting-edge
[real-time systems](http://storm.incubator.apache.org/) and
[flow-based programming](http://en.wikipedia.org/wiki/Flow-based_programming),
Vertigo allows real-time problems to be broken down into smaller tasks (as
Vert.x verticles) and distributed across a Vert.x cluster.

* Manages multi-step event processing systems, from simple pipelines to
  **complex networks of Vert.x modules/verticles**
* Supports deployment of networks **within a single Vert.x instance or across a
  Vert.x cluster** and provides automatic **failover** for failed components
* **Coordinates startup and shutdown** of networks across the Vert.x cluster
* **Promotes reusability** by abstracting communication details from verticle implementations,
  allowing components to be arbitrarily connected to form complex networks
* Guarantees **strong ordering** and **exactly-once** processing of messages
* Handles **event bus flow control** and **automatic retries** on failures
* Supports `Serializable` Java messages on the Vert.x event bus
* Facilitates distribution of messages between multiple verticle instances using
  **random, round-robin, mod hashing, fair, or fanout** methods
* Provides **cluster-wide shared data** structures for synchronization
* Supports **live network configuration changes** so networks do not have to
  be shutdown in order to be reconfigured
* Provides a simple **batch API** for efficiently batching messages between components
* Support **command line deployment** of networks
* Components can be written in **any Vert.x supported language**, with
  APIs for Vertigo in [Javascript][vertigo-js] and [Python][vertigo-python]

For an in-depth explanation of how Vertigo works, see [how it works](#how-it-works)

YourKit is generously supporting Vertigo and other open-source projects with its
full-featured Java Profiler. Take a look at YourKit's leading software products:
[YourKit Java Profiler](http://www.yourkit.com/java/profiler/index.jsp) and
[YourKit .NET Profiler](http://www.yourkit.com/.net/profiler/index.jsp).

# Getting Started
This is a brief tutorial that describes the basic components of Vertigo
along with a simple example.

## Networks
Networks are collections of Vert.x verticles and modules that are connected
together by input and output ports. Each component in a network contains processing
logic, and connections between components indicate how messages should be
passed between them. Networks can be created either in code or in JSON and can
be deployed in code or from the command line.

![Vertigo network](http://s21.postimg.org/ve93v28bb/Untitled_Diagram.png)

## Components

As with any Vert.x verticle, Vertigo components can be deployed with any number
of instances. Components communicate with each other over the Vert.x event bus.
But Vertigo doesn't use raw event bus addresses. Instead, components communicate
through named output and input ports. This allows components to be abstracted
from the relationships between them, making them reusable.

Messaging in Vertigo is inherently uni-directional. Components receive messages
from other components on input ports and send messages to other components on
output ports.

![Direct connections](http://s21.postimg.org/65oa1e3dj/Untitled_Diagram_1.png)

Messages are not routed through any central router. Rather, components
communicate with each other directly over the event bus.

```java
public class MyComponent extends ComponentVerticle {

  @Override
  public void start() {
    input.port("in").messageHandler(new Handler<String>() {
      public void handle(String message) {
        output.port("out").send(message);
      }
    });
  }

}
```

Ports do not have to be explicitly declared. Vertigo will lazily create ports
if they don't already exist. Messages can be of any type that is supported by the
Vert.x event bus. Vertigo guarantees that messages will always arrive in the order
in which they were sent.

In cases where a connection connects two components with multiple instances,
Vertigo facilitates special routing between the components with *selectors*.

![Selectors](http://s21.postimg.org/a3bjqsq6v/Untitled_Diagram_2.png)

While components receive messages on input ports and send messages to output ports,
the network configuration is used to define how ports on different components
relate to one another. Connections between components/ports in your network indicate
how messages will flow through the network.

```java
NetworkConfig network = vertigo.createNetwork("foo");
network.addComponent("bar", "bar.js", 2);
network.addComponent("baz", "baz.py", 4);
network.createConnection("bar", "out", "baz", "in");
```

## Cluster
Vertigo provides its own cluster abstraction within the Vert.x cluster. Vertigo
clusters are simple collections of verticles that manage deployment of networks,
allow modules and verticles to be deploye remotely (over the event bus)
and provide cluster-wide shared data structures. Clusters can be run either in
a single Vert.x instance (for testing) or across a Vert.x cluster.

## A Simple Network
Vertigo provides all its API functionality through a single `Vertigo` object.

```java
Vertigo vertigo = new Vertigo(this);
```

The `Vertigo` object supports creating and deploying networks. Each language
binding has an equivalent API.

```java
NetworkConfig network = vertigo.createNetwork("word-count");
network.addComponent("word-feeder", RandomWordCounter.class.getName());
network.addComponent("word-counter", WordCounter.class.getName(), 2);
network.createConnection("word-feeder", "word", "word-counter", "word", new HashSelector());
```<commit_msg>Add cluster diagram to README documentation.<commit_after>Vertigo
=======

**Need support? Check out the [Vertigo Google Group][google-group]**

**[Getting Started](#getting-started) | [Java User Manual](#java-user-manual) | [Javadoc](http://vertigo.kuujo.net/java)**

[Javascript][vertigo-js] | [Python][vertigo-python]

Vertigo is a durable polyglot event processing framework built on the
[Vert.x](http://vertx.io/) application platform. Combining concepts of cutting-edge
[real-time systems](http://storm.incubator.apache.org/) and
[flow-based programming](http://en.wikipedia.org/wiki/Flow-based_programming),
Vertigo allows real-time problems to be broken down into smaller tasks (as
Vert.x verticles) and distributed across a Vert.x cluster.

* Manages multi-step event processing systems, from simple pipelines to
  **complex networks of Vert.x modules/verticles**
* Supports deployment of networks **within a single Vert.x instance or across a
  Vert.x cluster** and provides automatic **failover** for failed components
* **Coordinates startup and shutdown** of networks across the Vert.x cluster
* **Promotes reusability** by abstracting communication details from verticle implementations,
  allowing components to be arbitrarily connected to form complex networks
* Guarantees **strong ordering** and **exactly-once** processing of messages
* Handles **event bus flow control** and **automatic retries** on failures
* Supports `Serializable` Java messages on the Vert.x event bus
* Facilitates distribution of messages between multiple verticle instances using
  **random, round-robin, mod hashing, fair, or fanout** methods
* Provides **cluster-wide shared data** structures for synchronization
* Supports **live network configuration changes** so networks do not have to
  be shutdown in order to be reconfigured
* Provides a simple **batch API** for efficiently batching messages between components
* Support **command line deployment** of networks
* Components can be written in **any Vert.x supported language**, with
  APIs for Vertigo in [Javascript][vertigo-js] and [Python][vertigo-python]

For an in-depth explanation of how Vertigo works, see [how it works](#how-it-works)

YourKit is generously supporting Vertigo and other open-source projects with its
full-featured Java Profiler. Take a look at YourKit's leading software products:
[YourKit Java Profiler](http://www.yourkit.com/java/profiler/index.jsp) and
[YourKit .NET Profiler](http://www.yourkit.com/.net/profiler/index.jsp).

# Getting Started
This is a brief tutorial that describes the basic components of Vertigo
along with a simple example.

## Networks
Networks are collections of Vert.x verticles and modules that are connected
together by input and output ports. Each component in a network contains processing
logic, and connections between components indicate how messages should be
passed between them. Networks can be created either in code or in JSON and can
be deployed in code or from the command line.

![Vertigo network](http://s21.postimg.org/ve93v28bb/Untitled_Diagram.png)

## Components

As with any Vert.x verticle, Vertigo components can be deployed with any number
of instances. Components communicate with each other over the Vert.x event bus.
But Vertigo doesn't use raw event bus addresses. Instead, components communicate
through named output and input ports. This allows components to be abstracted
from the relationships between them, making them reusable.

Messaging in Vertigo is inherently uni-directional. Components receive messages
from other components on input ports and send messages to other components on
output ports.

![Direct connections](http://s21.postimg.org/65oa1e3dj/Untitled_Diagram_1.png)

Messages are not routed through any central router. Rather, components
communicate with each other directly over the event bus.

```java
public class MyComponent extends ComponentVerticle {

  @Override
  public void start() {
    input.port("in").messageHandler(new Handler<String>() {
      public void handle(String message) {
        output.port("out").send(message);
      }
    });
  }

}
```

Ports do not have to be explicitly declared. Vertigo will lazily create ports
if they don't already exist. Messages can be of any type that is supported by the
Vert.x event bus. Vertigo guarantees that messages will always arrive in the order
in which they were sent.

In cases where a connection connects two components with multiple instances,
Vertigo facilitates special routing between the components with *selectors*.

![Selectors](http://s21.postimg.org/a3bjqsq6v/Untitled_Diagram_2.png)

While components receive messages on input ports and send messages to output ports,
the network configuration is used to define how ports on different components
relate to one another. Connections between components/ports in your network indicate
how messages will flow through the network.

```java
NetworkConfig network = vertigo.createNetwork("foo");
network.addComponent("bar", "bar.js", 2);
network.addComponent("baz", "baz.py", 4);
network.createConnection("bar", "out", "baz", "in");
```

## Cluster
Vertigo provides its own cluster abstraction within the Vert.x cluster. Vertigo
clusters are simple collections of verticles that manage deployment of networks,
allow modules and verticles to be deploye remotely (over the event bus)
and provide cluster-wide shared data structures. Clusters can be run either in
a single Vert.x instance (for testing) or across a Vert.x cluster.

![Cluster](http://s8.postimg.org/n535zhv9h/cluster.png)

Rather than communicating over the unreliable event bus, Vertigo uses Hazelcast
data structures to coordinate between the cluster and components.
This is one of the properties that makes Vertigo networks fault-tolerant. Even if
a failure occurs, the network configuration will remain in the cluster and Vertigo
will automatically failover any failed components.

## A Simple Network
Vertigo provides all its API functionality through a single `Vertigo` object.

```java
Vertigo vertigo = new Vertigo(this);
```

The `Vertigo` object supports creating and deploying networks. Each language
binding has an equivalent API.

```java
NetworkConfig network = vertigo.createNetwork("word-count");
network.addComponent("word-feeder", RandomWordCounter.class.getName());
network.addComponent("word-counter", WordCounter.class.getName(), 2);
network.createConnection("word-feeder", "word", "word-counter", "word", new HashSelector());
```
{'fraction_non_alphanumeric': 0.08363259117388876, 'fraction_numerical': 0.0037506982682946294, 'mean_word_length': 5.262868565717142, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1705333', 'n_tokens_mistral': 3300, 'n_tokens_neox': 3157, 'n_words': 1456}