<commit_before># Topology protocol

## Coordinator

When started, it reads and compiles the topology settings.

It also opens specified port for coordination requests.

## Worker

When started, needs to be given:

- method of coordination
- logical name

It opens connection to the coordinator, awaiting instructions. When registration with coordinator succeeds, it starts local topology.

## Local topology

This object:

- starts nodes (spouts and bolts)
- sends them commands, e.g. `init`, `run`, `pause`, `shutdown`

## Sequence between coordinator, worker and local topology

The worker first performs the initialization sequence and the runs two sequences in parallel, infinite loops:

- Leadership sequence
- Normal active-worker sequence

### Initial sequence

| Coordination storage | Worker | Local topology |
|-------------|--------|-----|
|  | Register worker | 
| Puts worker in worker list  |  | 

### Leadership sequence

| Coordination storage | Worker
|-------------|--------|
|  | Checks if leadership is established |
| Returns leadership status |  |
|  | If leadership is ok, do nothing more |
|  | Send leadership candidacy |
| Register candidacy |  |
|  | Check candidacy |
| Send true if candidadcy sucessfull |  |
|  | If not elected leader, do nothing more |
|  | Get worker statuses |
| Return worker statuses after marking overdue pings as `dead` | |
| | For all `dead` workers unassign their topologies |
| Update statuses for these topologies | |
| | Pronounce `dead` workers as `unloaded` |
| Store new worker statuses | |
| | Get topology statuses |
| Get topology statuses after setting overdue `waiting` status to `unassigned` and setting topologies of `dead` worker to `unassigned`  | |
| | Assign `unassigned` and `stopped` topologies to new workers by setting the to status `waiting` |<commit_msg>More docs<commit_after># Topology protocol

## Coordinator

When started, it reads and compiles the topology settings.

It also opens specified port for coordination requests.

## Worker

When started, needs to be given:

- method of coordination
- logical name

It opens connection to the coordinator, awaiting instructions. When registration with coordinator succeeds, it starts local topology.

## Local topology

This object:

- starts nodes (spouts and bolts)
- sends them commands, e.g. `init`, `run`, `pause`, `shutdown`

## Sequence between coordinator, worker and local topology

The worker first performs the initialization sequence and the runs two sequences in parallel, infinite loops:

- Leadership sequence
- Normal active-worker sequence

### Initial sequence

| Coordination storage | Worker |
|-------------|--------|
|  | Register worker | 
| Puts worker in worker list  |  | 

### Leadership sequence

| Coordination storage | Worker
|-------------|--------|
|  | Checks if leadership is established |
| Returns leadership status |  |
|  | If leadership is ok, do nothing more |
|  | Send leadership candidacy |
| Register candidacy |  |
|  | Check candidacy |
| Send true if candidadcy sucessfull |  |
|  | If not elected leader, do nothing more |
|  | Get worker statuses |
| Return worker statuses after marking overdue pings as `dead` | |
| | For all `dead` workers unassign their topologies |
| Update statuses for these topologies | |
| | Pronounce `dead` workers as `unloaded` |
| Store new worker statuses | |
| | Get topology statuses |
| Get topology statuses after setting overdue `waiting` status to `unassigned` and setting topologies of `dead` worker to `unassigned`  | |
| | Assign `unassigned` and `stopped` topologies to new workers by setting the to status `waiting` |
{'fraction_non_alphanumeric': 0.10524869469634515, 'fraction_numerical': 0.0, 'mean_word_length': 3.7769028871391077, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3694719', 'n_tokens_mistral': 1001, 'n_tokens_neox': 956, 'n_words': 431}