<commit_before># Specification

## Introduction

### Problem Statement

TChannel was developed at Uber during a period of explosive growth in its
[SOA][soa] ecosystem. Services living in a large distributed system like this
usually encounter similar classes of problems. Those problems often include:

 - Service discovery -- how do I discover *and consume* the services around me?

 - Fault tolerance -- how can I best insulate myself from failures outside of
   my control?

 - [Dapper][dapper]-style tracing -- how can I identify and monitor bottlenecks
   across the entire system?

[soa]: https://en.wikipedia.org/wiki/Service-oriented_architecture "SOA"
[dapper]: http://research.google.com/pubs/pub36356.html "Dapper"

### Design Goals

TChannel aims to solve these problems by providing a protocol for clients and
servers with an *intelligent routing mesh* (referred to as *Hyperbahn*)
connecting the two.

Consider those [SOA][soa] problems again:

 - Service discovery: All producers and consumers register themselves with the
   routing mesh. Consumers access producers only by their name; no need to know
   about hosts or ports.

 - Fault tolerance: The routing mesh tracks things like failure rates and
   [SLA][sla] violations. It can intelligently detect unhealthy hosts and
   remove them from the pool of available hosts.

 - Request tracing: Built into the protocol as a first class citizen.

[sla]: https://en.wikipedia.org/wiki/Service-level_agreement "SLA"

Consolidating logic *between* producers and consumers also allows us to push
core features to our entire SOA without requiring applications to update
libraries.

Additionally, when developing this protocol and routing mesh we had the
following goals in mind:

- The protocol must be easy to implement in multiple languages, especially
  Javascript, Python, and Go. We do not want to restrict ourselves to one
  language.

- Async behavior is fundamental. We follow a traditional request / response
  model with out of order responses; slow requests must not block subsequent,
  faster requests at the head of the line.

- Large requests/responses may/must be broken into fragments to be sent
  progressively (i.e., "streaming" requests are supported).

- TChannel can be used with arbitrary serialization schemes, e.g. JSON and
  Thrift, although Thrift is preferred for its [IDL][idl], type-safety, and
  validation guarantees.

- The routing mesh needs a high-performance forwarding path. Intermediaries
  must be able to make a forwarding decision quickly.

- Optional checksums for request/reponse integrity.

[idl]: https://thrift.apache.org/docs/idl "IDL"

## Field Length Conventions

In this document, all numeric values are unsigned and in big-endian byte order.
All strings are UTF-8. All keys and values used in headers are strings.

The schema for describing field lengths and counts is the same as finagle-mux:

- the schema `size:4 body:10` defines the field `size` to be 4 bytes, followed
  by 10 bytes of the field `body`.
- the schema `key~4` defines the field `key` to be 4 bytes of a size, followed
  by that many bytes of data.
- `key~4` is shorthand for `keysize:4 key:keysize`

Groups are denoted by parentheses. A group's repetition count is either {\*} for
0 or more repetitions, or {n} for exactly n repetitions.

## Message Flow

Tchannel is a bi-directional request/response protocol. Each connection between
peers is considered equivalent, regardless of which side initiated it. It's
possible, perhaps even desirable, to have multiple connections between the same
pair of peers. Message ids are scoped to a connection. It is not valid to send
a request down one connection and a response down another. It is certainly
possible to send one request down one connection and then a subsequent request
down another, for whatever reason you might have.

Initiating a new connection works like this:

1. node A initiates TCP connection to node B
2. B accepts TCP, must not send any data until receiving "init req"
3. A sends "init req" with desired version, must not send any data until
   receiving "init res"
4. B sends "init res", with selected version V. B may now send requests with
   version V.
5. A receives "init res". A may now send requests with version V.

Each message is encapsulated in a frame with some additional information that
is common across all message types. Part of that framing information is an id.
This id is chosen by the requestor when sending a request message. When
responding to a request, the responding node uses the message id in the request
frame for the response.

Each frame has a type which describes the format of the frame's body. Depending
on the frame type, some bodies are 0 bytes.

The "call req" frame type has a "ttl" field which is used to specify the
deadline for this request. The transmitter manages the ttl to account for time
in transit or time to failover, and send it to receivers only for deadline
propagation purposes.  This allows receivers to know how much time they have
remaining to complete this portion of the request. Intermediaries are free to
make retries with backoff as necessary, as long as they are still within the
ttl.

Here are some details of the message flow from a service A instance A1, through
a service router R1, into a service B instance B1, and then back. These are not
the full details of every single field, but they are intended to illustrate
some less obvious behaviors.

The flow in this example is: `A1 -> R1 -> B1 -> R1 -> A1`

A1 sends "call req" (type 0x03) to R1:

1. select a message id M1 for this frame, in a predictable sequence for easier
   debugging. This will be used to match up the response to this request. The
   scope of M1 exists only between A1 and R1 and only for this connection.
2. generate a unique traceid which will be propagated to any dependent
   requests. This should only be done if A1 is the edge-most service in the
   call chain. The point of this unique traceid is to build a span tree that
   covers all RPCs supporting a given client-visible transaction, not just
   tracing of a single logical request through multiple intermediaries.
3. generate a unique spanid which is used to refer to this specific step in the
   network
4. set the ttl to the maximum allowable time for this request. If a response
   cannot be sent within this time, downstream nodes will cancel the request.
5. A1 doesn't need any headers, so defaults are used
6. a "call req" body is generated with service\_name "service B", application
   payload args, and checksum

R1 receives "call req" from A1, sends "call req" (0x03) to B1:

1. select message id M2 for this frame
2. copy traceid from incoming message to traceid of new message
3. copy spanid from incoming message to parentid of new message
4. generate unique spanid
5. copy ttl from incoming message to ttl of new message
6. copy service name, args, and csum data to new message

B1 receives "call req" from R1, sends "call res" (0x04) to R1:

1. use message id M2 for this frame
2. send args from application response and compute csum

R1 receives "call res" from B1, send "call res" (0x03) to A1:

1. match incoming message id M2 with existing "call req"
2. use message id M1 for new message
3. copy args/csum from incoming message

A1 receives "call res" from R1:

1. match incoming message M1 with existing "call req"
2. deliver args to application

## Framing

All frames of all types use this structure:<commit_msg>Use correct reference to call req's `service` field<commit_after># Specification

## Introduction

### Problem Statement

TChannel was developed at Uber during a period of explosive growth in its
[SOA][soa] ecosystem. Services living in a large distributed system like this
usually encounter similar classes of problems. Those problems often include:

 - Service discovery -- how do I discover *and consume* the services around me?

 - Fault tolerance -- how can I best insulate myself from failures outside of
   my control?

 - [Dapper][dapper]-style tracing -- how can I identify and monitor bottlenecks
   across the entire system?

[soa]: https://en.wikipedia.org/wiki/Service-oriented_architecture "SOA"
[dapper]: http://research.google.com/pubs/pub36356.html "Dapper"

### Design Goals

TChannel aims to solve these problems by providing a protocol for clients and
servers with an *intelligent routing mesh* (referred to as *Hyperbahn*)
connecting the two.

Consider those [SOA][soa] problems again:

 - Service discovery: All producers and consumers register themselves with the
   routing mesh. Consumers access producers only by their name; no need to know
   about hosts or ports.

 - Fault tolerance: The routing mesh tracks things like failure rates and
   [SLA][sla] violations. It can intelligently detect unhealthy hosts and
   remove them from the pool of available hosts.

 - Request tracing: Built into the protocol as a first class citizen.

[sla]: https://en.wikipedia.org/wiki/Service-level_agreement "SLA"

Consolidating logic *between* producers and consumers also allows us to push
core features to our entire SOA without requiring applications to update
libraries.

Additionally, when developing this protocol and routing mesh we had the
following goals in mind:

- The protocol must be easy to implement in multiple languages, especially
  Javascript, Python, and Go. We do not want to restrict ourselves to one
  language.

- Async behavior is fundamental. We follow a traditional request / response
  model with out of order responses; slow requests must not block subsequent,
  faster requests at the head of the line.

- Large requests/responses may/must be broken into fragments to be sent
  progressively (i.e., "streaming" requests are supported).

- TChannel can be used with arbitrary serialization schemes, e.g. JSON and
  Thrift, although Thrift is preferred for its [IDL][idl], type-safety, and
  validation guarantees.

- The routing mesh needs a high-performance forwarding path. Intermediaries
  must be able to make a forwarding decision quickly.

- Optional checksums for request/reponse integrity.

[idl]: https://thrift.apache.org/docs/idl "IDL"

## Field Length Conventions

In this document, all numeric values are unsigned and in big-endian byte order.
All strings are UTF-8. All keys and values used in headers are strings.

The schema for describing field lengths and counts is the same as finagle-mux:

- the schema `size:4 body:10` defines the field `size` to be 4 bytes, followed
  by 10 bytes of the field `body`.
- the schema `key~4` defines the field `key` to be 4 bytes of a size, followed
  by that many bytes of data.
- `key~4` is shorthand for `keysize:4 key:keysize`

Groups are denoted by parentheses. A group's repetition count is either {\*} for
0 or more repetitions, or {n} for exactly n repetitions.

## Message Flow

Tchannel is a bi-directional request/response protocol. Each connection between
peers is considered equivalent, regardless of which side initiated it. It's
possible, perhaps even desirable, to have multiple connections between the same
pair of peers. Message ids are scoped to a connection. It is not valid to send
a request down one connection and a response down another. It is certainly
possible to send one request down one connection and then a subsequent request
down another, for whatever reason you might have.

Initiating a new connection works like this:

1. node A initiates TCP connection to node B
2. B accepts TCP, must not send any data until receiving "init req"
3. A sends "init req" with desired version, must not send any data until
   receiving "init res"
4. B sends "init res", with selected version V. B may now send requests with
   version V.
5. A receives "init res". A may now send requests with version V.

Each message is encapsulated in a frame with some additional information that
is common across all message types. Part of that framing information is an id.
This id is chosen by the requestor when sending a request message. When
responding to a request, the responding node uses the message id in the request
frame for the response.

Each frame has a type which describes the format of the frame's body. Depending
on the frame type, some bodies are 0 bytes.

The "call req" frame type has a "ttl" field which is used to specify the
deadline for this request. The transmitter manages the ttl to account for time
in transit or time to failover, and send it to receivers only for deadline
propagation purposes.  This allows receivers to know how much time they have
remaining to complete this portion of the request. Intermediaries are free to
make retries with backoff as necessary, as long as they are still within the
ttl.

Here are some details of the message flow from a service A instance A1, through
a service router R1, into a service B instance B1, and then back. These are not
the full details of every single field, but they are intended to illustrate
some less obvious behaviors.

The flow in this example is: `A1 -> R1 -> B1 -> R1 -> A1`

A1 sends "call req" (type 0x03) to R1:

1. select a message id M1 for this frame, in a predictable sequence for easier
   debugging. This will be used to match up the response to this request. The
   scope of M1 exists only between A1 and R1 and only for this connection.
2. generate a unique traceid which will be propagated to any dependent
   requests. This should only be done if A1 is the edge-most service in the
   call chain. The point of this unique traceid is to build a span tree that
   covers all RPCs supporting a given client-visible transaction, not just
   tracing of a single logical request through multiple intermediaries.
3. generate a unique spanid which is used to refer to this specific step in the
   network
4. set the ttl to the maximum allowable time for this request. If a response
   cannot be sent within this time, downstream nodes will cancel the request.
5. A1 doesn't need any headers, so defaults are used
6. a "call req" body is generated with service "service B", application
   payload args, and checksum

R1 receives "call req" from A1, sends "call req" (0x03) to B1:

1. select message id M2 for this frame
2. copy traceid from incoming message to traceid of new message
3. copy spanid from incoming message to parentid of new message
4. generate unique spanid
5. copy ttl from incoming message to ttl of new message
6. copy service name, args, and csum data to new message

B1 receives "call req" from R1, sends "call res" (0x04) to R1:

1. use message id M2 for this frame
2. send args from application response and compute csum

R1 receives "call res" from B1, send "call res" (0x03) to A1:

1. match incoming message id M2 with existing "call req"
2. use message id M1 for new message
3. copy args/csum from incoming message

A1 receives "call res" from R1:

1. match incoming message M1 with existing "call req"
2. deliver args to application

## Framing

All frames of all types use this structure:
{'fraction_non_alphanumeric': 0.04934649239797279, 'fraction_numerical': 0.011469725260069352, 'mean_word_length': 4.0376217668794085, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6295553', 'n_tokens_mistral': 3964, 'n_tokens_neox': 3822, 'n_words': 2362}