<commit_before>#Fastnet 1.0 Protocol Specification

##Introduction and Goals

Fastnet is a connection-based, channel-based protocol intended for games and realtime applications.
It intensionally forgoes much of the automatic management of TCP and assumes that settings will be provided by the application developer.

Unlike TCP, Fastnet works at the level of a message: a chunk of binary data of any size.
Messages are either fully received or never received at all, and the application need not provide further separation logic.
One Fastnet connection is logically 32768 smaller multiplexed connections called channels.

Messages can be sent reliably or unreliably.
Unreliable messages may or may not arrive at the other end, but are extremely fast and capable of representing frequently updated data such as position.
Reliable messages are intended for chat messages, status updates, or other information that absolutely must arrive.
A reliable message stalls all other messages in the queue behind it until such time as it arrives on the other end.

When considering the advantages of Fastnet, it is helpful and mostly accurate to think of  TCP as a train.
If one train car derails, everything behind it stops.
The primary advantage of Fastnet, then, is that you have the ability to use multiple train tracks at once.
Furthermore, for applications which can tolerate data loss, it is possible to mark some of the metaphorical cars as unimportant;
if these unimportant cars derail, the stream continues regardless.

Other semireliable UDP-based protocols exist, but usually only for one language.
Fastnet aims to be fully documented to the point where it can be recoded from scratch.
The other two features that Fastnet aims to support are the ability to fall back to other transports (TCP, HTTP, WebRTC) and the ability to support UDP hole-punching.
In my experience, it is difficult to find networking libraries that offer all three of these benefits.

##Definition of Terms##

The key words "must", "must not", "required", "shall", "shall not", "should", "should not", "recommended",  "may", and
"optional" in this document are to be interpreted as described in
[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).

A client refers to an entity who requests a connection from a server.
A server refers to an entity capable of accepting multiple connections.
All entities involved in Fastnet are either a client or a server, never both.

A sender is an entity capable of sending messages.
A receiver is an entity capable of receiving messages.
Clients and servers are both senders and receivers.

A packet refers to a chunk of data of any size.

A message refers to a chunk of data specified by the application.  This specification places no semantic meaning on the content of messages provided by applications.

A transport means a method of moving packets from one destination to another.  Transports usually involve the internet.

A transport's implementation refers to the concrete implementation of a transport.

UDP hole-punching refers to a technique for  circumventing network Address Translation and connecting two computers directly without the use of an intermediate server.

##Transports##

A transport is a method for moving packets from one destination to another.
Transports must have the following characteristics.

- An ordered transport must guarantee that, if packets arrive, they arrive in order.

- An incorruptible transport must guarantee that, if packets arrive, they match the sent packet.

- A reliable transport must guarantee that packets arrive if sent.  Reliable transports must be ordered and incorruptible transports.

- A connection-based transport must handle all logic of forming and keeping a connection open.  This specification refers to transports which are not connection-based as connectionless.

All implementations must provide at least the TCP and UDP transports.
The TCP transport must be reliable and connection-based.
The UDp transport must be unordered, unreliable, and connectionless.
Over IPV4, UDP's built-in checksum is optional.
To this end, UDP must also be corruptible.

Every transport must advertise the maximum packet size it supports in a way that allows the user of the implementation to obtain the value.
Every transport must support a maximum packet size of at least 4096 bytes.
Note that Fastnet will perform packet MTU discovery.

##Basic Packet Format##

A Fastnet packet must consist of a 2-byte channel identifier as a signed 16-bit integer followed by a packet payload of no more than the maximum packet size of the transport minus 2 bytes.
All integers are sent in big-endian.

This specification specifies per-channel packet formats.
Channels 0 to 32767 must be reserved for the application and are referred to as "message channels."
All other channels must be reserved for Fastnet's protocol and used as specified.

An implementation must prevent the user from using negative channel numbers for any purpose.
This may be done via a type system (use unsigned integers, for example) or by generating an error.

##Status Queries##

Channel -1 is the status query and connection handshake channel.
Query packets on channel -1 are ascii literals.
The following query operations must be supported:

- If a client sends the string "query:fastnet" as a packet payload on channel -1, the server must respond with "query:fastnet=yes"

- If the client sends "query:version", the server must respond with "query:version=1.0"

- If the client sends "query:extension:name", where name is an extension, the server must respond with "query:extension:name=yes" if the server supports the extension or "query:extension:name=no" if it does not.

An implementation shall not place limits on the number of times that a query may be requested from the server.
An implementation must continue to respond to queries even after a connection is established.

Extension names must be of the form "manufacturer_extensionname" and no longer than 64 characters.  "manufacturer" should be replaced with a string unique to the person or organization implementing the extension and "extensionname" with a string unique to the extension itself.
Extension and manufacturer names must be ASCII and lower-case.
This specification reserves the manufacturer string "fastnet" for official fastnet extensions.
An implementor must refrain from use of extensions with names of the form "fastnet_xxx".
An implementation must not make use of an extension without first receiving a positive query as to the server's support for said extension.

This specification defines two extensions: "fastnet_p2p" and "fastnet_file".
In order to insure uniqueness, implementors wishing to implement extensions must open a pull request or issue against the GitHub repository containing this specification in order to have a name and description added to the extension index.
Implementors are encouraged in the strongest terms to specify their extensions publicly so that other Fastnet implementations may make use of them.

##Connection Establishment##

Connections are established from client to server with the exception of UDP hole punching, described later in this specification and using a different algorithm from that described here.

A Fastnet server must allow only one connection from a specific IP and port.

Before beginning connection establishment, an implementation must use the above query interface to establish that Fastnet is listening in an implementation-defined manner.
This specification suggests that this be done by sending "query:fastnet" in a similar manner to the following connection handshake algorithm and looking for "query:fastnet=yes".

An implementation must not consider the connection of a connection-based transport to be the establishment of a fastnet connection.

The following must always take place on channel -1 before a connection is considered established.

NOTE: because reliable transports are ordered, the following algorithm cannot cause data loss on a reliable transport even though the client intensionally drops packets.

There are three packets involved in the connection handshake protocol, all of which are sent on channel -1:

- The connection request packet consists of the string "connect?"

- The connect packet consists of the string "connected" followed by a 4-byte integer.  This integer is the connection's identifier.

- The abort packet is the string "abort" followed immediately by a UTF8-encodedd error string without a trailing null.

To begin a connection, a client must:

- Send the connect request packet.

- begin waiting for either the connect packet or the abort packet with a timeout of 5000 MS.  If the transport is unreliable, the client must resend the connection request packet every 200 MS during this process; otherwise, it must not.

If the client receives the connect packet, it must parse the connection id, notify the application that the connection has been established, and begin processing packets.
The client must disregard all other packets until it manages to receive the connect packet.

If the client receives the abort packet, it must report the error string.

if the client times out before receiving either the connect or abort packet, the implementation must report an implementation-defined error.

When the server sees the connection request packet, it begins establishing a connection.
To establish a connection, a server must generate an integer ID.
This ID must be unique among all currently-established connections.
It must then encode and send the connect packet and immediately notify the application that a connection was established.
If the server continues to receive the connection request packet, it must continue to respond with the connect packet but do nothing further; it is possible for the client to not yet know that it is connected due to packet loss.

To refuse a connection, a server must send the abort packet with an implementation-defined string as to the reason.
This string must not be empty.
This specification suggests "unspecified error" as a reasonable default for situations in which a string is not available, i.e. because a game does not wish to advertise that a player has been banned.

Both clients and servers must expose the Fastnet connection ID for a connection in a manner that can be reached by the application developer; this ID is part of the UDP hole-punching algorithm.

Servers must ignore any packets not involved in an active connection.

##The Heartbeat Channel, Connection Breaking, and Round-trip Estimation##

Channel -2 must be the heartbeat channel.<commit_msg>Make queries binary packets. Make connection handshakes binary packets. Define an extension query and instructions for registering an extension.<commit_after>#Fastnet 1.0 Protocol Specification

##Introduction and Goals

Fastnet is a connection-based, channel-based protocol intended for games and realtime applications.
It intensionally forgoes much of the automatic management of TCP and assumes that settings will be provided by the application developer.

Unlike TCP, Fastnet works at the level of a message: a chunk of binary data of any size.
Messages are either fully received or never received at all, and the application need not provide further separation logic.
One Fastnet connection is logically 32768 smaller multiplexed connections called channels.

Messages can be sent reliably or unreliably.
Unreliable messages may or may not arrive at the other end, but are extremely fast and capable of representing frequently updated data such as position.
Reliable messages are intended for chat messages, status updates, or other information that absolutely must arrive.
A reliable message stalls all other messages in the queue behind it until such time as it arrives on the other end.

When considering the advantages of Fastnet, it is helpful and mostly accurate to think of  TCP as a train.
If one train car derails, everything behind it stops.
The primary advantage of Fastnet, then, is that you have the ability to use multiple train tracks at once.
Furthermore, for applications which can tolerate data loss, it is possible to mark some of the metaphorical cars as unimportant;
if these unimportant cars derail, the stream continues regardless.

Other semireliable UDP-based protocols exist, but usually only for one language.
Fastnet aims to be fully documented to the point where it can be recoded from scratch.
The other two features that Fastnet aims to support are the ability to fall back to other transports (TCP, HTTP, WebRTC) and the ability to support UDP hole-punching.
In my experience, it is difficult to find networking libraries that offer all three of these benefits.

##Definition of Terms##

The key words "must", "must not", "required", "shall", "shall not", "should", "should not", "recommended",  "may", and
"optional" in this document are to be interpreted as described in
[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).

A client refers to an entity who requests a connection from a server.
A server refers to an entity capable of accepting multiple connections.
All entities involved in Fastnet are either a client or a server, never both.

A sender is an entity capable of sending messages.
A receiver is an entity capable of receiving messages.
Clients and servers are both senders and receivers.

A packet refers to a chunk of data of any size.

A message refers to a chunk of data specified by the application.  This specification places no semantic meaning on the content of messages provided by applications.

A transport means a method of moving packets from one destination to another.  Transports usually involve the internet.

A transport's implementation refers to the concrete implementation of a transport.

UDP hole-punching refers to a technique for  circumventing network Address Translation and connecting two computers directly without the use of an intermediate server.

##Transports##

A transport is a method for moving packets from one destination to another.
Transports must have the following characteristics.

- An ordered transport must guarantee that, if packets arrive, they arrive in order.

- An incorruptible transport must guarantee that, if packets arrive, they match the sent packet.

- A reliable transport must guarantee that packets arrive if sent.  Reliable transports must be ordered and incorruptible transports.

- A connection-based transport must handle all logic of forming and keeping a connection open.  This specification refers to transports which are not connection-based as connectionless.

All implementations must provide at least the TCP and UDP transports.
The TCP transport must be reliable and connection-based.
The UDp transport must be unordered, unreliable, and connectionless.
Over IPV4, UDP's built-in checksum is optional.
To this end, UDP must also be corruptible.

Every transport must advertise the maximum packet size it supports in a way that allows the user of the implementation to obtain the value.
Every transport must support a maximum packet size of at least 4096 bytes.
Note that Fastnet will perform packet MTU discovery.

##Basic Packet Format##

A Fastnet packet must consist of a 2-byte channel identifier as a signed 16-bit integer followed by a packet payload of no more than the maximum packet size of the transport minus 2 bytes.
All integers are sent in big-endian.

This specification specifies per-channel packet formats.
Channels 0 to 32767 must be reserved for the application and are referred to as "message channels."
All other channels must be reserved for Fastnet's protocol and used as specified.

An implementation must prevent the user from using negative channel numbers for any purpose.
This may be done via a type system (use unsigned integers, for example) or by generating an error.

##Status Queries##

Channel -1 is the status query and connection handshake channel.
Queries must be responded to in both directions.
A query packet consists of the channel specifier, the byte 0, the number of the query to be performed as a 16-bit integer, and an optional argument.
A response packet consists of the byte 1, the number sent in the query packet, and a response as specified in the following table.
All of the following query operations must be implemented:

number | description | Arguments | response
------ | ----------- | --------- | --------
0 | Is Fastnet listening on this port? | None | one byte, 0 if no, 1 iif yes.
1 | version query | None | The version of the protocol as a string without terminating null. Currently this is "1.0"
2 | extension query. | Name of the extension as an ASCII string without terminating NULL | the byte 0 (not supported) or the byte 1 (supported) followed by the extension's name as sent by the client without terminating NULL.

An implementation shall not place limits on the number of times that a query may be sent and must always respond.
An implementation must continue to respond to queries even after a connection is established.

Extension names must be of the form "manufacturer_extensionname" and no longer than 64 characters.  "manufacturer" should be replaced with a string unique to the person or organization implementing the extension and "extensionname" with a string unique to the extension itself.
Extension and manufacturer names must be ASCII and lower-case.
This specification reserves the manufacturer string "fastnet" for official fastnet extensions.
An implementor must refrain from use of extensions with names of the form "fastnet_xxx".
An implementation must not make use of an extension without first receiving a positive query as to the server's support for said extension.

This specification defines two extensions: "fastnet_p2p" and "fastnet_file".
In order to insure uniqueness, implementors wishing to implement extensions must open a pull request or issue against the GitHub repository containing this specification in order to have a name and description added to the extension index.
Implementors are encouraged in the strongest terms to specify their extensions publicly so that other Fastnet implementations may make use of them.

##Connection Establishment##

Connections are established from client to server with the exception of UDP hole punching, described later in this specification and using a different algorithm from that described here.

A Fastnet server must allow only one connection from a specific IP and port.

Before beginning connection establishment, an implementation must use the above query interface to establish that Fastnet is listening in an implementation-defined manner.
This specification suggests that this be done in a similar manner to the following connection handshake algorithm but using the is Fastnet listening query (query number 0).

An implementation must not consider the connection of a connection-based transport to be the establishment of a fastnet connection.

The following must always take place on channel -1 before a connection is considered established.

NOTE: because reliable transports are ordered, the following algorithm cannot cause data loss on a reliable transport even though the client intensionally drops packets.

There are three packets involved in the connection handshake protocol, all of which are sent on channel -1:

- The connection request packet consists of the single byte 2.

- The connect packet consists of the single byte 3 followed by a 4-byte integer.  This integer is the connection's identifier.

- The abort packet consists of the single byte 4 followed immediately by a UTF8-encodedd error string without a trailing null.

To begin a connection, a client must:

- Send the connect request packet.

- begin waiting for either the connect packet or the abort packet with a timeout of 5000 MS.  If the transport is unreliable, the client must resend the connection request packet every 200 MS during this process; otherwise, it must not.

If the client receives the connect packet, it must parse the connection id, notify the application that the connection has been established, and begin processing packets.
The client must disregard all other packets including queries until it manages to receive the connect packet.

If the client receives the abort packet, it must report the error string.

if the client times out before receiving either the connect or abort packet, the implementation must report an implementation-defined error.

When the server sees the connection request packet, it begins establishing a connection.
To establish a connection, a server must generate an integer ID.
This ID must be unique among all currently-established connections.
It must then encode and send the connect packet and immediately notify the application that a connection was established.
If the server continues to receive the connection request packet, it must continue to respond with the connect packet but do nothing further; it is possible for the client to not yet know that it is connected due to packet loss.

To refuse a connection, a server must send the abort packet with an implementation-defined string as to the reason.
This string must not be empty.
This specification suggests "unspecified error" as a reasonable default for situations in which a string is not available, i.e. because a game does not wish to advertise that a player has been banned.

Both clients and servers must expose the Fastnet connection ID for a connection in a manner that can be reached by the application developer; this ID is part of the UDP hole-punching algorithm.

Servers must ignore any packets not involved in an active connection.

##The Heartbeat Channel, Connection Breaking, and Round-trip Estimation##

Channel -2 must be the heartbeat channel.
{'fraction_non_alphanumeric': 0.03257373885877056, 'fraction_numerical': 0.005191583203160893, 'mean_word_length': 4.683289817232376, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '459875', 'n_tokens_mistral': 4877, 'n_tokens_neox': 4645, 'n_words': 3323}