Transparent Inter-process Communication

Transparent Inter Process Communication (TIPC) is an Inter-process communication (IPC) service in Linux designed for cluster wide operation. It is sometimes presented as "Cluster Domain Sockets", in contrast to the well-known "Unix Domain Socket" service; the latter working only on a single kernel.
Features.
Some features of TIPC:
Implementations.
The TIPC protocol is available as a module in the mainstream Linux kernel, and hence in most Linux distributions. The TIPC project also provides open source implementations of the protocol for other operating systems including Wind River's VxWorks and Sun Microsystems' Solaris. TIPC applications are typically written in C (or C++) and utilize sockets of the AF_TIPC address family. Support for Go, D, Perl, Python, and Ruby is also available.
Service Addressing.
A TIPC application may use three types of addresses.
A socket can be bound to several different service addresses or ranges, just as different sockets can be bound to the same service address or range. Bindings are also qualified with a "visibility scope", i.e., node local or cluster global visibility.
Datagram Messaging.
Datagram messages are discrete data units between 1 and 66,000 byte of length, transmitted between non-connected sockets. Just like their UDP counterparts, TIPC datagrams are not guaranteed to reach their destination, but their chances of being delivered are still much better than for the former. Because of the link layer delivery guarantee, the only limiting factor for datagram delivery is the socket receive buffer size. The chances of success can also be increased by the sender, by giving his socket an appropriate delivery "importance" priority. Datagrams can be transmitted in three different ways.
Connection Oriented Messaging.
Connections can be established the same way as with TCP, by means of accept() and connect() on SOCK_STREAM sockets. However, in TIPC the client and server use service addresses or ranges instead of port numbers and IP addresses. TIPC does also provide two alternatives to this standard setup scenario.
The most distinguishing property of TIPC connections is still their ability to react promptly to loss of contact with the peer socket, without resorting to active neighbor heart-beating.
Group Messaging.
Group messaging is similar to datagram messaging, as described above, but with end-to-end flow control, and hence with delivery guarantee. There are however a few notable differences.
When joining a group, a member may indicate if it wants to receive "join" or "leave" events for other members of the group. This feature leverages the "service tracking" feature, and the group member will receive the events in the member socket proper.
Service Tracking.
An application accesses the tracking service by opening a connection to the TIPC internal topology server, using a reserved service address. It can then send one or more "service subscription messages" to the tracking service, indicating the service address or range it wants to track. In return, the topology service sends "service event messages" back to the application whenever matching addresses are bound or unbound by sockets within the cluster. A service event contains the found matching service range, plus the port and node number of the bound/unbound socket. There are two special cases of service tracking:
Although most service subscriptions are directed towards the node local topology server, it is possible to establish connections to other nodes' servers and observe their local bindings. This might be useful if e.g., a connectivity subscriber wants to create a matrix of all connectivity across the cluster, - not limited to what can be seen from the local node.
Cluster.
A TIPC network consists of individual processing elements or "nodes". Nodes can be either physical processors, virtual machines or network namespaces, e.g., in the form of Docker Containers. Those nodes are arranged into a "cluster" according to their assigned "cluster identity". All nodes having the same cluster identity will establish links to each other, provided the network is set up to allow mutual "neighbor discovery" between them. It is only necessary to change the cluster identity from its default value if nodes in different clusters potentially may discover each other, e.g., if they are attached to the same subnet. Nodes in different clusters cannot communicate with each other using TIPC.
Before Linux 4.17, nodes must be configured a unique 32-bit "node number" or address, which must comply with certain restrictions. As from Linux 4.17, each node has a 128 bit "node identity" which must be unique within the node's cluster. The node number is then calculated as a guaranteed unique hash from that identity.
If the node will be part of a cluster, the user can either rely on the "auto configuration capability" of the node, where the identity is generated when the first interface is attached, or he can set the identity explicitly, e.g., from the node's host name or a UUID. If a node will not be part of a cluster its identity can remain at the default value, zero.
Neighbor discovery is performed by UDP multicast or L2 broadcast, when available. If broadcast/multicast support is missing in the infrastructure, discovery can be performed by explicitly configured IP addresses.
Inter Node Links.
A cluster consists of nodes interconnected with one or two links. A link constitutes a reliable packet transport service, sometimes referred to as an "L2.5" data link layer.
Cluster Scalability.
Since Linux 4.7, TIPC comes with a unique, patent pending, auto-adaptive hierarchical neighbor monitoring algorithm. This "Overlapping Ring Monitoring" algorithm, in reality a combination of ring monitoring and the Gossip protocol, makes it possible to establish full-mesh clusters of up to 1000 nodes with a failure discovery time of 1.5 seconds, while it in smaller clusters can be made much shorter.
Performance.
TIPC provides outstanding performance, especially regarding round-trip latency times. Inter-node it is typically 33% faster than TCP, intra-node 2 times faster for small messages and 7 times faster for large messages. Inter-node, it provides a 10-30% lower maximal throughput than TCP, while its intra-node throughput is 25-30% higher. The TIPC team is currently studying how to add GSO/GRO support for intra node messaging, in order to match TCP even here.
Transport Media.
While designed to be able to use all kinds of transport media, implementations support UDP, Ethernet and Infiniband. The VxWorks implementation also supports shared memory which can be accessed by multiple instances of the operating system, running simultaneously on the same hardware.
Security.
Security must currently be provided by the transport media carrying TIPC. When running across UDP, IPSec can be used, when on Ethernet, MACSec is the best option. The TIPC team is currently looking into how to support TLS or DTLS, ether natively or by an addition to OpenSSL.
History.
This protocol was originally developed by Jon Paul Maloy at Ericsson during 1996-2005 and was used by that company in cluster applications for several years, before subsequently being released to the open source community and integrated in the mainstream Linux kernel. It has since then undergone numerous improvements and upgrades, all performed by a dedicated TIPC project team with participants from various companies. The management tool for TIPC is part of the "iproute2" tool package which comes as standard with all Linux distributions.