In recent years, implementation of “cloud-based” services, high-performance computing (HPC) and other activities employing data centers and the like have seen widespread adoption. Under a typical data center installation, a large number of servers installed in server chasses and server racks are interconnected in communication using network links (e.g., Ethernet or InfiniBand) and various switching mechanisms, such as switch blades/modules and “top-of-rack” (ToR) switches.
Under aspects of HPC, a very large number of compute nodes are implemented to solve various tasks in a parallel or substantially parallel manner. In essence, each compute node performs one part of a larger, more complex task. In order to implement this scheme, there needs to be input data to and output data must be exchanged among compute nodes. This data is communicated using one or more interconnects.
Various types of interconnects are used to interconnect the computer nodes in an interconnect hierarchy. For example, at the top of the hierarchy are interconnects between computing cores in the same processor. At the next level are interconnects between processors in the same platform, such as server blade or module. Next are interconnects between platforms, such as a backplane in a blade server. This is followed by interconnects between server chassis and/or ToR switches, interconnects between server racks, and finally interconnects between data centers. Generally, the communication bandwidth between nodes is reduced as one moves farther down the hierarchy.
In addition to latencies corresponding to transfers across the interconnect links themselves (which is a function of the link bandwidth and length and switching speed), significant latencies result from operations performed at the interfaces to the interconnects and/or additional processing/required to prepare the data for transfer across various interconnect links in the interconnect hierarchy. These data transfer latencies collectively reduce the communication performance and therefore the performance of the overall HPC implementation, and may represent a significant portion of the total latency (processing and data transfer) for the compute nodes.
Another important aspect of HPC is the software architecture. Implementing software using 10's of thousands of compute nodes in a parallel manner requires a significantly different software architecture than that employed for conventional applications. In addition, specific software modules have been developed for using corresponding types of interconnects, such as software modules used for communicating over InfiniBand.