TECHNICAL FIELD

Embodiments relate generally to distributed computing systems, and, more specifically, to network switch architectures for distribute artificial intelligence systems and other applications.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to: U.S. patent application Ser. No. 16/409,695, filed on May 10, 2019, entitled “Network Switch with Integrated Compute Subsystem for Distributed Artificial Intelligence and Other Applications,” by Matthews, et al.; U.S. patent application Ser. No. 16/409,699, filed on May 10, 2019, entitled “Egress-Based Compute Architecture for Network Switches in Distributed Artificial Intelligence and Other Applications,” by Matthews, et al.; U.S. patent application Ser. No. 16/409,701, filed on May 10, 2019, entitled “Parallelized Ingress Compute Architecture for Network Switches in Distributed Artificial Intelligence and Other Applications,” by Matthews, et al. The entire contents of each of these applications are hereby incorporated by reference for all purposes as if fully set forth herein.

BACKGROUND

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

Computationally-intense applications in computing systems can often be implemented by dividing the applications into distinct tasks that can be performed in parallel, and distributing those tasks amongst a number of computing devices. These computing devices are typically interconnected by a communications network via which they share data related to the computations, and are said to form a distributed computing system. Distributed computing systems may be used in a large variety of complex computing applications, such as, without limitation, simulations, language translation, image recognition, fraud detection, and so forth, as well as emerging applications.

For example, machine learning algorithms, and deep learning algorithms in particular, are commonly used to perform mission critical computing functions. Such algorithms may rely on a variety of frameworks, such as neural networks or belief networks, that input and process a set of values (referred to herein as an “input vector,” or simply “vector”) through one or more layers of interconnected nodes (referred to herein as “neurons”), to generate an output value or values. The output value(s) generally classify the input vector in some manner. For instance, the output of a neural network used for image recognition might classify an input vector of pixels, image attributes, or other image data as being either a dog or cat, depending on the purpose of the neural network.

Each neuron of a neural network accepts one or more inputs from the vector and/or other neurons. These inputs form the connections of the network. Each neuron is associated with an output function that computes the value output from the neuron based on the inputs to the neuron. The connections may be assigned weights. The weight of a connection, generally speaking, controls how heavily the input associated with that connection factors into the output function. For instance, a neuron might have an input p0 with a weight of 0.4 and an input p1 with a weight of 0.2. The value of the input p0 may therefore more heavily impact the output of the neuron (e.g., in the case of a simple summation of the products of each input and their weights, twice as much). In some embodiments, the neurons may be organized into two or more layers, including an input layer wherein each neuron outputs a value of the input vector, zero or more intermediate layers in which each neuron inputs one or more values output by an immediately previous layer and then output values to one or more neurons of an immediately subsequent layer, and a final output layer that inputs values output by an immediately previous layer and outputs the results of the neural network.

By carefully setting the weights of a neural network, the network may be configured to accurately or semi-accurately make classifications or other determinations based on input vectors. Suitable weights for a network configured to make a certain type of determination based on a certain type of data may be “learned” through various training algorithms. These training algorithms iteratively adjust the weights over time through a series of steps, including a forward pass, loss computation, and backward pass, until arriving at an “optimal” set of weights for the network, or until all training data has been processed.

The forward pass through the neural network processes an input vector selected from a suitable set of vectors (e.g., a set of “training data”) through the neurons using a test set of weights to produce an output referred to herein as a “prediction.” The loss computation computes the error in that prediction, which may be referred to as a gradient. Finally, the backward pass updates the test weights in the neural network using a gradient descent algorithm, so as to arrive at a new set of weights to test. The gradient descent algorithm selects the new weights in a manner intended to minimize the computed loss in subsequent iterations, until arriving at an acceptable or optimal loss, depending on the embodiment.

Distributed deep learning techniques have been developed in which training tasks are spread out across any number of networked computing devices, each implementing one or more “compute nodes” configured to perform a specific training task. For example, each compute node may be assigned a different set of input vectors (e.g., from a set of training data) to process with the neural network. The compute nodes share the gradients from their results over the network, which they combine together in a “reduction” phase to determine new weights to test on the network. Another approach is model parallelism, where portions of the model are assigned to each compute node and the interconnections (e.g., activations) between the various model layers are sent via the network as part of the forward pass.

Early distributed deep learning approaches made use of a centralized parameter server to reduce gradients and return results to compute nodes. In such approaches, the centralized parameter server is typically implemented using the same hardware as any other compute node, having significant computing power but limited connectivity. The centralized parameter server is therefore often a significant bottleneck, on account of network latency and/or limited network bandwidth to and from the centralized parameter server.

Another common distributed approach is known as “All Reduce.” A first compute node in such a system may be configured to pass on its gradients to a second node, which may be configured to aggregate or otherwise combine the result with its gradient. The second node then passes the result on to a third node, and this process repeats in a ring, tree, or other suitable pattern until all of the nodes have combined their gradients together. The final result is then passed back through the topology. The amount of time needed to reduce the gradients and propagate the result throughout the system is a significant bottleneck to the learning process.

More generally, the communication of data and other information between nodes of distributed computing systems has consistently proven to be a significant bottleneck in the performance of the distributed system.

