FIELD

This disclosure relates generally to techniques for accelerated data processing in a high-performance computing environment.

BACKGROUND

Various types of special-purpose processors, such as graphics processing units (GPUs) for general purpose computing and other types of hardware accelerators, have been developed for accelerated processing of specific types of workloads. The processing capabilities of GPU devices and other types of hardware accelerators are currently being utilized in various applications to accelerate the processing of highly-parallelized computational workloads in various technical fields. In particular, general-purpose computing on GPU (GPGPU) is utilized for high-throughput, accelerated processing of compute kernels for workloads (e.g., vector-based computations, matrix-based computations, etc.) that exhibit data-parallelism. For example, GPIUs are used to accelerate data processing in high-performance computing (HPC) and embedded computing systems, for various applications such as financial modeling, scientific research, machine learning (ML), deep learning (DL), data mining, video data transcoding, image analysis, image recognition, virus pattern matching, augmented reality, encryption/decryption, weather forecasting, big data analytics and comparisons, and other applications with computational workloads that have an inherently parallel nature.

A distributed computing environment which comprises a large scale of shared computing resources over a cluster of computing nodes is typically utilized to support emerging applications such as big data analytics and DL model training applications. A distributed DL model training task requires the collection, storage, and processing of a significantly large amount of data, wherein the data includes training data to build and optimize DL models, as well as model parameters of the deep learning models which are utilized for inference processing. Implementing an efficient distributed computing environment for DL training (and other HPC applications) is not trivial as the intensive computational workloads, and the massive volume of data that must be stored, streamed, prefetched, and coordinated between the shared computing resources of the distributed computing platform presents a significant challenge and practical limit on system performance and scalability.

For distributed DL training applications, an HPC system can implement a parameter server (PS) system to enable distributed and parallelized training of a deep neural network model using a cluster of worker nodes (e.g., accelerator devices such as GPU devices). A parameter server system provides a communication synchronization protocol in which multiple worker nodes involved in a parallel distributed DL training process have shared access to a recent set of model parameters of a given DL model being trained. The parameter server system provides a synchronization mechanism which involves performing an “all-reduce” operation by aggregating and averaging the processing results (subsets of computed parameters) from all parallel worker nodes, and then distributing the aggregated/averaged processing results to all worker nodes for subsequent computations. Depending on various factors such as, e.g., the amount of data that needs to be synchronized between the worker nodes, the number of worker nodes involved in the training process, the inter-node/intra-node network and hardware topology of the distributed systems, etc., the communication synchronization operations of the parameter server system can consume a significant amount of bus/network communication bandwidth, which can adversely impact the performance of the distributed DL training process.

SUMMARY

Illustrative embodiments of the invention include methods for implementing a parameter server within a networking infrastructure of a computing system to reduce the communication bandwidth and latency for performing communication synchronization operations of the parameter server. For example, one embodiment includes a method which comprises executing a distributed DL model training process to train model parameters of a DL) model using a plurality of worker nodes executing on one or more server nodes of a computing system, and executing a parameter server within a networking infrastructure of the computing system to aggregate local model parameters computed by the plurality of worker nodes and to distribute aggregated model parameters to the plurality of worker nodes using the networking infrastructure of the computing system.

Other embodiments of the invention include, without limitation, systems and articles of manufacture comprising processor-readable storage media, which are configured to implement a parameter server within a networking infrastructure of a computing system to reduce the communication bandwidth and latency for performing communication synchronization operations of the parameter server.

