Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs
Cloud datacenters are exponentially growing both in numbers and size. This
increase results in a network activity surge that warrants better congestion
avoidance. The resulting challenge is two-fold: (i) designing algorithms that
can be custom-tuned to the complex traffic patterns of a given datacenter; but,
at the same time (ii) run on low-level hardware with the required low latency
of effective Congestion Control (CC). In this work, we present a Reinforcement
Learning (RL) based CC solution that learns from certain traffic scenarios and
successfully generalizes to others. We then distill the RL neural network
policy into binary decision trees to achieve the desired $\mu$sec decision
latency required for real-time inference with RDMA. We deploy the distilled
policy on NVIDIA NICs in a real network and demonstrate state-of-the-art
performance, balancing all tested metrics simultaneously: bandwidth, latency,
fairness, and packet drops.