We present the design of a new large scale orchestration layer for accelerators. Our system,Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models.Pathwaysuses ashardeddataflow graph ofasynchronousoperators that consume and produce futures, and efficiently gang-schedulesheterogeneousparallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects.Pathwaysmakes use of a novelasynchronous distributed dataflowdesign that lets the control plane execute in parallel despite dependencies in the data plane. This design, with careful engineering, allowsPathwaysto adopt a single-controller model that makes it easier to express complex new parallelism patterns.
We demonstrate thatPathwayscan achieve performance parity (accelerator utilization)  with state-of-the-art systems when running SPMD computations over 2048 TPUs, while also delivering throughput comparable to the SPMD case for Transformer models that are pipelined acrossstages, or sharded across two islands of accelerators connected over a data center network.