In addition to parallel evaluation of deep networks, we also provide several solutions for rapid training of CNTK models. CPU Spark clusters excel at large scale extract, transform, and load (ETL) operations. However these clusters lack GPU acceleration and InfiniBand support, which degrades the performance of high communication network training. As a result, we have investigated hybrid systems consisting of dedicated multi-gpu machines, tethered to the Spark cluster over a virtual network. These systems use several CNTK distributed training modes such as the highly optimized 1-bit SGD [1]}. We provide Azure Resource Manager templates so that users can deploy these solutions.
