Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learning
To break the bottlenecks of mainstream cloud-based machine learning (ML)
paradigm, we adopt device-cloud collaborative ML and build the first end-to-end
and general-purpose system, called Walle, as the foundation. Walle consists of
a deployment platform, distributing ML tasks to billion-scale devices in time;
a data pipeline, efficiently preparing task input; and a compute container,
providing a cross-platform and high-performance execution environment, while
facilitating daily task iteration. Specifically, the compute container is based
on Mobile Neural Network (MNN), a tensor compute engine along with the data
processing and model execution libraries, which are exposed through a refined
Python thread-level virtual machine (VM) to support diverse ML tasks and
concurrent task execution. The core of MNN is the novel mechanisms of operator
decomposition and semi-auto search, sharply reducing the workload in manually
optimizing hundreds of operators for tens of hardware backends and further
quickly identifying the best backend with runtime optimization for a
computation graph. The data pipeline introduces an on-device stream processing
framework to enable processing user behavior data at source. The deployment
platform releases ML tasks with an efficient push-then-pull method and supports
multi-granularity deployment policies. We evaluate Walle in practical
e-commerce application scenarios to demonstrate its effectiveness, efficiency,
and scalability. Extensive micro-benchmarks also highlight the superior
performance of MNN and the Python thread-level VM. Walle has been in
large-scale production use in Alibaba, while MNN has been open source with a
broad impact in the community.