Active-Learning-as-a-Service: An Automatic and Efficient MLOps System for Data-Centric AI

The success of today's AI applications requires not only model training (Model-centric) but also data engineering (Data-centric). In data-centric AI, active learning (AL) plays a vital role, but current AL tools 1) require users to manually select AL strategies, and 2) can not perform AL tasks efficiently. To this end, this paper presents an automatic and efficient MLOps system for AL, named ALaaS (Active-Learning-as-a-Service). Specifically, 1) ALaaS implements an AL agent, including a performance predictor and a workflow controller, to decide the most suitable AL strategies given users' datasets and budgets. We call this a predictive-based successive halving early-stop (PSHEA) procedure. 2) ALaaS adopts a server-client architecture to support an AL pipeline and implements stage-level parallelism for high efficiency. Meanwhile, caching and batching techniques are employed to further accelerate the AL process. In addition to efficiency, ALaaS ensures accessibility with the help of the design philosophy of configuration-as-a-service. Extensive experiments show that ALaaS outperforms all other baselines in terms of latency and throughput. Also, guided by the AL agent, ALaaS can automatically select and run AL strategies for non-expert users under different datasets and budgets. Our code is available at \url{https://github.com/MLSysOps/Active-Learning-as-a-Service}.


Introduction
Data-centric AI is an emerging topic that focuses on engineering data to develop AI applications with off-the-shelf machine learning (ML) models [14]. Previous efforts are mainly model-centric AI that assumes a static environment. In this environment, 1) the data collection and engineering are done, 2) and continuously developing ML models to achieve high performance on test sets is the main target [16]. However, real-world AI applications are facing more complicated scenarios, which can not be adequately addressed by model-centric AI. For instance, researchers and practitioners have to spend a lot of time on data preparation, including data labeling [9], and error detection [27]. Meanwhile, they also need to monitor the data, so that they can update models in time when a distribution drift is detected [23]. Treating these issues only from a model-centric view will lead to a sub-optimal solution. Therefore, to further improve and democratize AI applications, a lot of efforts are now turning to data-centric or combining model-centric and data-centric [14].
Though the concept of data-centric AI has been proposed very recently, many pioneering studies whose core contributions lie in data engineering have already been proposed [43,50]. Among them, one vital direction is active learning (AL) [37]. The motivation of AL is to reduce manual labeling efforts while maintaining and even improving ML models' performance [49,43,17,15,6,2,1,56,31]. Specifically, it is well-known that ML models are very data-hungry. Therefore, to reach a high performance (e.g., accuracy) that meets application requirements, people always need to label a large amount of data. This process is extremely time-consuming and labor-intensive and thus often becomes the bottleneck of ML application development. To cope with the issue, AL selects the most representative yet diverse training samples from a large training data pool by utilizing AL strategies. Then, it only sends the selected samples to an oracle (e.g., human annotators) to label. Next, ML models will only be trained on these labeled sub-datasets. By doing so, we can still obtain an ML model with competitive performance but save labeling and training costs a lot.
However, utilizing AL is a non-trivial task. First, selecting a suitable AL strategy for a specific scenario given the budget and target accuracy is hard for both experts and non-experts. As a result, users have to run AL in a trial-and-error manner, resulting in huge time and monetary waste. Second, applying AL to AI application development is not simply searching for, selecting, or implementing AL strategies. Instead, users have to build a backend to run the AL pipeline, tailored for their applications in their environment (e.g., a private cluster and AWS). In other words, they need to undertake much repetitive engineering work with boilerplate code. Third, users have to consider the efficiency and cost issues, as AL often runs on a vast dataset, and some AL strategies (e.g., committee-based [11,32]) require running more than one ML model for data selection. Under-consideration will result in a long process time and additional cost. Though several open-source AL tools [12,20,51,47] lower the barrier of applying AL, and they can meet neither automation nor efficiency requirements.
To address these issues, we propose an automatic and efficient backend for AL. Our AL system, named Active-Learning-as-a-Service (ALaaS) (Figure 1). It can select AL strategies automatically given the budget and target accuracy without specifying an AL strategy. Also, it can run AL strategies on large datasets efficiently by utilizing single or distributed multiple devices. Specifically, ALaaS adopts the server-client architecture to perform AL tasks and can be deployed easily on both laptops and public clouds as a service. To achieve high efficiency, it implements a stage-level parallelism method to run AL tasks by fully utilizing hardware resources and reducing waiting time. Meanwhile, more acceleration techniques such as data cache and batching [10,54,53] are utilized to further speed up the AL process. Besides, for users who have difficulty selecting a suitable AL strategy, ALaaS provides a predictive-based successive halving early-stop (PSHEA) procedure to automatically select and run AL with an only budget and target accuracy inputs. In addition to that, our system also considers accessibility and modularity, so that users can use many AL strategies in our AL zoo with ease, and experts can propose more advanced AL strategies for new scenarios. Experiments show that 1) our ALaaS outperforms all other baselines in terms of latency and throughput, and 2) the PSHEA procedure can predict future accuracy and select suitable AL strategies under different settings. Further ablation studies show the effectiveness of our design and reveal more insightful conclusions.

Related Work
This section presents the related work, including three categories: Active Learning (AL) algorithms and tools, Data-centric AI, and MLOps.

AL Algorithms and Tools
We categorize AL strategies into three classes, namely, diversity-based, uncertainty-based, and hybrid sampling. Diversity-based methods [52,43] are designed to select the most diverse samples from the whole dataset to represent it. Uncertainty-based methods [49,40,17] aim to select the samples that can not be identified confidently by current ML models and then use these samples to further  Figure 1: An overview of our ALaaS with human-in-the-loop. Our system adopts a server-client architecture. The data scientist first collects the unlabeled data and sends them to the AL server using an AL client, where an AL agent, a data cache and multiple workers will select data samples collaboratively and efficiently. The selected samples will be sent to a human oracle for labeling and then for updating models.
improve ML models. Hybrid methods [22,4] combine both the above-mentioned methods. Our system supports all of these methods and runs them more efficiently.
Many open-source AL tools have been developed to benefit both academia and industry, including ModAL [12], DeepAL [20], Libact [51], and ALiPy [47]. Our ALaaS is inspired by these tools and further improves AL efficiency and accessibility by adopting the MLOps concept. The detailed comparison is summarized in Table1.

Data-centric AI
Data-centric AI is proposed to improve AI application performance by engineering datasets rather than only focusing on models. Recent Data-centric AI competitions and workshops [14] demonstrate many exciting studies from academia and industry. Inspired by the pioneering work, many datacentric methods have been proposed for different areas, including Natural Language Processing (NLP) [50,44], Computer Vision (CV) [21,7], Robotics [30], etc. Also, a new benchmark [16] has been built for pushing forward data-centric AI research. To the best of our knowledge, ALaaS is the first MLOps system for efficient AL from the data-centric view.

MLOps
MLOps (Machine Learning Operation) aims to streamline the ML model development and reduce the AI application maintenance cost. Many MLOps systems have been proposed for both data-centric AI and model-centric AI [39]. From a data-centric view, labeling tools (e.g., labelme [41]), data cleaning tools (e.g., ActiveClean [26]), data drift monitors and many others, can all be regarded as MLOps systems. From a model-centric view, there are model store systems [48], model continuous integration [54,38] tools, training platforms [24], and deployment platforms [8], etc. Different from these systems, ALaaS is designed specifically for executing human-involved AL tasks more efficiently.

System Design and Architecture
This section first highlights our Active-Learning-as-a-Service (ALaaS) with two key features, then details the workflow and the core optimizations of the system as shown in Figure1. protocol: "grpc" host: "0.0.0.0" port: 60035 replicas: 1 Figure 2: An AL service can be easily configured and started with YML files.

ALaaS Highlights
We highlight two key features provided by our system, namely efficiency and automation. These features are also our design principles, leading the implementation to consider both experts (e.g., data scientists and machine learning engineers) and non-experts (e.g., customers with little domain knowledge) all the time. Besides, we list the main differences between our system and other existing tools in the table1.
Efficiency. Different from previous ALaaS python tools, our system is developed as a machine learning service. As a service, in addition to providing various AL strategies, our ALaaS offers efficiency to users by employing a lot of optimization technologies, including a pipeline process [34], ML serving backend adoption [36], and caching.
Automation. To further lower the barrier to use, an AL system should ensure that it can be utilized by non-AL experts with minimal effort and without writing much code. To this end, we design an AL agent to automate the AL strategy selection and the data selection procedures. With the AL agent, non-experts only need to input target accuracy and budget, then sit and wait for the final results.

ALaaS Workflow
As shown in Figure 1, ALaaS adopts the service-client architecture to abstract complex AL algorithms into web-based services, enabling an out-of-the-box user experience. Specifically, first, users only need to prepare a configuration file including basic settings like model name and serving device by following the provided templates, as shown in Figure 2. Then, with very few lines of code (LoCs), users can start both the AL client and AL server. Next, users will push their unlabeled datasets to the AL server. The dataset can be stored either in the local disk or AWS S3 [45]. Meanwhile, if users do not specify an AL strategy, ALaaS will invoke an AL agent to automatically select suitable strategies to perform data selection tasks to approach users' accuracy targets constrained by budget. Finally, the AL server will return a selected dataset for human labeling as well as further model updating.

ALaaS Optimization
Different from previous methods only providing AL strategy for accessibility, our ALaaS achieves high efficiency and automation by implementing a pipeline processing with a data cache, and an AL agent including a performance predictor and a loop controller.
Pipeline. As shown in Figure3, we divide the processing into three stages, downloading (red color), pre-processing (blue color), and AL (green color). Once users start to send datasets, the ALaaS server will parse the datasets' Uniform Resource Identifier (URI) [5] in the AL client and pipeline the data downloading and data pre-processing to reduce the hardware waiting time and improve the processing speed. Meanwhile, we implement a data cache to temporarily store the processed samples. By doing so, we further improve the processing speed. The reason is public clouds usually adopt the computation and storage separation design, and transferring the data back and forth between the current computation node to the storage node is very time-consuming. After the AL server receive a batch (batch size depends on users' settings) of raw samples, it will send them to optimized inference workers with ML serving backend for pre-processing (e.g., embedding calculation). Finally, those workers can return the processed data to the AL server for final data selection, then send the selected results back to the AL client. We observe that even with such a simple pipeline optimization, ALaaS's processing speed can become 10x times faster than the other open-source platforms (see Section 4.2).
AL agent. The AL agent implements a Predictive-based Successive Halving Early-stop (PSHEA) procedure (as shown in the Algorithm 1) to help users who are unsure about selecting AL strategies. Specifically, it includes a performance predictor and a loop controller. The performance predictor is a negative exponential forecasting model [25]. It predicts the next-round accuracy that can be achieved by a certain AL strategy under certain budget settings. The loop controller is to run many AL strategies as 'candidates' first and then decide which one should be eliminated at the current round (i.e., an early-stopping mechanism). After several rounds of reaching pre-defined targets (e.g., budget or accuracy), most AL strategies will be eliminated and the left strategies with the selected data samples will be suggested to users. This design is mainly based on the conclusion that no AL strategy can outperform all others under all settings [18]. With this AL agent, non-experts can utilize ALaaS in a cost-efficient manner rather than trying all AL strategies in a brute force manner.

System Evaluation
This section presents the quantitative evaluation of our systems. We first compare our system with other open-source platforms. Then we benchmark our system from different perspectives to demonstrate its efficiency and accessibility. 2. unlabeled data set ξ with size τ ; 3: 3. the maximum labeling budget b max (b max ≤ τ ); 4: 4. active learning strategy set L; 5: a 0 ← pre-train the deep active learning model and get the initial evaluation accuracy. 6: initialize 1. the maximum evaluation accuracy a max = a 0 ; 7: 2. active learning round r = 0; 8: 3. labeled data set d l ← ∅ for strategy l; a * l ← train a negative exponential forecasting model on historical a l and predict the 18: active learning accuracy a * l in the next round; 19: b total ← b total + b l r calculate the consumed total budget. 20: end for 21: r ← r + 1 22: a max ← update the best evaluation accuracy among all strategies; 23: if number of strategy in L > 1 then /* strategy-level early-stopping */ 24: stop and remove the strategy l with the least a * l from L; 25: end if 26: end while

Evaluation setup
We implement our ALaaS with Python, gRPC, etc., and conduct a set of benchmark studies with the following settings.
Hardware & Software. We conduct experiments on AWS EC2 and an Apple Mac Mini (M1 Chip). Our integrated data processing worker is NVIDIA Triton Inference Server [36].
Dataset. We evaluate our system on CIFAR-10 [28] and SVHN datasets. CIFAR-10 includes 50,000 training images and 10,000 test images. SVHN contains 600,000 images and we randomly sample 3,000 images as the test set and 10,000 images as the AL set to be labeled.
Model. We use the widely deployed ResNet-18 [19] model to benchmark systems under many settings. We only fine-tune ResNet-18's last layer with the AL-selected and human-labeled samples.

Comparison with other AL open source tools
The first experiment compares the efficiency of ALaaS with that of other baselines.
Settings. In this experiment, we simulate a one-round AL process, which applies AL methods to scan the whole dataset to generate a sub-pool at one-time. This sub-pool includes samples that will be used to improve an existing ML model further. Specifically, we first train an ML model with randomly selected 10,000 images from the CIFAR-10 training set as the initial model. Next, we use different AL tools to serve the whole AL pipeline on an AWS 3x.large CPU/GPU EC2. For all tools, we use the same AL strategy named least confidence sampling [29]. Finally, these tools will select 10,000 samples from the rest 40,000 images in the training set and we will compare their latency and throughput for an efficiency evaluation.
Results & Insights. The results are shown in Table2. Compared to other tools, our ALaaS achieves the lowest latency and highest throughput while still maintaining the same updated model evaluation accuracy. This efficiency improvement can be attributed to two sides. First, our ALaaS implements stage-level parallelism which reduces the device idle time extremely. Second, ALaaS allows us- ing multi-threads and distributed processing workers (e.g., Triton inference server for embedding extraction) to accelerate the data selection.

ALaaS Characterization
We further benchmark our ALaaS with different system settings. The first experiment is to evaluate different AL strategies re-implemented in our system. The second experiment explores the system efficiency on different batch sizes. The third experiment evaluates the proposed AL strategy autoselection procedure with an early-stopping algorithm.

AL strategy impact
Our ALaaS already provides many out-of-the-box AL strategies in AL Strategy Zoo for users. This experiment evaluates these strategies re-implemented by ALaaS from accuracy and efficiency views to provide more insights. All settings are the same as in the previous experiment.

Results & Insights. The evaluation model accuracy of different methods is shown in Figure4a.
Core-Set [43] achieves the highest accuracy with no surprise as it is designed for CNNs in Computer Vision (CV) tasks. Meanwhile, Diverse Mini-Batch (DBAL) [55] and Margin Confidence sampling (MC) [29] have the second and the third highest accuracy respectively, though they have proposed for a long time. This indicates that even in the deep learning (DL) era, conventional methods (e.g., calculating margins) still play a vital role and can cooperate with DL well.
The throughput comparison is shown in Figure4b. The least confidence sampling (LC) has the highest throughput while Core-Set selection achieves the lowest throughput. Given the accuracy 4aand the throughput 4b results, we can conclude that the accuracy improvement of Core-Set comes from its heavy design while conventional sampling methods like MC balances the trade-off between the accuracy and the efficiency well.

Batch size impact.
Settings. We evaluate the batch size (BS) impact on the public cloud (i.e., AWS with S3). We first store the CIFAR-10 dataset on an AWS S3 bucket. We then start ALaaS on a laptop to simulate an end-to-end AL process, including downloading data from other devices, pre-processing data, and selecting samples with an AL strategy. The other settings are the same as the first experiment.
Results & Insights. Our ALaaS can manage the whole process steadily and efficiently with different batch sizes, as shown in Figure4c. We can observe many interesting phenomena. First, BS = 1 and BS = 2 have very close throughput. Second, the increasing trend from BS = 4 to BS = 16 is dramatic. Third, after BS = 16, the increasing trend will stop. We attribute the reason that the transmission time accounts for a large proportion of the total processing time when the batch size is small. Therefore, the throughput improvement is marginal at the beginning. Then, as the batch computation time becomes the largest part of the total processing time, the improvement is dramatic. Finally, when the batch size reaches the computation capacity, this increasing diminishes.

PSHEA procedure evaluation.
Settings. We evaluate our AL agent with the PSHEA procedure on both CIFAR-10 and SVHN datasets. For both CIFAR-10 and SVHN experiments, we simulate an 8-round AL procedure.
Results & Insights. We first use the least confidence sampling strategy to evaluate the effectiveness of our accuracy prediction model mentioned in the Algorithm1. From Figure5a, we observe that the prediction model can foresee the accuracy very accurately, laying a solid foundation for later AL strategy auto-selection. Then, we run our AL agent with PSHEA. The AL agent will launch all 7 AL strategies as candidates in the first round and then eliminate strategies round by round. As shown in Figure5b, for two different datasets, our AL agent selects different AL strategies under different budges (i.e., the percentage of the labeled data in this experiment). This observation aligns with the conclusion from previous studies [18] and further shows the necessity of multi-round auto-selection, as no AL method can always out perform others under different budgets and datasets. Our method early stops many of the AL selection processes and helps to save running costs. INIT

Conclusion
This paper presents a new MLOps system, named ALaaS, for data-centric AI. ALaaS adopts the philosophy of Machine-Learning-as-Service and implements a server-client architecture, so users can use AL as a web service. Meanwhile, our system employs stage-level parallelism, data cache, and batching to improve AL running efficiency. Furthermore, it includes an AL agent with a predictivebased successive halving early-stop (PSHEA) procedure to select suitable AL strategies for users under different budgets and accuracy targets. Experiments show that our system has lower latency and higher throughput compared to all other baselines. Experiments also show that the auto-selection process can help users eliminate low-performing AL strategies earlier and save cost. We release our code at GitHub to facilitate the AL research.