EasyScale: Accuracy-consistent Elastic Training for Deep Learning

Distributed synchronized GPU training is commonly used for deep learning. The resource constraint of using fixed GPUs makes large-scale deep learning training jobs suffer, and also lowers the cluster utilization. However, incorporating resource elasticity often introduces non-determinism in model accuracy, which is mainly due to the lack of capability to isolate the model training procedure from hardware resources. We introduce EasyScale, an elastic framework that scales distributed training on heterogeneous GPUs while producing deterministic deep learning models. EasyScale follows the data-parallel training flow strictly, traces the accuracy-relevant factors carefully, utilizes the deep learning characteristics for context switching efficiently, thus achieving elastic accuracy-consistent model training. To saturate the computation capability of heterogeneous GPUs, EasyScale dynamically assigns workers based on our intra-job and inter-job scheduling policies, minimizing GPU idle time and maximizing aggregated job throughput accordingly. Deployed in an online serving cluster of CompanyA, EasyScale powers elastic deep learning training jobs to utilize free GPUs opportunistically, improving the overall cluster utilization by 62.1% without violating SLA.


Introduction
Deep learning (DL) is now playing a vital role to support a wide range of production pipelines, such as advertising for online shopping, computer vision for autonomous driving, natural language processing for searching, etc. Recognizing the great power of DL, large companies have built large-scale shared GPU clusters to expedite the intelligent transformation of the entire society. In order to process the growing sizes of industry datasets, the common practice today often adopts distributed deep learning training (DLT), where each worker typically processes training data in mini-batches and adopts synchronized stochastic gradient descent (Sync-SGD) to compute gradients for model update. For example, PyTorch [43] training typically allocates a GPU per worker and uses Distributed Data Parallel (DDP) to perform gradient synchronization in every mini-batch. However, a DLT job will not start until all resources become available simultaneously due to gang-scheduling [31,58]. Besides, the job is executed in a fixed degree of parallelism (DoP), therefore it can never scale out/in to more/fewer GPUs when cluster load changes. The fixed DoP prevents the DLT jobs from adapting to the resource elasticity that is common in shared GPU clusters [54,55].
Recently, a series of research works (e.g., TorchElastic [9], ElasticDL [4], VirtualFlow [42], KungFu [38]) have proposed elastic deep learning frameworks, which allow a DLT job to continue its training process with dynamic scaling resources (i.e., number of GPUs). Several cluster management approaches (e.g., Gandiva [57], Optimus [44], Pollux [47]) have also utilized the resource elasticity to maximize cluster utilization or allocate job resources for fast convergence. Despite the well-known benefit of elasticity, the proposed elastic deep learning frameworks have rarely been used in the industry so far. The fundamental obstacle to their adoption is the inconsistent model accuracy when training with different resources. The resource elasticity introduces non-determinism to both training procedure and model convergence. Converging to similar accuracy through tuning hyperparameters (e.g., learning rate or batch size) in specific benchmarks cannot convince users, since the expected computation flow is changed implicitly. The concerns of non-determinism in model accuracy remain unresolved when changing datasets or model structures, which makes deep learning practitioners hesitate in embracing resource elasticity (Section 2).
In this work, we present EasyScale, the first training framework that achieves consistent model accuracy over resource elasticity for both homogeneous and heterogeneous GPUs, thereby improving the overall cluster efficiency by utilizing the idle GPUs of inference production clusters at best effort for elastic model training. We regard the deep learning model training as scientific experiments throughout the design of EasyScale, treating determinism and reproducibility as the first-class goal. EasyScale explores the possibilities of producing bitwise-consistent models regardless of different numbers and types of GPUs allocated, by decoupling the distributed model training procedure from hardware resources. This is done by an abstraction called EasyScaleThread, which encapsulates all the stages from data loading, sampling, computation, to communication and keeps them exactly the same as executed in fixed GPUs. EasyScale utilizes the characteristics of deep learning for fast context-switching, addresses the potential non-determinism in training worker states, and performs tracing and checkpointing efficiently at resource reconfiguration.
To further improve the utilization of allocated GPUs, EasyScale introduces an intra-job policy to schedule training workers across heterogeneous GPUs in a load-balanced way, minimizing resource wastage. Moreover, as workloads have different capabilities of using heterogeneous GPUs, our cluster scheduler further optimizes inter-job resource allocations to maximize aggregated throughput.
We have implemented EasyScale by modifying a popular framework, PyTorch, to provide the elastic scaling capability without compromising model accuracy, and also implemented the proposed scheduling policies on top of our Kubernetes scheduler (Section 4). We evaluate EasyScale on a cluster with 64 heterogeneous GPUs to demonstrate the achievement of accuracy-consistent model training using micro-benchmarks on typical workloads, and show the advantage of resource elasticity through real workload trace. The trace experiment shows that EasyScale can generate the model accuracy same as using fixed GPUs for all DLT jobs. In addition, EasyScale improves the average Job Completion Time (JCT) by 13.2× and makespan by 2.8× thanks to the resource elasticity. We have deployed EasyScale in a production cluster equipped with 3,000+ heterogeneous GPUs for online model serving. The cluster result demonstrates that EasyScale improves the GPU allocation ratio by 17.1% and the average GPU utilization by 62.1%. The evaluation also shows that using EasyScale, the DLT jobs can automatically scale in seconds when co-located with online model serving, without violating Service Level Agreements (SLAs).
The key contributions of the paper are as follows.
• We investigate the non-deterministic behaviors in existing elastic deep learning frameworks and in heterogeneous environments, and source the bitwise differences scattered in the entire DLT software stack.
• We introduce EasyScale for elastic distributed model training, which achieves consistent accuracy deterministically. EasyScale utilizes EasyScaleThread to preserve the training behaviors over elasticity same as the Py-Torch DDP, and conducts context-switching efficiently with negligible overhead.
• We introduce new intra-job and inter-job scheduling policies for improving individual EasyScale job throughput and the aggregated cluster throughput thereby utilizing the heterogeneous GPUs elastically and efficiently.
• We fully deploy EasyScale in production clusters to colocate elastic training jobs with online model serving, significantly improving cluster utilization under the constraints of meeting the model serving SLA.

Motivation
We first briefly summarize the demands of embracing elasticity observed when operating large-scale shared GPU clusters. We then analyze the key drawbacks of using existing elastic deep learning frameworks to motivate the design of EasyScale.

Demands for Resource Elasticity
Adapting to resource elasticity benefits both individual DLT job performance and cluster utilization. Distributed training is widely adopted in production clusters for processing massive data and large models, however, in sub-optimal performance, especially in shared clusters. In addition to the common difficulties of large-scale DLT jobs in long gang-scheduling delays [31,32,55,58,64], they get preempted constantly because of resource revocation. Our 2-day statistic in a GPU production cluster of CompanyA shows that jobs requesting more than 8 GPUs contribute 61.7% of resource revocation failures among all jobs. As a comparison, the ratio of 1-GPU jobs is only 5.3%. This is caused by the inherent Sync-SGD training behavior, where terminating any one worker ends the whole DL training. Through resource elasticity, the dilemma of frequent failure in large DLT jobs can be avoided. A multi-GPU job can start the training with incremental numbers of GPUs, eliminate the mandatory waiting of gang scheduling, and survive from resource revocation by scaling in to fewer GPUs. Moreover, job elasticity also offers more opportunities for training jobs to utilize the idle resources of the online model serving cluster. The 2-day GPU allocation statistics of an online service cluster ( Figure 1) show that the difference of required GPUs between idle and peak hours is up to 2,000. Ideally, the idle GPUs can be shared by both model training and online serving to maximize GPU utilization, similar to the big-data workloads [54]. Exploiting such an opportunity further suggests that the scaling of elastic DLT jobs needs to be efficient to meet the SLA of online serving jobs.

Non-determinism over Elasticity
To address the aforementioned problems of poor job performance and low cluster utilization, a series of research works [4,9,42,47] have proposed elastic deep learning frameworks that enable resource scaling at runtime through optimized synchronization methods (e.g., gradient accumula-   TE-1GPU  TE-2GPU  TE-4GPU  TE- C 0 C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 Total  tion [42,47]), hyper-parameters tuning [38], batch size adjustment [9], etc., to eventually reach similar model quality (i.e., test accuracy) compared to the synchronized distributed training with fixed resources. However, the above-mentioned frameworks are rarely used in industry due to the following drawbacks.
Inconsistent Model Accuracy. The multiple runs on model training with elastic frameworks fail to generate consistent model accuracy using different amounts of resources. Figure 2 illustrates the validation accuracy of ResNet18 on CIFAR10, an elastic model trained with varying numbers of V100 GPUs. We keep all hyper-parameters and random seed same as default but only with different allocated GPUs. TorchElastic (TE) [9] is configured with linear scaling rule for adjusting learning rates [25], and Pollux [47] can automatically decide the learning rate and batch size accordingly. Therefore, resource elasticity comes with different training behaviors compared to distributed model training on fixed GPUs. The result shows Pollux introduces relatively more minor variance in producing model quality, however, the difference is still non-ignorable, up to 5.8% at epoch 10.
We further continue the training to convergence at 100 epochs, and report the overall and per-class (10 classes in total) accuracies in Figure 3. The overall accuracy variances are still notable, 0.6% and 2.8% for TorchElastic and Pollux, respectively. The per-class accuracy variances are even larger, reaching up to 7.4% and 17.3%, and on average 3.9% and 7.4%, which indicates that the elastically trained models are different in bias compared to the ones using fixed GPUs. The per-class accuracy variances can be problematic and may destroy the model usability, in scenarios like the "pedestrian" detection on self-driving cars [24]. What's more, other metrics are still not disclosed, and the upper-bound of the model quality gap remains unknown, making DL researchers hesitate in using elastic training.
Difficult to Understand Hyper-parameter Effect. DL model is becoming a critical step in the production pipeline, not only its model accuracy but also hyper-parameters reasoning plays an important role, covering profit-critical models in the recommendation and advertising system [62] and even life-critical models in medical diagnostics and autonomous driving [24]. Figure 4 shows the experiment of ResNet50 on CIFAR10 by comparing the DDP training on fixed 4 GPUs to the elastic model training on 1/2/4 GPUs using Pollux [47]. The configurations are consistent except a hyper-parameter of the learning rate scheduler, gamma, which decides the learning rate reduction ratio after certain steps of model training (20 epochs in this experiment). The DDP experiment runs the 4-GPU training three times with the same setting, but the gamma value is 0.1, 0.3, and 0.5, respectively. Pollux also runs the same experiment, however, since Pollux is capable of using elastic numbers of GPUs, we configure it with 1 GPU for gamma as 0.1, 2 GPUs for gamma as 0.3, and 4 GPUs for gamma as 0.5. As illustrated in Figure 4, using DDP, it is clear to reason how the hyper-parameter gamma affects the model training procedure. The training loss keeps consistent in three runs for the first 20 epochs, and after that, the smaller gamma leads to the smaller train loss. However, given the different GPU numbers in Pollux, researchers can hardly reason about how gamma affects the training loss curve.
To summarize, we believe that DL models (i.e., the output of a DL training job) are now the result of a combination of algorithms, frameworks, and computational resources, which is the fundamental reason limiting the use of elastic resources for DLT jobs. Specially, the existing elastic DL frameworks [3,4,9,38,47] lack the capability to decouple the resources (e.g., GPU number, GPU type) from the model hyper-parameters, and thus fail to provide consistent model accuracy with elastic training. To resolve users' concern about model accuracy, enhance the performance of distributed DLT jobs, and improve the cluster utilization, we need to support elastic training while maintaining consistent model accuracy.

Overview
Data analysis systems, such as batch processing systems [19,30,59] and streaming processing systems [17,35], guarantee the consistent output regardless of the allocated resources. Analyzing training data samples to build neural network models, deep learning framework, known as a data analysis system for artificial intelligence, should also produce consistent model after iterative training. Previous elastic training approaches  fail to maintain the model accuracy because the behavior of training workers or even application semantic is changed upon different number of GPUs. Instead, we take a conservative approach where elastic training should generate model parameters bitwise identical compared to the data parallel training (i.e., DDP) over fixed number of GPUs. Figure 5 shows an example of scaling distributed 4-GPU model training from four GPUs to two GPUs. Four training workers can execute in-parallel on two ideal GPUs with unlimited memory and computation capability (Figure 5b), preserving training behavior and states to achieve the consistent training model. However, by multiplexing multiple DLT workers on a GPU, the concurrent memory consumption increases in the forward pass of multiple workers, which easily results in either out-of-memory failures [34] or significant overhead in memory swapping [12,58]. Besides, the aggregated GPU memory consumption of CUDA contexts of training workers are considerable. For instance, executing 16 workers on a 16GB V100 GPU costs 12GB GPU memory for CUDA context (around 750MB per CUDA context), leaving only 4GB for model training.
The key challenge in achieving elastic accuracy-consistent model training is to find a practical way of sharing a GPU to multiple training workers efficiently. Next, we introduce EasyScaleThread to utilize deep learning characteristics for lightweight and efficient context switching (Section 3.2). The sources of non-determinism during model training are identified to minimize the states affecting the final model accuracy (Section 3.3). We balance the training load allocation among heterogeneous GPUs and propose a scheduling policy for coordination among elastic jobs (Section 3.4).

EasyScaleThread
To process multiple training workers under resource constraints, we introduce EasyScaleThread as a key abstraction in our system. EasyScaleThread is inspired by the classical threading concept in operating systems and the "single program multiple data" (SPMD) model adopted commonly in deep learning [14], to capture the deep learning training procedure and decouple it from hardware resources. As shown in Figure 5c, each GPU is launched with a EasyScale PyTorch worker. The execution of original training work-ers are treated as the execution of EasyScaleThreads (EST), which can be allocated to a PyTorch worker process dynamically. In a worker, multiple EasyScaleThreads take turns to occupy the GPU for computation (i.e., model forwardbackward). Using a white-box approach, EasyScale hooks the key steps of model training, such as data loading, model backward (i.e., loss.backward in PyTorch), and model update (i.e., optimizer.step in PyTorch) through user annotations, therefore to perform accuracy-consistent context switch at mini-batch boundaries. The user-defined model training semantics, including model structure, data augmentation, batch size, optimizations, etc., are retained as usual. As for programming, users consider the number of total logical training workers to decide the hyperparameters (e.g., global batch size and learning rate), which is same as their experiences using fixed GPUs but automatically benefits from the elasticity provided by EasyScale.
Execution. For each mini-batch, training samples are processed by conducting forward-backward computation over the current model to generate gradients as output. After gradient synchronization, the model is updated. Figure 6 illustrates the situation where the training of 4 EasyScaleThreads scales from 2-GPU to 1-GPU. In EasyScale, a training worker has one GPU executor (one CUDA context) to share among EasyScaleThreads, therefore, it does not consume extra GPU memory. For each mini-batch, the input data is split across all EasyScaleThreads; at a time, EasyScale runtime schedules an EasyScaleThread, actively executing on a GPU, while freezing the others. The mini-batch is completed after all EasyScaleThreads finish.
When switching between two EasyScaleThreads, the training state of an EasyScaleThread needs to be saved to outside of the GPU, therefore to ensure enough GPU memory for the next EasyScaleThread, which can be costly. The key to switching EasyScaleThreads efficiently on a GPU for every minibatch is to reduce the states required for context switching. EasyScale leverages the unique characteristics of deep learning jobs to achieve that. Most of the deep learning components are shared and reused among different EasyScaleThreads. And EasyScale chooses to switch EasyScaleThreads after the finish of forward-backward computation, minimizing GPU-CPU memory copy.
We minimize the size of states by i) locating the source of non-determinism that affects the final accuracy and minimizing the necessary states (including the states of dataloader workers) to record, and ii) leveraging the data-parallel behavior of DL to minimize the working set for data swapping. The working set in GPU memory of an EasyScaleThread can be classified into temporal tensors and activations, model parameters and optimization states, and gradients [42], which we treat differently. First, for temporal tensors and activations, they are created in the forward step and destroyed in backward step after completing the gradients generation [57,58].
Therefore, they will automatically free up as expected at the end of mini-batches. Second, regarding the model parameters and optimizer states, each data-parallel worker will keep a replica during the training, and they are updated at the end of a mini-batch. Therefore, in EasyScale, they can be reused when switching EasyScaleThreads. Finally, the gradients are calculated based on the different data inputs across EasyScaleThreads, and thus cannot be reused. However, the gradients are usually small [58] and will be used only in distributed gradient synchronization at the end of the minibatch. Thus, in EasyScale, we migrate the gradients to host DRAM when context switch and overlap it with the computation of next EasyScaleThread. In this way, we execute the EasyScaleThreads alternately until all computations are finished. After that, distributed synchronization is triggered, the model update is conducted once to complete the mini-batch.
Reconfiguration. When resource reconfiguration is triggered, EasyScale adopts the on-demand checkpointing to persist minimal and necessary states, as shown in Figure 6. The checkpoint contains the contexts of all EasyScaleThreads, the extra states (including the training progress and other states for achieving accuracy-consistency (Section 3.3)), and deep learning parameters (e.g., model, optimizer, and learning rate scheduler). Different from the EasyScaleThread contexts, only one replica of the extra states and parameters is required as they are shared as the same for all EasyScaleThreads at the end of mini-batches. Note that, after the rebooting of model training, the runtime of EasyScale for each GPU loads a copy of extra states and model parameters, as well as the corresponding contexts of re-distributed EasyScaleThreads.
Optimization. To overlap data loading and GPU training, data workers are executed in standalone processors (i.e., loader worker processes in PyTorch), asynchronously loading training samples and performing data augmentations (e.g., crop or rotate images) to build training batches. The number of data workers is usually configured by users for each training worker to ensure the GPU training is not being blocked by data preprocessing. Naively paralleling the EasyScaleThreads requires a significant number of data worker processors (e.g., it's 128 processors when multiplexing 16 EasyScaleThreads on a GPU if 8 data workers per training worker), resulting in heavy system load and CPU overhead. In EasyScale, we optimize to share data workers among all EasyScaleThreads, since at each time only an EasyScaleThread is training on the GPU. The data consuming rate is similar to what it is in dedicated GPUs despite the sharing of multiple EasyScaleThreads.
To enable data worker sharing, EasyScale employs a distributed data sampler that jointly considers the global indices of EasyScaleThreads and the time-slicing pattern, to generate data indices in a queue. The data indices are then processed by data workers orderly. Figure 7 shows the case of sharing three data workers to two EasyScaleThreads, where the total EasyScaleThread number is four (i.e., training with 2-GPU in Figure 6). The training batches of EST 0 and EST 1 are b0 and b1 for mini-batch 0 and b4 and b5 for mini-batch 1. The state of data worker j to process data indices for EasyScaleThread i in the dedicated GPU is denoted as Ri-j, shown in Figure 7. To balance the load, data workers in EasyScale take turns to get the corresponding state (i.e., Ri-j) of given data indices from a queuing buffer for preprocessing, and the state is committed back to the queuing buffer once finished. Note that, due to the asynchronous execution of data workers, the progress of data workers (i.e., mini-batch index) is usually afore the training progress. To track and keep the consistent state for elasticity, a queuing buffer is introduced to record the necessary states (e.g., random generator state) for mini-batches that are not consumed. The worker states are dequeued from queuing buffer according to the training progress, which is then treated as part of the extra state in checkpointing.

Sourcing and Solving Non-determinism
Recall that the primary goal of EasyScale is accuracyconsistency of elastic training. Although EasyScaleThreads can accomplish the execution of elastic training, naïvely using them can still lead to inconsistent results compared to multiple processes using DDP. Besides, supporting heterogeneous GPUs can also introduce non-trivial non-deterministic behaviors, which have never been studied yet, to the best of our knowledge. Therefore, we apply a top-down method to compare EasyScale to DDP using the same number of workers, using changing configurations, to reveal the factors that affect the training behavior. Surprisingly, we find that the root causes of the non-determinism are scattered in almost the entire software stack of the training pipeline, from training frameworks, to communication, and to GPU kernels.
First, at the training framework level, frameworks have some states that need to be kept consistent throughout the training for determinism. Although deep learning training organizes operators (e.g., convolution, batch normalization) in a DAG graph [10], some operators implicitly rely on some states other than the outputs of their predecessors. For instance, Dropout depends on the random number generator (RNG) states in GPU; BatchNorm tracks its running status by considering the rank of the worker; data loaders and transformers of data augmentation rely on random states of Python, NumPy, and PyTorch, etc. Second, at the communication level, gradient synchronization via all-reduce is non-deterministic under resource elasticity. During synchronization, gradients are gathered into communication buckets to optimize communication performance. The gradient-to-bucket mapping is determined by the static reversed topological order of DAG graph initially. It is reconstructed at the end of the first mini-batch according to the order of gradient tensors received. However, worker restarts during elastic training will rebuild the communication channels, which may affect the gradient aggregation order of the first resuming mini-batch. This will eventually introduce non-determinism because of the ring allreduce implementation [5,6].
Finally, at the GPU kernel level, different kernels selected for the same operator can also lead to subtle differences in the results. There are two causes of different kernel selections. The first is that some profiling-based optimizations in frameworks [8], compilers [53], or vendor libraries [1] apply different kernel implementations across mini-batches to collect performance statistics to find a best-fit. The other is that kernel implementations can be hardware-relevant. For example, some kernel implementations are based on the number of stream processor units, hardware-specific low-bit components, etc., and hence cannot be applied to all types of GPUs.
Levels of determinism and treatments. EasyScale defines different levels of determinism of elastic training to provide clear consistency guarantees to users, and devises corresponding treatments to realize them.
D0: Fixed-DoP determinism -multiple training runs with fixed GPU resources should produce identical models. Achieving D0 requires consistent behaviors of training frameworks and their selected kernels. For the framework, we fix the random seeds of RNGs at the beginning of the training and record the RNG states in the data-loading worker states and those of EasyScaleThreads in the context, so that the states are kept consistent automatically by EasyScaleThreads. We also disable the best-fit algorithm selection and select the deterministic algorithms (e.g., without atomic instructions). This solution is officially recommended by DL frameworks [6,7], such as PyTorch and TensorFlow.
D1: Elasticity determinism -multiple training runs with changing numbers of homogeneous GPUs with checkpointrestarts should produce identical models. Beyond D0, D1 requires addressing the communication-level non-determinism. To this end, we assign a fixed virtual communication rank to each EasyScaleThread. We also record the indices forming the gradient buckets into the checkpoint. After restarting, buckets are reconstructed with recorded indices first before the training. The later communication channel reconstruction is disabled.
D2: Heterogeneity determinism -Multiple training runs with different types of GPUs should produce identical models. To achieve D2, we develop hardware-agnostic GPU kernels. Specifically, 1) we modify kernel implementations (e.g., reduce, dropout in PyTorch) limiting the number of SMs and threads, and 2) we force to select the same low-level implementation (e.g., convolution in cuDNN, and gemm, gemv in cuBLAS) by passing algo_id to the high-level calls.
Determining level of determinism. In EasyScale, D0 and D1 are enabled by default because our implementation has negligible overhead to achieve them (refer to Section 5.1.2). Achieving D2 can have a higher overhead for certain types of models (e.g., CV models) because they cannot use some vendor-optimized kernels (e.g., convolutions) for some GPU types. EasyScale can transparently analyze a model (by scanning the PyTorch nn.Modules) and identify whether it relies on operators that demand hardware-specific kernel optimizations. If not, we enable D2 and allow it to use heterogeneous GPUs, and restrict it to use homogeneous GPUs otherwise.

Scheduling Policy
In this section, we describe the scheduler design, which leverages the elasticity of EasyScale to assign EasyScaleThreads onto available GPUs and makes the best use of both homogeneous and heterogeneous GPUs in a cluster. Users submit an EasyScale job by specifying a maxP, the number of maximal workers to launch, which is also the number of EasyScaleThreads during job execution. The user can also specify a minP (>=0) that denotes the required guaranteed GPUs. A setting of minP == maxP means that the job falls back to using the same fixed DoP as DDP.
With the abstraction of EasyScaleThreads (EST) (Section 3.2) introduced above, we have decoupled the DL training and underlying GPU resources, thus the DDP-compatible training jobs can continually run over an elastic number of homogeneous GPUs. Together with the determinism treatments (Section 3.3), we have also achieved accuracy consistency under elastic training, even on heterogeneous GPUs.
The key challenge of scheduling ESTs on heterogeneous GPUs lies in computing capability heterogeneity and GPU memory heterogeneity. The method of scaling the batch size for each kind of GPU individually, which is adopted by Pollux [47] and VirtualFlow [42], is completely unacceptable for EasyScale, because it changes the inherent training hyperparameters and thus breaks accuracy consistency, as illustrated in Section 2.2. During the EST execution, each EST consumes a fixed amount of computing capability, where the micro-architecture characteristics (i.e., SM number and cache size) of GPUs determine the theoretical capability. The GPU with higher computing capability can execute more ESTs in a given time. And each EST has the fix-sized peak GPU memory usage. Moreover, since the memory of ESTs in the same GPU executor is totally reused/shared, the memory usage of the executor can represent that of ESTs. For brevity, we define the fixed computing capability and memory usage as the computation unit (CU) and memory unit (MU), respectively. However, without sophisticated analytical planning, mapping the specified CUs (maxP in total) and MUs (maxP at most) of an EasyScale job to heterogeneous GPUs (e.g., 1 V100-32G, 1 P100-16G, and 2 T4-16G) could cause significant performance waste due to severe load imbalance.

Heterogeneity-aware EST Planning
Planning CUs. When the allocated GPUs are homogeneous and the quantity is a factor of maxP, we can assign CUs evenly on those GPUs, because all CUs have the same computing time. With heterogeneous GPUs, the even-distribution strategy becomes inefficient. The different computing capabilities of heterogeneous GPUs result in the different forwardbackward computation times of CUs. Obviously, their actual capabilities cannot always be represented as the integer multiples of CU (e.g., ResNet50 is 2.45× faster on V100 than on T4), which becomes quite unachievable if with more types of GPUs. Meanwhile, Sync-SGD enforces a barrier on all GPUs to wait for the slowest one, which results in the waste of long idle GPU cycles, as shown in Figure 8a. Considering that, we need to allocate the CUs according to the GPU's computing capability for balance and to eliminate the idle cycles at most, as illustrated in Figure 8b.
We propose a new metric called waste, which is the wasted computing capability due to the mismatching between integer multiples of CUs and actual continuous capabilities of GPUs, or load imbalance, in other words. The waste is mainly twofold: load imbalance across heterogeneous GPUs, caused by inaccurate approximating of continuous actual computing capabilities with integer multiples of CUs, and load imbalance across homogeneous GPUs, caused by insufficient CU allocation to fully utilize all GPUs, since the total quantity of CUs is bounded by maxP. Therefore, we build an analytical model to quantify waste. The number of available GPUs is denoted as N i , where the subscript i represents the GPU type. The workload-related computing capability C i is estimated as number of mini-batches per second. The maximum number  of CUs assigned to GPU type i is denoted as A i . To ensure all EasyScaleThreads to be executed, the total maximum CU number over heterogeneous GPUs (CU_capacity) should be greater or equal to maxP (Equation 1a). The overload factor f overload represents the maximum overload of requested heterogeneous GPUs, where the overload is defined as CUs per computation capability (Equation 1b). If one GPU type undertakes too many CUs, it becomes the performance bottleneck and slows down other GPUs due to Sync-SGD. Therefore, the waste is formulated as the gap between C i and A i scaled by f overload , and the over-provisioned CU_capacity scaled by f overload (Equation 1c). The normalized waste percentage (Equation 1d) is derived to further distinguish the efficiency of current CU allocation under current N i , C i , and A i . The estimated performance is also derived (Equation 1e).
Planning MUs. There is also heterogeneity in memory capacity among heterogeneous GPUs. Take the data center GPUs for example, V100 has 32GB or 16GB, P100 and T4 have 16GB. Allocating MUs to a single executor of each GPU can minimize the overall memory footprint, since EasyScaleThreads introduce negligible memory overhead and their MUs are fully reused. Consequently, all GPUs show the same peak memory usage equal to MU, leading to idle memory for GPUs with larger memory capacity.
We propose a multiple executor design, which allows allocating more than one executor on a GPU so that multiple ESTs can be executed simultaneously. Therefore, GPUs with larger memory can tradeoff the executor number and the EST number per executor, while keeping # of executor × # of EST constant (e.g., with two allocated ESTs, there are two options: a) <1 executor × 2 ESTs>, and b) <2 executors × 1 EST>). It broadens the efficient scenarios that running more executors does not exceed the GPU resources (SM cores, memory) and still contributes to performance improvement even considering interference. The training of recommendation models (e.g., Wide&Deep) usually shows under-utilization of computing capability on GPU, which is usually less than 50% [63]. In this scenario, allocating multiple executors can utilize the leftover computing capability, and executors do not adversely impact each other, thereby improving the aggregated throughput.
We also adjust Equation 1 to modeling waste of multiple executors. The workload-related computing capability C i is replaced by MC i = m × C i × I i , representing the overall capability of m executors accounting for interference I i . The CU number assigned to GPU A i is replaced by MA i = m × A i , representing the total CU number of m executors.

EasyScale Scheduler
As shown in Figure 9, EasyScale adopts a hierarchical scheduling architecture. Each job contains an intra-job scheduler, named AIMaster, that is responsible for a) allocating the EasyScaleThreads among homogeneous and heterogeneous GPUs with minimized waste to maximize the resource utilization, and b) proposing the desired resources to scale out through estimated potential speedup. Besides, a cluster scheduler acts in a global mode to coordinate resources among jobs.  Figure 9: Scheduling workflow of EasyScale jobs.
Intra-job scheduler. The fundamental responsibility of intra-job scheduler (i.e., AIMaster in Figure 9) is generating the EST allocating configurations under given GPUs. First, under current available GPUs, it selects the top-1 configuration whose estimated throughput is the highest, and allocates ESTs accordingly. Second, it tries to scale out with one incremental GPU, thereby generating new configurations and selecting top-K configurations as proposals submitted to the cluster scheduler. Specifically, each configuration is composed of <nums, executors, threads, waste, per f >. nums, executors, and threads are arrays with the same length of GPU types, representing the GPU numbers, executor number, and EST number per executor, respectively. waste and per f represent this configuration's estimated waste and performance by the analytical model (Section 3.4.1). These configurations should satisfy the constraints of total cluster resources, minP, maxP, and the threshold (30% in practice) of normalized waste. Otherwise, they are ruled out.
Workloads show differences in throughput using heterogeneous GPUs (e.g., ResNet50 is 2.45× faster on V100 compared to that on T4, while Bert is 1.55×), which is hardly predictable without real execution. Therefore, the AIMaster module uses the runtime execution statistics of jobs to be aware of workload differences and determine the workloadrelated computing capability C i for each GPU type i. When the job is executed for the first time without profiling information, we initialize C i based on the historical data [55]. Given the profiled computing capability C i for each GPU type i, we calculate their integer approximations (e.g., ceil(t × C i ), floor(t ×C i )) assuming each capability undertakes k EST, and forms their combinations. Then we traverse the combinations and find available configurations. For configurations with the same <nums, executors, threads>, the one with minimized waste is selected, and others are filtered out.
The estimation of waste might sometimes be incorrect, which could probably result in worse training performance. Once the performance slowdown is observed after reconfiguration, we fall back to using previous resources and releasing the newly allocated ones.
Inter-job cluster scheduler. It responds to AIMaster proposals by considering resource availability and proposal priorities, as shown in Figure 9. In order to improve the overall cluster utilization and aggregated job throughput, it adopts a heuristic algorithm that tends to accept the proposals with a higher speedup per GPU, as shown in Algorithm 1. The inter-job scheduler sorts the proposals by the reported average speedup ratio in descending order. It then loops over the suggestions to start accepting the highest one. If multiple proposals introduce the same average GPU speedup, our scheduler prioritizes the one with more GPUs.
Algorithm 1 Inter-job cluster scheduler. if remaining satisfies p then 5: Approve(p) 6: availables -= # of GPU in p 7: proposals.pop() 8: end if 9: end while The cluster scheduler allows elastic jobs to make the best use of the free resources, which usually belong to others but are temporarily idle. However, preemption might be triggered if those GPUs need to be returned. In this situation, the cluster scheduler will try to allocate the same GPUs as preempted ones to elastic jobs. When timeout, the EasyScale job fallbacks to utilizing the available GPUs it currently owns.

Implementation
DLT jobs are running in Docker containers with EasyScale implementation. A prototype custom cluster scheduler is implemented on Kubernetes [16] for evaluation. EasyScale is fully implemented in our internal GPU cluster scheduler, which is an optimized version of Kubernetes scheduler, to serve the daily production GPU jobs. So far, we have deployed EasyScale in two of our internal GPU production clusters, which used to serve DL development (i.e., Jupyter Notebook) and model inference especially. One of the deployed clusters consists of more than 10K GPUs.
The implementation of EasyScale in DL framework is compatible with PyTorch 1.8 LTS [43]. It requires around 1,200 lines of code in Python and 2,000 lines of code in C++ modifications in PyTorch and an add-on library implemented based on PyTorch. The C++ implementation for the PyTorch framework includes an elasticity-capable distributed data-parallel communication library, ElasticDDP, which can support communication among multiple EasyScaleThread to all-reduce the gradients and build communication buckets consistently across job restarting when resource elasticity is triggered. The execution flow control and context switch are implemented in a Python module as an add-on for PyTorch.
To decide the load-balance allocation and control the job to scale-out with more GPUs, we implement AIMaster, which takes around 2,000 lines of Python code. It has three components. First, we collect the performance profiling reported by EasyScale runtime through an rpc library. Second, we raise resource proposals and monitor the status thereby being aware of the resource allocation timeout through a Kubernetes Python informer. Third, a policy controller is implemented to calculate and submit incremental resource requests to the cluster scheduler. In order to support the continual job training when resource elasticity, we take on-demand checkpoint [57] to record the user-defined model, epoch and minibatch states, and essential context switch states as mentioned in Section 3.3.
Alongside the implementation of EasyScale, we also implement a semi-automatic profiling tool to perform bitwise comparison among tensors, therefore to locate the inconsistent results of operators, identifying the sources of non-determinism in resource elasticity. We plan to open-source the EasyScale framework as well as our profiling tool to share the knowledge we learn when implementing EasyScale for accuracyconsistent elastic DL training.

Evaluation
In this section, we first show micro-benchmark results to demonstrate the accuracy-consistency and efficiency of EasyScale. We then evaluate the benefits of EasyScale in a small cluster with 32 V100 GPUs, 16 P100 GPUs, and 16 T4 GPUs to compare the scheduling policy using real workloads. Finally, we present the evaluation results on one of our production clusters equipped with thousands of GPUs.
The micro-benchmark experiments and trace experiments are conducted on a cloud GPU cluster with 16 servers, specially, 4 servers each with 8 V100 GPUs, 8 servers each with 2 P100 GPUs, and 4 servers each with 4 T4 GPUs. Each server runs CentOS 7.8, and their GPUs are powered by NVIDIA driver 450.102.04, CUDA 10.1, and CUDNN 7. The cloud GPU cluster is managed by Kubernetes; jobs are submitted through KubeFlow, and are executed in Docker containers. As for the workloads, eight state-of-the-art deep learning models are selected from Github, together with open datasets, as summarized in Table 1. The eight models are implemented based on PyTorch 1.8 LTS, and they are ported using EasyScaleThread with a few lines of code changing.

Micro-benchmark
In this section, we evaluate EasyScale with a series of microbenchmarks to understand its capabilities. We first demonstrate accuracy-consistency of EasyScale using typical DL workloads, and we measure and break down the overhead of enforcing accuracy-consistency. We then demonstrate that EasyScale is efficient in multiplexing multiple workers on a GPU by comparing it to worker packing [57], and analyze the overhead of the proposed EasyScaleThread abstraction.

Ensuring accuracy-consistency
In this section, we demonstrate the accuracy-consistency of EasyScale, which guarantees to produce bitwise-identical DL models under an elastic number of heterogeneous GPUs. We use EasyScale to train the DL models listed in Table 1 in three different stages with different GPU configurations: stage 0 with 4 V100 GPUs, stage 1 with 2 V100 GPUs, and stage 2 with 1 V100 and 2 P100 GPUs. Changing from from stage 0 to 1 represents the resource elasticity, and changing from from stage 1 to 2 represents the resource heterogeneity. In each stage, the models are trained for 100 mini-batches. We use PyTorch DDP to train the models with fixed GPUs ( i.e., 4 V100 GPUs). In the experiments, both EasyScale and DDP have 4 workers (4 EasyScaleThreads for EasyScale) in total. We test EasyScale with four determinism configurations as different combinations of behaviors defined in Section 3.3: two homogeneous determinism configurations (D0 and D1) and two heterogeneous ones (D0+D2 and D1+D2). DDP has two corresponding configurations, DDP-homo with fixed random seeds and deterministic algorithms to ensure the reproducibility, and DDP-heter with heterogeneous deterministic kernels (belong to D2) selected additionally. We compare the train loss curves of the last worker of EasyScale with DDP. Figure 10 shows the train loss differences of ResNet50 and VGG19 under (a) homogeneous configurations and (b) heterogeneous configurations. The train loss of EasyScale-D1 is identical to that of DDP-homo in stage 0 and stage 1, and the train loss of EasyScale-D1+D2 is identical to that of DDP-heter in all stages, showing the ability of EasyScale to preserve consistent accuracy.
Comparing the results of D0 vs. D1 (also D0+D2 vs. D1+D2) highlights the elasticity determinism of EasyScale. We observe that both EasyScale-D0 and EasyScale-D0+D2 start suffering from loss differences since stage 1, due to checkpointing and restart. D0 ignores the states of gradient synchronization in the checkpoint, and thus these states are lost after restarting. In comparison, D1 records these states in the checkpoint and achieves accuracy-consistency.
Further, comparing D1 vs. D1+D2 shows the heterogeneity determinism. Specifically, EasyScale-D1 starts having loss differences from stage 2 because different low-level kernel implementations are automatically selected on heterogeneous GPUs. But when we turn on the D2 option to fix the kernel selection for EasyScale and DDP, there is no loss difference in stage 2. The results of the other 6 models are similar and are hence omitted due to space limit. In summary, EasyScale with D1 and D2 options enabled, can ensure the accuracyconsistency with DDP, as long as they train for the same number of mini-batches.

Overhead of ensuring accuracy-consistency
In this section, we examine the overhead of ensuring accuracyconsistency by measuring the runtime of eight models with i) EasyScale with D1 option enabled and ii) EasyScale with D1 and D2 options enabled. We adopt the original version of PyTorch as the baseline. The experiment is conducted on V100, P100, and T4 GPUs. We run the experiment 10 times each to collect the runtime. As illustrated in Figure 11, the runtime is normalized to the baseline of each type of GPU.
We observe that the models can be classified into two categories in terms of the overheads. As for models such as NeuMF, Bert, Electra, SwinTransformer, introducing determinism brings less than 1% cost even for different GPUs, which means they can conduct model training elastically to achieve the bitwise results using different number of heterogeneous GPUs. For ShuffleNetV2, ResNet50, VGG19, YOLOv3, producing consistent results with different number of homogeneous GPUs can achieve similar performance compared to that without enforcing deterministic results. However, trying to reproduce the results using heterogeneous GPUs will introduce considerable performance cost (i.e., 236% on average). This is because EasyScale turns off vendor-optimized convolution kernels for these models for determinism. EasyScale can automatically identify jobs not relying on such kernels and lets them enjoy resource elasticity with heterogeneous GPUs, and use homogeneous GPUs instead for other jobs.

EasyScaleThread vs. Worker Packing
Working packing [57], i.e., multiplexing the same GPU across multiple workers, is another potential way to achieve accuracyconsistency with resource elasticity. We use two typical models, ResNet50 and ShuffleNetV2 on a V100 to compare EasyScale against worker packing. The batch size of ResNet50 model is configured to 32 as it is usually the com-  Figure 12: The peak GPU memory usage (curves) and throughput (bars) of EasyScale and worker packing when executing multiple EasyScaleThreads/workers on a V100. mon one in benchmarks, and the batch size of ShuffleNetV2 is set to 512 to saturate the 32GB V100 memory using one worker, which is usually the typical approach for DL researchers to utilize a GPU's capacity. EasyScale is configured as EasyScale-D1, and worker packing is implemented with DDP-homo to ensure the reproducibility.
In this experiment, we run worker packing and EasyScale on two models using different worker numbers 10 times and measure the training throughput (by dividing the batch size by average mini-batch time) and the peak GPU memory usage, as shown in Figure 12. All throughput numbers are normalized to the value of one worker using worker packing. As expected, when running only 1 worker, EasyScale and worker packing have almost the same memory usage and throughput. With the worker number increases, the GPU memory usage of EasyScale remains constant for both ResNet50 and ShuffleNetV2. As for worker packing, running multiple workers for ResNet50 increases the GPU memory usage gradually. It comes to out-of-memory (OOM) after 8 workers for ResNet50 and 2 workers for ShuffleNetV2. This is because EasyScaleThreads are executed in a time-slicing method, but packed workers are running simultaneously. EasyScale carefully reuses the deep learning components, especially the model parameters and optimizer states, and minimize the EasyScaleThread context, therefore, the GPU memory usage remains almost the same. Regarding GPU throughput, EasyScale generates an almost constant value regardless of the worker number, and worker packing grows at the beginning and reaches 1.11× compared to EasyScale. This throughput advantage of worker packing comes from higher GPU utilization due to concurrent execution of multiple kernels, but also at the price of higher memory usage as shown above. It becomes constant later because it quickly reaches the upper bound of the computation capacity of the GPU.

Overhead of EasyScaleThread
To demonstrate the efficiency of EasyScaleThread, we evaluate the overhead of its context switch and gradient syncing, using VGG19, YOLOv3, Bert, and SwinTransformer on a server equipped with 8 V100 GPUs. In this experiment, EasyScale uses only 1 GPU, and DDP uses 8 GPUs.
To measure the overhead of context switch for EasyScaleThread, we run different workloads using one EasyScaleThread per GPU, with and without context switch for EasyScale. Note that, with context switch disabled, EasyScale is not able to produce the accuracy-consistent result same as DDP. As shown in Figure 13a, the overheads of context switch for EasyScale in most cases are almost negligible, the maximum being 1% for Bert. This is because EasyScale carefully sources the non-determinism and records only the determinism-critical states rather than tensors or operators, as described in Section 3.3.
We further measure the gradient copy and synchronization overhead that EasyScale might introduce by comparing EasyScale and DDP. In this experiment, DDP is configured to run on 8 V100 GPUs, while EasyScale executes 8 EasyScaleThreads on one V100 GPU. Each configuration runs for 50 mini-batches. We skip the first 10 mini-batches for warm-up and record the average mini-batch time for both DDP and EasyScale. Besides, EasyScale further records the time of executing the first 7 EasyScaleThreads and divide it by 7 for plotting (i.e., EST 0-6 in Figure 13b). For the first 7 EasyScaleThreads, they only perform computation and copy the generated gradients through D2H operations. For the last EasyScaleThread (EST 7), the gradients require synchronization, which is similar to PyTorch DDP. We normalize the recorded times by that of DDP for each workload and the results are shown in Figure 13b.
It is somewhat surprising that EasyScale is always faster or compatible in performance, for both the first 7 EasyScaleThreads and the last one. For EST 0-6, this is due to the overlapping of gradient copy and the backward pass as well as the forward pass of next EasyScaleThread. The time of the last EasyScaleThread is also shorter than DDP mainly because in EasyScale, when the last EasyScaleThread is doing gradient synchronization, the other replicas of gradients are already ready there. However, in DDP, it is hard to ensure the gradients are simultaneously produced among all workers, which could induce potential delays. Therefore, the gradient copy and synchronization of EasyScale is efficient compared to PyTorch DDP. Besides, if EasyScale uses the same number of GPUs as DDP, it shows comparable performance, because only one EasyScaleThread resides on each GPU and the gradient copy is not needed.   What's more, we further measure the advantage of data worker sharing optimization among the above models with 8 EasyScaleThreads. With data worker sharing enabled, the training time of the first mini-batch has decreased to 32.9% on average. Because this optimization can reduce the number of required data workers distinctly (e.g., reduced from 32 to 4), thus lowering the launching time at elasticity.

Trace Experiment
Workloads. The training jobs are configured as the DL models listed in Table 1. The job runtime distribution of the trace is configured according to Microsoft [57], and the job arrival time is down-sampled from our production GPU training jobs. All jobs are considered to be submitted to the same tenant with the same priority.
Settings. The experiment compares EasyScale to Apache YARN's capacity scheduler (YARN-CS), a production GPU cluster scheduler used in Microsoft Philly [31], which enforces production jobs to be processed in FIFO mode to ensure inter-job fairness. All jobs in the experiment are using gang-schedule for allocating GPU resources and minP is set as 0 for EasyScale. YARN-CS fully respects the requirement of GPUs of DLT jobs and allocates the number of GPUs in the same type (e.g., all V100 GPUs) for a job. We configure EasyScale in two settings, EasyScale homo and EasyScale heter , both following the FIFO mode without changing the job order. In EasyScale homo , a job will only use homogeneous GPUs, which is implemented by constraining the proposals of aimaster to only raise those using homogeneous GPUs. And in EasyScale heter , a job can accept heterogeneous GPUs.
Results. Figure 14 shows the average job completion time (JCT) and the makespan for the three schedulers when executing the same job trace in a cluster with 64 heterogeneous GPUs (i.e., V100, P100, T4). Compared to the capacity scheduler, EasyScale homo improves average JCT by 8.3× and makespan by 2.5×. And EasyScale heter speedups by 13.2× in average JCT and 2.8× in makespan. Once enabling elasticity, the DLT jobs can get rid of the gang-schedule requirement, which significantly improves the performance by uti-lizing spare GPUs in an incremental approach, and finally resulting in the speed up as shown in EasyScale homo . With the ability to utilize heterogeneous hardware, EasyScale heter has more opportunities to use more available GPU resources. As illustrated in Figure 15, during the execution, the allocated GPUs of EasyScale heter is generally higher than that of EasyScale homo . This is because by guaranteeing consistent model accuracy in using heterogeneous GPUs for elastic training, EasyScale jobs can further utilize more idle available GPU resources to achieve better performance.

Cluster Experiment
We have deployed EasyScale in a shared heterogeneous GPU cluster with more than 3,000 GPUs. The production cluster used to dedicate for online GPU inference serving or development (e.g., Jupyter Notebook). Similar to Borg [54], which classifies service jobs as prod ones (i.e., high-priority) and batch jobs as non-prod, in this hybrid GPU cluster that mixes both production training and inference jobs, we treat inference serving jobs as high-priority with guaranteed quota and allow EasyScale DLT jobs to utilize the spare GPUs.
To illustrate the cluster efficiency improvement from EasyScale, one-day statistic is collected in Nov. 2021, right after EasyScale is fully deployed in this cluster, compared to the number collected in one-day of Dec. 2021. As shown in Figure 16, the first 1,440 minutes indicate the statistic collected before the deployment of EasyScale, while the last 1,440 minutes show how EasyScale utilize the idle GPUs by collecting the statistics. On average, EasyScale improves the GPU allocation ratio by 17.1% and improves the GPU SM utilization by 62.1%. During the one-day statistic, the elastic EasyScale jobs use 459 temporally idle GPUs on average and they can quickly scale in to free GPUs for high-priority online serving jobs in seconds. After the leaving of those inference jobs, the elastic DLT jobs full up the idle GPUs within 5 minutes. Our cluster statistic records a total number of 362 preemptions on that day and no EasyScale job fails.

Related Work and Discussion
Determinism and reproducibility. Determinism, reproducibility, ablation study are important for deep learning research [39,45,46]. Deep learning frameworks [7] and NVIDIA [6] have been studied deterministic model training using single GPU. However, it is a hard problem due to the float dominating execution, complicated software stack, hardware optimized implementation as deep learning adopts. The design of EasyScale derives from the understanding of non-determinism, considers bitwise identical in every step of model training, and extends the reproducibility to elastic training over heterogeneous GPUs. In Ampere GPU (e.g., A100), cuBLAS supports only internal heuristics approach without public interface to select GPU kernel algorithm, which can hardly produce accuracy-consistent results compared to that of using previous generation of GPUs [2].
Elastic deep learning. TorchElastic [9], ElasticDL [4], and Horovod Elastic [3] support elastic training and fault tolerance, however, they introduce non-determinism in model accuracy. KungFu [38] and Pollux [47] support adaptive training algorithms, including both adaptive batch sizes and learning rates, allowing both customized and build-in adaptation policies for efficient scaling. VirtualFlow [42] and Varuna [11] leverage the gradient accumulation approach to achieve elasticity. Those works cannot guarantee the trained model accuracy consistently among different runs, introducing concerns in embracing resource elasticity. As our parallel works, AutoPS [26], Singularity [51], and Pathways [14] also explore elastic training in different ways, including the model aggregations in parameter server architecture, CUDA calls analytics, and heterogeneous interconnects. EasyScale utilizes the characteristics of deep learning to achieve efficient elastic model training, and achieves accuracy-consistent trained model, which is bitwise identical to the one trained by fixed GPUs, therefore to remove the potential user concerns on elastic training. EasyScale currently focuses on data-parallel, however, new parallel strategies are proposed recently for giant model training [23,33,40,48]. We believe that elastically adaptive capability is essential for all deep learning frameworks, and we consider the support of advanced parallel strategies as future works.
Cluster scheduling. Resource management for deep learning jobs has been studied to improve utilization [55,57,58] and fairness [41]. SLAQ [60] and Pollux [47] prioritize resources by considering model convergence. To improve cluster utilization, ONES [15] tunes batch size of training jobs. Optimus [44] and EDL [56] adjust the number of parameterservers and workers. PipeSwitch [13] overlaps computation with layered model loading. Retiarii [61] dynamically allocates resources among AutoML jobs and applies cross-job optimization. Gandiva [57] and AntMan [58] utilize the unique deep learning characteristic to optimize scheduling at minibatch boundaries. EasyScale conducts lightweight context switching among EasyScaleThreads for every mini-batch. It ensures the model quality is consistent with the ones of using distributed synchronous training while supporting resource elasticity to improve cluster utilization, through the intra-job and inter-job scheduling polices.

Conclusion
Through EasyScale, we advocate decoupling deep learning training from fixed GPU resources. EasyScale represents a critical step toward serverless deep learning by showing not only the feasibility but also the potential. EasyScale achieves elastic accuracy-consistent model training by 1) introducing the EasyScaleThread abstraction to preserve the training behaviors over elasticity and heterogeneity, 2) sourcing the non-deterministic behaviors scattered in the DLT software stack and solving them, and 3) developing new intra-job and inter-job scheduling policies utilizing the heterogeneous GPU cluster. Going forward, we hope EasyScale can draw attention to the deterministic computation of deep learning, and we should not always trade determinism for performance when designing elastic deep learning systems.