WattsApp: Power-Aware Container Scheduling

Containers are becoming a popular workload deployment mechanism in modern distributed systems. However, there are limited software-based methods (hardware-based methods are expensive requiring hardware level changes) for obtaining the power consumed by containers for facilitating power-aware container scheduling, an essential activity for efficient management of distributed systems. This paper presents WattsApp, a tool underpinned by a six step software-based method for power-aware container scheduling to minimize power cap violations on a server. The proposed method relies on a neural network-based power estimation model and a power capped container scheduling technique. Experimental studies are pursued in a lab-based environment on 10 benchmarks deployed on Intel and ARM processors. The results highlight that the power estimation model has negligible overheads for data collection - nearly 90% of all data samples can be estimated with less than a 10% error, and the Mean Absolute Percentage Error (MAPE) is less than 6%. The power-aware scheduling of WattsApp is more effective than Intel's Running Power Average Limit (RAPL) based power capping for both single and multiple containers as it does not degrade the performance of all containers running on the server. The results confirm the feasibility of WattsApp.


INTRODUCTION
Container technology is a lightweight virtualization technique that has low overheads when compared to Virtual Machines (VMs) [28]. erefore, they are becoming popular for deploying workloads on clusters and clouds [30,35] and for upcoming distributed systems that use the edge of the network [33,34].
Container scheduling is an important avenue explored in the literature for distributed systems. Existing container scheduling strategies consider a number of relevant parameters, including resource demand, service level agreements and hardware/so ware requirements [19,38]. However, container scheduling like any deployment strategy need to be power-aware so that the total power consumption of a system does not exceed prede ned power cap limits.
Modern processors are equipped with power capping techniques, such as Dynamic Voltage and Frequency Scaling (DVFS) and Running Average Power Limit (RAPL) [22,37]. ese are hardwarebased and reduce the CPU frequency and voltage to lower processor power consumption. However, this degrades the entire system performance and consequently the deployed application.
It is valuable to gather the power consumption of individual containers running in a system so that they can be scheduled in a power-aware manner. However, there are limited so ware-based methods that measure container power consumption. Commercial vendors of Uninterrupted Power Supply (UPS) employ so warebased power estimation techniques based on static information, such as input voltage for di erent types of devices (for example, laptops) 1 , but ignore the workload-level granularity of power estimation. Other so ware based approaches, such as cWa s [26], cWa s++ [27] and SmartWa s [13] are either CPU architecture speci c, do not capture all components of the system that contribute to container power consumption, and are intrusive methods (further considered in Section 7). is fundamental gap is addressed in this paper by developing WattsApp, underpinned by a six step so ware-based (not hardware-based since they are expensive and require hardware level changes), hardware architecture agnostic and relatively non-intrusive power-aware container scheduling method. e aim is to estimate power consumption of containers and schedule power capped containers to stay within safe power budgets.
e research contributions of WattsApp are as follows: (i) A six-step so ware-based power-aware container scheduling method that accurately predicts power consumption of containers. e proposed method has negligible overheads (in relation to system power consumption) for collecting data required for estimating container power consumption. Additionally, nearly 90% of all data samples can be estimated using the power model with less than a 10% error. e Mean Absolute Percentage Error (MAPE) is observed to be between 1%-6%, which is relatively low. WattsApp is the rst prototype that builds power models and enforces power capping for parallel applications that execute on a cluster of containers.
(ii) e proposed power-aware method of WattsApp implements power capped scheduling for both single and multiple containers on the same server. e proposed power capping method is more bene cial than when no power cap or Intel's RAPL power cap is employed since the performance of all running containers on the system is not degraded (only containers that violate the budget are penalized). Potential approaches based on containers to achieve the power cap are to migrate the container to another server or deallocate resources of the container that violates the power cap. Experimentally, deallocating resources is a more viable approach than migration due to the inherent limitations of migrating containers. e remainder of the paper is organized as follows. Section 2 discusses the motivation for WattsApp. Section 3 proposes the power-aware container scheduling method. e underlying power model is discussed in Section 4. Section 5 presents the power capped container scheduling approach. Section 6 presents experimental studies. Section 7 discusses the related work. Section 8 concludes the paper by considering future work.

BACKGROUND
Predicting container power consumption is complex because it depends on the resource allocated to it and the workload running in the container. It is di erent when compared to the power prediction of VMs, other processes and hardware (processors, memory etc) because of the limited availability of data about the resource utilization and hardware performance counters speci c to containers.
is is because containers create multiple processes on the host operating system (OS). e number of processes varies depending on the activity that is performed within the containers. is paper observes that containers with more allocated resources consume more power for the same workload than on containers with fewer resources. However, in all cases the power consumed does not correspond to the increase in resources (CPU cores, memory). For example, the average power of an application running on twice the resources as another container, may not necessarily directly correspond to a factor of two.
is hypothesis is veri ed on 10 di erent scienti c workloads that are listed in Table 1. ese workloads are obtained from two sources. e rst is DCBench, a bench marking suite [17] from which ve MPI based workloads are chosen. e second is a collection of C/C++ based scienti c programs 2 . ese workloads are 2 h p://people.sc.fsu.edu/ ∼ jburkardt/c src/c src.html a combination of CPU-bound, I/O-bound and memory-bound scienti c workloads that execute to completion. is paper does not consider alternate classes of workloads, such as Internet-of-ings, stream processing, or sensor-based applications. e workloads considered in this paper may have di erent power consumption pa erns during execution. is is captured in the resource utilization and power data that is collected at a ne granularity and used for building the power model. is ensures that estimation can be carried out for di erent potential phases of a workload. Figure 1 highlights the average, minimum and maximum power consumed by a container with 3 CPU cores and 4GB RAM. Figure 2 provides results for the same workloads for twice the resources (6 CPU cores and 8GB RAM) It is evident that although the resources allocated are doubled the average power consumed does not necessarily double for all workloads (for example, refer to the workloads HEATED, MD, POISSON, PRIME and SGEFA).
Similar trends are obtained when the power consumption is noted for the above workloads over time (the results are exhaustive and are not presented in this paper). When more resources are added parallel workloads (applications running within a single container) execute faster, but reach their peak power consumption at di erent times. is paper does not aim to explain individual power pro les of workloads, but notes that a server that executes multiple containers could violate the power cap; speci cally, when multiple large size containers are multi-tenant on a server. ese large containers may consume high power and when they are multitenant their combined total power consumption could be higher and close to the maximum power consumption of the server.
Power cap violations are undesirable and need to be e ectively managed on servers running di erent workloads. ey occur when the total power consumed by a server exceeds a threshold de ned by the server administrators. When power cap violations occur, the server performance starts degrading since power management techniques like Dynamic Voltage and Frequency Scaling (DVFS) that are bundled with processors come in to play. DVFS reduces the server power consumption by using two power saving techniques, namely dynamic voltage scaling and dynamic frequency scaling [22]. Power saving is achieved by lowering the frequency and/or voltage of the CPU and other system resources. is reduction negatively impacts the performance of workloads executed on the server. For example, the performance of a container running on a server will drop when there is a power cap violation.
In order to avoid the above, a power aware container scheduling strategy is required. It is observed that power/energy saving bene ts of techniques like DVFS are diminishing because of the complexity in hardware technologies used for processors and memory (for example, increased memory performance and multi-core processors) [22]. erefore, a so ware-based power capping technique is desirable in addition to speci c hardware-based techniques.
is motivates the power capping technique proposed in this paper.

THE WATTSAPP METHOD
is section presents a method for so ware-based power aware scheduling of containers to minimize power cap violations on a server, which is fundamental in developing WattsApp. Power aware container scheduling is the distribution/consolidation of containers such that the total power consumed by a server does not cross a prede ned threshold (or cap) speci ed by an administrator.
A primary requirement for the WattsApp power aware container scheduling approach is obtaining information on the power consumed by an individual container. Resource utilization statistics of each container running on a server is used to calculate its power consumption. is information is used for container scheduling, such that the maximum power consumed does not violate any power restriction on an individual server.
Currently, there are no hardware methods for obtaining the power consumed by containers. Moreover, there are a few so ware methods to measure the container power consumption and these methods have concerns like they are architecture speci c, ignores essential system resources or intrusive as discussed in Section 7.
is article aims to bridge this gap. Hardware-based methods will require modi cation of the hardware (such as additional probes) resulting in more expensive processors. Hence, a so ware-based method is adopted to develop a model of container power consumption that depends on resource utilization information of the container. e model uses linear regression-based neural network to correlate container resource utilization statistics with system power consumption information to estimate the container power consumption. e model is further presented in Section 5.
e WattsApp method relies on two activities: (i) Container power prediction, which is estimating the power consumed by an individual container using so ware-based methods, and (ii) Power capped container scheduling, which is using the estimated power values to place containers equitably on a server. e importance of such a power-aware container scheduling method is that if the power cap exceeds on any server, then the performance of all containers running on the server will be degraded. To mitigate this, any power cap violation is periodically detected on each server by observing the total power consumption of the server. If the power consumed is above the threshold, then it is considered as a power cap violation. When a violation is detected, power capping is performed on the server without signi cantly a ecting the run time performance of all running containers.
Power capping in WattsApp is performed using two approaches, namely container migration and container resource (CPU cores) reduction. e method aims to nd a server on to which a container causing the server power capping violation can be migrated. If such a server is available, then the container is migrated to the identi ed server thereby avoiding any power cap violation.
If no servers are available to migrate a container, then a second approach is used by reducing the resources allocated to the container. e processor subsystem consumes nearly 85% power of the total system power [24]. erefore, to achieve power capping the CPU cores allocated to the container are rstly reduced one at a time until the power cap limit is restored. WattsApp uses technology speci c commands (such as docker update command) to change the cores allocated to a container; the change is immediately re ected. As a result the container will use fewer CPU cores, which eventually reduces the server power consumption.
When a container is migrated to a di erent server, the power consumption of the source server is again reduced below the power cap limit. A detailed discussion on both container migration and CPU core reduction is provided in Section 5. e proposed method of WattsApp shown in Figure 3 comprises six steps: namely Data Collection, Model Building, Power Estimation, Power Capped Allocation, Violation Detection and Power Capping. e rst three steps are for container power prediction, and the remaining steps are for power capped container scheduling.
Step 1 -Data Collection: e training data for supervised learning is collected to estimate the power consumption of containers. It collects system power consumption and resource usage statistics for each container from the host OS. is is correlated with the system power consumption to obtain container power consumption.
e system power consumption information is necessary as the regression techniques require labeled data for building a model. System power consumption data is collected from Wa s Up .net hardware power meter. It facilitates the power consumption sampling at a one second granularity. Real time system power consumption data can be obtained by connecting it through a USB interface. It is reported that the accuracy of this hardware meter is +/-1.5%+0.3W 3 [16]. e readings provided are also considered to be generally reliable although a high error rate is observed for readings below 1 Wa [16]. A series of power meters from Wa s Up are used in research reported in the literature [23][1][6] [3].
Step 2 -Model Building: e collected data is used to train data and build a neural network for individual containers running on the server. We build a model for each container on the server as di erent applications exhibits di erent properties. ese models will be utilized during run time for predicting the power consumption of the container. e input to the neural network is container resource utilization statistics (including the percentage of host CPU and memory the container is using, the total memory the container is using and the maximum allocated memory, the amount of data the container sent and received on its network interface, and the amount of the data read from and wri en to the block input output devices) and system power consumption.
e input data to the model is obtained in the previous step. e model (linear regressionbased) was developed using the Keras 4 deep learning library.
Step 3 -Power Estimation: Consider there are n containers (C 1 , C 2 , C 3 , · · · C n ) running on a server. e resource utilization statistics (CPU usage, memory usage, amount of block I/O and network data transfer) are collected for each container. e models developed (previous step) and the run-time statistics are used to predict the power consumed by the container.
Step 4 -Power Capped Allocation: e models are used for power capped container scheduling. It uses the predicted power consumption of the container and current total power consumption of the server before scheduling the container on the server. e power consumption information of the containers are obtained by using the power estimation model that uses the power pro le of each container for power capped allocation.
Step 5 -Violation Detection: To perform power capped container scheduling a power cap violation has to be detected. A er initial scheduling of a container, this step is executed at a ve minute interval to check the server for any power cap violation. If there is a violation, then the nal step enforces the power capping limit on the server.
Step 6 -Power Capping: is nal step adopts two techniques to enforce power capping. e rst is referred to as migrationanother server that can accommodate the container causing the power cap violation on a current server is identi ed; migration should not violate the power cap of the recipient server. If such a server is available, then the power cap violating container is migrated to the identi ed server. If no such server is identi ed, then a second technique, referred to as resource deallocation, is performed in which the number of CPU cores allocated to the container is reduced until the power cap limit is reached. is rst prototype of WattsApp only considers a single container causing power cap violations. However, if multiple containers cause power 3 h ps://www.vernier.com/ les/manuals/wu-pro.pdf 4 h ps://keras.io/ cap violation, then a priority based container selection approach is required, which is not considered in this paper. e underlying approaches of the rst three steps are presented in Section 4. e last three steps are further discussed in Section 5.

WATTSAPP CONTAINER POWER ESTIMATION
is section describes the Data Collection (Step 1), Model Building (Step 2), and Power Estimation (Step 3) steps of WattsApp.
e Data Collection step gathers (i) the system power consumption data, and (ii) resource utilization data of running containers.
is is a black box technique as the data is collected from the host operating system and no pro ling data is obtained from within the container. e other approaches (referred to as white box) collect pro ling information inside the container and should be avoided to maintain the integrity of the containers [14].
Power consumption data is collected from the Wa s Up .net power meter. e data obtained contains the time stamp and power consumption (in Wa s). e resource utilization data of the container is collected using the docker stats command, which provides the following data: (i) Id of the container and the name of the container, (ii) Percentage of host CPU and memory the container is using, (iii) Total memory the container is using and the maximum allo ed memory, (iv) e amount of data the container has sent and received on its network interface, (v) e amount of the data read from and wri en to the block input/output devices, and (vi) e number of processes/threads created by container. e data collection time stamp is also added to the output of docker stats. Both the power consumption data and the resource utilization data of the container is concatenated with respect to the time stamp. Multiple CPU cores allocated to the container are taken care of by the host CPU usage (for example, it ranges from 0 to 100% for one core, up to 200% for two cores and so on. Resource utilization and system power consumption data are obtained once per second during the execution of the benchmarks. e sequence of steps is presented in Algorithm 1. e inputs to the model are the container resource utilization (including percentage of host CPU and memory the container is using, total memory the container is using, maximum allo ed memory, amount of data the container has sent and received on its network interface, amount of the data read from and wri en to the block input output devices), and system power consumption. e output is container power consumption.
In the Power Estimation step, the power consumption of containers is rstly modeled by WattsApp. e power consumption of a server (P ser er ) comprises static power and dynamic power. Static power (P st at ic ) is de ned as the power consumption of system when there is no active container. is power is measured by using the Wa s Up .net power meter. If there is only one running container then the total power consumption is sum of idle power consumption and dynamic power consumption of the server. e dynamic power consumption (P d namic ) is de ned as the power consumption of the running container.
If there are n active containers on the server, then the dynamic power consumption of the server is the aggregate power consumption of all the containers.
where P cont ainer is the power consumption of the container. e dynamic power consumption of the system is considered as the sum of the power consumed by the CPU, the memory (RAM), the disk and the network.
where a, b, c, and d are constants, n is the number of running containers in a server, l represents the number of CPU cores allocated to a container, U cpu is the CPU utilization factor, U ram is the RAM utilization factor, U disk is the disk utilization factor, and U net is the network utilization factor.
e case when a single workload is executed across a cluster of containers is considered by WattsApp. In this case, the workload's power consumption will be the aggregate power consumption of all the containers of the cluster.
where Container k is element of C, the set of containers in the cluster C = (Container 1 , Container 2 , · · · Container n ).

WATTSAPP POWER CAPPED CONTAINER SCHEDULING
is section presents the use of the estimated power values at run time for a proposed power capped container scheduling method of

Algorithm 2: Power Capped Scheduling of Containers
Input :S, C i j , md WattsApp. e approach adopted is to initially allocate containers using a best t or rst t strategy, and subsequently when there is a power cap violation on the server, migrate the container elsewhere or reduce the allocated CPU cores of the running containers until the power capping is not violated. Power Capped Allocation (Step 4), Violation Detection (Step 5), and Power Capping (Step 6) proposed in Section 3 is considered in this section. Power Capped Allocation uses the estimated power consumption of containers to schedule containers by calculating the total power consumption of the candidate server a er adding the estimated power consumption of the container ready for deployment. If the power consumption of the server is anticipated to be below the power cap limit, then the container will be placed on the candidate server. is is repeated for all containers that are ready for placement. Algorithm 2 highlights this and Table 2 presents the notation used in this algorithm and the other algorithms (Algorithm 3 and Algorithm 4) presented in this section. It is assumed that there are n servers, and each server may have up to m containers.
Algorithm 2 executes for all containers ready for placement (line 1). e ag variable is initialized to false (line 2); this variable will be used to identify the case when no suitable server for power capped placement. Each server is checked one by one whether it can accommodate the container under consideration (line 3). e container power consumption is added to the candidate server power (line 4) to check if it can accommodate the container (line 5). If the candidate server can deploy the container, then it is allocated to the server (line 6). When container placement is successful, the ag is updated to true (line 7). If it is not possible to allocate on a given server, then the remaining servers are processed. When no suitable server is found for power capped placement (line 12), the container is allocated to the server with the lowest power consumption (line 13 and 14). A er this, the Algorithm 3 will work to detect the possibility of power cap violation, and if required, the power cap is applied using Algorithm 4. Available memory in the server s i C i j List of containers deployed in server S i for j = 1, 2, · · · , m C List of all the containers ready to be placed ACr i List of CPU core allocation to each container AM i List of memory allocation to each container md Power consumption Model PC i j Power consumption of container c i j PS i Calculated total power consumption of server s i , ps i ε PS c j , pc j e candidate container and its power consumption causing the server power increasing beyond the power cap threshold cap Power cap A process to determine any power cap violation is executed on the servers, ve minutes a er initially scheduling containers (pro ling data is collected for the rst ve minutes). is process uses the power prediction model to estimate the power consumption of each running container on the server. If the power consumption of any server (sum of power consumption of all the running containers) is beyond the power cap, then there is a power cap violation caused by the newly placed container). is container will be considered as the candidate for migration or CPU core reduction.
Power capping is achieved in two ways. e rst is by migrating the candidate container from the source to a destination server whose current power consumption is below the cap and would not be violated if it accepted the container. A stateful migration method, namely 'CRIU (Check-point/Restore In Userspace)' is employed for migrating containers. e second is by reducing the resources allocated to the candidate container, speci cally the number of CPU cores (reduce one at a time) as CPU usage signi cantly a ects the power consumption of Docker containers [31]. e Docker update command is used to change the number of allo ed CPU cores to the container. e performance of the container will be degraded when using this approach (further considered in Section 6).
Currently, there is support for power capping on the hardware. However, most hardware power capping techniques tweak the voltage and processor frequency, which a ects the potential performance of the entire system and is detrimental to all containers running on the system [22]. However, the proposed so ware power capping technique achieves the power cap without signi cantly a ecting the entire system's performance and only negatively impacts the container that causes the power cap violation.
Violation Detection is to detect when a power cap violation occurs on any server under consideration. e detection algorithm is given in Algorithm 3 and runs on each of the servers. When a violation is detected, Algorithm 4 performs power capping. e model used to compute the power consumption of the container at run-time is considered in Section 4.

Algorithm 3: Power Cap Violation Detection
Input :S, C i j , md is is achieved by collecting the resource usage statistics for each container (line 4) and then passing the data to the power model for predicting power consumed (line 5). e power consumption of all containers running on a server are summed to obtain server power consumption (line 6) a er which power cap violation (if any) is checked for on each individual server (line 9 and 10). If a server crosses the power cap limit then Algorithm 4 is performed with the required inputs. If the power consumed by all containers on the servers is below power cap, then no changes are made (line 13). e last step is Power Capping that uses Algorithm 4 and the input provided by Algorithm 3. A server (line 1) that can accommodate the container without crossing the power cap (line 2) is searched for by checking with a network process that runs on each server. If successful, then the container that violates the power cap on a source server is migrated to the destination server (line 3). If no candidate destination servers are found, the algorithm uses the second option of reducing the allocated CPU cores to a container until the server power consumption falls below the power cap (line 7). For this, the allocated cores are rst reduced by 1 (line 8) and then the resource usage statistics are collected (line 9) and the power consumption is predicted (line 10). Again, if the current power consumption falls below the power cap (line 11) then the algorithm is successful and returns (line 12).
CPU core reduction to achieve power capping will degrade performance of the selected container. is can be compensated for by increasing the CPU cores at a later stage when it may be feasible to do so without exceeding the power cap. is scenario is considered in experimental studies to demonstrate that the impact of CPU core reduction of a container can be compensated when running the parallel component of an application in a cluster.
ere may be a delay in enforcing the power capping limit since the detection algorithm is only executed once every ve minutes (this time is a con gurable parameter of the algorithm to suit any bespoke requirements). Experimentation on the impact of this limit is not presented in this article. It was empirically observed that a er a power cap violation was detected, a few minutes were required for migrating the container to another server. e time taken for migration is due to: (i) Creating checkpoints -a checkpoint freezes a running container and turns its state into a collection of les on disk, (ii) Compressing the checkpoint and transferring it to the selected server, and (iii) Creating a new container and restoring the checkpoint. e majority of the migration time (depends on the size of the container image) is for transferring the checkpoint to the destination server.

PERFORMANCE EVALUATION
e experiments highlight that for WattsApp: (i) the overheads for data collection to estimate power do not signi cantly impact system power consumption, (ii) there is limited error in estimating power using the model, (iii) WattsApp operates across multiple processors, and (iv) the proposed power capping method is more e ective for scheduling than alternate methods, such as Intel's RAPL. e experiments are conducted on two systems with di erent form factors. e rst is workstation Dell Precision 3630 with an Intel Xeon E-2174G processor and 16GB memory. e second is a small form factor Odroid N2 Board with a quad-core ARM Cortex-A73 CPU cluster and a dual core Cortex-A53 cluster and 4GB memory. e system power consumption data is collected using Wa s Up .net hardware power meter. e systems run on Ubuntu 18.04. e containers are created with Ubuntu 18.04 LTS image. Each container is allocated three CPU cores, 4GB memory on workstation and 2 CPU cores and 2GB memory on Odroid. Docker 17.12.0-ce version is used to deploy the containers. Keras 2.2.0 that runs on TensorFlow 1.8.0 is used to build the power model. e hardware power meter Wa s Up .net is used to obtain system power consumption in real-time. Wa s Up .net power is used to collect the power readings using USB from the host OS by a Python script that reads instantaneous power data.  Table 1 are used for evaluating the WattsApp method. ese are scienti c workloads that execute to completion. is paper does not consider alternate workloads, such as Internet-of-ings, stream processing, or sensor-based applications.
e experiments are carried out for single workloads on single containers, multiple workloads on multiple containers, and single workload across multiple containers (a cluster of containers) to thoroughly evaluate the WattsApp method.
Results: e data collection overheads and estimation error in container power prediction is rstly presented. en, the results from scheduling for power capping obtained for single and multiple containers are considered. e average system power consumption overhead when collecting data (CPU, memory, and disk utilization along with power) for a 89 second time period is shown in Figure 4. e blue plot shows the average system power consumption when only power data is collected, and the orange plot shows when both resource utilization and power metrics are collected. On an average, nearly 0.2 Wa s are spent. e graph illustrates that the overhead in terms of system power is negligible (less than 1% of system power consumption).
is is an indicator that Step 1 of the proposed power-aware scheduling method is feasible. Figure 5 shows the data collection overhead on the ARM processor; the overhead is nearly 1.7% of the system power consumption. Figure 6 shows the distribution of error on the Intel Xeon processor in the power values that is estimated for 444 samples using the neural network model. In this experiment, data collected from all workloads (Table 1) is consolidated to build the model that is validated using repeated random sampling by spli ing data into 75% and 25% as training and testing dataset respectively. More than 90% of the samples have an error of less than 10% and nearly 49%  of the samples have less than a 6% error. is highlights that the power model built in Step 2 of the method will have a reasonable accuracy for prediction in Step 3. Figure 7 highlights the Mean Absolute Percentage Error (MAPE) of the model for estimating power of individual workloads executing in a container on the Intel Xeon processor. For this experiment each workload is executed in a single container and the power is estimated for each container. MAPE indicates the average of percentage errors (a lower value indicates that the model estimates the power consumed with a higher accuracy). e average percentage error is between 1% and just over 6%.
Similar experiments are performed on the Odroid board with ARM processor using six workloads from (Table 1) (the four workloads, namely FUZZYKMEANS, KMEANS, KPCA and PCA, are distributed in DCBench with the binaries for the x86 platform). Figure 8 shows the error distribution of estimating power values for 400 samples using the neural network model. In this experiment, data collected from the six workloads is consolidated to build the model that is validated using repeated random sampling by spli ing data into 75% and 25% as training and testing dataset, respectively. More than 80% of the samples have an error of less than 15% and more than 60% of the samples have less than a 10% error. is highlights that the power model from Step 2 will have a reasonable prediction accuracy in Step 3. Figure 9 highlights the Mean Absolute Percentage Error (MAPE) of the model for estimating power of individual workloads executing in a container on the ARM processor. For this experiment each workload is executed in a single container and the power is estimated for each container. MAPE indicates the average of percentage errors (a lower value indicates that the model estimates the power consumed with a higher accuracy). e average percentage error is between 1% and just over 4%. Figure 10 shows the overheads associated with the two techniques adopted in power capping, namely migration and deallocation of resources for di erent sizes of containers. e time taken to migrate using the Checkpoint/Restore in Userspace approach provided by Docker is directly proportional to the size of the container as the container needs to be checkpointed and migrated to an alternate server. However, using the time taken to deallocate resources on a container on the server takes approximately 180 milliseconds. Although migration is a potential option to achieve the power cap, the results show that deallocating resources is a more viable option given the inherent overheads in container migration. In the next set of experiments, power capping results based on only resource deallocation is presented. Migration using containers is a less viable option based on existing technology (if a critical application has to be executed) given large migration overheads although it may be lower than VMs (also not suited for single parallel application executed across a cluster of containers).
In another experiment, a cluster of containers (four on the Intel processor and two on ARM) was created for running the MPI applications from Table 1. Figure 11 shows the results on the Intel processor to demonstrate the feasibility of container power prediction for parallel applications executed across multiple containers. e average MAPE is 3 with error between 1 % and around 5.5%. Figure 12 show the MAPE on ARM processor. e average MAPE 2.6 with error between 1 % and 4 %. e accuracy of WattsApp power estimation is considered for di erent input parameters when the benchmark is executed in a e data for the input parameters were not used during training. e results highlight that the average error percentage is between 0.8% and 4% for both Intel and ARM processors.
Two further experiments were carried out on the Intel processor to identify the potential bene t of the proposed power capping based container scheduling compared against when no power caps and Intel's RAPL-based power cap is employed. e rst experiment is when a single container executes on the server with a given workload. In this experiment, there is only one container running on the server that is likely to violate the power cap. e second experiment is when multiple (three) containers that run the same workload execute on the server. ere are multiple containers running on a given server, and any one of them may violate the power cap. Each container executes the same workload. Figure 15 shows the results for the rst experiment in which a single container executes on the server with a given workload. e graph shows the workload execution time for the proposed power cap method, no power cap, and RAPL's power cap is adopted. In this case, it is noted that the proposed power capping method is more e ective than RAPL's power cap since the total workload execution time is lower in every case. is highlights the bene t of using the proposed power-aware container scheduling method. Figure 16 shows the results for the second experiment in which three containers (C1, C2, C3) with the same workloads execute on the server. ese workloads are more representative of a real world scenario. e graph shows the workload execution time when the proposed power cap method, no power cap, and RAPL's power cap is adopted. It can be clearly seen that when RAPL is employed the workload execution time of all containers increase.
is is because RAPL achieves power capping by reducing the CPU frequency of the server, which in turn a ects the performance of all containers running on the server. On the other hand, it is noted that the proposed power cap technique reduces the allocated number of CPU cores, thus degrading the performance of only one container as opposed to all the running containers. erefore, 'C3' in many cases is noted to take longer than the other containers. In short, only one container out of many (that potentially violates the power cap) is penalized when using the proposed power cap technique. Figure 17 shows the peak power consumption on the Intel Xeon processor of applications (from Table 1) when there is no power (a) For DCBench programs from Table 1 (b) For MPI-C programs from Table 1 Figure 15: Execution time of individual workloads when a single container executes on a server using the power cap technique on the Intel Xeon processor (a) For DCBench programs from Table 1 (b) For MPI-C programs from Table 1  is experiment uses the power cap limit of 55W. e average peak power consumption of the proposed power capping technique is 56.4W which is close to the power cap limit where as the average peak power of RAPL's power cap is 60.2W and signi cantly higher than the power capping limit. e peak power consumption for WattsApp is 60.3W in comparison to the peak power consumption of RAPL's power cap is 65.9W.
Similar experiments are carried out on the Odroid board to demonstrate the e ectiveness of WattsApp power capping on ARM processors. Again only six workloads from Table 1 are used. As RAPL is speci c to Intel, these experiments only compare the WattsApp power cap with no power cap.  Figure 18 shows the results of the experiment in which a single container executes on the ARM processor. e workloads running under WattsApp power cap takes slightly longer time and executes within the power budget de ned by power cap limit of 7W. Figure 19 shows the results when two containers with the same workload executes on the ARM processor. e graph shows the workload execution time under the no power cap and WattsApp power cap. Again workloads running under WattsApp power cap takes longer time and the power budget remains below the power cap limit of 9W. is demonstrate that WattsApp power capping in also e ective on ARM processors. Figure 20 shows the peak power consumption of the six workloads from Table 1 when there is no power capping and under the WattsApp power capping regime. is experiment uses the power cap limit of 9W. e average peak power of the proposed power capping technique is 8.1W (below the power cap limit), but is 9.2W when there is no power cap. Figure 21 and Figure 22 demonstrates another aspect -when a single MPI application is executed across a cluster of containers.
is experiment considers that workloads are running on a cluster of two or more di erent servers. When the CPU cores of the workload need to be reduced on one server, then it is compensated for by increasing the CPU cores allocated on the other server for the workload. e result shows that power capping on one server with compensatory allocation on another server does not signi cantly impact performance. Summary: In short, the experimental results highlight that: (i) e data collection overhead in the proposed power-aware container scheduling method of WattsApp only a ects the system power consumption negligibly. (ii) Nearly 90% of data samples are estimated with less than 10% error. (iii) e MAPE of power estimation using the model that is employed by the proposed power-aware container scheduling method of WattsApp is between 1%-6%. is is relatively low and accurate estimations can be expected from the model. (iv) e power estimation method of WattsApp is also validated on the parallel applications across a cluster of containers. e proposed model estimate the power consumption with MAPE between 1% to 5.5%. e impact of power capping (CPU core reduction) for parallel workloads is minimized during the workload runtime by applying compensation. (v) Deallocation of resources are found to be a more feasible approach than migrating containers for the power capping technique given that the overheads for migration increase with the size of the container. e overheads for deallocating resources is negligible. (vi) When both single and multiple containers are executed on the server, it is noted that the proposed power cap method is more bene cial than when no power cap or RAPL's power cap is employed since the proposed method does not degrade the performance of all running containers to keep the power budget below the cap. WattsApp's power capping is also e ective since the peak power allowed by the WattsApp method is less than that of RAPL's power cap and does not violate the so power cap that may be imposed by administrators.

RELATED WORK
Power modeling of processors and VMs are well explored, but power modeling of containers is still in its early stages. In this section, the impact of virtualization techniques on system power consumption that do not use estimation models is discussed. en container power modeling techniques and power models for processors/servers and VMs are considered. Finally, research on container power capping is discussed.
Impact of virtualization techniques on system power: Power consumed by servers running containers has been experimentally measured without developing power models. e CPU usage of the container has a signi cant contribution to the overall power consumption [31]. Empirical investigation on four virtualization technologies, namely Xen, KVM, Docker and LXC is noted [25]. It was observed that for CPU workloads there is no signi cant di erence in power consumption among the above technologies. However, containers consume less power than other virtualization technologies for network-based workloads.
A comparison of the server power consumption [32] and energy comparison [18] has been presented for virtualization and containerization technologies. e power and energy characteristics of four hypervisors and a container engine including VMware ESXi, Microso Hyper-V, KVM, XenServer and Docker on six different hardware (three mainstream 2U rack servers, one emerging ARM64 server, one desktop server, and one laptop) is considered. It is observed that hypervisors exhibit di erent power and energy consumption pro les when the same workload is executed on the hardware. Although containers are light weight, they are not necessarily more power-e cient than VMs. Similar comparisons of running workloads on containers and bare-metal servers are considered [29]. It is observed that running Docker has an inherent power cost and thus energy consumed is higher than bare-metal.
Container Power Modeling: SmartWa s [13] is a self calibrating so ware power model for containers that relies on hardware performance counters and RAPL's power measurement of CPU and DRAM for estimating power. Using RAPL limits the applicability of SmartWa s to Intel architectures. It also does not capture the impact of disk access and network usage on power that may be the main activity of an I/O or a network based application. e power model of WattsApp on the other hand uses architecture agnostic parameters to model system power and its feasibility on multiple hardware platforms is demonstrated.
Lightweight power models, such as cWa s+ [26] and cWa s++ [27] are developed for containers. cWa s++ is a virtual power model that has two components: a client back-end and a server front-end. e client back-end is installed in the container and accesses the CPU event counters. cWa s++ uses two models, namely an event-based and RAPL-based models.
e event-based model uses CPU performance counters and RAPL-based models uses only RAPL event counters. e evaluation shows that the two power models are useful on workloads obtained from the PARSEC and in-house benchmarks. However, cWa s only uses CPU related metrics to compute container power from server power. CPUs are a major power consuming component of a typical server (nearly one-third [10] and even up to 40% [24] of the total server power), but other components need to be considered. Hence, WattsApp considers memory, IO and the network to account for container power. Moreover, cWa s is intrusive and requires client installations and access to containers (this may not be always feasible depending on access permissions, ownership models, business models etc).
ere is research that accounts for the power consumption of individual threads and application containers [8].
e research relies on power estimation of each CPU core obtained from Intel's RAPL and hardware performance counters (related to CPU events) obtained from the OS. A power-aware consolidation technique of containerized data centers based on a model built using CPU utilization is presented [21]. Both approaches are based on CPU related metrics and do not account for the power consumed by other components [24]). WattsApp on the other hand considers CPU, memory, disk and network related metrics.
Power Models for VM and Processor Power Models: Joulemeter [20] is a so ware power meter for VMs. Joulemeter estimates VM power using linear regression with ordinary least square to correlate VM pro ling data to system power consumption. BITWATTS uses a two-level approach to estimate the process level power estimation based on CPU power consumption is developed [10]. is approach performs pro ling at both system and VM level. e system-level pro ling estimates the system-level power consumption whereas, the VM level pro ling estimates the power consumption of hosted applications. BITWATTS uses a regression technique for learning the process level power models.
Another approach uses a tree regression-based technique for VM power metering [15]. It is suggested that the linear regression methods are not su ciently accurate and therefore an approach that recursively partitions the collected data into easy modeling pieces has been proposed. is method rst, builds the server model using the observation for server and then fairly divides this server power consumption among the virtual machines.
iMeter is a performance counters based VM power model based on polynomial kernel based support vector regression [36]. Principal component analysis is performed to select performance counters that have the most impact on power consumption.
Similarly, there are power models for processors and the components of a server. A two stage cross architecture power model is discussed in [9]. is model takes advantage of both linear regression and support vector machines to provide power estimation with a high accuracy across multiple hardware architectures.
A con gurable learning framework named PowerAPI that automatically builds the power model of a CPU is presented [11]. PowerAPI automatically explores the available hardware performance counters of a CPU and selects the performance counter having the most impact on the power consumption of the server.
ere are a number of other notable processor power consumption models proposed by researchers (for example, [5][6][7]12]), which are not considered within the scope of the discussion of this paper.
However, WattsApp is a so ware power model that uses container resource utilization statistics obtained from the host OS to estimate the container power consumption.
Container Power Capping: ere is limited research in the literature that focuses on power capping for containers. Two power capping techniques are proposed in literature: e rst is a power capping technique (DockerCap) for Docker containers [3]. e system power consumption is obtained from the hardware power meter and RAPL. e CPU quota of all the containers of di erent priority is reduced, thereby a ecting the performance of all the containers. e WattsApp method however uses two techniques, namely container migration, and CPU core reduction to achieve power capping. e merit of the proposed method is that the overall container performance is una ected and only the container that violates the power cap is degraded. e second power capping technique is proposed for Docker containers on the Kubernetes platform [2] by relying on DEEP-mon power monitoring [8]. is technique relies on RAPL and DVFS to manage power cap limits. It is demonstrated in this paper that using RAPL a ects the run-time performance of all containers running on the server. RAPL enforces a power cap on the processor package and DRAM by reducing the CPU frequency and thus degrades the overall system performance. However, WattsApp uses architecture independent metrics to measure resource utilization (CPU, memory, disk and network) making the approach portable. e proposed method is demonstrated to be e ective for both power-aware scheduling and capping.

CONCLUSIONS
is paper proposes WattsApp that is underpinned by a six-step power-aware scheduling method for containers to minimize power cap violations on a server in real-time.
e method relies on a neural network-based power estimation model. e trained model e ectively predicts over 90% of data samples with less than 10% error. By testing on 10 representative benchmark workloads, the approach is able to achieve a MAPE error of less than 6%, and displays minimal overhead during run time scheduling. Unlike hardware-based power capping techniques, such as Intel's RAPL, which are indiscriminate to workloads and degrade the overall performance of all containers running on a server, this so ware-based approach is able to target individual containers running workloads, minimizing overall processing degradation while maintaining a node's power budget. e proposed technique considers multiple scenarios, including (i) single/multiple application, single container and single application, multiple containers. WattsApp has been shown to be feasible and outperforms existing techniques.
Future Work: WattsApp is tested on two popular processor types. However, to deal with heterogeneous platforms the power model will need to be expanded further to accelerator architectures. e current method prioritizes the power budget of an individual server, but not the performance or SLA of the container workloads.
Be er understanding, pro ling, or feedback (for example, as presented in Barrel sh [4]) from an application would enable be er choices within the scheduler and is an avenue for future work.
Finally, this technique will be applied to edge computing, where power is a critical concern. As containers are being increasingly used in this space, this research is directly applicable. Alternate types of workloads, such as stream processing and sensor-based applications will be considered.