Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> With the increased use of "Virtual Machines" (VMs) as vehicles that isolate applications running on the same host, it is necessary to devise techniques that enable multiple VMs to share underlying resources both fairly and efficiently. To that end, one common approach is to deploy complex resource management techniques in the hosting infrastructure. Alternately, in this paper, we advocate the use of self-adaptation in the VMs themselves based on feedback about resource usage and availability. Consequently, we define "Friendly" VM (FVM) to be a virtual machine that adjusts its demand for system resources, so that they are both efficiently and fairly allocated to competing FVMs. Such properties are ensured using one of many provably convergent control rules, such as Additive-Increase/Multiplicative-Decrease (AIMD). By adopting this distributed application-based approach to resource management, it is not necessary to make assumptions about the underlying resources nor about the requirements of FVMs competing for these resources. To demonstrate the elegance and simplicity of our approach, we present a prototype implementation of our FVM framework in User-Mode Linux (UML)---an implementation that consists of less than 500 lines of code changes to UML. We present an analytic, control-theoretic model of FVM adaptation, which establishes convergence and fairness properties. These properties are also backed up with experimental results using our prototype FVM implementation. <s> BIB001 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> Over the past few years, the authors benchmarked 22 data center buildings. From this effort, we have determined that data centers can be over 40 times as energy intensive as conventional office buildings. Studying the more efficient of these facilities enabled us to compile a set of “best-practice” technologies for energy efficiency. These best practices include: improved air management, emphasizing control and isolation of hot and cold air streams; rightsizing central plants and ventilation systems to operate efficiently both at inception and as the data center load increases over time; optimized central chiller plants, designed and controlled to maximize overall cooling plant efficiency, central air-handling units, in lieu of distributed units; “free cooling” from either air-side or water-side economizers; alternative humidity control, including elimination of control conflicts and the use of direct evaporative cooling; improved uninterruptible power supplies; high-efficiency computer power supplies; on-site generation combined with special chillers for cooling using the waste heat; direct liquid cooling of racks or computers; and lowering the standby losses of standby generation systems. Other benchmarking findings include power densities from 5 to nearly 100 Watts per square foot; though lower than originally predicted, these densities are growing. A 5:1 variation in cooling effectiveness index (ratio of cooling power to computer power) was found, as well as large variations in power distribution efficiency and overall center performance (ratio of computer power to total building power). These observed variations indicate the potential of energy savings achievable through the implementation of best practices in the design and operation of data centers. <s> BIB002 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> Localized areas of high heat flux on microprocessors produce hot spots that limit their reliability and performance. With increasingly dense circuits and the integration of high power processors with low power memory, non-uniform thermal profiles will become more dramatic and difficult to manage. Chip scale thermal solutions designed to keep hot spots below a critical temperature unnecessarily overcool the rest of the CPU and add to heat-sink load. Localized hot spot cooling solutions, even active systems that contribute some additional heat, can do a better job controlling hot spot temperatures when efficiently integrated with a heat spreader. Embedded thermoelectric cooling (eTEC) is a promising approach to reduce the temperature of highly localized, high heat flux hot spots generated by today's advanced processors <s> BIB003 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> Power management has become increasingly necessary in large-scale datacenters to address costs and limitations in cooling or power delivery. This paper explores how to integrate power management mechanisms and policies with the virtualization technologies being actively deployed in these environments. The goals of the proposed VirtualPower approach to online power management are (i) to support the isolated and independent operation assumed by guest virtual machines (VMs) running on virtualized platforms and (ii) to make it possible to control and globally coordinate the effects of the diverse power management policies applied by these VMs to virtualized resources. To attain these goals, VirtualPower extends to guest VMs `soft' versions of the hardware power states for which their policies are designed. The resulting technical challenge is to appropriately map VM-level updates made to soft power states to actual changes in the states or in the allocation of underlying virtualized hardware. An implementation of VirtualPower Management (VPM) for the Xen hypervisor addresses this challenge by provision of multiple system-level abstractions including VPM states, channels, mechanisms, and rules. Experimental evaluations on modern multicore platforms highlight resulting improvements in online power management capabilities, including minimization of power consumption with little or no performance penalties and the ability to throttle power consumption while still meeting application requirements. Finally, coordination of online methods for server consolidation with VPM management techniques in heterogeneous server systems is shown to provide up to 34% improvements in power consumption. <s> BIB004 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> Current approaches to power management are based on operating systems with full knowledge of and full control over the underlying hardware; the distributed nature of multi-layered virtual machine environments renders such approaches insufficient. In this paper, we present a novel framework for energy management in modular, multi-layered operating system structures. The framework provides a unified model to partition and distribute energy, and mechanisms for energy-aware resource accounting and allocation. As a key property, the framework explicitly takes the recursive energy consumption into account, which is spent, e.g., in the virtualization layer or subsequent driver components. ::: ::: Our prototypical implementation targets hypervisor-based virtual machine systems and comprises two components: a host-level subsystem, which controls machine-wide energy constraints and enforces them among all guest OSes and service components, and, complementary, an energy-aware guest operating system, capable of fine-grained application-specific energy management. Guest level energy management thereby relies on effective virtualization of physical energy effects provided by the virtual machine monitor. Experiments with CPU and disk devices and an external data acquisition system demonstrate that our framework accurately controls and stipulates the power consumption of individual hardware devices, both for energy-aware and energyunaware guest operating systems. <s> BIB005 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> Modern mobile devices need to be extremely energy efficient. Due to the growing complexity of these devices, energy aware design exploration has become increasingly important. Current exploration tools often do not support energy estimation, or require the design to be very detailed before the estimate is possible. It is important to get early feedback on both performance and energy consumption during all phases of the design and at higher abstraction levels. This paper presents a unified optimization and exploration framework, from source level transformation to processor architecture design. The proposed retargetable compiler and simulator framework can map applications to a range of processors and memory configurations, simulate and report detailed performance and energy estimates. An accurate energy modeling approach is introduced, which can estimate the energy consumption of processor and memories at a component level, which can help to guide the design process. Fast energy-aware architecture exploration is illustrated using an example processor. The flow is demonstrated using a representative wireless benchmark on two state of the art processors and on a processor with advanced low power extensions for memories. The framework also supports exploration of various novel low power extensions and their combinations. We show that a unified framework enables fast feedback on the effect of source level transformations of the application code on the final cycle count and energy consumption. <s> BIB006 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> Virtualization technologies like VMware and Xen provide features to specify the minimum and maximum amount of resources that can be allocated to a virtual machine (VM) and a shares based mechanism for the hypervisor to distribute spare resources among contending VMs. However much of the existing work on VM placement and power consolidation in data centers fails to take advantage of these features. One of our experiments on a real testbed shows that leveraging such features can improve the overall utility of the data center by 47% or even higher. Motivated by these, we present a novel suite of techniques for placement and power consolidation of VMs in data centers taking advantage of the min-max and shares features inherent in virtualization technologies. Our techniques provide a smooth mechanism for power-performance tradeoffs in modern data centers running heterogeneous applications, wherein the amount of resources allocated to a VM can be adjusted based on available resources, power costs, and application utilities. We evaluate our techniques on a range of large synthetic data center setups and a small real data center testbed comprising of VMware ESX servers. Our experiments confirm the end-to-end validity of our approach and demonstrate that our final candidate algorithm, PowerExpandMinMax, consistently yields the best overall utility across a broad spectrum of inputs - varying VM sizes and utilities, varying server capacities and varying power costs - thus providing a practical solution for administrators. <s> BIB007 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> Power-manageable hardware can help save energy, but what can software developers do to address the problem? <s> BIB008 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> This paper investigates the effect of small cell deployments in urban and dense urban areas from the energy consumption perspective. Considering a parametric power model for the legacy macrocell networks and the new emerging small cells, this study quantifies the power reduction gain by deploying heterogeneous networks consisting of a mixture of both technologies. As a part of the study, a framework is developed which determines the optimal network architecture in terms of the combination of small cells and macrocells. While the optimality is merely decided based on the overall power consumption performance, both capacity and coverage requirements are simultaneously addressed. The results presented in this paper are based on the field measured traffic demand data from the urban area of the Wellington, NZ. Additionally, the paper investigates the impact of future traffic growth and provides a 5-year outlook for energy consumption of the network. The presented numerical results confirm substantial power reduction gain from deploying small cells. This becomes even more critical in the future when the traffic demand is increasing and the offloading effects of small cells are most beneficial. Furthermore, this study suggests that improving the idle mode power consumption of small cells is one of the key areas which can enable significant total power reduction. This study can serve as a guideline for operators when estimating the power efficiency of their network. <s> BIB009 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> This paper focuses on experiments investigating the quantity and quality of heat that can be captured from a liquid-cooled CPU on a computer server blade from a data center. This work also evaluates how effectively that waste heat energy dissipated from the CPUs residing on a server blade can be captured with a liquid-cooled thermal-extraction system. The ultimate goal behind this investigation is to determine the feasibility of using the thermal energy from a CPU to drive a cooling process. The unique application of this would be to capture the dissipated heat from many processors on many server blades within a data center and utilize that energy to drive a single-effect lithium bromide (Li-Br) refrigeration system. Another unique aspect of the investigation is the establishment of an interesting and useful relationship among CPU and system temperatures, thermal power, and CPU tasking levels. In response to the incoming system data, we explore the heat, temperature and power effects of adding insulation, varying water flow, tasking the CPU, and varying the cold plate-to-CPU clamping pressure. The aim is to provide an optimal and steady range of temperatures necessary for the chiller to operate. Results shown are for an IBM eServer xSeries 336 1U standalone server blade with dual Intel Xeon 3.6 GHz CPUs. Results are also shown for a simulation setup with an innovative cold plate that provides liquid cooling. <s> BIB010 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> In this work we propose a joint energy, thermal and cooling management technique (JETC) that significantly reduces per server cooling and memory energy costs. Our analysis shows that decoupling the optimization of cooling energy of CPU & memory and the optimization of memory energy leads to suboptimal solutions due to thermal dependencies between CPU and memory and non-linearity in cooling energy. This motivates us to develop a holistic solution that integrates the energy, thermal and cooling management to maximize energy savings with negligible performance hit. JETC considers thermal and power states of CPU & memory, thermal coupling between them and fan speed to arrive at energy efficient decisions. It has CPU and memory actuators to implement its decisions. The memory actuator reduces the energy of memory by performing cooling aware clustering of memory pages to a subset of memory modules. The CPU actuator saves cooling energy by reducing the hot spots between and within the CPU sockets and minimizing the effects of thermal coupling. Our experimental results show that employing JETC results in 50.7% average energy reduction in cooling and memory subsystems with less than 0.3% performance overhead. <s> BIB011 </s> Cloud Computing: Survey on Energy Efficiency <s> State of the Art <s> Large-Scale Antenna Systems (LSAS) is a form of multi-user MIMO technology in which unprecedented numbers of antennas serve a significantly smaller number of autonomous terminals. We compare the two most prominent linear pre-coders, conjugate beamforming and zero-forcing, with respect to net spectral-efficiency and radiated energy-efficiency in a simplified single-cell scenario where propagation is governed by independent Rayleigh fading, and where channel-state information (CSI) acquisition and data transmission are both performed during a short coherence interval. An effective-noise analysis of the pre-coded forward channel yields explicit lower bounds on net capacity which account for CSI acquisition overhead and errors as well as the sub-optimality of the pre-coders. In turn the bounds generate trade-off curves between radiated energy-efficiency and net spectral-efficiency. For high spectral-efficiency and low energy-efficiency zero-forcing outperforms conjugate beamforming, while at low spectral-efficiency and high energy-efficiency the opposite holds. Surprisingly, in an optimized system, the total LSAS-critical computational burden of conjugate beamforming may be greater than that of zero-forcing. Conjugate beamforming may still be preferable to zero-forcing because of its greater robustness, and because conjugate beamforming lends itself to a de-centralized architecture and de-centralized signal processing. <s> BIB012
Energy efficiency of both wireless and wired access networks has been the focus of several international initiatives and projects. Some prominent ones are the Bell Labsled GreenTouch TM consortium and the EU funded projects EARTH and ECONET. GreenTouch TM [2012] is a consortium of more than 50 leading ICT organizations, from both industry and the academy, dedicated to reducing the carbon footprint of communication networks. The goal of GreenTouch is to identify key components that can increase network energy efficiency by a factor of 1,000 by year 2015 compared to 2010 levels. This goal will be achieved by delivering improved architectures, specifications, and technologies. EARTH [2011] is a EU-funded IP research project in FP7 focused on mobile communication systems and their energy efficiency. The target of the project is a reduction of energy consumption by at least 50% focusing on LTE and LTE-A and existing 3G networks. The project ended in 2012 with tangible results in the areas of energy-efficient network architectures, deployment strategies, and optimization. ECONET [2013] is a EU-funded IP research project in FP7 investigating "dynamic adaptive technologies for wired network devices that allow saving energy when a device, or part of it, is not used." The objective of the project is "reducing the energy requirements of wired network equipment by 50% in the short to mid-term and by 80% in the long run." For wireless networks, recent research has focused on network architectures, scaling of energy consumption with load, and low-complexity processing. 3.2.1. Wireless Network Architectures. From a wireless architecture perspective, moving from traditional macrocellular networks to a HetNet architecture is one of the most impactful changes, one with a high potential for reducing energy consumption  ]. In macrocellular networks, due to the fact that energy is transmitted in a relatively unfocused way and the distance between base station and mobile device is typically large, a high amount of transmit power is required. Serving users with small cells can reduce this path loss by several orders of magnitude, to an extent at which the transmit power is no longer the limiting factor. In BIB009 , the authors showed that, for an urban area, energy consumption can be reduced by a factor of 46 by moving to a HetNet architecture with efficient idle modes. For macrocells, moving to remote radio heads can reduce cable losses and improve efficiency on the order of 3dB. Furthermore, increasing the number of antennas at the base station to large-scale antenna systems in combination with beamforming, which focuses energy to the user and also reduces interference, can significantly improve energy efficiency BIB012 . Server enclosures such as cabinets are important for optimal cooling and power supply systems, where later are out of scope of this survey. Although there is some research work in this field, most innovations come from the industry and production environments as best practices. Some important literature includes the book by  covering Google's practices inside data centers, whereas BIB002 provide best practices learned from benchmarking 22 data centers. 4.2.1. Server Cooling. Choosing an optimal enclosure design affects the efficiency of the power supply and cooling systems, and, via these systems, the server racks as well. As shown in BIB003 , localized cooling, specifically Embedded Thermoelectric Cooling (eTEC), can reduce the temperature of localized hot spots generated by modern processors and therefore reduce power consumption.  compare eTEC with a vapor compression refrigeration system for cooling microprocessors. They show how eTEC can achieve from 3% up to 10% in power savings and vapor up to 25%. The general conclusion is that localized cooling of the right component can produce some worthwhile improvements. Additionally, BIB010 suggest using the heat generated by CPUs to drive a cooling process, specifically, a single-effect lithium bromide (Li-Br) refrigeration system. BIB011 provide an overview of thermal management solutions for memory subsystems. They also present JETC, a management system for server memory and CPUs with a combined energy-efficient thermal and cooling solution. By applying such an approach, they consider dependencies between CPU and memory, as well as their shared cooling subsystem and achieve a 50.7% average energy reduction. Several research papers focus on different levels of potential actions at the CMS level to mitigate energy savings. We can distinguish four levels of actions. First, a VM can be reconfigured to change its resource requirements; in this way, the stress on the system is lower and energy consumption reduced. Furthermore, the physical machines themselves can be adjusted to their actual load so as to reduce their power consumption. Second, the placement of VMs can be optimized such that the most efficient physical machines are used. Third, VMs can be moved between physical machines, consolidating the load on fewer hosts and powering off unused machines. Finally, the scheduling of VMs over time can be adapted to reduce resource consumption at any given time. All these actions can be combined, and the use of several levers in the same framework is described in the following literature. Also, it must be taken into account the potential degradation of QoS induced. The approaches differ in the kind of constraints put on the QoS. PhD dissertations    or surveys are the primary sources of literature reviews. 5.2.1. VM Reconfiguration. Considering the first possibility, VM reconfiguration and hardware adjustment, BIB001 propose VMs that self-adapt their resource allocation to their demands. Similarly, Borgetto et al. [2012b] propose VM reconfiguration in which the middleware adapts the VM resources' demands to their needs. The authors propose proactive VM reconfiguration models, taking also into account the time needed to change the state of the physical machines (power on and off). BIB007 explore the problem by handling several parameters of CMS for resourcesharing VMs, including minimum, maximum, and proportion of CPU being allocated.  use DVFS-enabled infrastructure to adjust the hardware demands to actual real-time services needs. On the side of the hypervisors, BIB004 present VirtualPower, an extension that associates VMs with software CPU power state, as compared to the hypervisor conventional power states of a CPU. This allows hardware and software to be coordinated to use the best power mode and to use DVFS also in virtualized modes. BIB005 developed a low-level, fine-grained energy account system for hypervisors to allow power capping for guests. Additionally, working on the infrastructure itself in coordination with the VM management is also investigated. The energy efficiency of cloud appliances depends on a number of aspects including appliance development, compilation, deployment, and runtime phases. In addition, it is related to interaction with other elements of the cloud infrastructure, especially hardware, virtualization technology, and CMSs. 6.2.1. Design and Development.  cover many issues related to the energy efficiency of software products. They provide recommendations and techniques for developing energy-efficient software, especially concentrating on reducing power usage by idle appliances. The authors also show that limiting wake-up events and changing timer activities leads to significant energy savings. Some key principles to produce power-efficient software are proposed in BIB008 . First, the amount of resources consumed should directly correspond to the amount of useful work done by the software appliance. In particular, if the appliance's useful work is lower, then the system should run in a lower state and the power usage should be decreased to the extent related to the useful work reduction. This corresponds to achieving the W1 goal defined in this article. Second, the software should minimize power usage in an idle state by reducing the amount of unnecessary computation (e.g., using a push instead of a pull mechanism), which enables it to remain dormant until action is actually required. This corresponds to achieving the L2 goal defined in this article. Third, if possible, software requests to access additional resources should be done infrequently and in batches, thus decreasing the number of unnecessary wakeups. Additionally, as indicated in  , attention should be paid to details such as avoiding memory leaks or freeing unallocated memory. Otherwise, "these problems will cause increased interference from the host operating system, resulting in additional energy consumption"  . Fakhar et al. [2012] propose a green compiler that applies a number of techniques to make code more energy efficient. These techniques are split into strategies for compilers and software development. They include cache skipping, use of register operands, instruction clustering and reordering, loop optimization, and more. They address the problem of overheads related to the use of energy-efficiency optimizations in the compiler, which corresponds to the W2 goal. Other research work on compilers that take energy efficiency into account are the encc  or Coffee BIB006 compilers; however, these do not focus on software development for clouds.