Cloud Computing: Survey on Energy Efficiency <s> Context <s> Energy-aware design and evaluation of network protocols requires knowledge of the energy consumption behavior of actual wireless interfaces. But little practical information is available about the energy consumption behavior of well-known wireless network interfaces and device specifications do not provide information in a form that is helpful to protocol developers. This paper describes a series of experiments which obtained detailed measurements of the energy consumption of an IEEE 802.11 wireless network interface operating in an ad hoc networking environment. The data is presented as a collection of linear equations for calculating the energy consumed in sending, receiving and discarding broadcast and point-to-point data packets of various sizes. Some implications for protocol design and evaluation in ad hoc networks are discussed. <s> BIB001 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> The declining costs of commodity disk drives is rapidly changing the economics of deploying large amounts of online or near-line storage. Conventional mass storage systems use either high performance RAID clusters, automated tape libraries or a combination of tape and disk. In this paper, we analyze an alternative design using massive arrays of idle disks, or MAID. We argue that this storage organization provides storage densities matching or exceeding those of tape libraries with performance similar to disk arrays. Moreover, we show that with effective power management of individual drives, this performance can be achieved using a very small power budget. In particular, we show that our power management strategy can result in the performance comparable to an always-on RAID system while using 1/15th the power of such a RAID system. <s> BIB002 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> A large portion of the power budget in server environments goes into the I/O subsystem - the disk array in particular. Traditional approaches to disk power management involve completely stopping the disk rotation, which can take a considerable amount of time, making them less useful in cases where idle times between disk requests may not be long enough to outweigh the overheads. This paper presents a new approach called DRPM to modulate disk speed (RPM) dynamically, and gives a practical implementation to exploit this mechanism. Extensive simulations with different workload and hardware parameters show that DRPM can provide significant energy savings without compromising much on performance. This paper also discusses practical issues when implementing DRPM on server disks. <s> BIB003 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Recent research works have been presented on conserving energy for multi-disk systems either at a single disk drive level or at a storage system level and thereby having certain limitations. This paper studies several new redundancy-based, power-aware, I/O request scheduling and cache management policies at the RAID controller level to build energy-efficient RAID systems, by exploiting the redundant information and destage issues of the array for two popular RAID levels, RAID 1 and RAID 5. For RAID 1, we develop a Windowed Round Robin (WRR) request scheduling policy; for RAID 5, we introduce a N-chance Power Aware cache replacement algorithm (NPA) for writes and a Power-Directed, Transformable (PDT) request scheduling policy for reads. Trace-driven simulation proves EERAID saves much more energy than legacy RAIDs and existing solutions. <s> BIB004 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> In this paper, we study energy conservation techniques for disk array-based network servers. First, we introduce a new conservation technique, called Popular Data Concentration (PDC), that migrates frequently accessed data to a subset of the disks. The goal is to skew the load towards a few of the disks, so that others can be transitioned to low-power modes. Next, we introduce a user-level file server that takes advantage of PDC. In the context of this server, we compare PDC to the Massive Array of Idle Disks (MAID). Using a validated simulator, we evaluate these techniques for conventional and two-speed disks and a wide range of parameters. Our results for conventional disks show that PDC and MAID can only conserve energy when the load on the server is extremely low. When two-speed disks are used, both PDC and MAID can conserve significant energy with only a small fraction of delayed requests. Overall, we find that PDC achieves more consistent and robust energy savings than MAID. <s> BIB005 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> A virtualized system includes a new layer of software, the virtual machine monitor. The VMM's principal role is to arbitrate accesses to the underlying physical host platform's resources so that multiple operating systems (which are guests of the VMM) can share them. The VMM presents to each guest OS a set of virtual platform interfaces that constitute a virtual machine (VM). Once confined to specialized, proprietary, high-end server and mainframe systems, virtualization is now becoming more broadly available and is supported in off-the-shelf systems based on Intel architecture (IA) hardware. This development is due in part to the steady performance improvements of IA-based systems, which mitigates traditional virtualization performance overheads. Intel virtualization technology provides hardware support for processor virtualization, enabling simplifications of virtual machine monitor software. Resulting VMMs can support a wider range of legacy and future operating systems while maintaining high performance. <s> BIB006 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Over the past few years, the authors benchmarked 22 data center buildings. From this effort, we have determined that data centers can be over 40 times as energy intensive as conventional office buildings. Studying the more efficient of these facilities enabled us to compile a set of “best-practice” technologies for energy efficiency. These best practices include: improved air management, emphasizing control and isolation of hot and cold air streams; rightsizing central plants and ventilation systems to operate efficiently both at inception and as the data center load increases over time; optimized central chiller plants, designed and controlled to maximize overall cooling plant efficiency, central air-handling units, in lieu of distributed units; “free cooling” from either air-side or water-side economizers; alternative humidity control, including elimination of control conflicts and the use of direct evaporative cooling; improved uninterruptible power supplies; high-efficiency computer power supplies; on-site generation combined with special chillers for cooling using the waste heat; direct liquid cooling of racks or computers; and lowering the standby losses of standby generation systems. Other benchmarking findings include power densities from 5 to nearly 100 Watts per square foot; though lower than originally predicted, these densities are growing. A 5:1 variation in cooling effectiveness index (ratio of cooling power to computer power) was found, as well as large variations in power distribution efficiency and overall center performance (ratio of computer power to total building power). These observed variations indicate the potential of energy savings achievable through the implementation of best practices in the design and operation of data centers. <s> BIB007 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Most Ethernet interfaces available for deployment in switches and hosts today can operate in a variety of different low power modes. However, currently these modes have very limited usage models. They do not take advantage of periods of inactivity, when the links remain idle or under-utilized. In this study, we propose methods that allow for detection of such periods to obtain energy savings with little impact on loss or delay. We evaluate our methods on a wide range of real-time traffic traces collected at a high-speed backbone switch within our campus LAN. Our results show that Ethernet interfaces at both ends can be put in extremely low power modes anywhere from 40%-98% of the time observed. In addition, we found that approximately 37% of interfaces studied (on the same switch) can be put in low power modes simultaneously which opens the potential for further energy savings in the switching fabric within the switch. <s> BIB008 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Today's data centers may contain tens of thousands of computers with significant aggregate bandwidth requirements. The network architecture typically consists of a tree of routing and switching elements with progressively more specialized and expensive equipment moving up the network hierarchy. Unfortunately, even when deploying the highest-end IP switches/routers, resulting topologies may only support 50% of the aggregate bandwidth available at the edge of the network, while still incurring tremendous cost. Non-uniform bandwidth among data center nodes complicates application design and limits overall system performance. In this paper, we show how to leverage largely commodity Ethernet switches to support the full aggregate bandwidth of clusters consisting of tens of thousands of elements. Similar to how clusters of commodity computers have largely replaced more specialized SMPs and MPPs, we argue that appropriately architected and interconnected commodity switches may deliver more performance at less cost than available from today's higher-end solutions. Our approach requires no modifications to the end host network interface, operating system, or applications; critically, it is fully backward compatible with Ethernet, IP, and TCP. <s> BIB009 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> A fundamental challenge in data center networking is how to efficiently interconnect an exponentially increasing number of servers. This paper presents DCell, a novel network structure that has many desirable features for data center networking. DCell is a recursively defined structure, in which a high-level DCell is constructed from many low-level DCells and DCells at the same level are fully connected with one another. DCell scales doubly exponentially as the node degree increases. DCell is fault tolerant since it does not have single point of failure and its distributed fault-tolerant routing protocol performs near shortest-path routing even in the presence of severe link or node failures. DCell also provides higher network capacity than the traditional tree-based structure for various types of services. Furthermore, DCell can be incrementally expanded and a partial DCell provides the same appealing features. Results from theoretical analysis, simulations, and experiments show that DCell is a viable interconnection structure for data centers. <s> BIB010 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> The rapidly increasing energy consumption by computing and communications equipment is a significant economic and environmental problem that needs to be addressed. Ethernet network interface controllers (NICs) in the US alone consume hundreds of millions of US dollars in electricity per year. Most Ethernet links are underutilized and link energy consumption can be reduced by operating at a lower data rate. In this paper, we investigate adaptive link rate (ALR) as a means of reducing the energy consumption of a typical Ethernet link by adaptively varying the link data rate in response to utilization. Policies to determine when to change the link data rate are studied. Simple policies that use output buffer queue length thresholds and fine-grain utilization monitoring are shown to be effective. A Markov model of a state-dependent service rate queue with rate transitions only at service completion is used to evaluate the performance of ALR with respect to the mean packet delay, the time spent in an energy-saving low link data rate, and the oscillation of link data rates. Simulation experiments using actual and synthetic traffic traces show that an Ethernet link with ALR can operate at a lower data rate for over 80 percent of the time, yielding significant energy savings with only a very small increase in packet delay. <s> BIB011 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Todaypsilas CMP platforms are designed to be symmetric in terms of platform resources such as shared caches. However, it is becoming increasingly important to understand the performance implications of asymmetric caches for two key reasons: (a) multi-workload scenarios such as server consolidation are a growing trend and contention for shared cache resources between workloads causes logical cache asymmetry, (b) future CMP platforms may be designed to be physically asymmetric in hardware due to die area pressure, process variability or power/performance efficiency. Our focus in this paper is to understand the performance implications of both logical as well as physical asymmetric caches on server consolidation. Based on real measurements of a state-of-the-art CMP processor running a server consolidation benchmark (vConsolidate) we compare the performance implications as a function of (a) symmetric caches, (b) virtually asymmetric caches, (c) physically asymmetric caches and (d) a combination of logically and physically asymmetric caches. We analyze the performance behavior in terms of (i) performance of each of the individual workloads being consolidated and (ii) architectural components such as CPI and MPI. We believe that this asymmetric cache study is the first of its kind and provides useful data/insights on cache characteristics of server consolidation. We also present inferences on future optimizations in the VMM scheduler as well as potential hardware techniques for future CMPs with cache asymmetry. <s> BIB012 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> In the last few years, power dissipation has become an important design constraint, on par with performance, in the design of new computer systems. Whereas in the past, the primary job of the computer architect was to translate improvements in operating frequency and transistor count into performance, now power efficiency must be taken into account at every step of the design process. While for some time, architects have been successful in delivering 40% to 50% annual improvement in processor performance, costs that were previously brushed aside eventually caught up. The most critical of these costs is the inexorable increase in power dissipation and power density in processors. Power dissipation issues have catalyzed new topic areas in computer architecture, resulting in a substantial body of work on more power-efficient architectures. Power dissipation coupled with diminishing performance gains, was also the main cause for the switch from single-core to multi-core architectures and slowdown in frequency increase. This book aims to document some of the most important architectural techniques that were invented, proposed, and applied to reduce both dynamic power and static power dissipation in processors and memory hierarchies. A significant number of techniques have been proposed for a wide range of situations and this book synthesizes those techniques by focusing on their common characteristics. Table of Contents: Introduction / Modeling, Simulation, and Measurement / Using Voltage and Frequency Adjustments to Manage Dynamic Power / Optimizing Capacitance and Switching Activity to Reduce Dynamic Power / Managing Static (Leakage) Power / Conclusions <s> BIB013 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Today's storage systems place an imperative demand on energy efficiency. A storage system often places single-rotation- rate disks into standby mode by stopping them from spinning to conserve energy when the workload is not heavy. The major obstacle of this method is a high spin-up cost introduced by passively waking up the standby disk to service requests. In this paper, we propose a redundancy-based hierarchical I/O cache architecture called RIMAC to solve the problem. The idea of RIMAC is to enable data on the standby disk(s) to be recovered by accessing a two-level I/O cache and/or active disks if needed. In parity-based redundant disk arrays, RIMAC exploits parity redundancy to dynamically XOR-reconstruct the data being accessed toward standby disk(s) at both the cache and disk levels. By avoiding passive spin-ups, RIMAC can significantly improve both energy efficiency and performance. In RIMAC, we developed 1) two power-aware read request transformation schemes called Transformable Read in Cache (TRC) and Transformable Read on Disk (TRD), 2) a power-aware write request transformation policy for parity update, and 3) a second-chance parity cache replacement algorithm to favor the request transformation rate. We evaluated RIMAC by augmenting a validated storage system simulator, DiskSim, and tested three real-life server traces, including HP's cello99, UlUC's OLTP, and SPC's search engine. Comprehensive results indicate that RIMAC is able to reduce energy consumption by up to 18 percent and simultaneously improve the average response time by up to 34 percent compared with threshold-based power management schemes for single-rotation-rate disk- based RAIDs. <s> BIB014 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. Users specify the computation in terms of a map and a reduce function, and the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks. Programmers find the system easy to use: more than ten thousand distinct MapReduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand MapReduce jobs are executed on Google's clusters every day, processing a total of more than twenty petabytes of data per day. <s> BIB015 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> This paper presents BCube, a new network architecture specifically designed for shipping-container based, modular data centers. At the core of the BCube architecture is its server-centric network structure, where servers with multiple network ports connect to multiple layers of COTS (commodity off-the-shelf) mini-switches. Servers act as not only end hosts, but also relay nodes for each other. BCube supports various bandwidth-intensive applications by speeding-up one-to-one, one-to-several, and one-to-all traffic patterns, and by providing high network capacity for all-to-all traffic. BCube exhibits graceful performance degradation as the server and/or switch failure rate increases. This property is of special importance for shipping-container data centers, since once the container is sealed and operational, it becomes very difficult to repair or replace its components. Our implementation experiences show that BCube can be seamlessly integrated with the TCP/IP protocol stack and BCube packet forwarding can be efficiently implemented in both hardware and software. Experiments in our testbed demonstrate that BCube is fault tolerant and load balancing and it significantly accelerates representative bandwidth-intensive applications. <s> BIB016 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Energy efficiency is becoming increasingly important in the operation of networking infrastructure, especially in enterprise and data center networks. Researchers have proposed several strategies for energy management of networking devices. However, we need a comprehensive characterization of power consumption by a variety of switches and routers to accurately quantify the savings from the various power savings schemes. In this paper, we first describe the hurdles in network power instrumentation and present a power measurement study of a variety of networking gear such as hubs, edge switches, core switches, routers and wireless access points in both stand-alone mode and a production data center. We build and describe a benchmarking suite that will allow users to measure and compare the power consumed for a large set of common configurations at any switch or router of their choice. We also propose a network energy proportionality index, which is an easily measurable metric, to compare power consumption behaviors of multiple devices. <s> BIB017 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> The IBM iDataPlex data center solution is for Web 2.0, high performance computing (HPC) cluster, and corporate batch processing customers experiencing limitations of electrical power, cooling, physical space, or a combination of these. By providing a big picture approach to the design, iDataPlex uses innovative ways to integrate Intel-based processing at the node, rack, and data center levels to maximize power and cooling efficiencies while providing necessary compute density.An iDataPlex rack is built with industry-standard components to create flexible configurations of servers, chassis, and networking switches that integrate easily. Using technology for flexible node configurations, iDataPlex technology can configure customized solutions for applications to meet specific business needs for computing power, storage intensity, and the right I/O and networking.This IBM Redbooks publication is for customers who want to understand and implement the IBM iDataPlex solution. It introduces the iDataPlex solution and the innovations in its design, outlines its benefits, and positions it with IBM System x and BladeCenter servers. The book provides details of iDataPlex components and the supported configurations. It describes application considerations for an iDataPlex solution and aspects of data center design that are central to that solution. The book concludes by introducing the services offerings available from IBM for planning and installing an iDataPlex solution. <s> BIB018 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Main memory power in volume and mid-range servers is growing as a fraction of total system power. The resulting energy consumption increases system cost and the heat produced reduces reliability. Emergent memory technology will provide systems with the ability to dynamically turn-on (online) and turn-off (offline) memory devices at runtime. This technology, coupled with slack in memory demand, offers the potential for significant energy savings in servers. However, to gain general acceptance in the server community, power-aware techniques must maintain performance and scale to thousands of memory devices. We propose a memory management infra-structure for energy reduction (Memory MISER) that is transparent, performance-neutral, and scalable. Memory MISER provides: 1) a prototype Linux kernel that manages memory at device granularity, and 2) a user space daemon that tracks systemic memory demand and implements energy- and performance-constrained device controller policies. Experiments on an 8-node cluster of servers show our Memory MISER conserves memory energy up to 56.8 percent with no performance degradation for scientific codes that utilized the entire cluster. For multi-user workloads, we achieved memory energy savings of up to 67.94 percent with no performance degradation. Normalizing to total system energy consumption, our power-aware memory approach reduced energy between 18.81 percent and 39.02 percent. <s> BIB019 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Nowadays, power consumption of data centers has huge impacts on environments. Researchers are seeking to find effective solutions to make data centers reduce power consumption while keep the desired quality of service or service level objectives. Virtual Machine (VM) technology has been widely applied in data center environments due to its seminal features, including reliability, flexibility, and the ease of management. We present the GreenCloud architecture, which aims to reduce data center power consumption, while guarantee the performance from users' perspective. GreenCloud architecture enables comprehensive online-monitoring, live virtual machine migration, and VM placement optimization. To verify the efficiency and effectiveness of the proposed architecture, we take an online real-time game, Tremulous, as a VM application. Evaluation results show that we can save up to 27% of the energy when applying GreenCloud architecture. <s> BIB020 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Clusters provide powerful computing environments, but in practice much of this power goes to waste, due to the static allocation of tasks to nodes, regardless of their changing computational requirements. Dynamic consolidation is an approach that migrates tasks within a cluster as their computational requirements change, both to reduce the number of nodes that need to be active and to eliminate temporary overload situations. Previous dynamic consolidation strategies have relied on task placement heuristics that use only local optimization and typically do not take migration overhead into account. However, heuristics based on only local optimization may miss the globally optimal solution, resulting in unnecessary resource usage, and the overhead for migration may nullify the benefits of consolidation. In this paper, we propose the Entropy resource manager for homogeneous clusters, which performs dynamic consolidation based on constraint programming and takes migration overhead into account. The use of constraint programming allows Entropy to find mappings of tasks to nodes that are better than those found by heuristics based on local optimizations, and that are frequently globally optimal in the number of nodes. Because migration overhead is taken into account, Entropy chooses migrations that can be implemented efficiently, incurring a low performance overhead. <s> BIB021 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> In this paper, we look at two different approaches used to provide embedded system support for virtualization and virtual machine monitors for consumer electronics and mobile devices. We compare the micro-kernel approach, which has been a popular choice for building embedded operating systems with the Virtual Machine Monitor (VMM) or hypervisor approach widely deployed in general purpose computing environments such as desktops and data center servers. Comparison criteria are based on virtualization use cases that are typical of Consumer Electronics (CE) systems such as mobile devices and IPTV. These approaches are further evaluated based on performance and on their ability to allow re-use of existing (often real-time) software as well as modern open operating systems such as Linux while remaining as transparently as possible. Such transparency can come through different paths, including: leveraging of hardware virtualization support, minimal modifications to the original operating system internals (kernel, device drivers, etc.), and the ability to use existing operating system applications as-is and without the need to port them to a new environment. An analysis of the fundamental principles behind each approach is presented with a discussion of their impact on existing operating environments, together with practical performance results based on existing micro-kernels and real-time hypervisor benchmarks. We conclude that mapping the VMM (hypervisor) approach used in data centers to the needs of embedded systems is a better option for the support of complete operating systems (as guests) than extending micro-kernels for such functionality. <s> BIB022 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Numerous studies have shown that datacenter computers rarely operate at full utilization, leading to a number of proposals for creating servers that are energy proportional with respect to the computation that they are performing. In this paper, we show that as servers themselves become more energy proportional, the datacenter network can become a significant fraction (up to 50%) of cluster power. In this paper we propose several ways to design a high-performance datacenter network whose power consumption is more proportional to the amount of traffic it is moving -- that is, we propose energy proportional datacenter networks. We first show that a flattened butterfly topology itself is inherently more power efficient than the other commonly proposed topology for high-performance datacenter networks. We then exploit the characteristics of modern plesiochronous links to adjust their power and performance envelopes dynamically. Using a network simulator, driven by both synthetic workloads and production datacenter traces, we characterize and understand design tradeoffs, and demonstrate an 85% reduction in power --- which approaches the ideal energy-proportionality of the network. Our results also demonstrate two challenges for the designers of future network switches: 1) We show that there is a significant power advantage to having independent control of each unidirectional channel comprising a network link, since many traffic patterns show very asymmetric use, and 2) system designers should work to optimize the high-speed channel designs to be more energy efficient by choosing optimal data rate and equalization technology. Given these assumptions, we demonstrate that energy proportional datacenter communication is indeed possible. <s> BIB023 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> The concept of a user-deployed femtocell that operates in licensed spectrum and provides cellular coverage in the home has attracted much interest in the wireless industry and an extensive deployment of femtocells is expected in the future. However, several issues arise when femtocells become more widely deployed. One concern is the additional energy consumption caused by femtocells due to their continuous receiver processing and signal transmissions, which can be significant with a large number of femtocells. Secondly, the increase in the number of active cells can result in a high increase in core networks signaling for mobility procedures. Finally, the continuous pilot transmissions can cause interference, particularly in dense urban areas. In this paper all these issues are addressed by a novel idle mode procedure which allows the femtocell transmissions and associated processing to be switched off completely at all times when the femtocell does not need to support an active call. <s> BIB024 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Networks are a shared resource connecting critical IT infrastructure, and the general practice is to always leave them on. Yet, meaningful energy savings can result from improving a network's ability to scale up and down, as traffic demands ebb and flow. We present ElasticTree, a network-wide power1 manager, which dynamically adjusts the set of active network elements -- links and switches--to satisfy changing data center traffic loads. ::: ::: We first compare multiple strategies for finding minimum-power network subsets across a range of traffic patterns. We implement and analyze ElasticTree on a prototype testbed built with production OpenFlow switches from three network vendors. Further, we examine the trade-offs between energy efficiency, performance and robustness, with real traces from a production e-commerce website. Our results demonstrate that for data center workloads, ElasticTree can save up to 50% of network energy, while maintaining the ability to handle traffic surges. Our fast heuristic for computing network subsets enables ElasticTree to scale to data centers containing thousands of nodes. We finish by showing how a network admin might configure ElasticTree to satisfy their needs for performance and fault tolerance, while minimizing their network power bill. <s> BIB025 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> In this point-counterpoint discussion, Trevor Mudge argues for the combination of near-threshold voltage processors with techniques such as boosting to address the needs of datacenter workloads. Urs Holzle offers a cautionary note on the wisdom of giving up too much single-threaded performance to achieve energy-efficiency in large Internet service applications. <s> BIB026 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> An increasing number of large-scale server clusters are being deployed in data centers for supporting many different web-based application services in a seamless fashion. In this scenario, the rising energy costs for keeping up those web clusters is becoming an important concern for many business. In this paper we present an optimization solution for power and performance management in a platform running multiple independent web applications. Our approach assumes a virtualized server environment and includes an optimization model and strategy to dynamically manage the cluster power consumption, while meeting the application's workload demands. <s> BIB027 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Power trends in communication networks are analyzed using a transaction-based model. Traffic models are developed for North America and used to evaluate the relative power trends of wireline networks and mobile networks through 2020. An ideal case for aggressive network-efficiency improvement measures is evaluated within this framework and shown to lead to roughly unchanged consumption over the next decade. Implications for future technology requirements are discussed. <s> BIB028 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> To alleviate the growing concern of energy waste in networked devices, we present PCube, a server-centric data center structure that conserves energy by varying bandwidth availability based on traffic demand. PCube not only supports a low-power mode for existing data centers offering full bisection bandwidth without hardware modification or re-wiring, but also provides an alternative for new data centers at a lower construction cost. The bandwidth demand could be dynamic and scalable in the optimal way when considering conserving the energy waste. To further reduce the cost, we take advantage of traffic locality, and propose Hybrid PCube, in order to offer different bandwidth to different servers. We use samples of traffic from a real-world production data center with full bisection bandwidth across thousands of servers, and evaluate our proposed algorithm using these collected samples. Our experimental results have shown convincing evidence that the proposed algorithm is able to substantially reduce energy costs incurred by networked devices in data centers. <s> BIB029 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Large datacenter operators with sites at multiple locations dimension their key resources according to the peak demand of the geographic area that each site covers. The demand of specific areas follows strong diurnal patterns with high peak to valley ratios that result in poor average utilization across a day. In this paper, we show how to rescue unutilized bandwidth across multiple datacenters and backbone networks and use it for non-real-time applications, such as backups, propagation of bulky updates, and migration of data. Achieving the above is non-trivial since leftover bandwidth appears at different times, for different durations, and at different places in the world. For this purpose, we have designed, implemented, and validated NetStitcher, a system that employs a network of storage nodes to stitch together unutilized bandwidth, whenever and wherever it exists. It gathers information about leftover resources, uses a store-and-forward algorithm to schedule data transfers, and adapts to resource fluctuations. We have compared NetStitcher with other bulk transfer mechanisms using both a testbed and a live deployment on a real CDN. Our testbed evaluation shows that NetStitcher outperforms all other mechanisms and can rescue up to five times additional datacenter bandwidth thus making it a valuable tool for datacenter providers. Our live CDN deployment demonstrates that our solution can perform large data transfers at a much lower cost than naive end-to-end or store-and-forward schemes. <s> BIB030 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Cloud service providers use replication across geographically distributed data centers to improve end-to-end performance as well as to offer high reliability under failures. Content replication often involves the transfer of huge data sets over the wide area network and demands high backbone transport capacity. In this paper, we discuss how a Globally Reconfigurable Intelligent Photonic Network (GRIPhoN) between data centers could improve operational flexibility for cloud service providers. The proposed GRIPhoN architecture is an extension of earlier work [34] and can provide a bandwidth-on-demand service ranging from low data rates (e.g., 1 Gbps) to high data rates (e.g., 10-40 Gbps). The inter-data center communication network which is currently statically provisioned could be dynamically configured based on demand. Today's backbone optical networks can take several weeks to provision a customer's private line connection. GRIPhoN would enable cloud operators to dynamically set up and tear down their connections (sub-wavelength or wavelength rates) within a few minutes. GRIPhoN also offers cost-effective restoration capabilities at wavelength rates and automated bridge-and-roll of private line connections to minimize the impact of planned maintenance activities. <s> BIB031 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Power management is one of the biggest challenges facing current data centers. As processors consume the dominant amount of power in computer systems, power management of multicore processors is extremely significant. An efficient power model that accurately predict the power consumption of a processor is required to develop effective power management techniques. However, this challenge rises with using virtualization and increasing number of cores in the processors. In this paper, we analyze power consumption of a multicore processor, we develop three statistical CPU-Power models based on the number of active cores and average running frequency using a multiple liner regression. Our models are built upon a virtualized server. The models are validated statistically and experimentally. Statistically, our models cover 97\% of system variations. Furthermore, we test our models with different workloads and three benchmarks. The results show that our models achieve better performance compared to the recently proposed model for power management in virtualized environments. Our models provide highly accurate predictions for un-sampled combinations of frequency and cores, 95\% of the predicted values have less than 7\% error. Thus, we can integrate these models into power management mechanisms for a dynamic configuration of a virtual machine in terms of the number of its virtual-CPUs and the frequency of physical cores to achieve both performance and power constrains. <s> BIB032 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> This paper presents a fast array of wimpy nodes---FAWN---an approach for achieving low-power data-intensive data-center computing. FAWN couples low-power processors to small amounts of local flash storage, balancing computation and I/O capabilities. FAWN optimizes for per node energy efficiency to enable efficient, massively parallel access to data. The key contributions of this paper are the principles of the FAWN approach and the design and implementation of FAWN-KV---a consistent, replicated, highly available, and high-performance key-value storage system built on a FAWN prototype. Our design centers around purely log-structured datastores that provide the basis for high performance on flash storage, as well as for replication and consistency obtained using chain replication on a consistent hashing ring. Our evaluation demonstrates that FAWN clusters can handle roughly 350 key-value queries per Joule of energy---two orders of magnitude more than a disk-based system. <s> BIB033 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> This paper presents a virtual machine scheduling algorithm for Cloud Computing based on Green Computing concepts. The goal of this algorithm is the minimization of energy consumption in task executions in a cloud computing environment. This algorithm uses some features like shutdown of underutilized hosts, migration of loads of hosts that are operating below a certain threshold, and DVFS. It also applies the concept of active cooling control in order to minimize power consumption. The choice of which host will receive load is based on the concept of higher energy efficiency from the hosts, which is given by the ratio of MIPS by the energy consumed of each host. Results from simulation of this algorithm confronted with other surveyed algorithms have shown that it can improve the power consumption in clouds composed of heterogeneous datacenters while being equivalent to the best algorithms in homogeneous datacenters. <s> BIB034 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> This paper investigates the effect of small cell deployments in urban and dense urban areas from the energy consumption perspective. Considering a parametric power model for the legacy macrocell networks and the new emerging small cells, this study quantifies the power reduction gain by deploying heterogeneous networks consisting of a mixture of both technologies. As a part of the study, a framework is developed which determines the optimal network architecture in terms of the combination of small cells and macrocells. While the optimality is merely decided based on the overall power consumption performance, both capacity and coverage requirements are simultaneously addressed. The results presented in this paper are based on the field measured traffic demand data from the urban area of the Wellington, NZ. Additionally, the paper investigates the impact of future traffic growth and provides a 5-year outlook for energy consumption of the network. The presented numerical results confirm substantial power reduction gain from deploying small cells. This becomes even more critical in the future when the traffic demand is increasing and the offloading effects of small cells are most beneficial. Furthermore, this study suggests that improving the idle mode power consumption of small cells is one of the key areas which can enable significant total power reduction. This study can serve as a guideline for operators when estimating the power efficiency of their network. <s> BIB035 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> In multi-tenant datacenters, jobs of different tenants compete for the shared datacenter network and can suffer poor performance and high cost from varying, unpredictable network performance. Recently, several virtual network abstractions have been proposed to provide explicit APIs for tenant jobs to specify and reserve virtual clusters (VC) with both explicit VMs and required network bandwidth between the VMs. However, all of the existing proposals reserve a fixed bandwidth throughout the entire execution of a job. ::: In the paper, we first profile the traffic patterns of several popular cloud applications, and find that they generate substantial traffic during only 30%-60% of the entire execution, suggesting existing simple VC models waste precious networking resources. We then propose a fine-grained virtual network abstraction, Time-Interleaved Virtual Clusters (TIVC), that models the time-varying nature of the networking requirement of cloud applications. To demonstrate the effectiveness of TIVC, we develop Proteus, a system that implements the new abstraction. Using large-scale simulations of cloud application workloads and prototype implementation running actual cloud applications, we show the new abstraction significantly increases the utilization of the entire datacenter and reduces the cost to the tenants, compared to previous fixed-bandwidth abstractions. <s> BIB036 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> This paper focuses on experiments investigating the quantity and quality of heat that can be captured from a liquid-cooled CPU on a computer server blade from a data center. This work also evaluates how effectively that waste heat energy dissipated from the CPUs residing on a server blade can be captured with a liquid-cooled thermal-extraction system. The ultimate goal behind this investigation is to determine the feasibility of using the thermal energy from a CPU to drive a cooling process. The unique application of this would be to capture the dissipated heat from many processors on many server blades within a data center and utilize that energy to drive a single-effect lithium bromide (Li-Br) refrigeration system. Another unique aspect of the investigation is the establishment of an interesting and useful relationship among CPU and system temperatures, thermal power, and CPU tasking levels. In response to the incoming system data, we explore the heat, temperature and power effects of adding insulation, varying water flow, tasking the CPU, and varying the cold plate-to-CPU clamping pressure. The aim is to provide an optimal and steady range of temperatures necessary for the chiller to operate. Results shown are for an IBM eServer xSeries 336 1U standalone server blade with dual Intel Xeon 3.6 GHz CPUs. Results are also shown for a simulation setup with an innovative cold plate that provides liquid cooling. <s> BIB037 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> With the growing importance of the cloud computing paradigm, it is a challenge for cloud providers to keep the operational costs of the data centers in check, especially in the emerging markets, alongside catering to the customers' needs. It becomes essential to increase the operational efficiency of the data centers to be able to maximize VM (Virtual machine) offerings at minimal cost. To that end, energy-efficiency of the servers plays a critical role, as they influence the electrical and the cooling costs which constitute a major part of the total cost involved in the operation of a data center. Power-savings can be achieved at several different levels in a system: processors, memory, devices, and system-wide (involving powering down multiple components of a host all at once). At the processors level, depending on the workload trends, we can exploit technologies like DVFS (Dynamic Voltage and Frequency Scaling) or P-states when the CPU is running, and CPU sleep states (C-states) when the CPU is idle, to save power. Memory standards such as DDR3 have provisions for putting idle memory banks into low-power states. At the devices level, individual devices can be put into low-power states, controlled and co-ordinated by a run-time power management framework in the Operating System. This paper outlines the state-of-the-art in power- management technology on server hardware and describes how these raw features can be abstracted into a set of energy policies. We then explain how these policies or energy-profiles can be used to run a cloud datacener energy efficiently. Further, this paper also highlights some of the challenges involved in running cloud infrastructures in the emerging markets optimally despite some unique energy constraints. <s> BIB038 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> With a growing concern on the considerable energy consumed by data centers, research efforts are targeting toward green data centers with higher energy efficiency. In particular, server virtualization is emerging as the prominent approach to consolidate applications from multiple applications to one server, with an objective to save energy usage. However, little understanding has been obtained about the potential overhead in energy consumption and the throughput reduction for virtualized servers in data centers. In this research, we take the initiative to characterize the energy usage on virtualized servers. An empirical approach is adopted to investigate how server virtualization affects the energy usage in physical servers. Through intensive data collection and analysis, we identify a fundamental trade-off between the energy saving from server consolidation and the detrimental effects (e.g., energy overhead and throughput reduction) from server virtualization. This characterization lays a mathematical foundation for server consolidation in green data center architecture. <s> BIB039 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Nowadays, the cloud computing owners lack management and monitoring tools to ensure the performance, robustness, dependability, and security. To address this limitation, this paper described our experience with a lightweight monitoring framework using some extra development work. This framework performed end-to-end measurements at virtual machine instances and software in the public cloud. It monitors quality of service parameters of the IaaS and SaaS layer without modifying the implementation of the monitored object. In addition, we discussed the manager-agent and module-centralized architecture in details. All the modules make up the entire proposed framework to improve the performance of applications in public clouds. <s> BIB040 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Abstract Data centers can lower costs significantly by provisioning expensive electrical equipment (such as UPS, diesel generators, and cooling capacity) for the actual peak power consumption rather than server nameplate power ratings. However, it is possible that this under-provisioned power level is exceeded due to software behaviors on rare occasions and could cause the entire data center infrastructure to breach the safety limits. A mechanism to cap servers to stay within the provisioned budget is needed, and processor frequency scaling based power capping methods are readily available for this purpose. We show that existing methods, when applied across a large number of servers, are not fast enough to operate correctly under rapid power dynamics observed in data centers. We also show that existing methods when applied to an open system (where demand is independent of service rate) can cause cascading failures in the software service hosted, causing the service performance to fall uncontrollably even when power capping is applied for only a small reduction in power consumption. We discuss the causes for both these short-comings and point out techniques that can yield a safe, fast, and stable power capping solution. Our techniques use admission control to limit power consumption and ensure stability, resulting in orders of magnitude improvement in performance. We also discuss why admission control cannot replace existing power capping methods but must be combined with them. <s> BIB041 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> As processor and memory system speeds have significantly diverged, system designers have introduced ever larger caches in an effort to supply the processor with data at a rate it is capable of processing it. However, application characteristics vary and not all programs can effectively utilize large caches due to their inherent data reuse properties. The inability to use all the available cache capacity leads to wasted cache power dissipation. The rising specter of "dark silicon" makes it critical to avoid wasted power on a chip.In this paper, we develop a compile-time approach to analyze data reuse characteristics of affine computations and deduce the useful cache size(s) for a given system configuration. The non-useful cache can be power-gated to save power. Analysis of benchmarks shows that significant fractions of the last level cache of current processors may be turned off with no performance loss. <s> BIB042 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> Scientific community is one of the major driving forces for developing and utilizing IT technologies such as Supercomputers and Grid. Although, the main race has always been for bigger and faster infrastructures, an easier access to such infrastructures in recent years created a demand for more customizable and scalable environments. However, introducing new technologies and paradigms such as Cloud computing requires a comprehensive analysis of its benefits before the actual implementation. In this paper we introduce the TimeCap, a methodology for comparing IT infrastructures based on time requirements and resource capacity wastage. We go beyond comparing just the execution time by introducing the Time Complexity as part of TimeCap, a methodology used for comparing arbitrary time related tasks required for completing a procedure, i.e., obtaining the scientific results. Moreover, a resource capacity wastage is compared using the Discrete Capacity, a second methodology as part of TimeCap used for analyzing resource assignment and utilization. We evaluate our methodology by comparing a traditional physical infrastructure and a Cloud infrastructure using our local IT resources. We use a real world scientific application for calculating plasma instabilities for analyzing time and capacity required for computation. <s> BIB043 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> The popularization of cloud computing brings emergency concern to the energy consumption in big data centers. Besides the servers, the energy consumed by the network in a data center is also considerable. Existing works for improving the network energy efficiency are mainly focused on traffic engineering, i.e., consolidating flows and switching off unnecessary devices, which fails to comprehensively consider the unique features in data centers. In this paper, we advocate a joint optimization for achieving energy efficiency of data center networks by proposing a unified optimization framework. In this framework, we consider to take advantage of the application characteristics and topology features, and to integrate virtual machine assignment and traffic engineering. Under this framework, we then devise two efficient algorithms, TE VMA and TER, for assigning virtual machines and routing traffic flows respectively. Knowing the ncommunication patterns of the applications, the TE VMA algorithm is purposeful and can generate desirable traffic conditions for the next-step routing optimization. The TER algorithm makes full use of the hierarchical feature of the topology and is conducted on the multipath routing protocol. The performance of the overall framework is confirmed by both theoretical analysis and simulation results, where up to 50% total energy savings can be achieved, 20% more compared with traffic engineering only approaches. <s> BIB044 </s> Cloud Computing: Survey on Energy Efficiency <s> Context <s> With the introduction of femtocells, cellular networks are moving from the conventional centralized network architecture to a distributed one, where each network cell should make its own radio resource allocation decisions, while providing inter-cell interference mitigation. However, realizing such distributed network architecture is not a trivial task. In this paper, we first introduce a simple self-organization rule, based on minimizing cell transmit power, following which a distributed cellular network is able to converge into an efficient resource reuse pattern. Based on such self-organization rule and taking realistic resource allocation constraints into account, we also propose two novel resource allocation algorithms, being autonomous and coordinated, respectively. Performance of the proposed self-organization rule and resource allocation algorithms are evaluated using system-level simulations, and show that power efficiency is not necessarily in conflict with capacity improvements at the network level. The proposed resource allocation algorithms provide significant performance improvements in terms of user outages and network capacity over cutting-edge resource allocation algorithms proposed in the literature. <s> BIB045
The energy consumption of the Network domain consists of three main systems: the connections inside of a data center, the fixed network between data centers, and the end user network that increasingly provides the wireless last hop to end users who access services via smartphones, tablets, and laptops. Based on this breakdown, each system brings its own energy wastes and losses, as shown in Figure 5 . -DCN Data center network (N-1): Within a data center, the energy consumption of the network currently accounts for up to 5% of its total energy consumption [Fanara 2007] . As shown by BIB023 , network power accounts for approximately 20% of the total power when the servers are utilized at 100%. However, it goes up to 50% when utilization of servers decreases to 15%. This is due to the fact that many new cloud applications heavily rely on fast connectivity within a data center. As a result, this leads to an increasing share in energy consumption for these networks. : Connections between data centers are important for applications that run on a global scale, where instances that serve individual users are located in the data center closest to the end user but still need to communicate between each other. Another application of these networks is to migrate applications or data between data centers depending on the time of day, to minimize delay, energy consumption, or costs. As observed by BIB044 , this communication includes background, noninteractive, and bulk data transfers. -End user network (N-3): A connection to an end user who is accessing cloud services is usually made through a combination of wired and wireless networks. Since an increasing number of users access these services via mobile devices, the last hop of the connection is increasingly made through a wireless connection. Recent traffic predictions show that, compared to other kinds of network traffic, wireless traffic is increasing at the highest rate BIB028 , indicating that this trend will continue in the future. The wireless connection is significantly less energy efficient due to the high path loss, interference, and high processing involved in detection and error correction BIB001 , which all represent energy consumption overhead created by supporting tasks rather than the main task (i.e., data delivery). To reduce energy loss and waste, a number of actions can be taken to achieve the goals defined in the Section 2. These actions include: -L1. Reducing the heat load of network equipment inside a data center (N-1) would reduce its energy consumption and the consumption of its cooling subsystem as well. This can be achieved by adapting the network equipment design suggested by ASHRAE [2012] implementing front to rear air flow. This would also increase its reliability by a factor of 2. -L2. Goal L1 also brings benefits to goal L2: by reducing heat load, a smaller cooling subsystem can be installed, which consumes less energy. Although it comprises basic network equipment, failure handling supported by redundant equipment can also be considered a subsystem because it does not perform the main task of the system. Therefore, moving away from a traditional 2N tree topology toward the more flexible topologies currently being adopted by new data centers, such as Fat-Tree BIB009 , BCube BIB016 ], and DCell BIB010 , can provide benefits in terms of improved energy-efficient traffic management. -W1. Today's network equipment is not energy proportional, and simply turning on a switch can consume over 80% of its max power BIB017 ]. By implementing power saving modes BIB008 BIB024 BIB035 , rate adaptation BIB045 BIB011 , or simply turning off unused ports, links, and switches inside a data center (N-1) BIB025 would reduce idle energy consumption and therefore achieve this goal. In addition to tweaking communication equipment, utilizing more energy-efficient network topologies can also reduce power consumption BIB023 BIB029  ]. For D2D networks (N-2), solutions such as NetStitcher BIB030 can reduce network idle time by using unutilized network bandwidth for bulk transfers between data centers or exploiting the benefits of different data rates BIB031 ]. -W2. Achieving this goal depends mostly on how a network is used by the servers, as well as the Software and User domains. However, some optimization can still be done by observing communication patterns and reducing unnecessary traffic. Such an approach combines network traffic engineering and VM assignment  , as well as application profiling for network traffic BIB036 . In a perfect data center, the Server domain, along with the Network domain, would consist only of hardware equipment that consumes energy. Therefore, an obvious goal of every data center owner is to reduce the consumption of all supporting hardware equipment because it represents an energy loss. However, energy loss and waste do not stop there since servers can also contribute to energy waste due to poor server equipment usage policy, as well as energy loss due to a poor energy supply and internal subsystems. As shown in Figure 6 , systems of the Server domain include the server enclosure, such as server cabinets. Server racks represent another system, and components within a rack, such as CPU, memory, hard-disk, and the like, are the third system. -Enclosure (S-1): Enclosures may differ depending on the type of cooling applied to a data center. The most common air-based cooling, based on Computer Room Air Conditioners (CRACs), requires enclosures to have air inlets and outlets on opposite sides. The second type of cooling is indirect liquid cooling. Chilled water is delivered to the enclosure where it is used to absorb heat from the air that is used to cool servers. The enclosure can contain a closed loop of air or implement rear-door (or side-door) cooling, in which the cooled air is pushed back into the server room. Finally, direct liquid cooling solutions have been recently gaining interest BIB037 ]. This type of cooling is particularly efficient for powerful and heavily loaded servers, as in High Performance Computing (HPC) applications; however, it may be also useful for cloud infrastructures. In enclosures with direct liquid cooling, warm water is used to cool server components directly, most commonly through the use of cold plates  ] or microchannels BIB018 . Recently, other approaches based on the immersion of the whole server in a dielectric fluid have emerged (e.g., Iceotope system [Icetope 2013]) . Liquid cooling approaches provide significant energy savings (up to around 40% compared to air-based cooling), but have an impact on hardware cost, complexity, and compatibility with other equipment. -Racks (S-2): The idle power consumption of a server can be more than 50% of its peak power consumption BIB032 . Moreover, "most servers consume between 70 and 85 percent of full operational power"  ], which certainly does not represent a proportional increase of energy consumption with respect to system output. Consequently, "a facility operating at just 20 percent capacity may consume 80 percent of the energy as the same facility operating at 100 percent capacity"  ]. Additionally, this includes a huge energy waste by running servers idle without any useful output or with low utilization in the 10-50% utilization range, which is usually the case in typical data centers  . Finally, racks containing components that are not used at all (e.g., graphics cards) also contribute to energy loss. Another source of energy loss is fans, which have typical efficiency of around 60% (i.e., around 40% of power is lost due to heat dissipation). Additionally, if fan speed is not well adjusted to server load and temperature, a significant amount of energy is wasted. -Components (S-3): The energy efficiency of server components drastically affects the overall efficiency of a server. Most focus on components that take a bigger slice of the total energy consumption, such as the CPU, which can consume more than a third of total server energy consumption [Fan et al. 2007 ]. The typical Thermal Design Power (TDP) of today's processors can fall in the range of 80W-103W or 91 W on average  ]. However, this power is not proportional to its output. As a rule of thumb, CPU power increases by approximately k 2 when CPU frequency increases by k BIB026 . Practical experiment results are given in BIB032 , where a VM utilized 100% of a single physical core while consuming 26W. On the other hand, when a VM with the same performance ran on two physical cores, each being 50% utilized, it consumed only 17W. However, servers with large number of slower CPU cores can lead to lower utilization (i.e., a binpacking problem in which smaller bins cause a bigger bin-packing problem BIB026 ). In addition to underutilized CPU cores that affect dynamic power consumption, caches can also be poorly used or underutilized, which adds to the static power consumption of a processor. Since cache takes more than 40% of the processor die area BIB012 , it can significantly increase static power consumption, which in modern processors accounts for 20-40% of total power consumption BIB013 . Memory also creates energy overheads since it is built to provide high performance to meet increasing CPU demands and thus has grown in density, functionality, and scalability BIB019 ]. This has resulted in neglecting the energy efficiency of the memory subsystem. Finally, the disk system has proved to be another power drain that can generate an energy cost of as much as 25% annually while also occupying up to 75% of the floor space in a data center BIB014 ]. To mitigate all this energy loss and waste, a number of actions can be performed. Following our approach from Section 2, these actions include: -L1. Reducing the heat load of server components such as the CPU fulfils this goal. This can be achieved by using more energy-efficient components and their architectures; for example, using slower, so-called wimpy CPU cores that are more power efficient BIB026 , as in the FAWN project BIB033 where they utilize wimpy cores to build an energy-efficient key-value storage system. Another recognized approach is limiting input energy to a specific component (S-1) or an entire rack (S-2), also referred to as power capping BIB041 . Similarly, in the case of the memory subsystem, performance can be adjusted (i.e., throughput is used to mitigate high temperatures) and thus avoid energy loss through heat  . Energy loss can also be reduced by using compact server configurations that exclude components that are not used (e.g., Google uses such an approach in building its data centers). -L2. Following goal L1, goal L2 provides additional energy savings by reducing energy consumed by supporting systems, such as cooling and power supplies inside the server enclosure (S-3) and the servers themselves (S-2). For example, Google places backup batteries next to racks, thereby avoiding large UPS units that require their own cooling systems  . With this approach, goal L1 is also achieved since large UPS units leak electricity due to their low efficiency BIB007 . In addition to cooling and power supply systems, during idle run, subsystems such as cache can be turned off on most modern processors that employ more sophisticated hardware BIB038 ]. -W1. Using components that can automatically scale their power consumption based on current load would move toward achieving this goal; for example, using dynamic voltage and frequency scaling (DVFS)-capable CPUs that provide different P-states (power modes while being utilized), as well as sleep C-states (power modes while idle). BIB038 provide an overview of these techniques along with power capping. The same applies for other components, such as memory and storage disks, which can be put into a low power state while idle. However, this is beneficial only when there are frequent idle periods; otherwise, this can create even bigger power consumption overheads due to spin-up in the case of storage disks BIB014 . Using DRPM BIB003 ] with dynamic spin speed may represent a more flexible solution for gradually scaling the performance of a disk. Other energysaving techniques include MAID BIB002 , BUD  , EERAID BIB004 , and PDC BIB005 . Choosing the right processor architecture can also contribute to more efficient energy usage. Due to the nature of applications running in a cloud (e.g., web search, video hosting, and MapReduce), emphasis is placed on parallelism and thus on multicore processors instead of high-speed single-core processors BIB026 . However, using single-threaded operations still beats multithreaded operations on slower CPUs due to the higher software development and optimization costs BIB026 . Therefore, optimization should also be done in the Appliance domain to develop middleware for transparent workload parallelization. -W2. As shown by BIB042 , bigger cache size does not necessarily mean a lower miss rate. Therefore, choosing the right size cache can decrease energy waste and achieve this goal. Additionally, using cache subsystems for storage disks to reduce reads and writes from/to the disk and increase its idle time can also contribute to energy savings BIB014 . Such onboard controller caches can already be found on modern hardware. The CMS domain includes the scheduler, monitoring system, virtualization technology, and all other software components responsible for managing physical and virtual machines within a cloud (e.g., OpenStack  and Xen hypervisor [Citrix 2012]) . A scheduler's main function is to deploy resources for fulfilling customer requests. Its supporting task is providing a monitoring system that gives additional information about allocated and available resources, such as utilization, QoS, and the like. Additionally, virtualization technology is used for better resource management and on-demand deployment and offers high scalability for cloud infrastructures. Based on this, the energy efficiency of the CMS can be examined through its component systems and includes both energy loss and waste, as shown in Figure 7 . -Virtualization (C-1): Virtualization technology provides an additional infrastructure layer on top of which multiple VMs can be deployed BIB006 . Although virtualization technology can improve resource utilization BIB043 , it also consumes resources and thus creates an energy consumption overhead, mostly through a hypervisor BIB039 . As reported in Jin et al. [2012], a hypervisor based on full virtualization (i.e., KVM [RedHat 2012]) creates much higher overhead (11.6%) than one based on paravirtualization (0.47%; e.g., Xen [Citrix 2012] ) and as opposed to using physical machines. Additionally, overly large VM images are sources of additional losses (e.g., too large memory allocation and storage size). -Monitoring system (C-2): The monitoring system provides information used for managing an infrastructure and providing QoS . However, gathering monitoring metrics consumes resources (e.g., monitoring agents and probes) and thus creates an energy consumption that is considered a loss according to the model represented in Section 2. This can be due to cumbersome monitoring systems whose monitoring agents are heavyweight processes that consume lots of memory and CPU power even when idle.  give an overview of commercial and open-source monitoring tools, as well as monitoring systems in general. Database storage for metric values is another example of memory cluttered with a huge amount of unused data. -Scheduler (C-3): Managing cloud resources should not be overdone; for example, re-scheduling VMs every couple of minutes might give optimal deployment at the moment, but the rescheduling itself would probably consume more energy than it saves. Furthermore, migrations can lead to performance overhead as well as energy overhead. Although a performance loss can be avoided by using live migrations of VMs BIB020 ], the resulting energy overhead is often overlooked. When migrating a VM from one node to another, both nodes must be powered on until the migration is complete BIB027 . This includes both time and energy overheads for the migration, which is only rarely considered in the literature in the context of job placement BIB021 ]. In addition to these issues, H-0 in Figure 7 represents energy delivered to hardware equipment that was not fully utilized by the CMS domain (e.g., idle machines). Although H-0 can be directly related to the Hardware domain, it can also be minimized from the CMS perspective, for example, by consolidating underutilized machines and turning off idle ones . To reduce these wastes and loses, a number of actions can be taken, according to the goals defined in Section 2: -L1. Goal L1 can be achieved during the development phase of the CMS by implementing functions that can directly control hardware equipment because the CMS has "knowledge" of which resources are required and which are not (e.g., shutting down idle machines [Borgetto et al. 2012b] ). The CMS can go beyond controlling only servers to expand its control to the network system or even to the cooling and power supply systems BIB034 . In this way, energy delivered to hardware equipment that is not utilized by the CMS (H-0) could be significantly reduced. -L2. To meet goal L2, the CMS should use lightweight supporting subsystems, such as monitoring (C-2) and virtualization (C-1) technologies, and avoid cumbersome systems that provide large numbers of functionalities that are not utilized by the cloud manager. This includes lightweight monitoring systems BIB040 ] and the selection of appropriate virtualization technology, namely, full-virtualization vs. para-virtualization or even microkernel architectures BIB022 ]. -W1. Running the CMS supporting systems idle still consumes resources and therefore wastes energy (C-1 and C-2). For this reason, CMS subsystems should be implemented in a modular fashion, in which modules are loaded only when they are actually required (e.g., the monitoring agent that loads plugins for initialized metrics and removes them once they are no longer required  ). This also includes minimizing resource consumption while running in idle mode (e.g., using lightweight hypervisors). -W2. CMS system energy waste (C-3) can be avoided by optimizing the scheduler and measuring not only its results, but also its tradeoffs for achieving those results (e.g., how much resources a single scheduling action takes and how many actions are taken). This includes optimization of the scheduling algorithm and technology used for its implementation. The Appliance has a relatively smaller impact on the overall energy consumption than some other elements of the cloud infrastructure such as servers. On the other hand, appliances are responsible for the useful work, which is ultimately delivered to users. Hence, to adequately assess and manage the energy efficiency of the cloud, appliances must be taken into consideration. Three subsystems can be recognized in the appliance: an application that is used by the end user and that performs a main task of the appliance, a runtime environment required for running the application, and an operating system, which serves as a bridge between the physical or virtual machine and the software running on top of it. The energy efficiency of appliances affects both energy loss and waste according to the model presented in Section 2; these are shown in Figure 8 . -Application (A-1): The application is the core part of the software appliance. There are different types of applications used in clouds. One of the most common types is a portal delivering Web content to end users. Other typical applications include databases and Web services. Some of these applications may even include large distributed computations, graphical rendering, simulations, and complex workflows hidden behind cloud Web interfaces. A well-known example of more advanced processing is MapReduce BIB015 . An application provides the core functionality to a user. Nevertheless, even on this level, losses and wastes of energy may take place. First, energy is usually consumed by additional components of the application. These modules are integral parts of the application, but they are also responsible for its nonfunctional aspects, such as security, reliability, control, logging, and the like. When the appliance is not strongly utilized by end users, these modules can be a source of energy consumption unrelated to any useful work. Energy is also consumed by supporting subsystems that are responsible for the maximization of appliance utilization and for dynamic adaptation of the appliance to load (according to goal W2). These subsystems are needed to optimize appliance efficiency (e.g., by stopping some services in the case of low load), but energy used by them is wasted because it is not used to deliver the core functionality of the appliance. Finally, part of the energy consumed by the application is not fully utilized by users. Energy can be consumed by the application running threads, processes or services, and allocated data structures in memory and on hard disks without producing any output. For example, if a lower number of Web servers is sufficient for end users, then any additional servers waste energy. -Runtime environment (A-2): Applications usually need a runtime environment to be executed from Java VMs or software-interpreting script languages such as Python, through Web servers such as Apache or Tomcat, and on to more complex systems. From the perspective of the appliance main task, energy consumed by the runtime environment is a loss that should be minimized (goal L2). Loss can be also caused by runtime environment overheads (e.g., programming languages or lack of optimization for given application type and hardware). -Operating system (A-3): Both application and runtime environments must be executed on top of an operating system. The operating system can be an off-the-shelf system or a specific distribution tailored to appliance needs. Again, energy consumed purely by the operating system is a loss from the perspective of the appliance's main task. Especially heavy operating systems whose majority of functionality is not used by the appliance results in significant overheads (e.g., related to OS services, maintenance tasks, etc.). In addition to these listed issues from the Appliance perspective, energy spent for running the CMS (C-0) is considered as entirely lost because it performs supporting tasks rather than the main task of the appliance. A number of actions can be taken to reduce these energy losses and wastes according to the goals defined in Section 2. These goals with regards to appliances are as follow: -L1. Proper implementation of cloud appliances can help to reduce energy losses. This can be done during the development phase by optimizing the implementation, as well as by using lightweight programming languages and only required libraries (A-2). The first step to achieve this goal is the use of a fine-grained monitoring and estimation of power usage in order to identify processes responsible for high energy consumption. -L2. Although the Appliance subdomain represents IT software equipment, it can still have supporting systems that cause energy losses by creating a resource consumption overhead (e.g., a small application running on a heavy operating system) while using only a small percentage of its functions (A-3). Therefore, goal L2 includes reducing energy losses by proper implementation of applications and selection of an appropriate underlying technology. This should also include the use of reduced and customized operating systems and runtime environments. Similarly, as in the case of L1, precise information about the parts of a software responsible for high energy consumption must be identified to apply appropriate optimization. -W1. Optimization of the appliance can reduce energy consumption by decreasing its resource usage or increasing its performance. Such an approach targets a goal (W1) of trying to reduce resource consumption while performing the same task. Applied techniques can focus on smart appliance management by switching off or reducing specific functionality in case of low or no load (A-1). To achieve low wastes, decisions should take into account available hardware so that the number of threads/processes or internal load balancing are optimized with energy efficiency in mind. Additionally, to meet goal W1, appliances need to be highly scalable in order to fully utilize available resources. For example, they should provide performance proportional to the consumed energy by scaling to a high number of cores, big cache sizes, high CPU frequency, and more. Otherwise, use of these resources should be treated as energy waste because they cause higher power usage without a proportional increase of performance. In the worst case, many cores and a large amount of memory can be allocated to an appliance that produces very little useful work. -W2. Minimizing the unnecessary use of the appliance depends on the way users access it. Any smart functions applied must avoid breaking SLAs set with users. However, even with these constraints, a number of actions can be taken to reduce useless energy consumption. These techniques can include serving requests in batches, reducing the numbers of backups and checkpoints, limiting the number of service instances or threads, and adjusting the frequency of monitoring, polling, caching, and indexing. Overheads can be also related to the relevant functionality of appliances (e.g., security and resilience). Hence, applying most of these techniques requires finding a tradeoff between energy efficiency and other key aspect of appliances, such as performance, resilience, and security.