Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> Large-scale Internet services require a computing infrastructure that can beappropriately described as a warehouse-sized computing system. The cost ofbuilding datacenter facilities capable of delivering a given power capacity tosuch a computer can rival the recurring energy consumption costs themselves.Therefore, there are strong economic incentives to operate facilities as closeas possible to maximum capacity, so that the non-recurring facility costs canbe best amortized. That is difficult to achieve in practice because ofuncertainties in equipment power ratings and because power consumption tends tovary significantly with the actual computing activity. Effective powerprovisioning strategies are needed to determine how much computing equipmentcan be safely and efficiently hosted within a given power budget. In this paper we present the aggregate power usage characteristics of largecollections of servers (up to 15 thousand) for different classes ofapplications over a period of approximately six months. Those observationsallow us to evaluate opportunities for maximizing the use of the deployed powercapacity of datacenters, and assess the risks of over-subscribing it. We findthat even in well-tuned applications there is a noticeable gap (7 - 16%)between achieved and theoretical aggregate peak power usage at the clusterlevel (thousands of servers). The gap grows to almost 40% in wholedatacenters. This headroom can be used to deploy additional compute equipmentwithin the same power budget with minimal risk of exceeding it. We use ourmodeling framework to estimate the potential of power management schemes toreduce peak power and energy usage. We find that the opportunities for powerand energy savings are significant, but greater at the cluster-level (thousandsof servers) than at the rack-level (tens). Finally we argue that systems needto be power efficient across the activity range, and not only at peakperformance levels. <s> BIB001 </s> Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> Data center power consumption is growing to unprecedented levels: the EPA estimates U.S. data centers will consume 100 billion kilowatt hours annually by 2011. Much of this energy is wasted in idle systems: in typical deployments, server utilization is below 30%, but idle servers still consume 60% of their peak power draw. Typical idle periods though frequent--last seconds or less, confounding simple energy-conservation approaches. In this paper, we propose PowerNap, an energy-conservation approach where the entire system transitions rapidly between a high-performance active state and a near-zero-power idle state in response to instantaneous load. Rather than requiring fine-grained power-performance states and complex load-proportional operation from each system component, PowerNap instead calls for minimizing idle power and transition time, which are simpler optimization goals. Based on the PowerNap concept, we develop requirements and outline mechanisms to eliminate idle power waste in enterprise blade servers. Because PowerNap operates in low-efficiency regions of current blade center power supplies, we introduce the Redundant Array for Inexpensive Load Sharing (RAILS), a power provisioning approach that provides high conversion efficiency across the entire range of PowerNap's power demands. Using utilization traces collected from enterprise-scale commercial deployments, we demonstrate that, together, PowerNap and RAILS reduce average server power consumption by 74%. <s> BIB002 </s> Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> Data center power infrastructure incurs massive capital costs, which typically exceed energy costs over the life of the facility. To squeeze maximum value from the infrastructure, researchers have proposed over-subscribing power circuits, relying on the observation that peak loads are rare. To ensure availability, these proposals employ power capping, which throttles server performance during utilization spikes to enforce safe power budgets. However, because budgets must be enforced locally -- at each power distribution unit (PDU) -- local utilization spikes may force throttling even when power delivery capacity is available elsewhere. Moreover, the need to maintain reserve capacity for fault tolerance on power delivery paths magnifies the impact of utilization spikes. ::: In this paper, we develop mechanisms to better utilize installed power infrastructure, reducing reserve capacity margins and avoiding performance throttling. Unlike conventional high-availability data centers, where collocated servers share identical primary and secondary power feeds, we reorganize power feeds to create shuffled power distribution topologies. Shuffled topologies spread secondary power feeds over numerous PDUs, reducing reserve capacity requirements to tolerate a single PDU failure. Second, we propose Power Routing, which schedules IT load dynamically across redundant power feeds to: (1) shift slack to servers with growing power demands, and (2) balance power draw across AC phases to reduce heating and improve electrical stability. We describe efficient heuristics for scheduling servers to PDUs (an NP-complete problem). Using data collected from nearly 1000 servers in three production facilities, we demonstrate that these mechanisms can reduce the required power infrastructure capacity relative to conventional high-availability data centers by 32% without performance degradation. <s> BIB003 </s> Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> Power costs are very significant for data centers. To maximally utilize the provisioned power capacity, data centers often employ over-subscription, that is, the sum of peak consumptions of individual servers may be greater than the provisioned capacity. Power budgeting methods are employed to ensure that actual consumption never exceeds capacity. However, current power budgeting methods enforce capacity limits in hardware and are not well suited for virtualized servers because the hardware is shared among multiple applications. We present a power budgeting system for virtualized infrastructures that enforces power limits on individual distributed applications. Our system enables multiple applications to share the same servers but operate with their individual quality of service guarantees. It responds to workload and power availability changes, by dynamically allocating appropriate amount of power to different applications and tiers within applications. The design is mindful of practical constraints such the data center's limited visibility into hosted application performance. We evaluate the system using workloads derived from real world data center traces. <s> BIB004 </s> Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> Datacenter power consumption has a significant impact on both its recurring electricity bill (Op-ex) and one-time construction costs (Cap-ex). Existing work optimizing these costs has relied primarily on throttling devices or workload shaping, both with performance degrading implications. In this paper, we present a novel knob of energy buffer (eBuff) available in the form of UPS batteries in datacenters for this cost optimization. Intuitively, eBuff stores energy in UPS batteries during "valleys" - periods of lower demand, which can be drained during "peaks" - periods of higher demand. UPS batteries are normally used as a fail-over mechanism to transition to captive power sources upon utility failure. Furthermore, frequent discharges can cause UPS batteries to fail prematurely. We conduct detailed analysis of battery operation to figure out feasible operating regions given such battery lifetime and datacenter availability concerns. Using insights learned from this analysis, we develop peak reduction algorithms that combine the UPS battery knob with existing throttling based techniques for minimizing datacenter power costs. Using an experimental platform, we offer insights about Op-ex savings offered by eBuff for a wide range of workload peaks/valleys, UPS provisioning, and application SLA constraints. We find that eBuff can be used to realize 15-45% peak power reduction, corresponding to 6-18% savings in Op-ex across this spectrum. eBuff can also play a role in reducing Cap-ex costs by allowing tighter overbooking of power infrastructure components and we quantify the extent of such Cap-ex savings. To our knowledge, this is the first paper to exploit stored energy - typically lying untapped in the datacenter - to address the peak power draw problem. <s> BIB005 </s> Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> Power over-subscription can reduce costs for modern data centers. However, designing the power infrastructure for a lower operating power point than the aggregated peak power of all servers requires dynamic techniques to avoid high peak power costs and, even worse, tripping circuit breakers. This work presents an architecture for distributed per-server UPSs that stores energy during low activity periods and uses this energy during power spikes. This work leverages the distributed nature of the UPS batteries and develops policies that prolong the duration of their usage. The specific approach shaves 19.4% of the peak power for modern servers, at no cost in performance, allowing the installation of 24% more servers within the same power budget. More servers amortize infrastructure costs better and, hence, reduce total cost of ownership per server by 6.3%. <s> BIB006 </s> Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> Datacenters spend $10-25 per watt in provisioning their power infrastructure, regardless of the watts actually consumed. Since peak power needs arise rarely, provisioning power infrastructure for them can be expensive. One can, thus, aggressively under-provision infrastructure assuming that simultaneous peak draw across all equipment will happen rarely. The resulting non-zero probability of emergency events where power needs exceed provisioned capacity, however small, mandates graceful reaction mechanisms to cap the power draw instead of leaving it to disruptive circuit breakers/fuses. Existing strategies for power capping use temporal knobs local to a server that throttle the rate of execution (using power modes), and/or spatial knobs that redirect/migrate excess load to regions of the datacenter with more power headroom. We show these mechanisms to have performance degrading ramifications, and propose an entirely orthogonal solution that leverages existing UPS batteries to temporarily augment the utility supply during emergencies. We build an experimental prototype to demonstrate such power capping on a cluster of 8 servers, each with an individual battery, and implement several online heuristics in the context of different datacenter workloads to evaluate their effectiveness in handling power emergencies. We show that: (i) our battery-based solution can handle emergencies of short duration on its own, (ii) supplement existing reaction mechanisms to enhance their efficacy for longer emergencies, and (iii) battery even provide feasible options when other knobs do not suffice. <s> BIB007 </s> Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> One of the main driving forces of the growing adoption of virtualization is its dramatic simplification of the provisioning and dynamic management of IT resources. By decoupling running entities from the underlying physical resources, and by providing easy-to-use controls to allocate, deallocate and migrate virtual machines (VMs) across physical boundaries, virtualization opens up new opportunities for improving overall system resource use and power efficiency. While a range of techniques for dynamic, distributed resource management of virtualized systems have been proposed and have seen their widespread adoption in enterprise systems, similar techniques for dynamic power management have seen limited acceptance. The main barrier to dynamic, power-aware virtualization management stems not from the limitations of virtualization, but rather from the underlying physical systems; and in particular, the high latency and energy cost of power state change actions suited for virtualization power management. In this work, we first explore the feasibility of low-latency power states for enterprise server systems and demonstrate, with real prototypes, their quantitative energy-performance trade offs compared to traditional server power states. Then, we demonstrate an end-to-end power-aware virtualization management solution leveraging these states, and evaluate the dramatically-favorable power-performance characteristics achievable with such systems. We present, via both real system implementations and scale-out simulations, that virtualization power management with low-latency server power states can achieve comparable overheads as base distributed resource management in virtualized systems, and thus can benefit from the same level of adoption, while delivering close to energy-proportional power efficiency. <s> BIB008 </s> Green Data Centers: A Survey, Perspectives, and Future Directions <s> C. Power Distribution <s> Power infrastructure contributes to a significant portion of datacenter expenditures. Overbooking this infrastructure for a high percentile of the needs is becoming more attractive than for occasional peaks. There exist several computing knobs to cap the power draw within such under-provisioned capacity. Recently, batteries and other energy storage devices have been proposed to provide a complementary alternative to these knobs, which when decentralized (or hierarchically placed), can temporarily take the load to suppress power peaks propagating up the hierarchy. With aggressive under-provisioning, the power hierarchy becomes as central a datacenter resource as other computing resources, making it imperative to carefully allocate, isolate and manage this resource (including batteries), across applications. Towards this goal, we present vPower, a software system to virtualize power distribution. vPower includes mechanisms and policies to provide a virtual power hierarchy for each application. It leverages traditional computing knobs as well as batteries, to apportion and manage the infrastructure between co-existing applications in the hierarchy. vPower allows applications to specify their power needs, performs admission control and placement, dynamically monitors power usage, and enforces allocations for fairness and system efficiency. Using several datacenter applications, and a 2-level power hierarchy prototype containing batteries at both levels, we demonstrate the effectiveness of vPower when working in an under-provisioned power infrastructure, using the right computing knobs and the right batteries at the right time. Results show over 50% improved system utilization and scale-out for vPower's over-booking, and between 12-28% better application performance than traditional power-capping control knobs. It also ensures isolation between applications competing for power. <s> BIB009
Whereas virtual machine consolidation allows the total number of active physical servers to be reduced, power distribution approaches have also been proposed to reduce the high energy costs of idle systems. Driven by increasing power density, the increasing number of IT devices, and the capability of ondemand addition and removal of IT devices, there have been dramatic changes in how power is utilized in data centers  . In general, power distribution is over-provisioned in data centers to accommodate peaks and to allow for future expansion. However, based on the assumption that simultaneous peak draw across all equipment will occur only rarely, power over-subscription is intentionally exploited to improve power utilization in modern data centers  , BIB001 , BIB003 , BIB004 . Fan et al. BIB001 were the first to study power usage at the scale of data center workloads, and they reported the use of power modeling for power provisioning. Their work yielded several key findings: 1) The gap between the maximum power actually used by large groups of servers and their aggregate theoretical peak usage can be as large as 40% in data centers, which suggests a significant opportunity to host additional servers under the same power budget; 2) power capping is more useful as a safety mechanism to prevent overload situations; 3) large groups of servers are operating near peak power levels in certain time intervals, which suggests that power gaps and power management techniques can be more easily exploited at the data center level than at the rack level; and 4) when large groups of servers are considered, frequency scaling has the potential to be moderately effective at reducing peak power consumption. Their evaluation results indicated energy savings of 35-40%. Meisner et al. BIB002 proposed PowerNap, an energy-conservation approach that supported an entire system in transitioning rapidly between a high-performance active state and a near-zero-power idle state in response to instantaneous variations in load. Based on the PowerNap concept, the authors developed requirements and mechanisms to eliminate idle power waste in enterprise blade servers. The authors also introduced another power provisioning approach named RAILS to be applied when PowerNap was operating in the low-efficiency regions of current blade center power supplies with the intent of providing high conversion efficiency throughout the entire range. They reported that PowerNap and RAILS could reduce the average server power consumption by 74%. To overcome local utilization spikes that could force the throttling of server performance to enforce safe power budgets, Pelley et al. BIB003 developed mechanisms to better utilize the installed power infrastructure. They proposed Power Routing to centrally schedule servers dynamically across redundant power feeds, and they proved that the problem of assigning servers to power distribution units (PDUs) was NPcomplete. The authors first optimally solved the relaxation problem through linear programming, assuming the servers to be assigned fractionally across feeds, and then constructed an approximate solution to the original problem. Using traces from production systems, they demonstrated that capital costs could be reduced by 32% without performance degradation. For the case in which servers are shared by virtual machines belonging to different applications, a power budgeting solution named VPS was presented by BIB004 to guarantee the quality of service. VPS dynamically shifted power among various distributed components as workloads and power availability varied to efficiently utilize the power budget. The authors combined hardware-based and software-based power control knobs to optimize the performance. They used traces to evaluate the following performance metrics: data-center-level and application-level power budgeting errors, application performance differentiation, and performance achieved within the power budget. In BIB008 , the authors explored the feasibility of low-latency power states for enterprise server systems and proposed an end-to-end virtualization power management solution. They implemented the capability of a low-power and low-latency state for virtualization servers and demonstrated this technology could overcome the traditional barriers (e.g., incurred latencies) to achieve substantial energy savings via dynamic consolidation. Based on real-system evaluations, they reported that data center power efficiency could be improved by 30%. In addition to power capping, several studies have introduced Uninterruptible Power Supplies (UPSs) to store energy and reduce power budget violations BIB005 , BIB006 , BIB007 . In BIB005 , the authors pioneered the leveraging of a UPS in a data center as an energy buffer (eBuff) to address the peak power draw problem. They used the energy stored in UPS batteries to provide energy during peak demand, resulting in a reduction in capital expenses and operating expenses without performance degradation. They reported that eBuff yielded a 15-45% peak reduction, which corresponded to a 6-18% savings in operation expenses. Because eBuff was centralized and limited by the capacity of the UPS battery, Kontorinis et al. BIB006 presented an architecture for distributed per-server UPSs that stored energy during low-activity periods and used this energy during power spikes. They proposed details regarding the sizing as well as technological alternatives and approaches that managed battery charging and discharging behavior while addressing reliability and availability concerns. They demonstrated that their policies prolonged the duration of UPS batteries' usage and reduced their total cost of ownership per server by 6.3%. In BIB007 , the authors studied the possibility of leveraging stored energy to address power emergencies. Their work demonstrated that mechanisms for power capping had performance-degrading ramifications, and they proposed an offline theoretical framework and several online heuristics to temporarily augment utility supplies during emergencies. They reported that battery-based solutions 1) could cope with emergencies of short duration, 2) supplemented existing reaction strategies to enhance efficacy for longer emergencies, and 3) provided feasible options when other strategies (power states, migration, etc.) did not suffice. In the case of aggressive over-subscription, whereas the aforementioned work suggested power capping and stored energy to reduce the risk of power peaks, the authors of BIB009 suggested that the power hierarchy should be treated as a data center resource of comparable importance to that of other computing resources. To this end, the authors proposed vPower, a software system to virtualize power distribution. vPower combined computing parameters (such as DVFS, consolidation, etc.) and energy storage devices to establish a virtual power hierarchy for each application. The primary advantages of vPower were that it 1) allowed applications to specify their power needs, 2) performed admission control and assignment, 3) dynamically monitored power usage, and 4) enforced the allocation of power for fairness and system efficiency. The results of a power hierarchy prototype indicated the achieved improvement in system utilization was over 50% compared with the conservative scheduling approach based on peak load requirements.