In this paper, we design an analytically and experimentally better online energy and job scheduling algorithm with the objective of maximizing net profit for a service provider in green data centers. We first study the previously known algorithms and conclude that these online algorithms have provable poor performance against their worst-case scenarios. To guarantee an online algorithm's performance in hindsight, we design a randomized algorithm to schedule energy and jobs in the data centers and prove the algorithm's expected competitive ratio in various settings. Our algorithm is theoretical-sound and it outperforms the previously known algorithms in many settings using both real traces and simulated data. An optimal offline algorithm is also implemented as an empirical benchmark.
Batched stream processing is a new distributed data processing paradigm that models recurring batch computations on incrementally bulk-appended data streams. The model is inspired by our empirical study on a trace from a large-scale production data-processing cluster; it allows a set of effective query optimizations that are not possible in a traditional batch processing model. We have developed a query processing system called Comet that embraces batched stream processing and integrates with DryadLINQ. We used two complementary methods to evaluate the effectiveness of optimizations that Comet enables. First, a prototype system deployed on a 40-node cluster shows an I O reduction of over 40 using our benchmark. Second, when applied to a real production trace covering over 19 million machine-hours, our simulator shows an estimated I O saving of over 50 . Interest has been growing in powering data centers (at least partially) with renewable or "green" sources of energy, such as solar or wind. However, it is challenging to use these sources because, unlike the "brown" (carbon-intensive) energy drawn from the electrical grid, they are not always available. In this keynote talk, I will first discuss the tradeoffs involved in leveraging green energy today and the prospects for the future. I will then discuss the main research challenges and questions involved in managing the use of green energy in data centers. Next, I will describe some of the software and hardware that researchers are building to explore these challenges and questions. Specifically, I will overview systems that match a data center's computational workload to the green energy supply. I will also describe Parasol, the solar-powered micro-data center we have just built at Rutgers University. Finally, I will discuss some potential avenues for future research on this topic.