Computer science is experiencing an AI renaissance, in which machine learning models are expediting important breakthroughs in academic research and commercial applications. Effective training of these models, however, is not trivial due in part to hyperparameters: user-configured values that parametrize learning models and control their ability to learn from data. Existing hyperparameter optimization methods are highly parallel but make no effort to balance the search across heterogeneous hardware or to prioritize searching high-impact spaces. In this paper, we introduce a framework for massively Scalable Hardware-Aware Distributed Hyperparameter Optimization (SHADHO). Our framework calculates the relative complexity of each search space and monitors performance on the learning task over all trials. These metrics are then used as heuristics to assign hyperparameters to distributed workers based on their hardware. We demonstrate that our framework scales to 1400 heterogeneous cores and that it achieves a factor of 1.6 speedup in the necessary time to find an optimal set of hyperparameters over a standard distributed hyperparameter optimization framework.
Bag-of-Tasks (BoT) applications consisting of multiple tasks widely exist in numerous fields. As customers use cloud resources in a pay-as-you-go way, they are willing to execute BoT applications on clouds. When the private cloud has insufficient available resources to afford all tasks, the cloud provider has to outsource some tasks to public clouds with resource-used costs. The key challenge here is how to schedule tasks on hybrid clouds to minimize makespan given a limited budget. We study and formulate this problem as an Integer Programming problem. Accordingly, we propose an effective heuristic (EH) including two phases (task sequencing and task scheduling). EH uses a Longest Task First method (LTF) to generate a task sequence. A Task Assignment method (TA) is established to schedule all tasks in the obtained sequence one by one. Experimental results demonstrate that the proposed EH outperforms the baseline (RoundRobin) significantly. The use of utility on-demand computing infrastructures, such as Amazon's Elastic Clouds [1], is a viable solution to speed lengthy parallel computing problems to those without access to other cluster or grid infrastructures. With a suitable middleware, bag-of-tasks problems could be easily deployed over a pool of virtual computers created on such infrastructures. In bag-of-tasks problems, as there is no communication between tasks, the number of concurrent tasks is allowed to vary over time. In a utility computing infrastructure, if too many virtual computers are created, the speedups are high but may not be cost effective; if too few computers are created, the cost is low but speedups fall below expectations. Without previous knowledge of the processing time of each task, it is difficult to determine how many machines should be created. In this paper, we present an heuristic to optimize the number of machines that should be allocated to process tasks so that for a given budget the speedups are maximal. We have simulated the proposed heuristics against real and theoretical workloads and evaluated the ratios between number of allocated hosts, charged times, speedups and processing times. With the proposed heuristics, it is possible to obtain speedups in line with the number of allocated computers, while being charged approximately the same predefined budget. Bag-of-Tasks (BoT) applications consisting of multiple tasks widely exist in numerous fields. As customers use cloud resources in a pay-as-you-go way, they are willing to execute BoT applications on clouds. Cloud providers and customers establish contracts in which applications' due dates are specified. If an application cannot be finished before the due date, the cloud provider should pay a tardiness penalty. When the private cloud has insufficient available resources to afford all customer-submitted BoT applications, the cloud provider has to outsource some tasks to public clouds with resource-used costs. The key challenge here is how to schedule tasks on hybrid clouds to minimize the total cost, including all applications' tardiness penalties and the cost of using public clouds' resources. We study and formulate this problem as an Integer Programming. Accordingly, we propose an effective greedy heuristic (GH) including two phases (task ordering and task scheduling). GH uses an Earlier Latest Start Time First method (ELSTF) for task ordering with the result that a task sequence is obtained. A Task Dispatching method (TD) is established for the task scheduling, in which each task in the obtained task sequence is scheduled one by one. Experimental results demonstrate that the proposed GH outperforms the baseline (RoundRobin) remarkably. ELSTF and TD are also verified to be effective.