Dynamic Slot Allocation Optimization Framework for MapReduce Clusters
ARTICLE INFO The Hadoop MRv1 uses the slot based resource model with the static configuration of map/reduce slots. There is a strict utility constrain that map tasks can only run on map slots and reduce tasks can only use reduce slots. Due to the rigid execution order between map and reduce tasks in a MapReduce environment, slots can be severely under utilized, which significantly degrades the performance .In contrast to YARN that gives up the slot based resource model and propose a container based model to maximize the resource utilization via unawareness of the types of map/reduce tasks, we keep the slot based model and propose a dynamic slot utilization optimization system called DynamicMR to improve the performance of Hadoop by maximizing the slots utilization as well as slot utilization efficiency while guaranteeing the fairness across pools. We proposed dynamic slots can be severely under utilized, which significantly degrades the performance. To address proposes a dynamic slot allocation and scheduling system called DynamicMR consisting of three levels of schedulers, i.e. Dynamic Hadoop Fair Scheduler (DHFS) Dynamic Speculative Task Scheduler (DSTS) and Data Locality Maximization Scheduler (DLMS).The experimental results show that our DynamicMR can improve the system performance significantly while guaranteeing the fairness, by up to 32% 40% for a single job and 19% 28% for MapReduce workloads with multiple jobs, in comparison to Hadoop. Keywords Slot Allocation delay scheduler dynamicMR MapReduce hadoop fair sceduler slot preScheduling Article History Received :13 th March 2016 Received in revised form