Hierarchical Self-Tuning of Concurrency and Resource Units in Data-Analytics Frameworks

TitleHierarchical Self-Tuning of Concurrency and Resource Units in Data-Analytics Frameworks
Publication TypeConference Paper
Year of Publication2017
AuthorsLee, G, Fortes, JAB
Conference NameInternational Conference on Autonomic Computing (ICAC)
Date Published07/2017
Conference LocationColumbus, OH, USA
Keywordsautonomic control, data-analytics frameworks, fuzzy control, Hadoop, job completion time, makespan, MapReduce, Performance tuning, resource utilization, self-tunin
AbstractMany big-data processing applications use dataanalytics frameworks such as Apache Hadoop. Such frameworks have tunable configuration parameters set by experienced system administrators and/or application developers. However, tuning parameters manually can be hard and time-consuming because it requires domain-specific knowledge and understanding of complex inter-dependencies amongst parameters. Most of the frameworks seek efficient resource management by using slots or containers as resource units to be assigned to jobs or tasks, the maximum number of slots or containers in a system being part of the static configuration of the system. This static resource management has limited effectiveness in coping with jobs’ diversity and workload dynamics, even in the case of a single job. Seeking to improve performance (e.g., multiple-jobs makespan and job completion time) without modification of the framework, this paper proposes a hierarchical approach using a fuzzy-logic controller to dynamically adjust the number of concurrent jobs and additional controllers (one for each cluster node) to adjust the number of resource units assigned to jobs on each node. The fuzzylogic controller uses fuzzy rules based on a concave downward relationship between aggregate CPU usage and the number of concurrent jobs. The other controllers use a simple heuristic algorithm to adjust the number of resource units on the basis of resource usage by job tasks. A prototype of our approach was implemented for Apache Hadoop on a cluster running at CloudLab. The prototype was evaluated using realistic workloads generated by SWIM, a statistical workload injector for MapReduce. The evaluation shows that the proposed approach yields up to a 42% reduction of the jobs makespan that results from using Hadoop default settings.