Network-Aware Task Assignment for MapReduce Applications in Shared Clusters

Abstract

Running MapReduce applications in shared clusters is becoming increasingly compelling to improve the cluster utilization. However, the network sharing across diverse applications can make the network bandwidth for MapReduce applications constrained and heterogeneous, which inevitably increases the severity of network hotspots in racks, and makes the existing task assignment policies that focus on the data locality no longer effective. To deal with this issue, this paper proposes a lightweight networkaware task assignment strategy for MapReduce applications in shared clusters. By analyzing the relationship between job completion time and the assignment of both map and reduce tasks across racks, it devises and integrates two simple yet effective greedy heuristics, which can minimize the completion time of map phase and reduce phase, respectively. With extensive prototype experiments on a 12-node 3-rack MapReduce cluster and complementary large-scale simulations driven by Facebook job traces, we demonstrate that our network-aware strategy can shorten the completion time of MapReduce jobs, in comparison to the state-of-the-art task assignment strategies, yet with an acceptable computational overhead.