Open Access Open Access  Restricted Access Subscription Access

Network-Aware Task Assignment for MapReduce Applications in Shared Clusters

Fei Xu,
Fang-Ming Liu,
Peng Yin,
Hai Jin,

Abstract


Running MapReduce applications in shared clusters is becoming increasingly compelling to improve the cluster utilization. However, the network sharing across diverse applications can make the network bandwidth for MapReduce applications constrained and heterogeneous, which inevitably increases the severity of network hotspots in racks, and makes the existing task assignment policies that focus on the data locality no longer effective. To deal with this issue, this paper proposes a lightweight networkaware task assignment strategy for MapReduce applications in shared clusters. By analyzing the relationship between job completion time and the assignment of both map and reduce tasks across racks, it devises and integrates two simple yet effective greedy heuristics, which can minimize the completion time of map phase and reduce phase, respectively. With extensive prototype experiments on a 12-node 3-rack MapReduce cluster and complementary large-scale simulations driven by Facebook job traces, we demonstrate that our network-aware strategy can shorten the completion time of MapReduce jobs, in comparison to the state-of-the-art task assignment strategies, yet with an acceptable computational overhead.

Keywords


MapReduce; Task assignment; Shared clusters; Network hotspots

Citation Format:
Fei Xu, Fang-Ming Liu, Peng Yin, Hai Jin, "Network-Aware Task Assignment for MapReduce Applications in Shared Clusters," Journal of Internet Technology, vol. 16, no. 2 , pp. 325-333, Mar. 2015.

Full Text:

PDF

Refbacks

  • There are currently no refbacks.





Published by Executive Committee, Taiwan Academic Network, Ministry of Education, Taipei, Taiwan, R.O.C
JIT Editorial Office, Library and Information Center, National Dong Hwa University
No. 1, Sec. 2, Da Hsueh Rd. Shoufeng, Hualien 97401, Taiwan, R.O.C.
Tel: +886-3-931-7017  E-mail: jit.editorial@gmail.com