Joint Online Optimization of Task Rescheduling and Data Redistribution

Abstract

Wide-area distributed computing environment is the main platform for storing large amounts of data and conducting wide-area computing. Tasks and data are jointly scheduled among multiple computing platforms to improve system efficiency. However, large network latency and limited bandwidth in wide-area networks may cause a large delay in scheduling information and data migration, which brings low task execution efficiency and a long time waiting for data. Traditional works mainly focus on allocating tasks based on data locality or distributing data replications, but optimizing task allocation or data placement alone is insufficient from a global perspective. To mitigate the impact of large network latency and limited bandwidth on system performance, joint online optimization of task rescheduling and data redistribution is proposed in this study. The task allocation and data placement can be adjusted collaboratively during the system running process through the task stealing and backfilling mechanism and the data replication placement mechanism. The experimental results indicate that compared with the state-of-the-art method, the proposed method improves the system throughput and computing resource utilization by 20.67% and 20.26% respectively, and can significantly reduce the global data migration costs.