Inter-Block Multi-Erasure Coding Scheme for Cloud-Based Big Bulk Data Transmission

Abstract

Big data is one of the hottest topics in the area of Information Technology. The intersection between cloud computing and big data actually brings a lot of challenges to both the academia and the industry. Aiming at the GPU Clusters based cloud computing applications, high-speed big bulk data transmission usually becomes incorrect, leading to incorrect computations and communication. Erasure codes have been widely used to protect big data against errors or erasure during reliable transmission and storage by providing redundancy in big data storage and transmission in cloud systems. Although, most of the existing erasure codes designed originally for bit or symbol are based on the theory of Reed-Solomon codes. Due to the complexities of both encoding and decoding, they are not suitable to tolerate failures with respect to an entire big bulk data block transmission in a real-time. Therefore, we propose an effective inter-block multierasure coding scheme (IMEC), designed for big bulk data transmission of cloud computing applications based on our prior research, with relative simple encoding and decoding procedures. IMEC extends the traditional paritycheck codes, incorporating with the features of GF(2^8) field in Galois field, which can tolerate simultaneous failures (or losses) of 4 entire data blocks with only 4 redundancy data blocks for a group of continuous data and its encoding and decoding complexities are simple linear relationship with respect to the size of a data block. Furthermore, the IMEC scheme exhibits maximum distance separable (MDS) property, featuring an optimal erasure capability with respect to the same redundancy information.