PyRS: Cross-platform Data Fault-tolerant Storage Library Based on RS Erasure Code

Abstract

Erasure code has been used by more and more researchers to solve the problem of efficient, reliable, and fault-tolerant data storage. However, the existing libraries based on erase code can only run on the Linux platform, and some of them need GPU support. This paper implements a cross-platform data fault-tolerant storage library based on RS erasure code, PyRS, which is running on CPU without GPU support and developed in Python. PyRS uses Vandermonde matrix as the coding matrix and Numba and NumPy libraries to speed up and optimize the program. This paper compares PyRS with Jerasure on the same Linux platform. The results show that the encoding and decoding speed of PyRS is 8 times faster than that of Jerasure. The CPU usage rate of both is about 25% and the memory usage rate of PyRS is about 5% higher than that of Jerasure. The same experiments are carried out on PyRS on the Windows platform. Experimental results show that compared with running on the Linux platform, PyRS running on the Windows platform has almost the same speed of encoding and decoding, and its CPU usage rate increases by about 15%, while its memory usage rate decreases by about 5%.