A Two-Tier Replication Framework and Its Consensus Protocol for Big Data Storage of Internet of Things

Abstract

Big data generated from the Internet of Things (IoT) brings enormous challenges to data management. In this paper, we focus on fault tolerance and replication, especially consistency in IoT big data storage. A two-tier replication framework is presented for replication across multiple data centers in the context of big data storage. A consensus protocol with batching and logical pipelining optimizations, which is called L-Paxos, is introduced to achieve high throughput and to provide good scalability. An analytical model is developed to provide a good approximation to optimal batch sizes when L-Paxos reaches maximum throughput. Finally, we evaluate L-Paxos's performance and validate the analytical model experimentally.