Feature-Based High Availability Mechanism for Extreme Aggregation Tasks in Real-Time Data Stream Processing

Abstract

Data stream under Cloud or IoT (Internet of Things) implies real-time and continuous processing, during which availability guarantee is essential but expensive. For extreme aggregation tasks, traditional HA mechanisms require vast space at run-time and much longer recovery latency at fail-time especially under worse input. In this paper, feature-based high availability mechanism is proposed for extreme aggregation tasks, in which space-bounded feature is maintained through random sampling over time-based sliding window and failed tasks can be recovered precisely with high probability in an efficient way. The probabilistic effectiveness has been proved theoretically, and meanwhile the acceptable tradeoff between related overheads and performance has been demonstrated by comprehensive experiments on both synthetic and real data.