Hybrid Big Data Architecture for High-Speed Log Anomaly Detection

Abstract

Anomaly detection in network traffic can be very challenging, especially for environments with high-speed networks and lots of servers. In these environments, log data of network traffic is usually large, coming at high-speed, and have various formats, the classic case of big data problem. This makes anomaly detection very difficult due to the fact that to get good accuracy, large amount of data must be processed in real-time. To solve this problem, this paper proposes a hybrid architecture for network traffic anomaly detection using popular big data framework including Apache Spark and Apache Flume. To demonstrate the capabilities of our proposed solution, we implement a SARIMA-based anomaly detection as a case study. The experimental results clearly indicated that our proposed architecture allows anomaly detection with good accuracy in large-scale environment effectively.