Immutable ledger-based security bigdata analytic system
Immutable ledger-based security bigdata analytic system The proposed system has focused on both batch and real-time log analysis use cases. The audit trails can be analyzed in a batch mode at automated regular intervals or manually and in a near real-time manner using the application. The serverless architecture which has been adopted while developing the system has improved the scalability of the system while providing a cost-effective audit log processing environment. This also removes the additional overhead of maintaining servers to run the application. Using HiveQL on the EMR cluster, the data get queried from the storage and then stored on an external table to perform batch process against those data. The processed data then get stored on a cloud storage, in an output directory. Then the output data get queried and displayed on a dashboard for visualization. To provide real-time log analysis and real-time anomaly detection, a server has been monitored by using a monitoring agent
