Advanced Anomaly Detection in Server Logs

Abstract: This research develops an anomaly detection system for sequential log data from distributed services by integrating sequential and contextual features. Sequential features capture the temporal dependencies between log entries, allowing the model to identify unusual patterns in the sequence of events that may precede an anomaly. Contextual features, such as word embeddings, are used to extract the semantic meaning from the content of the log messages, helping the model understand the context behind each log entry. This is crucial for detecting anomalies that may not be evident from temporal patterns alone, such as subtle variations in system behavior or unusual error descriptions. The inclusion of semantic meaning enables the detection of early-stage anomalies, which may otherwise go unnoticed if only the sequence of events is considered. An ml model is trained on labeled anomaly data and evaluated on new logs from different servers to assess its generalization ability. By combining sequential patterns with contextual understanding, this approach enhances the accuracy and timeliness of anomaly detection, addressing both the timing and content of logs. The results demonstrate the effectiveness of using advanced feature extraction techniques with ml models for robust and early anomaly detection in dynamic, distributed server environments.

Machine Learning (ML) Deep Learning NLP Tokenization Long Short-Term Memory Network

ID Projekti: #39000858

Rreth projektit