Anomaly Detection

Overview

The Anomaly Detection feature is designed to help you detect and investigate anomalies within your system’s logs using machine learning algorithms on your historical data. By providing a detailed breakdown of abnormal activities across different services, this feature allows you to proactively identify and address potential issues.

How it works

A log anomaly is identified through a comprehensive analysis that considers both temporal and semantic patterns. Our system calculates an anomaly score using two key components:

1. Time Series Analysis: Detects unusual patterns in log occurrence frequency over time
2. Semantic Analysis: Identifies unusual or unexpected content patterns within the log messages

An anomaly is flagged when the ratio of anomalous logs to total logs within a one-minute window exceeds a defined threshold, calculated as: (Total Anomaly Counts per Minute / Total Log Counts per Minute) > Threshold

This dual approach helps identify both:

• Unusual spikes or drops in log frequency
• Logs with content that deviates from normal patterns

The combined analysis ensures robust anomaly detection while minimizing false positives and adapting to your system's normal behavior patterns.

Anomalies in Logs

When you are on the Log List screen, you will see a comprehensive breakdown of logs over time. To view anomalous logs, select the 'Anomalous logs' checkbox in the sidebar.

Anomalies Section

After selecting the anomalous logs checkbox, you will see a filtered log list showing only anomalous logs and a timeline chart displaying anomalous log counts over time. This visualization helps you identify when anomalies occurred and their frequency patterns.

Data Privacy and Security

Each customer has their own dedicated ML model for log anomaly detection, ensuring complete data isolation and privacy This means:

• Your log data is exclusively used to train your organization's specific model
• No log data is shared across different customers or organizations
• All data processing and model training occurs within secured environment

This dedicated model approach ensures that your anomaly detection system learns from and adapts to your unique logging patterns while maintaining the highest standards of data protection and confidentiality.

Important Note

Please note that the ML-based log anomaly detection system continuously learns and adapts to your organization's logging patterns. While it strives for accuracy, it may require some time to learn your system's normal behavior and may occasionally produce false positives. We encourage users to provide feedback to help improve the model's accuracy for your specific use case.

Next Steps

• Log Monitoring Overview
• Creating Log Filters
• Log Explorer
• Custom Metrics
• Transforming Logs into Transactions
• Creating Alerts
• Real User Monitoring (RUM)