Creating Alerts

The Alerts page is a tool for monitoring and responding to critical events across your entire infrastructure. Create high-level alerts to detect system-level anomalies or supervise a single Kubernetes node. Set up alerts using telemetry data your team collects to effectively manage system-wide issues.

Selecting a Monitor Type

When setting up an alert, the first step is to choose the appropriate monitor type. This determines the data source that the alert will be based on. Below are the available options for monitor types:

Metrics: Use this monitor type to create alerts based on numerical data points, such as CPU usage, memory consumption, or application performance metrics.
Host: Select this option to get alerted when a host is reported as down longer than a defined time period.
Logs: Choose this type to set up alerts on log data. This is useful for identifying patterns, errors, or specific events within your log streams.
APM (Application Performance Monitoring): Use this monitor type to track application-level performance metrics, such as request latency, requests, or error rates.
RUM (Real User Monitoring): Select this option to monitor real user interactions with your application, including page load times, errors encountered by users, and web vitals.
LLM (Large Language Model): Use this monitor type to create alerts based on performance and usage metrics related to large language models.
Custom: Select this option to create alerts based on custom metrics.

Defining the Query

The query configuration allows you to select metrics, apply functions, and set filters. Additionally, you can modify the aggregation window, apply additional functions over the time window, and perform arithmetic using two or more queries.

The query section is divided into two components: Individual Queries and the Formula Area.

Individual Queries

Metric: Select the desired metric to analyze.
From: Filter data using operators such as =, !=, REGEX, etc.
Function: Perform an aggregate function over histogram values. NOTE: This does not apply to scalar values like gauges or counters.
Rollup: Apply an aggregate function to scalar values within the defined time window.
Over: Define the time window for the rollup.

Formula Area

When defining two or more queries, you can perform arithmetic operations (e.g., addition, subtraction, multiplication, division) using the variables assigned to each query. Each query is automatically assigned a variable (e.g., a, b, c), which you can use in the formula area to construct expressions. For example:

(a + b) / c
a * b - c

Setting Alert Conditions

In this step, you configure the alert thresholds and conditions to define when an alert should trigger. The following options are available:

Thresholds: Set two distinct thresholds for alerts. The warning threshold serves as an early indicator, while the critical threshold signifies more severe conditions.
Query Evaluation Period: Define the time window for evaluating the query over a range of data points. If advanced options are enabled, the query evaluation period MUST BE greater than the value set for over.
Alert After: Specify the number of data points that must meet the defined threshold to trigger an alert.

Configuring Notifications

Configure how alerts are named, grouped, and delivered.

Monitor Name: Assign a human-readable name to the alert. This name will appear in the Middleware platform and in notification messages.
Tags: Add tags to group related alerts for easier organization and filtering.
Add Custom Alert Message: Create a custom message body for the alert. This message will be included in notifications sent to the configured recipients.

Aggregation and Medium

Renotification: Enable this option to continuously send alerts over a specified time period until the issue is resolved.
Medium: Select the notification delivery method, such as Slack, Email, Microsoft Teams, or PagerDuty.

Next Steps

Need assistance or want to learn more about Middleware? Contact our support team at [email protected].