OTEL-Native Filters
Otel-Native Filters are the agent-side processing layer in Middleware Pipelines. They let you filter, modify, redact, and drop telemetry before data leaves your cluster, using native OpenTelemetry processors.
Middleware has also moved from the old standalone Agent Pipeline to this newer model where Otel-Native Filters are built directly inside Cluster Pipelines with a visual pipeline graph.
Where OTEL-Native Filters Are Available
Otel-Native Filters appear only when the pipeline source is a Kubernetes Cluster. They do not appear for Hosts or Integrations because these filters run inside the OpenTelemetry Collector in the cluster.
| Source Type | OTEL-Native Filters Support | Signals |
|---|---|---|
| Host | No | Not available |
| Cluster | Yes | Logs, Metrics, Traces |
| Serverless | No | Not available |
How it works in the data path
These filters run before data is sent out of the cluster, so they reduce volume early.
Flow: Application → OpenTelemetry Collector (Cluster) → Otel-Native Filters → Middleware → Backend Control → Storage
How to Configure it
1 Go to Settings → Pipelines
2 Select a Cluster source
3 Click + Add Otel Native Filters
4 Configure per signal (Logs / Metrics / Traces)
One Filter Set Per Pipeline
Each pipeline can have:
- One Otel-Native filter set for Logs
- One for Metrics
- One for Traces
If you need different agent-side behaviour, create another pipeline and attach different clusters/rules.
Available Processors
Middleware exposes native OTel processors in a UI-friendly way.
| Processor | What it does | Example use |
|---|---|---|
| Attributes | Remove or rename attributes | Remove k8s.pod.uid, rename service to service_name |
| Compact | Remove attributes only when values match specific noisy patterns | Remove http.route only for /health and /metrics |
| Filter | Drop telemetry entirely based on conditions (attribute/operator/value, AND/OR) | Drop traces where service.name = "cron-job" |
| Redaction | Mask or remove sensitive data using allow-list, regex, and predefined PII patterns | Mask emails, cards, phone, SSN, IP, MAC, UUID |
Processor Execution Order
Processors run in this order in the graph:
Attributes → Compact → Filter → Redaction → Middleware
You can reorder processors, add multiple processors of the same type, and remove processors anytime.
OTEL-Native Filters vs Backend Control
| Area | OTEL-Native Filters | Backend Control |
|---|---|---|
| Runs at | Inside your cluster | Middleware backend |
| Main purpose | Reduce, secure, and optimise before sending | Drop before storage |
| Cost impact | Saves network + ingestion cost | Saves storage cost |
| Best for | Advanced OTel users | Most users |
Using both gives maximum control.
Key Limitations
- Only for Kubernetes clusters
- One filter set per pipeline and signal
- For multiple agent-side configurations, create multiple pipelines
For creating pipeline, user needs to add one or more than one processor. User can also add same processor multiple times.