---
title: "KEDA External Scaler"
description: "Use the Middleware KEDA external scaler to autoscale Kubernetes workloads on any metric in Middleware - infrastructure, Kubernetes, span-derived or custom metrics."
section: "Agent Installation"
breadcrumb: "Containers > Kubernetes > KEDA External Scaler"
canonical: "https://docs.middleware.io/get-started/agent/kubernetes/keda-external-scaler"
---
# KEDA External Scaler

The Middleware KEDA external scaler autoscales Kubernetes workloads on any **metric** in Middleware — infrastructure, Kubernetes, span-derived or custom metrics.

It is a [KEDA external scaler](https://keda.sh/docs/2.21/concepts/external-scalers/): a small gRPC service that KEDA calls on every polling interval to ask whether a workload is active and what its metric is worth. It runs alongside KEDA in your cluster and needs no changes to KEDA itself.

> **Note:** The scaler queries **metrics only**. Logs, traces and RUM sessions cannot be scaled on directly — but a metric *derived* from them can, such as the span metric `traces.span.metrics.duration_sum` used in the latency example below.

## How a query is written

The scaler builds the Middleware query for you, so a trigger reads as configuration rather than as an opaque payload:

```yaml
metric:   k8s.pod.cpu.utilization
resource: k8s.pod
agg:      avg
filters:  k8s.namespace.name:production
window:   300      # seconds
```

## Prerequisites

- A Kubernetes cluster running v1.23 or later.
- [KEDA](https://keda.sh/docs/2.21/deploy/) installed in the cluster. The Helm chart is the recommended route:

  ```bash
  helm repo add kedacore https://kedacore.github.io/charts
  helm repo update

  helm install keda kedacore/keda \
    --namespace keda \
    --create-namespace \
    --wait
  ```

  Installing KEDA needs cluster-admin, because it creates CRDs, an APIService for `external.metrics.k8s.io` and cluster-wide RBAC.
- A Middleware **project ingestion token** — the same key an agent uses. It is scoped to a single project and carries no user identity.

Verify KEDA before going further. The metrics APIService must report `Available: True`, since the HPA cannot read any external metric until it does:

```bash
kubectl get pods -n keda
kubectl get apiservice v1beta1.external.metrics.k8s.io
```

> **Warning:** Confirm which cluster you are pointed at before installing anything: `kubectl config current-context`.

## Read this first: `metricType`

This is the single most common way to misconfigure an external scaler.

KEDA defaults `metricType` to `AverageValue`, which **divides your metric by the replica count** before comparing it to the target. That is correct for a total that should be shared between pods, such as a queue backlog. It is wrong for anything already aggregated.

| Your metric | `metricType` | Why |
| --- | --- | --- |
| CPU %, memory %, error rate, mean latency | `Value` | Already an average; dividing it again is meaningless |
| Queue depth, total requests, backlog | `AverageValue` | A total that should be spread across replicas |

Scaling on mean latency with the default means that at 10 replicas a 500 ms average is treated as 50 ms, and the workload never scales up.

Two related rules:

- Activation is **strictly greater than**: a value of exactly 5 does not activate a trigger whose `activationValue` is 5.
- Activation applies **only while `minReplicaCount` is 0**. Above that, the workload is always considered active.

## Connecting to the scaler

The scaler is hosted by Middleware, so there is nothing to deploy. Point the trigger at your account endpoint over TLS:

```yaml
triggers:
  - type: external
    metadata:
      scalerAddress: <uid>.middleware.io:443
      enableTLS: "true"
```

Replace `<uid>` with your Middleware account UID — the same subdomain you use to sign in.

> **Note:** `enableTLS: "true"` is required on port 443. Without it KEDA opens a plaintext connection to a TLS listener and every poll fails.

Both `scalerAddress` and `enableTLS` are KEDA's own trigger fields, so they sit alongside the query metadata rather than being interpreted by the scaler.

## Where the API key comes from

The scaler authenticates with a project ingestion token — the same key an agent uses. It **stores no credential of its own**: each trigger names an environment variable **on the workload it scales**, and KEDA resolves it — through a `secretKeyRef` if that is how the workload gets it — substituting the real value before the metadata reaches the scaler.

Create the Secret in the workload's namespace:

```bash
kubectl -n <workload-namespace> create secret generic middleware-secret \
  --from-literal=api-key=<your-api-key>
```

The workload reads it into an environment variable:

```yaml
# The workload already has its key
env:
  - name: MW_API_KEY
    valueFrom:
      secretKeyRef:
        name: middleware-secret
        key: api-key
```

Its ScaledObject names the variable, not the key:

```yaml
triggers:
  - type: external
    metadata:
      apiKeyFromEnv: MW_API_KEY
```

The key stays in the workload's Secret, and one scaler can serve many workloads that each read a different project.

Order matters: `apiKeyFromEnv` is resolved from the workload's pod spec, so the workload has to exist before its ScaledObject, and the Secret before that.

Alternatively, `apiKey` sets the token directly on the trigger, which is handy for a quick test. `apiKeyFromEnv` takes precedence when both are present.

KEDA sends the resolved key in the trigger metadata on every poll, so `enableTLS: "true"` is required on the `:443` endpoint.

## Trigger metadata

### Query

| Key | Default | Description |
| --- | --- | --- |
| `metric` | *required* | Metric name, e.g. `k8s.pod.cpu.utilization` |
| `resource` | *required* | The metric's source dataset, e.g. `k8s.pod`, `host`, `trace`, `custom` |
| `agg` | `avg` | How the series are combined: `avg`, `sum`, `min`, `max`, `any` |
| `rollup` | `avg` | How the raw samples inside one bucket condense, before `agg` combines the series |
| `groupBy` | — | Comma-separated attributes, e.g. `k8s.pod.name` |
| `filters` | — | `key:value`, `key:[a,b]`, `key:~value`; see below |
| `window` | `300` | Lookback size, in seconds |
| `windowOffset` | `0` | Seconds to shift the window back, to allow for ingestion lag |

Only those five aggregations are accepted for `agg` and `rollup`. Anything else is rejected before it is sent, because the query backend treats an unknown aggregation as fatal.

### Filters

The same syntax as Middleware's search bar. Terms are separated by spaces and combined with AND:

```
service.name:checkout k8s.namespace.name:[production,staging] host.name:~web
```

| Form | Meaning |
| --- | --- |
| `key:value` | equals |
| `key:[a,b]` | matches any of the listed values |
| `key:~value` | contains |

Quote a value that contains a space: `service.name:"my service"`.

Terms are always combined with AND. Other operators and `or` groups are not expressible here.

### Reduction

Each series contributes its **latest usable point** — the metric's current value. When a query returns several series, `seriesAggregator` combines them.

```
seriesAggregator: max

pod-a: [10, 30] -> 30
pod-b: [20, 50] -> 50   ->  50
pod-c: [90, 10] -> 10
```

| Key | Default | Description |
| --- | --- | --- |
| `seriesAggregator` | — | `max` or `avg`. **Required when the query returns more than one series.** |
| `lastPointOffset` | `0` | Step back this many points from the newest usable one |

The newest bucket of a window is often still filling and reads low; set `lastPointOffset: "1"` to skip it. The offset counts points that have values, so unfilled buckets at the end of a window do not shift which point it lands on.

Two situations are errors rather than a quietly-wrong number:

- Several series with no `seriesAggregator`, so a `groupBy` that fans out further than expected says so.
- A series with fewer points than `lastPointOffset`, which means the offset is misconfigured for this query rather than that data is missing.

A series with nothing landed yet is different: that is a gap in the data, so it is skipped and the series that do have points still decide the metric.

### Scaling

| Key | Default | Description |
| --- | --- | --- |
| `targetValue` | *required* | HPA target; must be above 0 |
| `activationValue` | `0` | Scale above zero when the value exceeds this |
| `metricName` | the metric | HPA metric name. Defaults to the metric being queried. |
| `metricUnavailableValue` | — | Value to report when a window has no data |

### Connection

| Key | Default | Description |
| --- | --- | --- |
| `scalerAddress` | *required* | KEDA's own field: your account endpoint, `<uid>.middleware.io:443` |
| `enableTLS` | `false` | KEDA's own field. Must be `"true"` for the `:443` endpoint. |
| `apiKeyFromEnv` | — | Environment variable **on the scaled workload** holding the token. KEDA resolves it and substitutes the value. Preferred. |
| `apiKey` | — | The token set directly on the trigger. Used only when `apiKeyFromEnv` is absent. |
| `timeout` | `10` | Per-request timeout, in seconds |

## Trigger type

Use `type: external`, which polls. `external-push` is not supported: the scaler does not implement `StreamIsActive`, and KEDA reconnects to it indefinitely rather than giving up, so a push trigger would retry in a loop.

## Examples

### Scale on CPU utilisation

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api-cpu-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: my-api
  minReplicaCount: 1
  maxReplicaCount: 20
  pollingInterval: 30
  # If Middleware is unreachable, hold at a sane replica count instead of letting
  # the workload collapse. The scaler reports missing data as an error precisely
  # so that this engages.
  fallback:
    failureThreshold: 3
    replicas: 4
  triggers:
    - type: external
      # Serve the polled value to the HPA instead of re-querying.
      useCachedMetrics: true
      # CPU utilisation is already an average, so it must be compared directly.
      metricType: Value
      metadata:
        scalerAddress: <uid>.middleware.io:443
        enableTLS: "true"
        apiKeyFromEnv: MW_API_KEY

        metric: k8s.pod.cpu.utilization
        resource: k8s.pod
        agg: avg
        groupBy: k8s.pod.name
        filters: k8s.namespace.name:production

        window: "300"          # seconds
        # Several pods, so the busiest one decides.
        seriesAggregator: max

        targetValue: "70"
        activationValue: "1"
```

### Scale on request latency

This uses a span-derived **metric**, not a trace query — `traces.span.metrics.duration_sum` is produced from spans at ingest and lives in the metrics store like any other.

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: checkout-latency-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: checkout
  minReplicaCount: 2
  maxReplicaCount: 30
  pollingInterval: 30
  fallback:
    failureThreshold: 3
    replicas: 6
  triggers:
    - type: external
      useCachedMetrics: true
      # An average is already aggregated, so compare it directly.
      metricType: Value
      metadata:
        scalerAddress: <uid>.middleware.io:443
        enableTLS: "true"
        apiKeyFromEnv: MW_API_KEY

        metric: traces.span.metrics.duration_sum
        resource: trace
        agg: avg
        filters: service.name:checkout

        window: "300"          # seconds
        seriesAggregator: max
        # The newest bucket of a window is often still filling and reads low.
        lastPointOffset: "1"

        targetValue: "250"     # milliseconds
        activationValue: "1"
```

### Scale a worker to zero on a queue backlog

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-backlog-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: worker
  # Scale to zero when the queue is empty. Activation only applies while
  # minReplicaCount is 0; above that the workload is always considered active
  # and activationValue is ignored.
  minReplicaCount: 0
  maxReplicaCount: 50
  pollingInterval: 30
  fallback:
    failureThreshold: 3
    replicas: 2
  triggers:
    - type: external
      useCachedMetrics: true
      # A total backlog should be shared across workers, so AverageValue is
      # right here: 500 messages against a target of 100 asks for 5 replicas.
      metricType: AverageValue
      metadata:
        scalerAddress: <uid>.middleware.io:443
        enableTLS: "true"
        apiKeyFromEnv: MW_API_KEY

        metric: queue.messages.ready
        resource: rabbitmq
        agg: max
        filters: queue.name:jobs

        window: "120"          # seconds

        targetValue: "100"
        # Activation is strictly greater than, so a single queued message wakes
        # the workload while an empty queue lets it scale back to zero.
        activationValue: "0"
```

Apply one ScaledObject per Deployment. Several ScaledObjects on one workload fight over its replica count.

## Verify

```bash
kubectl get scaledobject -A
kubectl describe scaledobject api-cpu-scaler
kubectl get hpa -w
kubectl logs -n keda -l app=keda-operator -f
```

A healthy ScaledObject shows `READY=True`, with `ACTIVE` reflecting the metric. The HPA showing `<unknown>` means the scaler could not produce a value; the KEDA operator logs the reason it returned.

## Missing data is an error, not zero

When a query matches nothing, the scaler returns an error rather than 0. Zero would drive the workload straight to its floor, so the safe reading is "unknown". Set `fallback` on every ScaledObject to choose what happens then:

```yaml
fallback:
  failureThreshold: 3
  replicas: 4
```

Set `metricUnavailableValue` instead if an empty window genuinely means a specific number for your workload.

## Query volume

Every request queries Middleware directly, as the reference external scaler and KEDA's built-in scalers do. Two things multiply that:

- `GetMetricsAndActivity` issues **two** gRPC calls back to back, `GetMetrics` then `IsActive`, both wanting the same number. Built-in scalers avoid this by construction; the external-scaler protocol splits it, and nothing upstream deduplicates it.
- The HPA queries KEDA's metrics server roughly **every 15s**, regardless of your `pollingInterval`.

The second is the larger factor, and KEDA fixes it for you:

```yaml
spec:
  pollingInterval: 30
  triggers:
    - type: external
      useCachedMetrics: true    # serve the polled value to the HPA
```

> **Note:** `useCachedMetrics` is a field on the **trigger**, not on `spec.advanced`. Putting it under `advanced` is rejected by the admission webhook with `unknown field "spec.advanced.useCachedMetrics"`.

`useCachedMetrics` serves the operator's polled value to the HPA instead of re-querying, and is supported for external scalers (only `cpu`, `memory` and `cron` are excluded). Keeping freshness there rather than inside the scaler leaves one place that decides how stale a reading may be. Keep `pollingInterval` at 30s or more.

## Ingestion lag

KEDA scales on what the API returns now, and there is always some delay between an event and its being queryable. No polling interval can compensate for that.

- `windowOffset` moves the whole window behind the freshness horizon.
- `lastPointOffset` drops the newest bucket, which is often still filling and therefore reads low.

Both trade freshness for correctness: a reading arrives slightly late rather than being spuriously low. Keep `window` comfortably above your end-to-end lag, and keep `pollingInterval` at 30s or more.

## Checking a query before you apply it

A trigger's `metric`, `resource`, `filters` and `groupBy` are the same fields a metric widget uses. The quickest way to confirm one returns what you expect is to build it in a Middleware dashboard first: pick the metric, set the aggregation, add the filters, and read the value off the chart.

Two things to confirm before applying the ScaledObject:

- The query returns **data** over your `window`. An empty window is reported as an error, not zero.
- It returns the number of **series** you expect. More than one series requires `seriesAggregator`.

## Coming from the Datadog scaler

| Datadog | Here |
| --- | --- |
| `query` | `metric` + `resource` + `agg` + `filters` |
| `queryValue` / `targetValue` | `targetValue` |
| `activationQueryValue` | `activationValue` |
| `queryAggregator` | `seriesAggregator` (same `max` / `avg`) |
| `age` | `window` |
| `timeWindowOffset` | `windowOffset` |
| `lastAvailablePointOffset` | `lastPointOffset` (counts points that have values) |
| `metricUnavailableValue` | `metricUnavailableValue` |
| `type` | KEDA's own `metricType` |

## Troubleshooting

| Symptom | Likely cause |
| --- | --- |
| HPA shows `<unknown>` | The query returns no data over the window, or the API key is wrong. Check the KEDA operator logs and confirm the query in a dashboard. |
| Workload never scales up | `metricType` is `AverageValue` for an already-averaged metric. Use `Value`. |
| Nothing happens near the target | The HPA ignores differences within 10% of the target. |
| `InvalidArgument` on the ScaledObject | A metadata mistake; the message names the field. |
| Never scales to zero | Activation only applies while `minReplicaCount` is 0. |
| Value looks low | The newest bucket may still be filling. Try `lastPointOffset: "1"`. |
| Several series returned, request rejected | A `groupBy` fanned out further than expected. Set `seriesAggregator` to `max` or `avg`. |

## Uninstall

Delete your ScaledObjects **first**. Removing the CRDs while ScaledObjects still exist leaves HPAs orphaned and workloads stuck at their current replica count:

```bash
kubectl delete scaledobject --all -A
helm uninstall keda -n keda
```