KEDA External Scaler

The Middleware KEDA external scaler autoscales Kubernetes workloads on any metric in Middleware — infrastructure, Kubernetes, span-derived or custom metrics.

It is a KEDA external scaler: a small gRPC service that KEDA calls on every polling interval to ask whether a workload is active and what its metric is worth. It runs alongside KEDA in your cluster and needs no changes to KEDA itself.

How a query is written#

The scaler builds the Middleware query for you, so a trigger reads as configuration rather than as an opaque payload:

1metric:   k8s.pod.cpu.utilization
2resource: k8s.pod
3agg:      avg
4filters:  k8s.namespace.name:production
5window:   300      # seconds

Prerequisites#

• A Kubernetes cluster running v1.23 or later.

• KEDA installed in the cluster. The Helm chart is the recommended route:

  1helm repo add kedacore https://kedacore.github.io/charts
  2helm repo update
  3
  4helm install keda kedacore/keda \
  5  --namespace keda \
  6  --create-namespace \
  7  --wait

  Installing KEDA needs cluster-admin, because it creates CRDs, an APIService for external.metrics.k8s.io and cluster-wide RBAC.

• A Middleware project ingestion token — the same key an agent uses. It is scoped to a single project and carries no user identity.

Verify KEDA before going further. The metrics APIService must report Available: True, since the HPA cannot read any external metric until it does:

1kubectl get pods -n keda
2kubectl get apiservice v1beta1.external.metrics.k8s.io

Read this first: metricType#

This is the single most common way to misconfigure an external scaler.

KEDA defaults metricType to AverageValue, which divides your metric by the replica count before comparing it to the target. That is correct for a total that should be shared between pods, such as a queue backlog. It is wrong for anything already aggregated.

Scaling on mean latency with the default means that at 10 replicas a 500 ms average is treated as 50 ms, and the workload never scales up.

Two related rules:

• Activation is strictly greater than: a value of exactly 5 does not activate a trigger whose activationValue is 5.
• Activation applies only while minReplicaCount is 0. Above that, the workload is always considered active.

Connecting to the scaler#

The scaler is hosted by Middleware, so there is nothing to deploy. Point the trigger at your account endpoint over TLS:

1triggers:
2  - type: external
3    metadata:
4      scalerAddress: <uid>.middleware.io:443
5      enableTLS: "true"

Replace <uid> with your Middleware account UID — the same subdomain you use to sign in.

Both scalerAddress and enableTLS are KEDA's own trigger fields, so they sit alongside the query metadata rather than being interpreted by the scaler.

Where the API key comes from#

The scaler authenticates with a project ingestion token — the same key an agent uses. It stores no credential of its own: each trigger names an environment variable on the workload it scales, and KEDA resolves it — through a secretKeyRef if that is how the workload gets it — substituting the real value before the metadata reaches the scaler.

Create the Secret in the workload's namespace:

1kubectl -n <workload-namespace> create secret generic middleware-secret \
2  --from-literal=api-key=<your-api-key>

The workload reads it into an environment variable:

1# The workload already has its key
2env:
3  - name: MW_API_KEY
4    valueFrom:
5      secretKeyRef:
6        name: middleware-secret
7        key: api-key

Its ScaledObject names the variable, not the key:

1triggers:
2  - type: external
3    metadata:
4      apiKeyFromEnv: MW_API_KEY

The key stays in the workload's Secret, and one scaler can serve many workloads that each read a different project.

Order matters: apiKeyFromEnv is resolved from the workload's pod spec, so the workload has to exist before its ScaledObject, and the Secret before that.

Alternatively, apiKey sets the token directly on the trigger, which is handy for a quick test. apiKeyFromEnv takes precedence when both are present.

KEDA sends the resolved key in the trigger metadata on every poll, so enableTLS: "true" is required on the :443 endpoint.

Trigger metadata#

Query#

Only those five aggregations are accepted for agg and rollup. Anything else is rejected before it is sent, because the query backend treats an unknown aggregation as fatal.

Filters#

The same syntax as Middleware's search bar. Terms are separated by spaces and combined with AND:

1service.name:checkout k8s.namespace.name:[production,staging] host.name:~web

Quote a value that contains a space: service.name:"my service".

Terms are always combined with AND. Other operators and or groups are not expressible here.

Reduction#

Each series contributes its latest usable point — the metric's current value. When a query returns several series, seriesAggregator combines them.

1seriesAggregator: max
2
3pod-a: [10, 30] -> 30
4pod-b: [20, 50] -> 50   ->  50
5pod-c: [90, 10] -> 10

The newest bucket of a window is often still filling and reads low; set lastPointOffset: "1" to skip it. The offset counts points that have values, so unfilled buckets at the end of a window do not shift which point it lands on.

Two situations are errors rather than a quietly-wrong number:

• Several series with no seriesAggregator, so a groupBy that fans out further than expected says so.
• A series with fewer points than lastPointOffset, which means the offset is misconfigured for this query rather than that data is missing.

A series with nothing landed yet is different: that is a gap in the data, so it is skipped and the series that do have points still decide the metric.

Scaling#

Connection#

Trigger type#

Use type: external, which polls. external-push is not supported: the scaler does not implement StreamIsActive, and KEDA reconnects to it indefinitely rather than giving up, so a push trigger would retry in a loop.

Examples#

Scale on CPU utilisation#

1apiVersion: keda.sh/v1alpha1
2kind: ScaledObject
3metadata:
4  name: api-cpu-scaler
5  namespace: default
6spec:
7  scaleTargetRef:
8    name: my-api
9  minReplicaCount: 1
10  maxReplicaCount: 20
11  pollingInterval: 30
12  # If Middleware is unreachable, hold at a sane replica count instead of letting
13  # the workload collapse. The scaler reports missing data as an error precisely
14  # so that this engages.
15  fallback:
16    failureThreshold: 3
17    replicas: 4
18  triggers:
19    - type: external
20      # Serve the polled value to the HPA instead of re-querying.
21      useCachedMetrics: true
22      # CPU utilisation is already an average, so it must be compared directly.
23      metricType: Value
24      metadata:
25        scalerAddress: <uid>.middleware.io:443
26        enableTLS: "true"
27        apiKeyFromEnv: MW_API_KEY
28
29        metric: k8s.pod.cpu.utilization
30        resource: k8s.pod
31        agg: avg
32        groupBy: k8s.pod.name
33        filters: k8s.namespace.name:production
34
35        window: "300"          # seconds
36        # Several pods, so the busiest one decides.
37        seriesAggregator: max
38
39        targetValue: "70"
40        activationValue: "1"

Scale on request latency#

This uses a span-derived metric, not a trace query — traces.span.metrics.duration_sum is produced from spans at ingest and lives in the metrics store like any other.

1apiVersion: keda.sh/v1alpha1
2kind: ScaledObject
3metadata:
4  name: checkout-latency-scaler
5  namespace: default
6spec:
7  scaleTargetRef:
8    name: checkout
9  minReplicaCount: 2
10  maxReplicaCount: 30
11  pollingInterval: 30
12  fallback:
13    failureThreshold: 3
14    replicas: 6
15  triggers:
16    - type: external
17      useCachedMetrics: true
18      # An average is already aggregated, so compare it directly.
19      metricType: Value
20      metadata:
21        scalerAddress: <uid>.middleware.io:443
22        enableTLS: "true"
23        apiKeyFromEnv: MW_API_KEY
24
25        metric: traces.span.metrics.duration_sum
26        resource: trace
27        agg: avg
28        filters: service.name:checkout
29
30        window: "300"          # seconds
31        seriesAggregator: max
32        # The newest bucket of a window is often still filling and reads low.
33        lastPointOffset: "1"
34
35        targetValue: "250"     # milliseconds
36        activationValue: "1"

Scale a worker to zero on a queue backlog#

1apiVersion: keda.sh/v1alpha1
2kind: ScaledObject
3metadata:
4  name: worker-backlog-scaler
5  namespace: default
6spec:
7  scaleTargetRef:
8    name: worker
9  # Scale to zero when the queue is empty. Activation only applies while
10  # minReplicaCount is 0; above that the workload is always considered active
11  # and activationValue is ignored.
12  minReplicaCount: 0
13  maxReplicaCount: 50
14  pollingInterval: 30
15  fallback:
16    failureThreshold: 3
17    replicas: 2
18  triggers:
19    - type: external
20      useCachedMetrics: true
21      # A total backlog should be shared across workers, so AverageValue is
22      # right here: 500 messages against a target of 100 asks for 5 replicas.
23      metricType: AverageValue
24      metadata:
25        scalerAddress: <uid>.middleware.io:443
26        enableTLS: "true"
27        apiKeyFromEnv: MW_API_KEY
28
29        metric: queue.messages.ready
30        resource: rabbitmq
31        agg: max
32        filters: queue.name:jobs
33
34        window: "120"          # seconds
35
36        targetValue: "100"
37        # Activation is strictly greater than, so a single queued message wakes
38        # the workload while an empty queue lets it scale back to zero.
39        activationValue: "0"

Apply one ScaledObject per Deployment. Several ScaledObjects on one workload fight over its replica count.

Verify#

1kubectl get scaledobject -A
2kubectl describe scaledobject api-cpu-scaler
3kubectl get hpa -w
4kubectl logs -n keda -l app=keda-operator -f

A healthy ScaledObject shows READY=True, with ACTIVE reflecting the metric. The HPA showing <unknown> means the scaler could not produce a value; the KEDA operator logs the reason it returned.

Missing data is an error, not zero#

When a query matches nothing, the scaler returns an error rather than 0. Zero would drive the workload straight to its floor, so the safe reading is "unknown". Set fallback on every ScaledObject to choose what happens then:

1fallback:
2  failureThreshold: 3
3  replicas: 4

Set metricUnavailableValue instead if an empty window genuinely means a specific number for your workload.

Query volume#

Every request queries Middleware directly, as the reference external scaler and KEDA's built-in scalers do. Two things multiply that:

• GetMetricsAndActivity issues two gRPC calls back to back, GetMetrics then IsActive, both wanting the same number. Built-in scalers avoid this by construction; the external-scaler protocol splits it, and nothing upstream deduplicates it.
• The HPA queries KEDA's metrics server roughly every 15s, regardless of your pollingInterval.

The second is the larger factor, and KEDA fixes it for you:

1spec:
2  pollingInterval: 30
3  triggers:
4    - type: external
5      useCachedMetrics: true    # serve the polled value to the HPA

useCachedMetrics serves the operator's polled value to the HPA instead of re-querying, and is supported for external scalers (only cpu, memory and cron are excluded). Keeping freshness there rather than inside the scaler leaves one place that decides how stale a reading may be. Keep pollingInterval at 30s or more.

Ingestion lag#

KEDA scales on what the API returns now, and there is always some delay between an event and its being queryable. No polling interval can compensate for that.

• windowOffset moves the whole window behind the freshness horizon.
• lastPointOffset drops the newest bucket, which is often still filling and therefore reads low.

Both trade freshness for correctness: a reading arrives slightly late rather than being spuriously low. Keep window comfortably above your end-to-end lag, and keep pollingInterval at 30s or more.

Checking a query before you apply it#

A trigger's metric, resource, filters and groupBy are the same fields a metric widget uses. The quickest way to confirm one returns what you expect is to build it in a Middleware dashboard first: pick the metric, set the aggregation, add the filters, and read the value off the chart.

Two things to confirm before applying the ScaledObject:

• The query returns data over your window. An empty window is reported as an error, not zero.
• It returns the number of series you expect. More than one series requires seriesAggregator.

Coming from the Datadog scaler#

Troubleshooting#

Uninstall#

Delete your ScaledObjects first. Removing the CRDs while ScaledObjects still exist leaves HPAs orphaned and workloads stuck at their current replica count:

1kubectl delete scaledobject --all -A
2helm uninstall keda -n keda