KEDA External Scaler

The Middleware KEDA external scaler autoscales Kubernetes workloads on any metric in Middleware — infrastructure, Kubernetes, span-derived or custom metrics.

It is a KEDA external scaler: a small gRPC service that KEDA calls on every polling interval to ask whether a workload is active and what its metric is worth. It runs alongside KEDA in your cluster and needs no changes to KEDA itself.

How a query is written#

The scaler builds the Middleware query for you, so a trigger reads as configuration rather than as an opaque payload:

metric:   k8s.pod.cpu.utilization
resource: k8s.pod
agg:      avg
filters:  k8s.namespace.name:production
window:   300      # seconds

Prerequisites#

• A Kubernetes cluster running v1.23 or later.

• KEDA installed in the cluster. The Helm chart is the recommended route:

  bash

  helm repo add kedacore https://kedacore.github.io/charts
  helm repo update
  
  helm install keda kedacore/keda \
    --namespace keda \
    --create-namespace \
    --wait

  Installing KEDA needs cluster-admin, because it creates CRDs, an APIService for external.metrics.k8s.io and cluster-wide RBAC.

• A Middleware project ingestion token — the same key an agent uses. It is scoped to a single project and carries no user identity.

Verify KEDA before going further. The metrics APIService must report Available: True, since the HPA cannot read any external metric until it does:

kubectl get pods -n keda
kubectl get apiservice v1beta1.external.metrics.k8s.io

Read this first: metricType#

This is the single most common way to misconfigure an external scaler.

KEDA defaults metricType to AverageValue, which divides your metric by the replica count before comparing it to the target. That is correct for a total that should be shared between pods, such as a queue backlog. It is wrong for anything already aggregated.

Scaling on mean latency with the default means that at 10 replicas a 500 ms average is treated as 50 ms, and the workload never scales up.

Two related rules:

• Activation is strictly greater than: a value of exactly 5 does not activate a trigger whose activationValue is 5.
• Activation applies only while minReplicaCount is 0. Above that, the workload is always considered active.

Connecting to the scaler#

The scaler is hosted by Middleware, so there is nothing to deploy. Point the trigger at your account endpoint over TLS:

triggers:
  - type: external
    metadata:
      scalerAddress: <uid>.middleware.io:443
      enableTLS: "true"

Replace <uid> with your Middleware account UID — the same subdomain you use to sign in.

Both scalerAddress and enableTLS are KEDA's own trigger fields, so they sit alongside the query metadata rather than being interpreted by the scaler.

Where the API key comes from#

The scaler authenticates with a project ingestion token — the same key an agent uses. It stores no credential of its own: each trigger names an environment variable on the workload it scales, and KEDA resolves it — through a secretKeyRef if that is how the workload gets it — substituting the real value before the metadata reaches the scaler.

Create the Secret in the workload's namespace:

kubectl -n <workload-namespace> create secret generic middleware-secret \
  --from-literal=api-key=<your-api-key>

The workload reads it into an environment variable:

# The workload already has its key
env:
  - name: MW_API_KEY
    valueFrom:
      secretKeyRef:
        name: middleware-secret
        key: api-key

Its ScaledObject names the variable, not the key:

triggers:
  - type: external
    metadata:
      apiKeyFromEnv: MW_API_KEY

The key stays in the workload's Secret, and one scaler can serve many workloads that each read a different project.

Order matters: apiKeyFromEnv is resolved from the workload's pod spec, so the workload has to exist before its ScaledObject, and the Secret before that.

Alternatively, apiKey sets the token directly on the trigger, which is handy for a quick test. apiKeyFromEnv takes precedence when both are present.

KEDA sends the resolved key in the trigger metadata on every poll, so enableTLS: "true" is required on the :443 endpoint.

Trigger metadata#

Query#

Only those five aggregations are accepted for agg and rollup. Anything else is rejected before it is sent, because the query backend treats an unknown aggregation as fatal.

Filters#

The same syntax as Middleware's search bar. Terms are separated by spaces and combined with AND:

service.name:checkout k8s.namespace.name:[production,staging] host.name:~web

Quote a value that contains a space: service.name:"my service".

Terms are always combined with AND. Other operators and or groups are not expressible here.

Reduction#

Each series contributes its latest usable point — the metric's current value. When a query returns several series, seriesAggregator combines them.

seriesAggregator: max

pod-a: [10, 30] -> 30
pod-b: [20, 50] -> 50   ->  50
pod-c: [90, 10] -> 10

The newest bucket of a window is often still filling and reads low; set lastPointOffset: "1" to skip it. The offset counts points that have values, so unfilled buckets at the end of a window do not shift which point it lands on.

Two situations are errors rather than a quietly-wrong number:

• Several series with no seriesAggregator, so a groupBy that fans out further than expected says so.
• A series with fewer points than lastPointOffset, which means the offset is misconfigured for this query rather than that data is missing.

A series with nothing landed yet is different: that is a gap in the data, so it is skipped and the series that do have points still decide the metric.

Scaling#

Connection#

Trigger type#

Use type: external, which polls. external-push is not supported: the scaler does not implement StreamIsActive, and KEDA reconnects to it indefinitely rather than giving up, so a push trigger would retry in a loop.

Examples#

Scale on CPU utilisation#

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api-cpu-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: my-api
  minReplicaCount: 1
  maxReplicaCount: 20
  pollingInterval: 30
  # If Middleware is unreachable, hold at a sane replica count instead of letting
  # the workload collapse. The scaler reports missing data as an error precisely
  # so that this engages.
  fallback:
    failureThreshold: 3
    replicas: 4
  triggers:
    - type: external
      # Serve the polled value to the HPA instead of re-querying.
      useCachedMetrics: true
      # CPU utilisation is already an average, so it must be compared directly.
      metricType: Value
      metadata:
        scalerAddress: <uid>.middleware.io:443
        enableTLS: "true"
        apiKeyFromEnv: MW_API_KEY

        metric: k8s.pod.cpu.utilization
        resource: k8s.pod
        agg: avg
        groupBy: k8s.pod.name
        filters: k8s.namespace.name:production

        window: "300"          # seconds
        # Several pods, so the busiest one decides.
        seriesAggregator: max

        targetValue: "70"
        activationValue: "1"

Scale on request latency#

This uses a span-derived metric, not a trace query — traces.span.metrics.duration_sum is produced from spans at ingest and lives in the metrics store like any other.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: checkout-latency-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: checkout
  minReplicaCount: 2
  maxReplicaCount: 30
  pollingInterval: 30
  fallback:
    failureThreshold: 3
    replicas: 6
  triggers:
    - type: external
      useCachedMetrics: true
      # An average is already aggregated, so compare it directly.
      metricType: Value
      metadata:
        scalerAddress: <uid>.middleware.io:443
        enableTLS: "true"
        apiKeyFromEnv: MW_API_KEY

        metric: traces.span.metrics.duration_sum
        resource: trace
        agg: avg
        filters: service.name:checkout

        window: "300"          # seconds
        seriesAggregator: max
        # The newest bucket of a window is often still filling and reads low.
        lastPointOffset: "1"

        targetValue: "250"     # milliseconds
        activationValue: "1"

Scale a worker to zero on a queue backlog#

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-backlog-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: worker
  # Scale to zero when the queue is empty. Activation only applies while
  # minReplicaCount is 0; above that the workload is always considered active
  # and activationValue is ignored.
  minReplicaCount: 0
  maxReplicaCount: 50
  pollingInterval: 30
  fallback:
    failureThreshold: 3
    replicas: 2
  triggers:
    - type: external
      useCachedMetrics: true
      # A total backlog should be shared across workers, so AverageValue is
      # right here: 500 messages against a target of 100 asks for 5 replicas.
      metricType: AverageValue
      metadata:
        scalerAddress: <uid>.middleware.io:443
        enableTLS: "true"
        apiKeyFromEnv: MW_API_KEY

        metric: queue.messages.ready
        resource: rabbitmq
        agg: max
        filters: queue.name:jobs

        window: "120"          # seconds

        targetValue: "100"
        # Activation is strictly greater than, so a single queued message wakes
        # the workload while an empty queue lets it scale back to zero.
        activationValue: "0"

Apply one ScaledObject per Deployment. Several ScaledObjects on one workload fight over its replica count.

Verify#

kubectl get scaledobject -A
kubectl describe scaledobject api-cpu-scaler
kubectl get hpa -w
kubectl logs -n keda -l app=keda-operator -f

A healthy ScaledObject shows READY=True, with ACTIVE reflecting the metric. The HPA showing <unknown> means the scaler could not produce a value; the KEDA operator logs the reason it returned.

Missing data is an error, not zero#

When a query matches nothing, the scaler returns an error rather than 0. Zero would drive the workload straight to its floor, so the safe reading is "unknown". Set fallback on every ScaledObject to choose what happens then:

fallback:
  failureThreshold: 3
  replicas: 4

Set metricUnavailableValue instead if an empty window genuinely means a specific number for your workload.

Query volume#

Every request queries Middleware directly, as the reference external scaler and KEDA's built-in scalers do. Two things multiply that:

• GetMetricsAndActivity issues two gRPC calls back to back, GetMetrics then IsActive, both wanting the same number. Built-in scalers avoid this by construction; the external-scaler protocol splits it, and nothing upstream deduplicates it.
• The HPA queries KEDA's metrics server roughly every 15s, regardless of your pollingInterval.

The second is the larger factor, and KEDA fixes it for you:

spec:
  pollingInterval: 30
  triggers:
    - type: external
      useCachedMetrics: true    # serve the polled value to the HPA

useCachedMetrics serves the operator's polled value to the HPA instead of re-querying, and is supported for external scalers (only cpu, memory and cron are excluded). Keeping freshness there rather than inside the scaler leaves one place that decides how stale a reading may be. Keep pollingInterval at 30s or more.

Ingestion lag#

KEDA scales on what the API returns now, and there is always some delay between an event and its being queryable. No polling interval can compensate for that.

• windowOffset moves the whole window behind the freshness horizon.
• lastPointOffset drops the newest bucket, which is often still filling and therefore reads low.

Both trade freshness for correctness: a reading arrives slightly late rather than being spuriously low. Keep window comfortably above your end-to-end lag, and keep pollingInterval at 30s or more.

Checking a query before you apply it#

A trigger's metric, resource, filters and groupBy are the same fields a metric widget uses. The quickest way to confirm one returns what you expect is to build it in a Middleware dashboard first: pick the metric, set the aggregation, add the filters, and read the value off the chart.

Two things to confirm before applying the ScaledObject:

• The query returns data over your window. An empty window is reported as an error, not zero.
• It returns the number of series you expect. More than one series requires seriesAggregator.

Coming from the Datadog scaler#

Troubleshooting#

Uninstall#

Delete your ScaledObjects first. Removing the CRDs while ScaledObjects still exist leaves HPAs orphaned and workloads stuck at their current replica count:

kubectl delete scaledobject --all -A
helm uninstall keda -n keda