Datadog

Middleware supports the ingestion of APM traces, metrics and logs from the Datadog Agent's Dual Shipping feature.

This feature is only supported for Datadog's Linux and Kubernetes Agents

Prerequisites#

1 Traces#

Datadog Agent version 6.7.0 or above

2 HTTP Logs#

Datadog Agent version 6.13 or above

3 Metrics#

Datadog Agent version 6.17 or above

Linux Agent#

Add Trace Configuration#

Add the following additional_endpoints within the apm_config section in your datadog.yaml file:

If you already have the apm_config section in your datadog.yaml file, add an additional endpoint in the same section. Datadog agent does not support multiple apm_config sections.

The Linux Datadog Agent is configured by default in the following YAML file: /etc/datadog-agent/datadog.yaml

yaml
apm_config:
  enabled: true
  additional_endpoints:
    "https://<MW_UID>.middleware.io":
    - "<MW_API_KEY>"

Add Log Configuration#

Add the following additional_endpoints within the logs_config section in your datadog.yaml file:

Unlike the APM Traces, you only need to provide your hostname (<uid>.middleware.io) without the protocol scheme (https://)

yaml
logs_config:
    use_http: true
    additional_endpoints:
    - api_key: "<MW_API_KEY>"
        Host: "<MW_UID>.middleware.io"
        Port: 443
        is_reliable: true

Add Metric Configuration#

Add the following to the additional_endpoints section.

yaml
additional_endpoints:
  "https://<MW_UID>.middleware.io":
  - "<MW_API_KEY>"

Unlike traces and logs which are under apm_config and logs_config sections, the additional_endpoints section for metrics is at the top level (same level as apm_config and logs_config).

In order to get visibility into the processes running on your infrastructure, enable live processes collection under the process_config section and add additional_endpoints as shown below:

yaml
process_config:
  process_collection:
    enabled: true
  additional_endpoints:
    "https://<MW_UID>.middleware.io":
    - "<MW_API_KEY>"

Restart Datadog Agent#

Run the following code in your terminal to restart the Datadog Agent:

shell
sudo systemctl restart datadog-agent

Kubernetes Agent#

The Datadog Kubernetes agent supports Helm and Operator methods of installation. Middleware can ingest traces, metrics and logs from the Datadog agent installed using any of these methods.

Helm chart#

If you have installed the Datadog Agent using their Helm Chart, you must add the following section in your datadog-values.yaml file that you created in Step 3 of their guide:

You can remove either the logs_config or the apm_config from the below file depending on what data you would like to send to Middleware.

yaml
agents:
  useConfigMap: true
  customAgentConfig:
    logs_config:
      container_collect_all: true
      use_http: true
      additional_endpoints:
        - api_key: "<MW_API_KEY>"
          Host: "<MW_UID>.middleware.io"
          Port: 443
          is_reliable: true
    apm_config:
      additional_endpoints:
        "https://<MW_UID>.middleware.io":
          - "<MW_API_KEY>"
    # collects metrics
    additional_endpoints:
      "https://<MW_UID>.middleware.io":
       - "<MW_API_KEY>"
    # collects kubernetes object data
    orchestrator_explorer:
      orchestrator_additional_endpoints:
        "https://<MW_UID>.middleware.io":
         - "<MW_API_KEY>" 
    # collects process and container data
    process_config:
      additional_endpoints:
        "https://<MW_UID>.middleware.io":
         - "<MW_API_KEY>"

Below is an example of the full datadog-values.yaml file that will send logs,metrics and APM traces to the Middleware platform:

yaml
datadog:
  site: "us5.datadoghq.com"
  clusterName: <your cluster name>
  env:
    - name: DD_CHECKS_TAG_CARDINALITY
      value: "orchestrator"
    - name: DD_HOSTNAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
  kubelet:
    tlsVerify: false
  apiKeyExistingSecret: datadog-secret
  processAgent:
    enabled: true
    processCollection: true
  cluster-agent:
    enabled: true
  orchestratorExplorer:
    enabled: true
  kubeStateMetricsCore:
    enabled: true
  metricsProvider:
    enabled: true
    #collectApiServicesMetrics: false
  containerImageCollection:
    enabled: true
  clusterChecks:
    enabled: true
    useClusterChecksRunners: true
  targetSystem: linux
  logs:
    enabled: true
    containerCollectAll: true
  # Essential configuration for Kubernetes and container data correlation
  # This configuration enables the Agent to collect metrics from kubelet and containerintegration and 
  # associate them with the correct Kubernetes cluster.
  confd:
    kubelet.yaml: |-
      ad_identifiers:
        - _kubelet
      instances:
        - tags:
          - dd_cluster_name:<your cluster name>
    container.yaml: |-
      ad_identifiers:
        - _container
      instances:
        - tags:
          - dd_cluster_name:<your cluster name>
agents:
  env:
    - name: DD_HOSTNAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
  useConfigMap: true
  customAgentConfig:
    logs_config:
      container_collect_all: true
      use_http: true
      additional_endpoints:
        - api_key: "<MW_API_KEY>"
          Host: "<MW_UID>.middleware.io"
          Port: 443
          is_reliable: true
    apm_config:
      additional_endpoints:
        "https://<MW_UID>.middleware.io":
          - "<MW_API_KEY>"
    # collects metrics
    additional_endpoints:
      "https://<MW_UID>.middleware.io":
       - "<MW_API_KEY>"
    # collects kubernetes object data
    orchestrator_explorer:
      orchestrator_additional_endpoints:
        "https://<MW_UID>.middleware.io":
         - "<MW_API_KEY>" 
    # collects process and container data
    process_config:
      additional_endpoints:
        "https://<MW_UID>.middleware.io":
         - "<MW_API_KEY>"
clusterAgent:
  useConfigMap: true
  enabled: true
  # Kubenetes State related metrics
  datadog_cluster_yaml:
    orchestrator_explorer:
      orchestrator_additional_endpoints:
        "https://<MW_UID>.middleware.io":
         - "<MW_API_KEY>"
  env:
    - name: DD_ADDITIONAL_ENDPOINTS
      value: '{"https://<MW_UID>.middleware.io": ["<MW_API_KEY>"]}'
    - name: DD_HOSTNAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName

For cloud providers such as Amazon EKS and Azure AKS, you must override the DD_HOSTNAME environment variable to ensure consistent and stable host naming. If not overridden can lead to fragmented host-level metrics.

Kubernetes Operator#

If you have installed the Datadog Agent using their Kubernetes operator, you must add the following section in datadog-agent.yaml that you create in step 3 of their guide.

The current specification section only enables logs and traces. To enable metrics, traces, and logs, please refer to the complete YAML configuration later below.

yaml
spec:
  ...
  override:
    nodeAgent:
      customConfigurations:
        datadog.yaml:
          configData: |
            logs_enabled: true
            apm_config:
              enabled: true
              additional_endpoints:
                - "https://<MW_UID>.middleware.io":
                    - "<MW_API_KEY>"
            logs_config:
              use_http: true
              container_collect_all: true
              additional_endpoints:
                - api_key: "<MW_API_KEY>"
                  Host: "<MW_UID>.middleware.io"
                  Port: 443
                  is_reliable: true

Below is an example of full datadog-agent.yaml file that will send metrics, logs and APM traces to the Middleware platform:

yaml
kind: DatadogAgent
apiVersion: datadoghq.com/v2alpha1
metadata:
  name: datadog
spec:
  global:
    clusterName: "your-cluster-name"
    site: us5.datadoghq.com
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
    kubelet:
      tlsVerify: false
  features:
    orchestratorExplorer:
      enabled: true
      scrubContainers: true
    kubeStateMetricsCore:
      enabled: true
    processDiscovery:
      enabled: true
    oomKill:
      enabled: true
    liveContainerCollection:
      enabled: true
    eventCollection:
      collectKubernetesEvents: true
    logCollection:
      enabled: true
      containerCollectAll: true
    liveProcessCollection:
      enabled: true
    apm:
      enabled: true
      hostPortConfig:
        enabled: true
  override:
    clusterAgent:
      env:
      - name: DD_CLUSTER_NAME  # Add explicit cluster name env var
        value: "your-cluster-name"
      - name: DD_HOSTNAME
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
      - name: DD_ADDITIONAL_ENDPOINTS
        value: '{"https://<MW_UID>.middleware.io": ["<MW_API_KEY>"]}'
      - name: DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS
        value: '{"https://<MW_UID>.middleware.io": ["<MW_API_KEY>"]}'
    nodeAgent:
      env:
      - name: DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS
        value: '{"https://<MW_UID>.middleware.io": ["<MW_API_KEY>"]}'
      - name: DD_HOSTNAME
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
      customConfigurations:
        datadog.yaml:
          configData: |-
            additional_endpoints:
              "https://<MW_UID>.middleware.io":
                - "<MW_API_KEY>"
            logs_config:
              container_collect_all: true
              use_http: true
              additional_endpoints:
                - api_key: "<MW_API_KEY>"
                  Host: "<MW_UID>.middleware.io"
                  Port: 443
                  is_reliable: true
            apm_config:
              additional_endpoints:
                "https://<MW_UID>.middleware.io":
                  - "<MW_API_KEY>"
            process_config:
              additional_endpoints:
                "https://<MW_UID>.middleware.io":
                  - "<MW_API_KEY>"
      extraConfd:
        configDataMap:
          kubelet.yaml: |-
            ad_identifiers:
              - _kubelet
            instances:
              - tags:
                - dd_cluster_name:your-cluster-name
          container.yaml: |-
            ad_identifiers:
              - _container
            instances:
              - tags:
                - dd_cluster_name:your-cluster-name

Middleware only supports datadoghq.com/v2alpha1 apiVersion for Datadog agent resource.

To find out the resource version currently in use by Datadog Kubernetes operator, issue the command below:

bash
kubectl api-resources --api-group=datadoghq.com

The output of above command should look like below:

NAME              SHORTNAMES   APIVERSION               NAMESPACED   KIND
datadogagents     dd           datadoghq.com/v2alpha1   true         DatadogAgent
...
...

APIVERSION should show datadoghq.com/v2alpha1 for the Datadog agent.

Amazon ECS on EC2 Cluster#

Middleware can ingest logs and APM traces from the Datadog Agent running on ECS clusters that use EC2 instances.

For more information on ingesting Log and APM Traces using Amazon ECS with Datadog, navigate here.

Add Trace Config#

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping APM traces in the Datadog Agent.

The DD_APM_ENABLED and DD_APM_ADDITION_ENDPOINTS environment variables are in addition to existing environment variables (e.g. DD_API_KEY, DD_SITE, etc.) that may already be defined in your Datadog Agent’s task definition.

"environment": [
   ...
   {
     "name": "DD_APM_ENABLED",
     "value": "true"
   },
   {
     "name": "DD_APM_ADDITIONAL_ENDPOINTS",
     "value": "{\"https://<MW_UID>.middleware.io\": [\"<MW_API_KEY>\"]}"
   },
 ]

Add Log Config#

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping Logs in the Datadog Agent.

The four environment variables mentioned above are in addition to existing environment variables (e.g. DD_API_KEY, DD_SITE, etc.) that may already be defined in your Datadog Agent’s task definition.

"environment": [
  {
    "name": "DD_LOGS_CONFIG_USE_HTTP",
    "value": "true"
  },
  {
     "name": "DD_LOGS_ENABLED",
     "value": "true"
   },
   {
     "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
     "value": "true"
   },
   {
     "name": "DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS",
     "value": "[{\"api_key\": \"<MW_API_KEY>\", \"Host\": \"<MW_UID>.middleware.io\", \"Port\": 443, \"is_reliable\": true}]"
   }
 ]

Update Agent Service#

Once you have updated the ECS task definition for the Datadog Agent, update the relevant Datadog Agent Service to redeploy the agent with your new configuration. APM traces and logs will start flowing into your Middleware account.

Datadog PHP APM#

This integration lets you reuse your existing Datadog PHP APM instrumentation and forward the same traces, logs and profiling data into Middleware.

What it does#

  • Your PHP application is instrumented using the Datadog PHP tracer (dd-trace-php).
  • The tracer sends APM data to the Middleware Host Agent using the Datadog APM protocol on port 8126.
  • The Middleware Host Agent exposes a Datadog-compatible receiver, converts the data, and forwards it to the Middleware backend.
  • You can then explore traces in Middleware alongside other services and infrastructure.

How it works#

Middleware does not replace your Datadog PHP tracer. Instead, it relies on Datadog's tracing library to generate spans inside your PHP runtime, then points that tracer to the Middleware Host Agent.

The PHP tracer:

  • Instruments common PHP frameworks and libraries automatically
  • Measures request latency, throughput and errors per endpoint
  • Captures context like HTTP URLs, status codes, database queries and unhandled exceptions

By enabling the Datadog receiver in the Middleware Host Agent, the agent listens on a Datadog-compatible port and accepts traces and optional profiling data from your PHP application. From there, the data behaves like native APM telemetry inside Middleware.

Prerequisites#

1 Middleware Host Agent#

  • The Middleware Host Agent must be installed and running on the same machine as your PHP application, or reachable from it.
  • If not installed, follow the Host Agent installation guide and pick the correct installer for your environment.

2 Network connectivity#

  • The PHP application must reach the Middleware Host Agent on port 8126.
  • For non-container workloads, this is usually http://localhost:8126.
  • For container workloads, the agent is usually reachable at the Docker host IP or a Kubernetes service, which you configure through MW_AGENT_SERVICE.

3 Supported PHP environment#

  • PHP version 7 or newer.
  • A web server or runtime such as Apache, PHP-FPM, Nginx with PHP-FPM, or the PHP CLI server.

Installation#

1. Ensure the Middleware Host Agent is reachable#

Install and run the Middleware Host Agent on the same machine that hosts your PHP application.

If you are running your application in a containerized environment, you must expose the agent to the container and set MW_AGENT_SERVICE so the application can resolve the agent:

bash
MW_AGENT_SERVICE=172.17.0.1
bash
MW_AGENT_SERVICE=mw-service.mw-agent-ns.svc.cluster.local

Use the value that matches how you deployed the Middleware Host Agent in your cluster or host. Pass this environment variable into your container or deployment spec so that the PHP runtime can route traces to the Host Agent.

2. Download the Datadog PHP APM installer#

Download the latest Datadog PHP tracer installer directly from GitHub:

bash
curl -LO https://github.com/DataDog/dd-trace-php/releases/latest/download/datadog-setup.php

Commit the script or a pinned version of it to your deployment pipeline if you want a reproducible tracer version across environments.

3. Run the installer#

Run the installer so it detects all available PHP binaries on the host and configures the tracer for each of them:

bash
sudo php datadog-setup.php --php-bin=all

What this does:

  • Detects PHP installations such as the Apache PHP module, PHP-FPM and CLI binaries
  • Installs the Datadog tracer extension and required configuration snippets
  • Prepares the runtime so that future requests are traced automatically once you set the environment variables

4. Configure Datadog tracer environment variables#

Configure environment variables so the Datadog tracer knows:

  • Which logical service name to use
  • Where to send traces, which in this case is the Middleware Host Agent Datadog receiver at http://localhost:8126 or http://$MW_AGENT_SERVICE:8126

For Apache with PHP-FPM, use the env[] directive in your www.conf pool configuration:

ini
env[DD_SERVICE] = <service-name>
env[DD_TRACE_AGENT_URL] = "http://localhost:8126"

Replace <service-name> with a meaningful name such as checkout-service or billing-api. This name appears as the service name in Middleware.

If the agent is not on local loopback inside your PHP container, use the MW_AGENT_SERVICE host:

ini
env[DD_SERVICE] = <service-name>
env[DD_TRACE_AGENT_URL] = "http://mw-service.mw-agent-ns.svc.cluster.local:8126"

If PHP runs directly inside Apache, you can configure environment variables at server, virtual host, or directory level, including .htaccess, using SetEnv:

apache
SetEnv DD_SERVICE <service-name>
SetEnv DD_TRACE_AGENT_URL http://localhost:8126

Change the URL if your Middleware Host Agent runs on a different host or service name.

For the built-in PHP server, CLI scripts or artisan-style commands, you can set environment variables inline:

bash
DD_TRACE_DEBUG=1 DD_SERVICE=<service-name> DD_TRACE_AGENT_URL=http://localhost:8126 php -S localhost:8888

DD_TRACE_DEBUG=1 enables verbose tracer logs, which are useful while validating the integration. Turn this off in production for lower log noise.

Optional tagging for environments and versions

Datadog's PHP tracer supports additional environment variables like DD_ENV and DD_VERSION that help you filter traces by environment and application version.

You can optionally set these to improve how Middleware groups and filters your traces:

bash
DD_ENV=production
DD_VERSION=1.3.2
DD_SERVICE=<service-name>
DD_TRACE_AGENT_URL=http://localhost:8126

Middleware receives these tags, and you can use them to separate staging and production services, or to compare performance across releases.

Restart services after installation

After the installer completes, restart your PHP runtime:

  • For Apache with the PHP module: restart Apache
  • For PHP-FPM: restart the PHP-FPM service and reload the web server
  • For other servers: restart the relevant process so the new PHP configuration is loaded

5. Enable the Datadog receiver in the Middleware Host Agent#

The Middleware Host Agent must be configured to accept Datadog-formatted traces. This is controlled by the Datadog receiver configuration. You can enable it in two ways.

When installing the agent, pass the following environment variable so the Datadog receiver is enabled from the start:

bash
MW_ENABLE_DATADOG_RECEIVER=true

Follow the standard Host Agent installation guide for all supported flags and examples.

If the agent is already installed, edit the configuration file and append the Datadog receiver setting.

  • On Unix-based systems: /etc/mw-agent/agent-config.yaml
  • On Windows systems: <agent-installation-directory>\agent-config.yaml

Add or update the following line:

yaml
enable-datadog-receiver: true

Save the file and restart the agent:

bash
sudo systemctl restart mw-agent

After the restart, the agent listens for Datadog APM traffic on its configured Datadog receiver port, usually 8126.

Troubleshooting#

  • Confirm enable-datadog-receiver: true is present in agent-config.yaml and the agent is restarted.
  • Check that DD_TRACE_AGENT_URL points to the Middleware Host Agent and not to a Datadog Agent running elsewhere.
  • Verify there are no firewalls or network policies blocking port 8126.
  • Enable DD_TRACE_DEBUG=1 temporarily and inspect application logs for tracer errors.
  • Confirm DD_SERVICE is set for the runtime that actually serves traffic.
  • For Apache with PHP-FPM, make sure env[] directives are defined in the correct pool configuration and the pool was reloaded.
  • For Apache without PHP-FPM, verify SetEnv is not overridden in a deeper .htaccess or virtual host.
  • Run php -i | grep ddtrace or check the PHP info page to confirm the ddtrace extension is loaded.
  • Ensure the Datadog installer was run with the --php-bin=all flag if you have multiple PHP binaries.
  • CLI scripts require the same environment variables as web requests.
  • When running scheduled jobs or workers, set DD_SERVICE, DD_TRACE_AGENT_URL and other tags in the environment of the scheduler or worker process.

FAQ#

If you want to stop sending APM traces and logs to Datadog, you can change the api_key (or environment variable DD_API_KEY) in the /etc/datadog/datadog.yaml file to something invalid.

The Datadog Agent does not work if you comment out your api_key or set it to an empty value.

Below is an example of setting your api_key and site to invalid values:

yaml
api_key: usingmiddleware
site: example.com
  • Execute the following command to ensure the Datadog Linux Agent is in an active state:
shell
sudo systemctl status datadog-agent

  • Verify that your Middleware API key and target are correct. The apm_config target is http://<uid>.middleware.io whereas the logs_config Host field value is <uid>.middleware.io (without https://).

  • Make sure you only have one apm_config and logs_config section in the following configuration file: /etc/datadog/datadog.yaml

  • Check your Datadog Kubernetes Agent. Ensure the datadog-agent and cluster-agent pods are operational with a liveness and readiness check for both pods.

  • Check your Kubernetes Helm chart installation method by checking the datadog-values.yaml and ensure that the agent section is at the same level as the datadog section. The agent section should NOT be inside the datadog section.

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us or join our Slack channel.

What did you think of this content?

Ask AI about this page

Answers cite the Datadog pages they came from.