Installing the Datadog Agent in Middleware (Dual Shipping)

Middleware supports the ingestion of APM traces, metrics and logs from the Datadog Agent's Dual Shipping feature.

This feature is only supported for Datadog's Linux and Kubernetes Agents

Prerequisites#

1 Traces#

Datadog Agent version 6.7.0 or above

2 HTTP Logs#

Datadog Agent version 6.13 or above

3 Metrics#

Datadog Agent version 6.17 or above

Linux Agent#

Add Trace Configuration#

Add the following additional_endpoints within the apm_config section in your datadog.yaml file:

If you already have the apm_config section in your datadog.yaml file, add an additional endpoint in the same section. Datadog agent does not support multiple apm_config sections.

The Linux Datadog Agent is configured by default in the following YAML file: /etc/datadog-agent/datadog.yaml

1apm_config:
2  enabled: true
3  additional_endpoints:
4    "https://<MW_UID>.middleware.io":
5    - "<MW_API_KEY>"

Add Log Configuration#

Add the following additional_endpoints within the logs_config section in your datadog.yaml file:

Unlike the APM Traces, you only need to provide your hostname (<uid>.middleware.io) without the protocol scheme (https://)

1logs_config:
2    use_http: true
3    additional_endpoints:
4    - api_key: "<MW_API_KEY>"
5        Host: "<MW_UID>.middleware.io"
6        Port: 443
7        is_reliable: true

Add Metric Configuration#

Add the following to the additional_endpoints section.

1additional_endpoints:
2  "https://<MW_UID>.middleware.io":
3  - "<MW_API_KEY>"

Unlike traces and logs which are under apm_config and logs_config sections, the additional_endpoints section for metrics is at the top level (same level as apm_config and logs_config).

In order to get visibility into the processes running on your infrastructure, enable live processes collection under the process_config section and add additional_endpoints as shown below:

1process_config:
2  process_collection:
3    enabled: true
4  additional_endpoints:
5    "https://<MW_UID>.middleware.io":
6    - "<MW_API_KEY>"

Restart Datadog Agent#

Run the following code in your terminal to restart the Datadog Agent:

1sudo systemctl restart datadog-agent

Kubernetes Agent#

The Datadog Kubernetes agent supports Helm and Operator methods of installation. Middleware can ingest traces, metrics and logs from the Datadog agent installed using any of these methods.

Helm chart#

If you have installed the Datadog Agent using their Helm Chart, you must add the following section in your datadog-values.yaml file that you created in Step 3 of their guide:

You can remove either the logs_config or the apm_config from the below file depending on what data you would like to send to Middleware.

1agents:
2  useConfigMap: true
3  customAgentConfig:
4    logs_config:
5      container_collect_all: true
6      use_http: true
7      additional_endpoints:
8        - api_key: "<MW_API_KEY>"
9          Host: "<MW_UID>.middleware.io"
10          Port: 443
11          is_reliable: true
12    apm_config:
13      additional_endpoints:
14        "https://<MW_UID>.middleware.io":
15          - "<MW_API_KEY>"
16    # collects metrics
17    additional_endpoints:
18      "https://<MW_UID>.middleware.io":
19       - "<MW_API_KEY>"
20    # collects kubernetes object data
21    orchestrator_explorer:
22      orchestrator_additional_endpoints:
23        "https://<MW_UID>.middleware.io":
24         - "<MW_API_KEY>" 
25    # collects process and container data
26    process_config:
27      additional_endpoints:
28        "https://<MW_UID>.middleware.io":
29         - "<MW_API_KEY>"

Below is an example of the full datadog-values.yaml file that will send logs,metrics and APM traces to the Middleware platform:

1datadog:
2  site: "us5.datadoghq.com"
3  clusterName: <your cluster name>
4  env:
5    - name: DD_CHECKS_TAG_CARDINALITY
6      value: "orchestrator"
7    - name: DD_HOSTNAME
8      valueFrom:
9        fieldRef:
10          fieldPath: spec.nodeName
11  kubelet:
12    tlsVerify: false
13  apiKeyExistingSecret: datadog-secret
14  processAgent:
15    enabled: true
16    processCollection: true
17  cluster-agent:
18    enabled: true
19  orchestratorExplorer:
20    enabled: true
21  kubeStateMetricsCore:
22    enabled: true
23  metricsProvider:
24    enabled: true
25    #collectApiServicesMetrics: false
26  containerImageCollection:
27    enabled: true
28  clusterChecks:
29    enabled: true
30    useClusterChecksRunners: true
31  targetSystem: linux
32  logs:
33    enabled: true
34    containerCollectAll: true
35  # Essential configuration for Kubernetes and container data correlation
36  # This configuration enables the Agent to collect metrics from kubelet and containerintegration and 
37  # associate them with the correct Kubernetes cluster.
38  confd:
39    kubelet.yaml: |-
40      ad_identifiers:
41        - _kubelet
42      instances:
43        - tags:
44          - dd_cluster_name:<your cluster name>
45    container.yaml: |-
46      ad_identifiers:
47        - _container
48      instances:
49        - tags:
50          - dd_cluster_name:<your cluster name>
51agents:
52  env:
53    - name: DD_HOSTNAME
54      valueFrom:
55        fieldRef:
56          fieldPath: spec.nodeName
57  useConfigMap: true
58  customAgentConfig:
59    logs_config:
60      container_collect_all: true
61      use_http: true
62      additional_endpoints:
63        - api_key: "<MW_API_KEY>"
64          Host: "<MW_UID>.middleware.io"
65          Port: 443
66          is_reliable: true
67    apm_config:
68      additional_endpoints:
69        "https://<MW_UID>.middleware.io":
70          - "<MW_API_KEY>"
71    # collects metrics
72    additional_endpoints:
73      "https://<MW_UID>.middleware.io":
74       - "<MW_API_KEY>"
75    # collects kubernetes object data
76    orchestrator_explorer:
77      orchestrator_additional_endpoints:
78        "https://<MW_UID>.middleware.io":
79         - "<MW_API_KEY>" 
80    # collects process and container data
81    process_config:
82      additional_endpoints:
83        "https://<MW_UID>.middleware.io":
84         - "<MW_API_KEY>"
85clusterAgent:
86  useConfigMap: true
87  enabled: true
88  # Kubenetes State related metrics
89  datadog_cluster_yaml:
90    orchestrator_explorer:
91      orchestrator_additional_endpoints:
92        "https://<MW_UID>.middleware.io":
93         - "<MW_API_KEY>"
94  env:
95    - name: DD_ADDITIONAL_ENDPOINTS
96      value: '{"https://<MW_UID>.middleware.io": ["<MW_API_KEY>"]}'
97    - name: DD_HOSTNAME
98      valueFrom:
99        fieldRef:
100          fieldPath: spec.nodeName

For cloud providers such as Amazon EKS and Azure AKS, you must override the DD_HOSTNAME environment variable to ensure consistent and stable host naming. If not overridden can lead to fragmented host-level metrics.

Kubernetes Operator#

If you have installed the Datadog Agent using their Kubernetes operator, you must add the following section in datadog-agent.yaml that you create in step 3 of their guide.

The current specification section only enables logs and traces. To enable metrics, traces, and logs, please refer to the complete YAML configuration later below.

1spec:
2  ...
3  override:
4    nodeAgent:
5      customConfigurations:
6        datadog.yaml:
7          configData: |
8            logs_enabled: true
9            apm_config:
10              enabled: true
11              additional_endpoints:
12                - "https://<MW_UID>.middleware.io":
13                    - "<MW_API_KEY>"
14            logs_config:
15              use_http: true
16              container_collect_all: true
17              additional_endpoints:
18                - api_key: "<MW_API_KEY>"
19                  Host: "<MW_UID>.middleware.io"
20                  Port: 443
21                  is_reliable: true

Below is an example of full datadog-agent.yaml file that will send metrics, logs and APM traces to the Middleware platform:

1kind: DatadogAgent
2apiVersion: datadoghq.com/v2alpha1
3metadata:
4  name: datadog
5spec:
6  global:
7    clusterName: "your-cluster-name"
8    site: us5.datadoghq.com
9    credentials:
10      apiSecret:
11        secretName: datadog-secret
12        keyName: api-key
13    kubelet:
14      tlsVerify: false
15  features:
16    orchestratorExplorer:
17      enabled: true
18      scrubContainers: true
19    kubeStateMetricsCore:
20      enabled: true
21    processDiscovery:
22      enabled: true
23    oomKill:
24      enabled: true
25    liveContainerCollection:
26      enabled: true
27    eventCollection:
28      collectKubernetesEvents: true
29    logCollection:
30      enabled: true
31      containerCollectAll: true
32    liveProcessCollection:
33      enabled: true
34    apm:
35      enabled: true
36      hostPortConfig:
37        enabled: true
38  override:
39    clusterAgent:
40      env:
41      - name: DD_CLUSTER_NAME  # Add explicit cluster name env var
42        value: "your-cluster-name"
43      - name: DD_HOSTNAME
44        valueFrom:
45          fieldRef:
46            fieldPath: spec.nodeName
47      - name: DD_ADDITIONAL_ENDPOINTS
48        value: '{"https://<MW_UID>.middleware.io": ["<MW_API_KEY>"]}'
49      - name: DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS
50        value: '{"https://<MW_UID>.middleware.io": ["<MW_API_KEY>"]}'
51    nodeAgent:
52      env:
53      - name: DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS
54        value: '{"https://<MW_UID>.middleware.io": ["<MW_API_KEY>"]}'
55      - name: DD_HOSTNAME
56        valueFrom:
57          fieldRef:
58            fieldPath: spec.nodeName
59      customConfigurations:
60        datadog.yaml:
61          configData: |-
62            additional_endpoints:
63              "https://<MW_UID>.middleware.io":
64                - "<MW_API_KEY>"
65            logs_config:
66              container_collect_all: true
67              use_http: true
68              additional_endpoints:
69                - api_key: "<MW_API_KEY>"
70                  Host: "<MW_UID>.middleware.io"
71                  Port: 443
72                  is_reliable: true
73            apm_config:
74              additional_endpoints:
75                "https://<MW_UID>.middleware.io":
76                  - "<MW_API_KEY>"
77            process_config:
78              additional_endpoints:
79                "https://<MW_UID>.middleware.io":
80                  - "<MW_API_KEY>"
81      extraConfd:
82        configDataMap:
83          kubelet.yaml: |-
84            ad_identifiers:
85              - _kubelet
86            instances:
87              - tags:
88                - dd_cluster_name:your-cluster-name
89          container.yaml: |-
90            ad_identifiers:
91              - _container
92            instances:
93              - tags:
94                - dd_cluster_name:your-cluster-name

Middleware only supports datadoghq.com/v2alpha1 apiVersion for Datadog agent resource.

To find out the resource version currently in use by Datadog Kubernetes operator, issue the command below:

1kubectl api-resources --api-group=datadoghq.com

The output of above command should look like below:

1NAME              SHORTNAMES   APIVERSION               NAMESPACED   KIND
2datadogagents     dd           datadoghq.com/v2alpha1   true         DatadogAgent
3...
4...

APIVERSION should show datadoghq.com/v2alpha1 for the Datadog agent.

Amazon ECS on EC2 Cluster#

Middleware can ingest logs and APM traces from the Datadog Agent running on ECS clusters that use EC2 instances.

For more information on ingesting Log and APM Traces using Amazon ECS with Datadog, navigate here.

Add Trace Config#

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping APM traces in the Datadog Agent.

The DD_APM_ENABLED and DD_APM_ADDITION_ENDPOINTS environment variables are in addition to existing environment variables (e.g. DD_API_KEY, DD_SITE, etc.) that may already be defined in your Datadog Agent’s task definition.

1"environment": [
2   ...
3   {
4     "name": "DD_APM_ENABLED",
5     "value": "true"
6   },
7   {
8     "name": "DD_APM_ADDITIONAL_ENDPOINTS",
9     "value": "{\"https://<MW_UID>.middleware.io\": [\"<MW_API_KEY>\"]}"
10   },
11 ]

Add Log Config#

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping Logs in the Datadog Agent.

The four environment variables mentioned above are in addition to existing environment variables (e.g. DD_API_KEY, DD_SITE, etc.) that may already be defined in your Datadog Agent’s task definition.

1"environment": [
2  {
3    "name": "DD_LOGS_CONFIG_USE_HTTP",
4    "value": "true"
5  },
6  {
7     "name": "DD_LOGS_ENABLED",
8     "value": "true"
9   },
10   {
11     "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
12     "value": "true"
13   },
14   {
15     "name": "DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS",
16     "value": "[{\"api_key\": \"<MW_API_KEY>\", \"Host\": \"<MW_UID>.middleware.io\", \"Port\": 443, \"is_reliable\": true}]"
17   }
18 ]

Update Agent Service#

Once you have updated the ECS task definition for the Datadog Agent, update the relevant Datadog Agent Service to redeploy the agent with your new configuration. APM traces and logs will start flowing into your Middleware account.

FAQ#

How do I stop sending APM traces and logs to Datadog and only send them to Middleware?

If you want to stop sending APM traces and logs to Datadog, you can change the api_key (or environment variable DD_API_KEY) in the /etc/datadog/datadog.yaml file to something invalid.

The Datadog Agent does not work if you comment out your api_key or set it to an empty value.

Below is an example of setting your api_key and site to invalid values:

1api_key: usingmiddleware
2site: example.com

Why can't I see my APM traces and logs on Middleware?

Execute the following command to ensure the Datadog Linux Agent is in an active state:

1sudo systemctl status datadog-agent

Verify that your Middleware API key and target are correct. The apm_config target is http://<uid>.middleware.io whereas the logs_config Host field value is <uid>.middleware.io (without https://).
Make sure you only have one apm_config and logs_config section in the following configuration file: /etc/datadog/datadog.yaml
Check your Datadog Kubernetes Agent. Ensure the datadog-agent and cluster-agent pods are operational with a liveness and readiness check for both pods.
Check your Kubernetes Helm chart installation method by checking the datadog-values.yaml and ensure that the agent section is at the same level as the datadog section. The agent section should NOT be inside the datadog section.

Need assistance or want to learn more about Middleware? Contact our support team at [email protected] or join our Slack channel.