Installing the Datadog Agent in Middleware (Dual Shipping)

Middleware supports the ingestion of APM traces, metrics and logs from the Datadog Agent's Dual Shipping feature.

This feature is only supported for Datadog's Linux and Kubernetes Agents

Prerequisites

1 Traces

Datadog Agent version 6.7.0 or above

2 HTTP Logs

Datadog Agent version 6.13 or above

3 Metrics

Datadog Agent version 6.17 or above

Linux Agent

Add Trace Configuration

Add the following additional_endpoints within the apm_config section in your datadog.yaml file:

If you already have the apm_config section in your datadog.yaml file, add an additional endpoint in the same section. Datadog agent does not support multiple apm_config sections.

The Linux Datadog Agent is configured by default in the following YAML file: /etc/datadog-agent/datadog.yaml

1apm_config:
2  enabled: true
3  additional_endpoints:
4    "https://<uid>.middleware.io":
5    - "<MW_API_KEY>"

Add Log Configuration

Add the following additional_endpoints within the logs_config section in your datadog.yaml file:

Unlike the APM Traces, you only need to provide your hostname (<uid>.middleware.io) without the protocol scheme (https://)

1logs_config:
2    use_http: true
3    additional_endpoints:
4    - api_key: "<MW_API_KEY>"
5        Host: "<uid>.middleware.io"
6        Port: 443
7        is_reliable: true

Add Metric Configuration

Add the following to the additional_endpoints section.

1additional_endpoints:
2  "https://<uid>.middleware.io":
3  - "<MW_API_KEY>"

Unlike traces and logs which are under apm_config and logs_config sections, the additional_endpoints section for metrics is at the top level (same level as apm_config and logs_config).

In order to get visibility into the processes running on your infrastructure, enable live processes collection under the process_config section and add additional_endpoints as shown below:

1process_config:
2  process_collection:
3    enabled: true
4  additional_endpoints:
5    "https://<uid>.middleware.io":
6    - "<MW_API_KEY>"

Restart Datadog Agent

Run the following code in your terminal to restart the Datadog Agent:

1sudo systemctl restart datadog-agent

Kubernetes Agent

The Datadog Kubernetes agent supports Helm and Operator methods of installation. Middleware can ingest traces, metrics and logs from the Datadog agent installed using any of these methods.

Helm chart

If you have installed the Datadog Agent using their Helm Chart, you must add the following section in your datadog-values.yaml file that you created in Step 3 of their guide:

You can remove either the logs_config or the apm_config from the below file depending on what data you would like to send to Middleware.

1agents:
2  useConfigMap: true
3  customAgentConfig:
4    logs_config:
5      container_collect_all: true
6      use_http: true
7      additional_endpoints:
8        - api_key: "<MW_API_KEY>"
9          Host: "<uid>.middleware.io"
10          Port: 443
11          is_reliable: true
12    apm_config:
13      additional_endpoints:
14        "https://<uid>.middleware.io":
15          - "<MW_API_KEY>"
16    # collects metrics
17    additional_endpoints:
18      "https://<uid>.middleware.io":
19       - "<MW_API_KEY>"
20    # collects kubernetes object data
21    orchestrator_explorer:
22      orchestrator_additional_endpoints:
23        "https://<uid>.middleware.io":
24         - "<MW_API_KEY>" 
25    # collects process and container data
26    process_config:
27      additional_endpoints:
28        "https://<uid>.middleware.io":
29         - "<MW_API_KEY>"

Below is an example of the full datadog-values.yaml file that will send logs,metrics and APM traces to the Middleware platform:

1datadog:
2  site: "us5.datadoghq.com"
3  clusterName: <your cluster name>
4  env:
5    - name: DD_CHECKS_TAG_CARDINALITY
6      value: "orchestrator"
7  kubelet:
8    tlsVerify: false
9  apiKeyExistingSecret: datadog-secret
10  processAgent:
11    enabled: true
12    processCollection: true
13  cluster-agent:
14    enabled: true
15  orchestratorExplorer:
16    enabled: true
17  kubeStateMetricsCore:
18    enabled: true
19  metricsProvider:
20    enabled: true
21    #collectApiServicesMetrics: false
22  containerImageCollection:
23    enabled: true
24  clusterChecks:
25    enabled: true
26    useClusterChecksRunners: true
27  targetSystem: linux
28  logs:
29    enabled: true
30    containerCollectAll: true
31  # Essential configuration for Kubernetes and container data correlation
32  # This configuration enables the Agent to collect metrics from kubelet and containerintegration and 
33  # associate them with the correct Kubernetes cluster.
34  confd:
35    kubelet.yaml: |-
36      ad_identifiers:
37        - _kubelet
38      instances:
39        - tags:
40          - dd_cluster_name:<your cluster name>
41    container.yaml: |-
42      ad_identifiers:
43        - _container
44      instances:
45        - tags:
46          - dd_cluster_name:<your cluster name>
47agents:
48  useConfigMap: true
49  customAgentConfig:
50    logs_config:
51      container_collect_all: true
52      use_http: true
53      additional_endpoints:
54        - api_key: "<MW_API_KEY>"
55          Host: "<uid>.middleware.io"
56          Port: 443
57          is_reliable: true
58    apm_config:
59      additional_endpoints:
60        "https://<uid>.middleware.io":
61          - "<MW_API_KEY>"
62    # collects metrics
63    additional_endpoints:
64      "https://<uid>.middleware.io":
65       - "<MW_API_KEY>"
66    # collects kubernetes object data
67    orchestrator_explorer:
68      orchestrator_additional_endpoints:
69        "https://<uid>.middleware.io":
70         - "<MW_API_KEY>" 
71    # collects process and container data
72    process_config:
73      additional_endpoints:
74        "https://<uid>.middleware.io":
75         - "<MW_API_KEY>"
76clusterAgent:
77  useConfigMap: true
78  enabled: true
79  # Kubenetes State related metrics
80  datadog_cluster_yaml:
81    orchestrator_explorer:
82      orchestrator_additional_endpoints:
83        "https://<uid>.middleware.io":
84         - "<MW_API_KEY>"
85  env:
86    - name: DD_ADDITIONAL_ENDPOINTS
87      value: '{"https://<uid>.middleware.io": ["<MW_API_KEY>"]}'

Kubernetes Operator

If you have installed the Datadog Agent using their Kubernetes operator, you must add the following section in datadog-agent.yaml that you create in step 3 of their guide.

The current specification section only enables logs and traces. To enable metrics, traces, and logs, please refer to the complete YAML configuration later below.

1spec:
2  ...
3  override:
4    nodeAgent:
5      customConfigurations:
6        datadog.yaml:
7          configData: |
8            logs_enabled: true
9            apm_config:
10              enabled: true
11              additional_endpoints:
12                - "https://<uid>.middleware.io":
13                    - "<MW_API_KEY>"
14            logs_config:
15              use_http: true
16              container_collect_all: true
17              additional_endpoints:
18                - api_key: "<MW_API_KEY>"
19                  Host: "<uid>.middleware.io"
20                  Port: 443
21                  is_reliable: true

Below is an example of full datadog-agent.yaml file that will send metrics, logs and APM traces to the Middleware platform:

1kind: DatadogAgent
2apiVersion: datadoghq.com/v2alpha1
3metadata:
4  name: datadog
5spec:
6  global:
7    clusterName: "your-cluster-name"
8    site: us5.datadoghq.com
9    credentials:
10      apiSecret:
11        secretName: datadog-secret
12        keyName: api-key
13    kubelet:
14      tlsVerify: false
15  features:
16    orchestratorExplorer:
17      enabled: true
18      scrubContainers: true
19    kubeStateMetricsCore:
20      enabled: true
21    processDiscovery:
22      enabled: true
23    oomKill:
24      enabled: true
25    liveContainerCollection:
26      enabled: true
27    eventCollection:
28      collectKubernetesEvents: true
29    logCollection:
30      enabled: true
31      containerCollectAll: true
32    liveProcessCollection:
33      enabled: true
34    apm:
35      enabled: true
36      hostPortConfig:
37        enabled: true
38  override:
39    clusterAgent:
40      env:
41      - name: DD_CLUSTER_NAME  # Add explicit cluster name env var
42        value: "your-cluster-name"
43      - name: DD_HOSTNAME
44        valueFrom:
45          fieldRef:
46            fieldPath: spec.nodeName
47      - name: DD_ADDITIONAL_ENDPOINTS
48        value: '{"https://<uid>.middleware.io": ["<MW_API_KEY>"]}'
49      - name: DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS
50        value: '{"https://<uid>.middleware.io": ["<MW_API_KEY>"]}'
51    nodeAgent:
52      env:
53      - name: DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS
54        value: '{"https://<uid>.middleware.io": ["<MW_API_KEY>"]}'
55      - name: DD_HOSTNAME
56        valueFrom:
57          fieldRef:
58            fieldPath: spec.nodeName
59      customConfigurations:
60        datadog.yaml:
61          configData: |-
62            additional_endpoints:
63              "https://<uid>.middleware.io":
64                - "<MW_API_KEY>"
65            logs_config:
66              container_collect_all: true
67              use_http: true
68              additional_endpoints:
69                - api_key: "<MW_API_KEY>"
70                  Host: "<uid>.middleware.io"
71                  Port: 443
72                  is_reliable: true
73            apm_config:
74              additional_endpoints:
75                "https://<uid>.middleware.io":
76                  - "<MW_API_KEY>"
77            process_config:
78              additional_endpoints:
79                "https://<uid>.middleware.io":
80                  - "<MW_API_KEY>"
81      extraConfd:
82        configDataMap:
83          kubelet.yaml: |-
84            ad_identifiers:
85              - _kubelet
86            instances:
87              - tags:
88                - dd_cluster_name:your-cluster-name
89          container.yaml: |-
90            ad_identifiers:
91              - _container
92            instances:
93              - tags:
94                - dd_cluster_name:your-cluster-name

Middleware only supports datadoghq.com/v2alpha1 apiVersion for Datadog agent resource.

To find out the resource version currently in use by Datadog Kubernetes operator, issue the command below:

1kubectl api-resources --api-group=datadoghq.com

The output of above command should look like below:

1NAME              SHORTNAMES   APIVERSION               NAMESPACED   KIND
2datadogagents     dd           datadoghq.com/v2alpha1   true         DatadogAgent
3...
4...

APIVERSION should show datadoghq.com/v2alpha1 for the Datadog agent.

Amazon ECS on EC2 Cluster

Middleware can ingest logs and APM traces from the Datadog Agent running on ECS clusters that use EC2 instances.

For more information on ingesting Log and APM Traces using Amazon ECS with Datadog, navigate here.

Add Trace Config

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping APM traces in the Datadog Agent.

The DD_APM_ENABLED and DD_APM_ADDITION_ENDPOINTS environment variables are in addition to existing environment variables (e.g. DD_API_KEY, DD_SITE, etc.) that may already be defined in your Datadog Agent’s task definition.

1"environment": [
2   ...
3   {
4     "name": "DD_APM_ENABLED",
5     "value": "true"
6   },
7   {
8     "name": "DD_APM_ADDITIONAL_ENDPOINTS",
9     "value": "{\"https://<uid>.middleware.io\": [\"<MW_API_KEY>\"]}"
10   },
11 ]

Add Log Config

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping Logs in the Datadog Agent.

The four environment variables mentioned above are in addition to existing environment variables (e.g. DD_API_KEY, DD_SITE, etc.) that may already be defined in your Datadog Agent’s task definition.

1"environment": [
2  {
3    "name": "DD_LOGS_CONFIG_USE_HTTP",
4    "value": "true"
5  },
6  {
7     "name": "DD_LOGS_ENABLED",
8     "value": "true"
9   },
10   {
11     "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
12     "value": "true"
13   },
14   {
15     "name": "DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS",
16     "value": "[{\"api_key\": \"<MW_API_KEY>\", \"Host\": \"<uid>.middleware.io\", \"Port\": 443, \"is_reliable\": true}]"
17   }
18 ]

Update Agent Service

Once you have updated the ECS task definition for the Datadog Agent, update the relevant Datadog Agent Service to redeploy the agent with your new configuration. APM traces and logs will start flowing into your Middleware account.

FAQ

How do I stop sending APM traces and logs to Datadog and only send them to Middleware?

If you want to stop sending APM traces and logs to Datadog, you can change the api_key (or environment variable DD_API_KEY) in the /etc/datadog/datadog.yaml file to something invalid.

The Datadog Agent does not work if you comment out your api_key or set it to an empty value.

Below is an example of setting your api_key and site to invalid values:

1api_key: usingmiddleware
2site: example.com

Why can't I see my APM traces and logs on Middleware?

Execute the following command to ensure the Datadog Linux Agent is in an active state:

1sudo systemctl status datadog-agent

Verify that your Middleware API key and target are correct. The apm_config target is http://<uid>.middleware.io whereas the logs_config Host field value is <uid>.middleware.io (without https://).
Make sure you only have one apm_config and logs_config section in the following configuration file: /etc/datadog/datadog.yaml
Check your Datadog Kubernetes Agent. Ensure the datadog-agent and cluster-agent pods are operational with a liveness and readiness check for both pods.
Check your Kubernetes Helm chart installation method by checking the datadog-values.yaml and ensure that the agent section is at the same level as the datadog section. The agent section should NOT be inside the datadog section.

Need assistance or want to learn more about Middleware? Contact our support team at [email protected] or join our Slack channel.