Installing the Datadog Agent in Middleware (Dual Shipping)
Middleware supports the ingestion of APM traces, metrics and logs from the Datadog Agent's Dual Shipping feature.
This feature is only supported for Datadog's Linux and Kubernetes Agents
Prerequisites
1 Traces
Datadog Agent version 6.7.0
or above
2 HTTP Logs
Datadog Agent version 6.13
or above
3 Metrics
Datadog Agent version 6.17
or above
Linux Agent
Add Trace Configuration
Add the following additional_endpoints
within the apm_config
section in your datadog.yaml
file:
If you already have the apm_config
section in your datadog.yaml
file, add an additional endpoint in the same section. Datadog agent does not support multiple apm_config
sections.
The Linux Datadog Agent is configured by default in the following YAML
file: /etc/datadog-agent/datadog.yaml
1apm_config:
2 enabled: true
3 additional_endpoints:
4 "https://<uid>.middleware.io":
5 - "<MW_API_KEY>"
Add Log Configuration
Add the following additional_endpoints
within the logs_config
section in your datadog.yaml
file:
Unlike the APM Traces, you only need to provide your hostname (<uid>.middleware.io
) without the protocol scheme (https://
)
1logs_config:
2 use_http: true
3 additional_endpoints:
4 - api_key: "<MW_API_KEY>"
5 Host: "<uid>.middleware.io"
6 Port: 443
7 is_reliable: true
Add Metric Configuration
Add the following to the additional_endpoints
section.
1additional_endpoints:
2 "https://<uid>.middleware.io":
3 - "<MW_API_KEY>"
Unlike traces and logs which are under apm_config
and logs_config
sections, the additional_endpoints
section for metrics is at the top level (same level as apm_config
and logs_config
).
In order to get visibility into the processes running on your infrastructure, enable live processes collection under the process_config
section and add additional_endpoints
as shown below:
1process_config:
2 process_collection:
3 enabled: true
4 additional_endpoints:
5 "https://<uid>.middleware.io":
6 - "<MW_API_KEY>"
Restart Datadog Agent
Run the following code in your terminal to restart the Datadog Agent:
1sudo systemctl restart datadog-agent
Kubernetes Agent
The Datadog Kubernetes agent supports Helm and Operator methods of installation. Middleware can ingest traces, metrics and logs from the Datadog agent installed using any of these methods.
Helm chart
If you have installed the Datadog Agent using their Helm Chart, you must add the following section in your datadog-values.yaml
file that you created in Step 3 of their guide:
You can remove either the logs_config
or the apm_config
from the below file depending on what data you would like to send to Middleware.
1agents:
2 useConfigMap: true
3 customAgentConfig:
4 logs_config:
5 container_collect_all: true
6 use_http: true
7 additional_endpoints:
8 - api_key: "<MW_API_KEY>"
9 Host: "<uid>.middleware.io"
10 Port: 443
11 is_reliable: true
12 apm_config:
13 additional_endpoints:
14 "https://<uid>.middleware.io":
15 - "<MW_API_KEY>"
16 # collects metrics
17 additional_endpoints:
18 "https://<uid>.middleware.io":
19 - "<MW_API_KEY>"
20 # collects kubernetes object data
21 orchestrator_explorer:
22 orchestrator_additional_endpoints:
23 "https://<uid>.middleware.io":
24 - "<MW_API_KEY>"
25 # collects process and container data
26 process_config:
27 additional_endpoints:
28 "https://<uid>.middleware.io":
29 - "<MW_API_KEY>"
Below is an example of the full datadog-values.yaml
file that will send logs,metrics and APM traces to the Middleware platform:
1datadog:
2 site: "us5.datadoghq.com"
3 clusterName: <your cluster name>
4 env:
5 - name: DD_CHECKS_TAG_CARDINALITY
6 value: "orchestrator"
7 kubelet:
8 tlsVerify: false
9 apiKeyExistingSecret: datadog-secret
10 processAgent:
11 enabled: true
12 processCollection: true
13 cluster-agent:
14 enabled: true
15 orchestratorExplorer:
16 enabled: true
17 kubeStateMetricsCore:
18 enabled: true
19 metricsProvider:
20 enabled: true
21 #collectApiServicesMetrics: false
22 containerImageCollection:
23 enabled: true
24 clusterChecks:
25 enabled: true
26 useClusterChecksRunners: true
27 targetSystem: linux
28 logs:
29 enabled: true
30 containerCollectAll: true
31 # Essential configuration for Kubernetes and container data correlation
32 # This configuration enables the Agent to collect metrics from kubelet and containerintegration and
33 # associate them with the correct Kubernetes cluster.
34 confd:
35 kubelet.yaml: |-
36 ad_identifiers:
37 - _kubelet
38 instances:
39 - tags:
40 - dd_cluster_name:<your cluster name>
41 container.yaml: |-
42 ad_identifiers:
43 - _container
44 instances:
45 - tags:
46 - dd_cluster_name:<your cluster name>
47agents:
48 useConfigMap: true
49 customAgentConfig:
50 logs_config:
51 container_collect_all: true
52 use_http: true
53 additional_endpoints:
54 - api_key: "<MW_API_KEY>"
55 Host: "<uid>.middleware.io"
56 Port: 443
57 is_reliable: true
58 apm_config:
59 additional_endpoints:
60 "https://<uid>.middleware.io":
61 - "<MW_API_KEY>"
62 # collects metrics
63 additional_endpoints:
64 "https://<uid>.middleware.io":
65 - "<MW_API_KEY>"
66 # collects kubernetes object data
67 orchestrator_explorer:
68 orchestrator_additional_endpoints:
69 "https://<uid>.middleware.io":
70 - "<MW_API_KEY>"
71 # collects process and container data
72 process_config:
73 additional_endpoints:
74 "https://<uid>.middleware.io":
75 - "<MW_API_KEY>"
76clusterAgent:
77 useConfigMap: true
78 enabled: true
79 # Kubenetes State related metrics
80 datadog_cluster_yaml:
81 orchestrator_explorer:
82 orchestrator_additional_endpoints:
83 "https://<uid>.middleware.io":
84 - "<MW_API_KEY>"
85 env:
86 - name: DD_ADDITIONAL_ENDPOINTS
87 value: '{"https://<uid>.middleware.io": ["<MW_API_KEY>"]}'
Kubernetes Operator
If you have installed the Datadog Agent using their Kubernetes operator, you must add the following section in datadog-agent.yaml
that you create in step 3 of their guide.
The current specification section only enables logs and traces. To enable metrics, traces, and logs, please refer to the complete YAML configuration later below.
1spec:
2 ...
3 override:
4 nodeAgent:
5 customConfigurations:
6 datadog.yaml:
7 configData: |
8 logs_enabled: true
9 apm_config:
10 enabled: true
11 additional_endpoints:
12 - "https://<uid>.middleware.io":
13 - "<MW_API_KEY>"
14 logs_config:
15 use_http: true
16 container_collect_all: true
17 additional_endpoints:
18 - api_key: "<MW_API_KEY>"
19 Host: "<uid>.middleware.io"
20 Port: 443
21 is_reliable: true
Below is an example of full datadog-agent.yaml
file that will send metrics, logs and APM traces to the Middleware platform:
1kind: DatadogAgent
2apiVersion: datadoghq.com/v2alpha1
3metadata:
4 name: datadog
5spec:
6 global:
7 clusterName: "your-cluster-name"
8 site: us5.datadoghq.com
9 credentials:
10 apiSecret:
11 secretName: datadog-secret
12 keyName: api-key
13 kubelet:
14 tlsVerify: false
15 features:
16 orchestratorExplorer:
17 enabled: true
18 scrubContainers: true
19 kubeStateMetricsCore:
20 enabled: true
21 processDiscovery:
22 enabled: true
23 oomKill:
24 enabled: true
25 liveContainerCollection:
26 enabled: true
27 eventCollection:
28 collectKubernetesEvents: true
29 logCollection:
30 enabled: true
31 containerCollectAll: true
32 liveProcessCollection:
33 enabled: true
34 apm:
35 enabled: true
36 hostPortConfig:
37 enabled: true
38 override:
39 clusterAgent:
40 env:
41 - name: DD_CLUSTER_NAME # Add explicit cluster name env var
42 value: "your-cluster-name"
43 - name: DD_HOSTNAME
44 valueFrom:
45 fieldRef:
46 fieldPath: spec.nodeName
47 - name: DD_ADDITIONAL_ENDPOINTS
48 value: '{"https://<uid>.middleware.io": ["<MW_API_KEY>"]}'
49 - name: DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS
50 value: '{"https://<uid>.middleware.io": ["<MW_API_KEY>"]}'
51 nodeAgent:
52 env:
53 - name: DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS
54 value: '{"https://<uid>.middleware.io": ["<MW_API_KEY>"]}'
55 - name: DD_HOSTNAME
56 valueFrom:
57 fieldRef:
58 fieldPath: spec.nodeName
59 customConfigurations:
60 datadog.yaml:
61 configData: |-
62 additional_endpoints:
63 "https://<uid>.middleware.io":
64 - "<MW_API_KEY>"
65 logs_config:
66 container_collect_all: true
67 use_http: true
68 additional_endpoints:
69 - api_key: "<MW_API_KEY>"
70 Host: "<uid>.middleware.io"
71 Port: 443
72 is_reliable: true
73 apm_config:
74 additional_endpoints:
75 "https://<uid>.middleware.io":
76 - "<MW_API_KEY>"
77 process_config:
78 additional_endpoints:
79 "https://<uid>.middleware.io":
80 - "<MW_API_KEY>"
81 extraConfd:
82 configDataMap:
83 kubelet.yaml: |-
84 ad_identifiers:
85 - _kubelet
86 instances:
87 - tags:
88 - dd_cluster_name:your-cluster-name
89 container.yaml: |-
90 ad_identifiers:
91 - _container
92 instances:
93 - tags:
94 - dd_cluster_name:your-cluster-name
Middleware only supports datadoghq.com/v2alpha1 apiVersion
for Datadog agent resource.
To find out the resource version currently in use by Datadog Kubernetes operator, issue the command below:
1kubectl api-resources --api-group=datadoghq.com
The output of above command should look like below:
1NAME SHORTNAMES APIVERSION NAMESPACED KIND
2datadogagents dd datadoghq.com/v2alpha1 true DatadogAgent
3...
4...
APIVERSION
should show datadoghq.com/v2alpha1
for the Datadog agent.
Amazon ECS on EC2 Cluster
Middleware can ingest logs and APM traces from the Datadog Agent running on ECS clusters that use EC2 instances.
For more information on ingesting Log and APM Traces using Amazon ECS with Datadog, navigate here.
Add Trace Config
Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping APM traces in the Datadog Agent.
The DD_APM_ENABLED
and DD_APM_ADDITION_ENDPOINTS
environment variables are in addition to existing environment variables (e.g. DD_API_KEY
, DD_SITE
, etc.) that may already be defined in your Datadog Agent’s task definition.
1"environment": [
2 ...
3 {
4 "name": "DD_APM_ENABLED",
5 "value": "true"
6 },
7 {
8 "name": "DD_APM_ADDITIONAL_ENDPOINTS",
9 "value": "{\"https://<uid>.middleware.io\": [\"<MW_API_KEY>\"]}"
10 },
11 ]
Add Log Config
Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping Logs in the Datadog Agent.
The four environment variables mentioned above are in addition to existing environment variables (e.g. DD_API_KEY
, DD_SITE
, etc.) that may already be defined in your Datadog Agent’s task definition.
1"environment": [
2 {
3 "name": "DD_LOGS_CONFIG_USE_HTTP",
4 "value": "true"
5 },
6 {
7 "name": "DD_LOGS_ENABLED",
8 "value": "true"
9 },
10 {
11 "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
12 "value": "true"
13 },
14 {
15 "name": "DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS",
16 "value": "[{\"api_key\": \"<MW_API_KEY>\", \"Host\": \"<uid>.middleware.io\", \"Port\": 443, \"is_reliable\": true}]"
17 }
18 ]
Update Agent Service
Once you have updated the ECS task definition for the Datadog Agent, update the relevant Datadog Agent Service to redeploy the agent with your new configuration. APM traces and logs will start flowing into your Middleware account.
FAQ
How do I stop sending APM traces and logs to Datadog and only send them to Middleware?
If you want to stop sending APM traces and logs to Datadog, you can change the api_key
(or environment variable DD_API_KEY
) in the /etc/datadog/datadog.yaml
file to something invalid.
The Datadog Agent does not work if you comment out your api_key
or set it to an empty value.
Below is an example of setting your api_key
and site
to invalid values:
1api_key: usingmiddleware
2site: example.com
Why can't I see my APM traces and logs on Middleware?
- Execute the following command to ensure the Datadog Linux Agent is in an active state:
1sudo systemctl status datadog-agent
Verify that your Middleware API key and target are correct. The
apm_config
target ishttp://<uid>.middleware.io
whereas thelogs_config
Host
field value is<uid>.middleware.io
(withouthttps://
).Make sure you only have one
apm_config
andlogs_config
section in the following configuration file:/etc/datadog/datadog.yaml
Check your Datadog Kubernetes Agent. Ensure the
datadog-agent
andcluster-agent
pods are operational with a liveness and readiness check for both pods.Check your Kubernetes Helm chart installation method by checking the
datadog-values.yaml
and ensure that theagent
section is at the same level as thedatadog
section. Theagent
section should NOT be inside thedatadog
section.
Need assistance or want to learn more about Middleware? Contact our support team at [email protected] or join our Slack channel.