Installing the Datadog Agent in Middleware (Dual Shipping)

Middleware supports the ingestion of APM traces, metrics and logs from the Datadog Agent's Dual Shipping feature.

Prerequisites

Linux Agent

Add Trace Configuration

Add the following additional_endpoints within the apm_config section in your datadog.yaml file:

apm_config:
  enabled: true
  additional_endpoints:
    "https://<uid>.middleware.io":
    - "<MW_API_KEY>"

Add Log Configuration

Add the following additional_endpoints within the logs_config section in your datadog.yaml file:

logs_config:
    use_http: true
    additional_endpoints:
    - api_key: "<MW_API_KEY>"
        Host: "<uid>.middleware.io"
        Port: 443
        is_reliable: true

Add Metric Configuration

Add the following to the additional_endpoints section.

additional_endpoints:
  "https://<uid>.middleware.io":
  - "<MW_API_KEY>"

In order to get visibility into the processes running on your infrastructure, enable live processes collection under the process_config section and add additional_endpoints as shown below:

process_config:
  process_collection:
    enabled: true
  additional_endpoints:
    "https://<uid>.middleware.io":
    - "<MW_API_KEY>"

Restart Datadog Agent

Run the following code in your terminal to restart the Datadog Agent:

sudo systemctl restart datadog-agent

Kubernetes Agent

The Datadog Kubernetes agent supports Helm and Operator methods of installation. Middleware can ingest traces, metrics and logs from the Datadog agent installed using any of these methods.

Helm chart

If you have installed the Datadog Agent using their Helm Chart, you must add the following section in your datadog-values.yaml file that you created in Step 3 of their guide:

agents:
  useConfigMap: true
  customAgentConfig:
    logs_config:
      container_collect_all: true
      use_http: true
      additional_endpoints:
        - api_key: "<MW_API_KEY>"
          Host: "<uid>.middleware.io"
          Port: 443

Below is an example of the full datadog-values.yaml file that will send logs,metrics and APM traces to the Middleware platform:

datadog:
  site: "us5.datadoghq.com"
  clusterName: <your cluster name>
  env:
    - name: DD_CHECKS_TAG_CARDINALITY
      value: "orchestrator"
  kubelet:
    tlsVerify: false
  apiKeyExistingSecret: datadog-secret
  processAgent:

Kubernetes Operator

If you have installed the Datadog Agent using their Kubernetes operator, you must add the following section in datadog-agent.yaml that you create in step 3 of their guide.

You must add this section under the spec section.

spec:
  ...
  override:
    nodeAgent:
      customConfigurations:
        datadog.yaml:
          configData: |
            logs_enabled: true
            apm_config:
              enabled: true

Below is an example of full datadog-agent.yaml file that will send logs and APM traces to the Middleware platform:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    site: "us5.datadoghq.com"
    kubelet:
      tlsVerify: false
    credentials:

To find out the resource version currently in use by Datadog Kubernetes operator, issue the command below:

kubectl api-resources --api-group=datadoghq.com

The output of above command should look like below:

NAME              SHORTNAMES   APIVERSION               NAMESPACED   KIND
datadogagents     dd           datadoghq.com/v2alpha1   true         DatadogAgent
...
...

APIVERSION should show datadoghq.com/v2alpha1 for the Datadog agent.

Amazon ECS on EC2 Cluster

Middleware can ingest logs and APM traces from the Datadog Agent running on ECS clusters that use EC2 instances.

Add Trace Config

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping APM traces in the Datadog Agent.

"environment": [
   ...
   {
     "name": "DD_APM_ENABLED",
     "value": "true"
   },
   {
     "name": "DD_APM_ADDITIONAL_ENDPOINTS",
     "value": "{\"https://<uid>.middleware.io\": [\"<MW_API_KEY>\"]}"
   },

Add Log Config

Add the following JSON script to your ECS task definition. The below environment variables are required for dual shipping Logs in the Datadog Agent.

"environment": [
  {
    "name": "DD_LOGS_CONFIG_USE_HTTP",
    "value": "true"
  },
  {
     "name": "DD_LOGS_ENABLED",
     "value": "true"
   },
   {

Update Agent Service

Once you have updated the ECS task definition for the Datadog Agent, update the relevant Datadog Agent Service to redeploy the agent with your new configuration. APM traces and logs will start flowing into your Middleware account.

FAQ

api_key: usingmiddleware
site: example.com

sudo systemctl status datadog-agent