---
title: "Amazon ECS"
description: "Set up AWS ECS monitoring in Middleware to track container performance, metrics, and operational insights effectively."
section: "ECS Monitoring"
breadcrumb: "ECS Monitoring > Setup ECS"
canonical: "https://docs.middleware.io/get-started/agent/amazon-ecs"
---
# Amazon ECS

The Middleware Agent collects task and container telemetry from Amazon Elastic Container Service (ECS) workloads on Amazon EC2 or AWS Fargate. Log collection and application tracing provide additional context when configured.

Use ECS telemetry to investigate task performance and container resource utilization, and application behavior across your ECS clusters..

For example, task metrics and logs can help explain why a workload slows down or stops unexpectedly.

**Note**: Amazon ECS Anywhere and database integrations are not currently supported through this installation.

## How it works

Middleware supports two instrumentation methods:

- **Manual instrumentation**: Configure the Middleware Agent in your ECS task definitions. On Fargate, the Agent runs as a sidecar container. On EC2, it runs as a Daemon service on each container instance.
- **Auto-instrumentation**: Use the `mw-ecs` CLI to automatically add the Middleware Agent, OpenTelemetry instrumentation libraries, and Fluent Bit log routing to existing ECS task definitions.

The Middleware Agent collects ECS metrics through the ECS Task Metadata Endpoint. Container metrics on EC2 can also be collected through Docker.

## Prerequisites

Before installing the Middleware Agent, verify the following:

- An AWS account with an existing ECS cluster.
- ECS workloads running on EC2 or Fargate.
- Permissions to create, register, and update ECS task definitions and services.
- A Middleware account with an API key (`MW_API_KEY`) and target URL (`MW_TARGET`).
- Network connectivity from the Agent to your Middleware target URL over HTTPS.

For auto-instrumentation, additional AWS credentials and CLI requirements are listed in the relevant section.

## Installation

Choose the instrumentation method that matches your environment.

| Method | Use case |
| --- | --- |
| Manual — Fargate | Deploy the Middleware Agent alongside individual ECS application tasks |
| Manual — EC2 | Monitor ECS workloads by running one Agent per EC2 container instance |
| Auto-instrumentation | Add monitoring to existing ECS task definitions using the `mw-ecs` CLI |

## Manual instrumentation

### Method 1: Install on ECS Fargate

For Fargate, the Middleware Agent runs as a sidecar container within each ECS task.

A task configured for metrics and log collection contains three containers:

1. **Application container**: Runs your application.
2. **Middleware Agent**: Collects and forwards telemetry to Middleware.
3. **Fluent Bit**: Routes application logs to the Middleware Agent using FireLens.

#### 1 Add the Middleware Agent container

Open your ECS task definition and add the following container to the `containerDefinitions` array.

```json
{
  "name": "mw-agent",
  "image": "ghcr.io/middleware-labs/mw-host-agent:master",
  "cpu": 256,
  "portMappings": [
    {
      "name": "8006-tcp",
      "containerPort": 8006,
      "hostPort": 8006,
      "protocol": "tcp",
      "appProtocol": "http"
    }
  ],
  "essential": true,
  "environment": [
    {
      "name": "MW_API_KEY",
      "value": "<MW_API_KEY>"
    },
    {
      "name": "MW_TARGET",
      "value": "https://<MW_UID>.middleware.io:443"
    }
  ],
  "mountPoints": [],
  "volumesFrom": [],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "<YOUR_LOGS_GROUP>",
      "mode": "non-blocking",
      "awslogs-create-group": "true",
      "max-buffer-size": "25m",
      "awslogs-region": "<YOUR_REGION>",
      "awslogs-stream-prefix": "ecs"
    }
  }
}
```

Replace the placeholders with your Middleware credentials, AWS Region, and CloudWatch Logs group.

**Note**: For production deployments, use AWS Secrets Manager or another supported secret-management method instead of storing the API key in plaintext.

#### 2 Collect ECS metrics

After adding the Middleware Agent sidecar, ECS task and container metrics are collected automatically.

The Agent uses the ECS Task Metadata Endpoint to retrieve task and container resource information.

No additional metrics configuration is required for this step.

#### 3 Configure task log collection

To forward application logs to Middleware, add a Fluent Bit sidecar container and configure AWS FireLens.

Add the following container to your ECS task definition:

```json
{
  "name": "log_router",
  "image": "amazon/aws-for-fluent-bit:stable",
  "cpu": 0,
  "portMappings": [],
  "essential": true,
  "environment": [],
  "mountPoints": [],
  "volumesFrom": [],
  "user": "0",
  "firelensConfiguration": {
    "type": "fluentbit",
    "options": {
      "enable-ecs-log-metadata": "true"
    }
  }
}
```

Next, add the following `logConfiguration` to your primary application container:

```json
"logConfiguration": {
  "logDriver": "awsfirelens",
  "options": {
    "Host": "127.0.0.1",
    "Name": "forward",
    "Port": "8006"
  }
}
```

FireLens forwards application logs to the Middleware Agent on port `8006`.

Make sure the Middleware Agent sidecar is running before verifying log collection.

#### 4 Parse structured JSON logs (optional)

By default, FireLens forwards JSON logs as a structured payload, which Middleware displays in the log body.

To display the `Message` field as the log body and keep other JSON fields as attributes, update the Fluent Bit configuration:

```json
"firelensConfiguration": {
  "type": "fluentbit",
  "options": {
    "enable-ecs-log-metadata": "true",
    "config-file-type": "file",
    "config-file-value": "/fluent-bit/configs/parse-json.conf"
  }
}
```

The application must emit structured JSON logs containing a Message field for this parsing configuration.

#### 5 Deploy the updated task definition

After configuring the containers:

1. Register a new revision of the ECS task definition.
2. Update your ECS service to use the new revision.
3. Deploy the updated task.
4. Verify that the application, Middleware Agent, and Fluent Bit containers are running.

**Complete task definition example**

{Retain the complete Fargate task definition JSON from the existing documentation, including the application container, Middleware Agent, and Fluent Bit log router.}

### Method 2: Install on ECS EC2

For ECS workloads running on EC2, deploy the Middleware Agent as a Daemon service.

This runs one Middleware Agent task on each eligible EC2 container instance in the ECS cluster.

#### 1 Create the Middleware Agent task definition

Create a task definition named `mw-agent-task`.

Choose the network mode based on your ECS environment:

| Network mode | Description |
| --- | --- |
| `host` | Uses the EC2 host network directly |
| `bridge` | Uses Docker bridge networking |
| `awsvpc` | Assigns an independent network interface to the Agent task |

The Agent task definition must include:

- Middleware Agent image: `ghcr.io/middleware-labs/mw-host-agent:master`
- `MW_API_KEY` and `MW_TARGET` environment variables
- Required host filesystem mounts
- Container ports `9319`, `9320`, and `8006`
- EC2 launch compatibility

Use the appropriate task definition for your network mode.

**Host network mode**

{Retain the complete Host Network Mode JSON from the existing documentation.}

**Bridge network mode**

{Retain the complete Bridge Network Mode JSON from the existing documentation.}

**AWSVPC network mode**

{Retain the complete AWSVPC Network Mode JSON from the existing documentation.}

**Note**: The EC2 configuration requires host-level mounts and privileged container access. Review these permissions before deploying the Agent.

#### 2 Register the task definition

You can register the Middleware Agent task definition using the AWS CLI or AWS Management Console.

**Using AWS CLI**

Save your task definition as `mw-ecs-agent.json` and run:

```bash
aws ecs register-task-definition \
  --cli-input-json file://mw-ecs-agent.json
```

**Using AWS Management Console**

1. Open **Amazon ECS → Task Definitions**.
2. Click **Create new task definition**.
3. Select the JSON configuration editor.
4. Paste your Middleware Agent task definition.
5. Register the task definition.

#### 3 Deploy the Agent as a Daemon service

1. Open **Amazon ECS → Clusters**.
2. Select the ECS cluster you want to monitor.
3. Click **Create service**.
4. Under **Compute options**, select **Launch type** and choose **EC2**.
5. Under **Deployment configuration**, select:

   - **Application type**: Service
   - **Task definition**: mw-agent-task
   - **Revision**: Latest registered revision
   - **Service type**: DAEMON
6. Enter a service name and create the service.

The Daemon scheduling strategy runs one Middleware Agent task per eligible EC2 container instance.

#### 4 Configure service discovery for AWSVPC (optional)

If you use awsvpc network mode, configure service discovery so application tasks can communicate with the Middleware Agent.

1. Enable **Service discovery** in the ECS service configuration.
2. Create or select an existing namespace.
3. Set the service discovery name to `mw-agent`.
4. Enable ECS task health propagation.
5. Configure DNS records as required by your VPC.

For example, if the service name is `mw-agent` and the namespace is `mw-agent-ns`, the service discovery endpoint is:

```bash
mw-agent.mw-agent-ns
```

This endpoint is used by application tasks to send telemetry to the Middleware Agent.

#### 5 Configure application traces and logs

Update the task definition for each application you want to monitor.

**For Host or Bridge network mode**

Set the following environment variable in your application container:

```bash
MW_AGENT_SERVICE=172.17.0.1
```

**For AWSVPC network mode**

Use the Middleware Agent service discovery endpoint:

```bash
MW_AGENT_SERVICE=mw-agent.mw-agent-ns
```

For log collection, configure your application's logging settings to forward logs to the Agent on port `8006`.

Use the following destinations:

| Network mode | Log destination |
| --- | --- |
| Host / Bridge | `172.17.0.1:8006` |
| AWSVPC | `mw-agent.mw-agent-ns:8006` |

After updating the application task definition, register a new revision and update the ECS service to use it.

#### 6 Verify the Agent installation

Go to **Amazon ECS → Clusters → [Your Cluster] → Services**.

Verify that:

- The Middleware Agent Daemon service is running.
- One Agent task is scheduled on each eligible EC2 container instance.
- The Agent tasks report a healthy running status.
- Application tasks can communicate with the Agent.

## Auto-instrumentation

Middleware provides the `mw-ecs` CLI to add observability components to existing ECS task definitions without manually configuring sidecars.

Auto-instrumentation supports **Java**, **Node.js**, and **Python**, with automatic language detection.

The CLI adds:

- **Middleware Agent sidecar**: Collects and forwards telemetry.
- **APM init container**: Provides OpenTelemetry instrumentation libraries.
- **Fluent Bit sidecar**: Routes application logs using FireLens.

## Prerequisites

Before using auto-instrumentation, make sure you have:

- An existing ECS task definition.
- AWS credentials configured through environment variables, an AWS CLI profile, or an IAM role.
- Middleware API key and target URL.
- An active Middleware AWS integration if you want to discover task definitions from the Middleware UI.
- Go 1.21 or later, only if building the mw-ecs CLI from source.

#### 1 Install the mw-ecs CLI

Run the following command on the machine used to manage your ECS deployments:

```bash
MW_API_KEY=<MW_API_KEY> \
MW_TARGET=https://<MW_UID>.middleware.io:443 \
bash -c "$(curl -L https://install.middleware.io/scripts/mw-ecs-install.sh)"
```

The installation script downloads the `mw-ecs` CLI and configures it with your Middleware credentials.

You can also build the CLI from the [mw-ecs GitHub repository](https://github.com/middleware-labs/mw-ecs-instrumentation).

#### 2 Discover ECS task definitions

List active ECS task definition families and their instrumentation status:

```bash
mw-ecs discover --region <YOUR_REGION>
```

The output includes:

- Task definition family
- Latest available revision
- MW agent sidecar enabled status
- APM enabled status
- Logs enabled status

#### 3 Instrument task definitions

To instrument a specific task definition:

```bash
mw-ecs instrument \
  --task-definition <FAMILY_NAME>:<REVISION> \
  --enable-apm \
  --enable-logs \
  --register \
  --region <YOUR_REGION>
```

This command adds the required Middleware components and registers a new ECS task definition revision.

#### Instrument multiple task definitions

You can instrument multiple task definitions in a single command:

```bash
mw-ecs instrument \
  --task-definition my-api-service:3 \
  --task-definition my-worker-service:5 \
  --task-definition my-web-app:2 \
  --enable-apm \
  --enable-logs \
  --register \
  --region us-east-1
```

#### Instrument all active task definitions

To instrument all active task definition families:

```bash
mw-ecs instrument \
  --all \
  --enable-apm \
  --enable-logs \
  --register \
  --region <YOUR_REGION>
```

**Note**: Use --dry-run to review the changes before registering new task definition revisions.

#### Instrumentation options

| Flag | Description |
| --- | --- |
| `--task-definition` | Task definition family and revision, or full ARN |
| `--all` | Instrument all active task definition families |
| `--language` | Override language detection (`java`, `node`, `python`) |
| `--libc` | Override C library detection (`glibc`, `musl`) |
| `--enable-apm` | Enable application trace collection |
| `--enable-logs` | Add FireLens log routing |
| `--register` | Register a new ECS task definition revision |
| `--run` | Run the task after registration |
| `--dry-run` | Preview changes without applying them |
| `--region` | AWS Region |
| `--cluster` | ECS cluster name |
| `--subnets` | Subnets for AWSVPC networking |
| `--security-groups` | Security groups for AWSVPC networking |

#### 4 Verify instrumentation

After registering the instrumented task definition, run:

```bash
mw-ecs discover --region <YOUR_REGION>
```

Verify that the task definition reports Middleware instrumentation.

Next, update your ECS service to use the newly registered revision.

**Note**: Registering a task definition does not automatically update an existing ECS service. Deploy the new revision to start collecting application telemetry.

## Auto-instrument from the Middleware UI

You can also generate instrumentation commands directly from the Middleware UI.

1. Go to [Installation → Amazon ECS](https://app.middleware.io/installation/agent/aws-ecs).
2. Select **Auto-Instrumentation**.
3. Configure the AWS integration if it is not already connected.
4. Review the discovered ECS task definitions.
5. Select the task definitions you want to instrument.
6. Enable the required options:

   - **Enable APM**
   - **Enable Logs**
   - **Register**
7. Copy the generated `mw-ecs instrument` command.
8. Run the command on the machine where the `mw-ecs` CLI is installed.

After registration, update the ECS service to use the new task definition revision.

## Verify data collection

After deploying the Middleware Agent and updating your ECS services, allow a few minutes for telemetry to appear.

Go to [Infrastructure Monitoring → ECS](https://app.middleware.io/infrastructure/aws.ecs.task/list) and verify that your ECS resources are reporting data.

After data collection starts, you can:

- View ECS clusters, services, tasks, task definitions, and containers.
- Monitor container CPU, memory, and other collected resource metrics.
- View application and container logs in [Log Monitoring](https://app.middleware.io/logs).
- View application traces in [APM](https://docs.middleware.io/features/apm/traces), if instrumentation is enabled.
- [Create alerts](https://docs.middleware.io/features/alerts/manage-alerts) for collected metrics.
- Create custom dashboard or view predefined dashboard, go to Dashboard Builder and select ECS Metrics Dashboard.

## Roll back instrumentation

To revert an instrumented task definition to a previous revision, run:

```bash
mw-ecs rollback --task-definition <FAMILY_NAME>:<REVISION>
```

Verify the resulting task definition configuration and update the ECS service to use the intended revision.

## Uninstall

#### Remove the mw-ecs CLI

To uninstall the `mw-ecs` CLI:

```bash
bash -c "$(curl -L https://install.middleware.io/scripts/mw-ecs-uninstall.sh)"
```

This removes the CLI tool from your machine.

**Note**: Removing the CLI does not remove Middleware components from task definitions that were previously instrumented.

To stop collecting telemetry, roll back the instrumented task definitions and deploy the corresponding revisions.

#### Remove manually installed Agents

For Fargate deployments, remove the Middleware Agent and Fluent Bit containers from the task definition, register a new revision, and deploy it.

For EC2 deployments, remove the Middleware Agent Daemon service after confirming that application tasks no longer depend on its telemetry endpoints.

## Troubleshooting

### Middleware Agent is not running

If the Agent task fails to start:

- Verify that the ECS task definition contains the required Agent configuration.
- Check the ECS task stopped reason and container exit codes.
- Verify that MW_API_KEY and MW_TARGET are configured correctly.
- Review Agent logs in the configured CloudWatch Logs group.
- Confirm that the task has sufficient CPU and memory resources.

### No ECS metrics in Middleware

If the Agent is running but ECS metrics are not available:

- Verify outbound connectivity to your Middleware target URL on port 443.
- Check that the Agent can access the ECS Task Metadata Endpoint.
- Verify that the Agent is deployed using the correct launch type and network mode.
- Confirm that ECS tasks are running and reporting resource data.

### Application logs are missing

If application logs do not appear:

- Verify that the Fluent Bit container is running.
- Check that the application uses the `awsfirelens` log driver.
- Verify that the configured log destination points to the Middleware Agent on port `8006`.
- Check Fluent Bit logs for forwarding or connection errors.

### Application traces are missing

If auto-instrumentation is enabled but traces do not appear:

- Verify that the application language is supported.
- Confirm that the instrumented task definition was registered successfully.
- Verify that the ECS service is running the new task definition revision.
- Check that the APM init container completed successfully.
- Review the application and Middleware Agent logs for instrumentation errors.

### Task definitions are not discovered

If task definitions do not appear in the Middleware UI:

- Verify that the AWS integration is connected.
- Check that the AWS account and Region are correct.
- Confirm that the integration has permission to discover ECS task definitions.
- Verify that the task definitions are active.

### Installation menu is missing

If you cannot access the ECS installation page, your user role may not have the required `Installation` permission.

Contact your Middleware administrator to update your permissions in **Settings**.

If the issue persists, contact Middleware Support at support@middleware.io.

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us page or join our Slack channel.