--- title: "Amazon Web Services" description: "Step-by-step guide to configure Middleware integration with AWS for real-time monitoring, log tracking, and error detection." section: "Integrations" breadcrumb: "Integrations > Cloud Providers > Amazon Web Services (AWS)" canonical: "https://docs.middleware.io/integrations/aws-integration" --- # Amazon Web Services The Middleware AWS integration collects CloudWatch metrics from one or more AWS accounts across 90+ AWS namespaces and shows them in default dashboards. Connect through a CloudWatch Metric Stream or by polling the CloudWatch API. ## AWS Metrics Middleware’s AWS integration is able to collect metrics from multiple AWS accounts across approximately 90+ namespaces, such as EC2, S3, Lambda, Kinesis, RDS, ECS, and more. Middleware is able to collect metrics from your account either through a **[Kinesis Stream](/integrations/aws-integration#using-a-kinesis-stream)** or through **[API Polling](/integrations/aws-integration#api-polling)**. Each has their benefits which will be described below, and you can choose either one or use both simultaneously depending on your use case. ## Using a Kinesis Stream This approach uses CloudWatch Metric Streams and Amazon Data Firehose to fetch metrics. By default Cloudwatch Metric Streams send data with roughly two to three minute lag. Using this approach provides more granular control of what metrics you'd like sent to Middleware, from any AWS Namespace. See AWS' doc on **[CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html)** for more detail on how they work. The below diagram shows the required resources for Middleware's case, which you can either set up automatically using our Cloudformation template, or manually yourself. ![Kinesis Stream Approach](/images/aws/kinesis-stream-flow.webp) ### CloudFormation Template (Recommended) Middleware provides a CloudFormation template to automatically create all of the necessary resources in your AWS account. The set up will take 5 - 10 minutes and will immediately begin sending data upon completion. #### 1 Launch Stack You can access this template by clicking the **Launch Stack** button in your installation page. ![Kinesis Stream - Auto Method](/images/aws/kinesis-stream-approach.png) #### 2 Create IAM Role In order to run the Cloudformation stack, you will need an IAM Role that has permissions to deploy this stack. Middleware will prompt you to input an existing IAM Role if you have one. ![Kinesis Stream IAM Role Permission](/images/aws/kinesis-iam-role.png) If you do not have the necessary **IAM Role**, you will need to create one with the following policy, which you can do by using the below JSON, or via the admin console UI: ```yaml { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:*", "organizations:DescribeAccount", "organizations:DescribeOrganization", "organizations:DescribeOrganizationalUnit", "organizations:DescribePolicy", "organizations:ListChildren", "organizations:ListParents", "organizations:ListPoliciesForTarget", "organizations:ListRoots", "organizations:ListPolicies", "organizations:ListTargetsForPolicy" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "cloudformation:*" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "autoscaling:Describe*", "cloudwatch:*", "logs:*", "sns:*", "iam:GetPolicy", "iam:GetPolicyVersion", "iam:GetRole", "oam:ListSinks" ], "Resource": "*" }, { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "arn:aws:iam::*:role/aws-service-role/events.amazonaws.com/AWSServiceRoleForCloudWatchEvents*", "Condition": { "StringLike": { "iam:AWSServiceName": "events.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "oam:ListAttachedLinks" ], "Resource": "arn:aws:oam:*:*:sink/*" }, { "Action": [ "firehose:*" ], "Effect": "Allow", "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:*", "s3-object-lambda:*" ], "Resource": "*" } ] } ``` If using the admin console UI, follow the below steps: 1. Go to the **IAM page -> Roles** page, and click on **Create Role** 2. Select **AWS service** as the Trusted entity type and choose **CloudFormation** as the Use case 3. In the Permissions policies step, choose **IAMFullAccess**, **CloudWatchFullAccess**, **AWSCloudFormationFullAccess**, **AmazonKinesisFirehoseFullAccess**, and **AmazonS3FullAccess**, then click Next 4. Set a **Role Name** and click on **Create Role** #### 3 Create the Stack Once the IAM role is created and linked to Middleware, deploy the CloudFormation stack using the **Create Stack** button. This will create all the necessary resources needed to collect and send AWS metrics to the Middleware platform. ### Manual Creation We only recommend manual creation of the stack for teams that have strict policies around new components in their AWS account. If you run into issues during Manual creation please review each step diligently, and if you're stuck reach out to our support team for assistance. All steps required are outlined in this doc below, and are also accessible via the [Installation page](https://app.middleware.io/installation/integrations/aws) in your account. For more detail on the Firehose delivery stream and metric stream, see [AWS CloudWatch Metric Streams](/integrations/kinesis-streams). #### 1 Setup Firehose Stream 1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/kinesis) and open the Kinesis console 2. Choose **Data Firehose** in the navigation panel 3. Choose **Create delivery stream** 4. The stream’s Source should be **Direct PUT** and Destination set to **HTTP Endpoint** with the following details: - **Kinesis endpoint:** https://{ACCOUNT-UID}.middleware.io/v1/metrics/cloudwatch - **Access key:** Access Key is found in the [Installation page](https://app.middleware.io/installation/integrations/aws). 5. In the Backup settings, select an S3 backup bucket to receive any failed events that exceed the retry duration. > **Note:** Your `{ACCOUNT-UID}` is the subdomain of your Middleware account, for example the account `demo.middleware.io` has an `{ACCOUNT-UID}` = `demo`. #### 2 Create a CloudWatch Metric Stream 1. Open the [CloudWatch AWS console](https://console.aws.amazon.com/cloudwatch) 2. In the left navigation pane, choose **Metrics → Streams** 3. Click the **Create metric stream button** 4. Choose the **All namespaces** in the metric stream 5. Choose **Select an existing Firehose owned by your account**, and select the Firehose Delivery Stream you created earlier 6. Specify an **Output Format of OpenTelemetry 1.0** 7. Optionally, specify a name for this metric stream under **Metric Stream Name** 8. Finally, click on **Create metric stream** > **Note:** To ensure that your request metrics are accurately reported, it is important to [Enable Requests metrics](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) for your Amazon S3 buckets through the AWS console. > **Warning:** Keep in mind that Cloudwatch Metric Stream will only be fetching data from the namespaces which are in same region. > Ex: If you want to fetch metrics data from your EC2 machine running in us-east-2, consider creating the Cloudwatch Metric Stream in the us-east-2 as well. ## API Polling Middleware's API Polling only requires users to provide Middleware with API keys or a specific IAM Role to enable automatic polling of the selected resources in their AWS account. Although easier to set up, this approach has a five minute lag on all metrics compared to the Kinesis streams' three minute lag. The major benefits of this approach is that it requires no additional resources to be deployed in your AWS account, unlike our Kinesis Stream approach or other approaches used by our competitors. ![API Polling Approach](/images/aws/api-polling-flow.webp) To get started, access the AWS integration page inside of the installation pages, and choose from one of two approaches - **[Role Delegation](/integrations/aws-integration#role-delegation)** or **[Access Keys](/integrations/aws-integration#access-keys)**. Which method you choose will depend on your team's security policies. ![API Polling Approach](/images/aws/api-polling-approach.png) > **Note:** API Polling is achieved using Cloudwatch APIs which may incur costs on your AWS account. Please refer to [AWS pricing](https://aws.amazon.com/cloudwatch/pricing/) for further information about expected costs for CloudWatch APIs. ### Role Delegation Using Role Delegation is best if your team requires you to keep API keys internal or have strict policies around sharing. You can set up the required role using our CloudFormation Template or by manually creating the user. > **Note:** If you'd like to create the user Manually you can skip to step 3 below to find the required permissions. You will then input that user's ARN into step 4. ![Role Delegation Method](/images/aws/role-delegation-method.png) #### CloudFormation Template #### 1 Launch the CloudFormation Template To Access the template, click **Create Role** from within the Middleware app on the [Installation -> Integration -> API Polling](https://app.middleware.io/installation/integrations/aws) page. This will redirect you to CloudFormation with a pre-filled template, where you will need to select a IAM Role in the **Permissions** section. ![Stack for Create Role](/images/aws/create-role-stack.png) #### 2 Create IAM Role or Choose Admin You may either use the default Admin Role, or create a new one manually following the below steps: - Go to the [IAM -> Roles page](https://console.aws.amazon.com/iam/home?region=us-east-1#/roles), and click on **Create Role** - Select **AWS service** as the Trusted entity type, choose **CloudFormation** as the Use case, and proceed to the next step - Search for and check the box next to **IAMFullAccess**, then click Next - Set a **Role Name** and press **Create Role** - Set the permissions to below JSON ```yaml { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:*", "organizations:DescribeAccount", "organizations:DescribeOrganization", "organizations:DescribeOrganizationalUnit", "organizations:DescribePolicy", "organizations:ListChildren", "organizations:ListParents", "organizations:ListPoliciesForTarget", "organizations:ListRoots", "organizations:ListPolicies", "organizations:ListTargetsForPolicy" ], "Resource": "*" } ] } ``` #### 3 Run Template Once an IAM Role is selected, click **Run Stack** which will create a Role named **MiddlewareRoleIntegration** that Middleware will use to trigger AWS APIs on your behalf. This Role includes the following permissions: ```yaml { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:Get*", "cloudwatch:List*", "cloudwatch:Describe*", "rds:List*", "rds:Describe*", "ecs:List*", "ecs:Describe*", "ec2:List*", "ec2:Describe*", "s3:List*", "s3:GetBucketTagging", "pi:Get*", "pi:Describe*", "pi:List*", "logs:ListTagsLogGroup", "logs:FilterLogEvents", "logs:DescribeLogGroups", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeTags" ], "Resource": "*" } ] } ``` #### 4 Enter the Account ID and ARN Copy the **AWS Account ID** and **AWS Role ARN** of the created role into Middleware ![Role Delegation Details](/images/aws/role-delegation-details.png) #### 4 Choose Regions Choose your desired Namespaces that you'd like to poll metrics from ![Selection of Regions and Namespaces](/images/aws/regions-namespaces.png) #### 5 Save Configuration Input the desired regions and toggle on the Namespaces you'd like to poll metrics from, and click **Save Configuration** #### Manual Role Creation Manual Role Creation is not recommended as it has the most room for error. It is required however for some teams that have their AWS permissions heavily managed. If you hit issues please reach out to support. #### 1 Create Policy with Required Permissions Create a policy with the below permissions, and specify a suitable name for your policy, e.g., **MiddlewareRolePolicy**: ```yaml { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:Get*", "cloudwatch:List*", "cloudwatch:Describe*", "rds:List*", "rds:Describe*", "ecs:List*", "ecs:Describe*", "ec2:List*", "ec2:Describe*", "s3:List*", "s3:GetBucketTagging", "pi:Get*", "pi:Describe*", "pi:List*", "logs:ListTagsLogGroup", "logs:FilterLogEvents", "logs:DescribeLogGroups", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeTags" ], "Resource": "*" } ] } ``` ![Selection of Regions and Namespaces](/images/aws/manual-role-specify-policy.png) #### 2 Create Role with Trusted Relationships for Middleware Choose the **Custom Trust Policy** option while creating the Role, Paste the trust policy provided below into your **Trusted Relationships**, and select the policy created in the previous step (e.g., **MiddlewareRolePolicy**), Then, specify an appropriate role name, e.g.: **MiddlewareIntegrationRole** ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::378448125786:root" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "your-initial-token" } } } ] } ``` ![Selection of Regions and Namespaces](/images/aws/manual-role-trusted-entity.png) #### 3 Add Account ID and ARN in Middleware Click **Create Role**, Afterward, copy your **Role's ARN** and **Account ID**. Go through the UI prompts in to copy your Account ID and the Role's ARN, toggle namespaces and logs on/off and save the configuration. ### Access Keys Using Access Keys allow you to simply link an IAM User or User Group to Middleware. Each team will have different security poilicies that will determine if Role or Key based permission is required. #### 1 Create IAM User - Using the AWS Console, create an **IAM User** - Click on the **Create Policy** under the **Attach policies directly** section ![Middleware Amazon Web Services: Set Policies](/images/aws/access-keys-policies.png) - Use the following JSON to specify the permissions: ```yaml { "Version": "2012-10-17", "Statement": [ { "Action": [ "cloudwatch:Get*", "cloudwatch:List*", "cloudwatch:Describe*", "rds:List*", "rds:Describe*", "ecs:List*", "ecs:Describe*", "ec2:List*", "ec2:Describe*", "s3:List*", "s3:GetBucketTagging", "pi:Get*", "pi:Describe*", "pi:List*", "logs:ListTagsLogGroup", "logs:FilterLogEvents", "logs:DescribeLogGroups", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeTags" ], "Resource": "*", "Effect": "Allow" } ] } ``` - If using an IAM User skip to the next step, otherwise add the IAM User to the desired User Group #### 2 Create API Access Keys - Go to the User’s page and click on the **Create access key** - Select **Command Line Interface(CLI)** - Click **Next** and copy the **AWS Access Key ID** and **AWS Secret Access Key** ![Access Keys Method](/images/aws/access-keys-screen.png) #### 3 Input keys into Middleware Navigate to the the [Middleware AWS Integration Page](https://app.middleware.io/installation/integrations/aws), click on **API Polling -> Access Keys**, input the related information, and click **Save Details** ![Access Keys Method](/images/aws/access-keys-method.png) #### 4 Toggle on Namespaces and Save Input the desired regions and toggle on the Namespaces you'd like to poll metrics from, and click **Save Configuration** ![Access Keys' Regions & Namespaces](/images/aws/access-keys-selections.png) ### Supported Namespaces We currently collect metrics from the following AWS Namespaces: EC2, S3, RDS, ECS, EBS, Firehose, Lambda, WAF, and Lex. We are rapidly expanding to include other AWS Namespaces and will release them as built. If you have a specific need please reach out to support and we will prioritze accordingly. ![API Polling Approach](/images/aws/new-data-collected.png) ## Accessing Data Both options will allow you to create custom widgets with all metrics collected. The only difference between the two is that the **API Polling** method will auto-generate dashboards for toggled on Namespaces which are filterable as depicted in the below screenshot. The **Kinesis Stream** method does not currently have this capability. Visit the [Dashboards](https://demo2.middleware.io/dashboards) page to begin viewing your metrics or creating new widgets. ![Middleware Amazon Web Services: AWS Dashboards](/images/aws/aws-dashboards.png) --- ## AWS Logs Along with metrics, Middleware can also collect logs from certain namespaces in your AWS account. We currently support collecting logs via AWS CloudTRail or CloudWatch. The logs will be sent to the Middleware platform using the **API Polling** approach, similar to the metrics collection, with the same two methods: **Role Delegation** and **Access Keys**. ![AWS Logs Details](/images/aws/aws-logs-filled.png) After setting up [API Polling](/integrations/aws-integration#api-polling), you will see the AWS Logs section at the bottom of the Integration page. Here, you can enter a list of **AWS Log Groups** along with their **prefixes** (optional) with desired regions. With these details, Middleware will start API polling and display your logs on its Logs dashboard. Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us page or join our Slack channel.