Amazon Web Services (AWS)

AWS Metrics

Middleware’s AWS integration is able to collect metrics from multiple AWS accounts across approximately 90+ namespaces, such as EC2, S3, Lambda, Kinesis, RDS, ECS, and more.

Middleware is able to collect metrics from your account either through a Kinesis Stream or through API Polling. Each has their benefits which will be described below, and you can choose either one or use both simultaneously depending on your use case.

Using a Kinesis Stream

This approach uses CloudWatch Metric Streams and Amazon Data Firehose to fetch metrics. By default Cloudwatch Metric Streams send data with roughly two to three minute lag.

Using this approach provides more granular control of what metrics you'd like sent to Middleware, from any AWS Namespace. See AWS' doc on CloudWatch Metric Streams for more detail on how they work.

The below diagram shows the required resources for Middleware's case, which you can either set up automatically using our Cloudformation template, or manually yourself.

CloudFormation Template (Recommended)

Middleware provides a CloudFormation template to automatically create all of the necessary resources in your AWS account. The set up will take 5 - 10 minutes and will immediately begin sending data upon completion.

1 Launch Stack

You can access this template by clicking the Launch Stack button in your installation page.

2 Create IAM Role

In order to run the Cloudformation stack, you will need an IAM Role that has permissions to deploy this stack. Middleware will prompt you to input an existing IAM Role if you have one.

If you do not have the necessary IAM Role, you will need to create one with the following policy, which you can do by using the below JSON, or via the admin console UI:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:*",
                "organizations:DescribeAccount",
                "organizations:DescribeOrganization",
                "organizations:DescribeOrganizationalUnit",

If using the admin console UI, follow the below steps:

Go to the IAM page -> Roles page, and click on Create Role
Select AWS service as the Trusted entity type and choose CloudFormation as the Use case
In the Permissions policies step, choose IAMFullAccess, CloudWatchFullAccess, AWSCloudFormationFullAccess, AmazonKinesisFirehoseFullAccess, and AmazonS3FullAccess, then click Next
Set a Role Name and click on Create Role

3 Create the Stack

Once the IAM role is created and linked to Middleware, deploy the CloudFormation stack using the Create Stack button.

This will create all the necessary resources needed to collect and send AWS metrics to the Middleware platform.

Manual Creation

We only recommend manual creation of the stack for teams that have strict policies around new components in their AWS account. If you run into issues during Manual creation please review each step diligently, and if you're stuck reach out to our support team for assistance.

All steps required are outlined in this doc below, and are also accessible via the Installation page in your account.

1 Setup Firehose Stream

Sign in to the AWS Management Console and open the Kinesis console
Choose Data Firehose in the navigation panel
Choose Create delivery stream
The stream’s Source should be Direct PUT and Destination set to HTTP Endpoint with the following details:
- Kinesis endpoint: https://{ACCOUNT-UID}.middleware.io/v1/metrics/cloudwatch
- Access key: Access Key is found in the Installation page.
In the Backup settings, select an S3 backup bucket to receive any failed events that exceed the retry duration.

Your {ACCOUNT-UID} is the subdomain of your Middleware account, for example the account demo.middleware.io has an {ACCOUNT-UID} = demo.

2 Create a CloudWatch Metric Stream

Open the CloudWatch AWS console
In the left navigation pane, choose Metrics → Streams
Click the Create metric stream button
Choose the All namespaces in the metric stream
Choose Select an existing Firehose owned by your account, and select the Firehose Delivery Stream you created earlier
Specify an Output Format of OpenTelemetry 1.0
Optionally, specify a name for this metric stream under Metric Stream Name
Finally, click on Create metric stream

To ensure that your request metrics are accurately reported, it is important to Enable Requests metrics for your Amazon S3 buckets through the AWS console.

Keep in mind that Cloudwatch Metric Stream will only be fetching data from the namespaces which are in same region. Ex: If you want to fetch metrics data from your EC2 machine running in us-east-2, consider creating the Cloudwatch Metric Stream in the us-east-2 as well.

API Polling

Middleware's API Polling only requires users to provide Middleware with API keys or a specific IAM Role to enable automatic polling of the selected resources in their AWS account. Although easier to set up, this approach has a five minute lag on all metrics compared to the Kinesis streams' three minute lag.

The major benefits of this approach is that it requires no additional resources to be deployed in your AWS account, unlike our Kinesis Stream approach or other approaches used by our competitors.

To get started, access the AWS integration page inside of the installation pages, and choose from one of two approaches - Role Delegation or Access Keys.

Which method you choose will depend on your team's security policies.

API Polling is achieved using Cloudwatch APIs which may incur costs on your AWS account. Please refer to AWS pricing for further information about expected costs for CloudWatch APIs.

Role Delegation

Using Role Delegation is best if your team requires you to keep API keys internal or have strict policies around sharing. You can set up the required role using our CloudFormation Template or by manually creating the user.

If you'd like to create the user Manually you can skip to step 3 below to find the required permissions. You will then input that user's ARN into step 4.

CloudFormation Template

1 Launch the CloudFormation Template

To Access the template, click Create Role from within the Middleware app on the Installation -> Integration -> API Polling page.

This will redirect you to CloudFormation with a pre-filled template, where you will need to select a IAM Role in the Permissions section.

2 Create IAM Role or Choose Admin

You may either use the default Admin Role, or create a new one manually following the below steps:

Go to the IAM -> Roles page, and click on Create Role
Select AWS service as the Trusted entity type, choose CloudFormation as the Use case, and proceed to the next step
Search for and check the box next to IAMFullAccess, then click Next
Set a Role Name and press Create Role
Set the permissions to below JSON

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:*",
                "organizations:DescribeAccount",
                "organizations:DescribeOrganization",
                "organizations:DescribeOrganizationalUnit",

3 Run Template

Once an IAM Role is selected, click Run Stack which will create a Role named MiddlewareRoleIntegration that Middleware will use to trigger AWS APIs on your behalf.

This Role includes the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:Get*",
        "cloudwatch:List*",
        "cloudwatch:Describe*",
        "rds:List*",

4 Enter the Account ID and ARN

Copy the AWS Account ID and AWS Role ARN of the created role into Middleware

4 Choose Regions

Choose your desired Namespaces that you'd like to poll metrics from

5 Save Configuration

Input the desired regions and toggle on the Namespaces you'd like to poll metrics from, and click Save Configuration

Manual Role Creation

Manual Role Creation is not recommended as it has the most room for error. It is required however for some teams that have their AWS permissions heavily managed. If you hit issues please reach out to support.

1 Create Policy with Required Permissions

Create a policy with the below permissions, and specify a suitable name for your policy, e.g., MiddlewareRolePolicy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:Get*",
        "cloudwatch:List*",
        "cloudwatch:Describe*",
        "rds:List*",

2 Create Role with Trusted Relationships for Middleware

Choose the Custom Trust Policy option while creating the Role, Paste the trust policy provided below into your Trusted Relationships, and select the policy created in the previous step (e.g., MiddlewareRolePolicy), Then, specify an appropriate role name, e.g.: MiddlewareIntegrationRole

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::378448125786:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {

3 Add Account ID and ARN in Middleware

Click Create Role, Afterward, copy your Role's ARN and Account ID.

Go through the UI prompts in to copy your Account ID and the Role's ARN, toggle namespaces and logs on/off and save the configuration.

Access Keys

Using Access Keys allow you to simply link an IAM User or User Group to Middleware. Each team will have different security poilicies that will determine if Role or Key based permission is required.

1 Create IAM User

Using the AWS Console, create an IAM User
Click on the Create Policy under the Attach policies directly section

Use the following JSON to specify the permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "cloudwatch:Get*",
        "cloudwatch:List*",
        "cloudwatch:Describe*",
        "rds:List*",
        "rds:Describe*",

If using an IAM User skip to the next step, otherwise add the IAM User to the desired User Group

2 Create API Access Keys

Go to the User’s page and click on the Create access key
Select Command Line Interface(CLI)
Click Next and copy the AWS Access Key ID and AWS Secret Access Key

3 Input keys into Middleware

Navigate to the the Middleware AWS Integration Page, click on API Polling -> Access Keys, input the related information, and click Save Details

4 Toggle on Namespaces and Save

Input the desired regions and toggle on the Namespaces you'd like to poll metrics from, and click Save Configuration

Supported Namespaces

We currently collect metrics from the following AWS Namespaces: EC2, S3, RDS, ECS, EBS, Firehose, Lambda, WAF, and Lex. We are rapidly expanding to include other AWS Namespaces and will release them as built.

If you have a specific need please reach out to support and we will prioritze accordingly.

Accessing Data

Both options will allow you to create custom widgets with all metrics collected. The only difference between the two is that the API Polling method will auto-generate dashboards for toggled on Namespaces which are filterable as depicted in the below screenshot. The Kinesis Stream method does not currently have this capability.

Visit the Dashboards page to begin viewing your metrics or creating new widgets.

AWS Logs

Along with metrics, Middleware can also collect logs from certain namespaces in your AWS account. We currently support collecting logs via AWS CloudTRail or CloudWatch. The logs will be sent to the Middleware platform using the API Polling approach, similar to the metrics collection, with the same two methods: Role Delegation and Access Keys.

After setting up API Polling, you will see the AWS Logs section at the bottom of the Integration page. Here, you can enter a list of AWS Log Groups along with their prefixes (optional) with desired regions.

With these details, Middleware will start API polling and display your logs on its Logs dashboard.

Need assistance or want to learn more about Middleware? Contact our support team in Slack.