Heroku Buildpack for Middleware

The Heroku buildpack for Middleware is a buildpack for the Middleware Agent (mw-agent). The buildpack installs and runs mw-agent on a Heroku Dyno to receive, process and export metric and trace data to Middleware Observability platform.

To collect custom application metrics or traces, include the language appropriate Middleware APM package in your application.

Installation

Install the Heroku CLI, login, and create an app.

Step 1: Setup APPNAME environment variable.

export APPNAME=<YOUR_HEROKU_APPLICATION_NAME>

Step 2: Add the Middleware buildpack to your Heroku project:

cd <HEROKU_APPLICATION_DIRECTORY>

# Configure Heroku to expose Dyno metadata inside the app.
heroku labs:enable runtime-dyno-metadata -a $APPNAME

# Set required environment variables
heroku config:add MW_API_KEY=<MW_API_KEY>
heroku config:add MW_TARGET=https://<MW_UID>.middleware.io:443

# Set hostname in Middleware as appname.dynotype.dynonumber for metrics continuity across dyno ID changes.

mw-agent is installed by default at /app/mw-agent directory and is automatically started on every Dyno start.

mw-agent provides local OpenTelemetry endpoints on port 9313 for gRPC and 9320 for HTTP. It also provides a local fluentforward endpoint on port 8006.

Buildpack Ordering

Typically, your application buildpack should be the last buildpack in the heroku buildpacks command output. Heroku uses the last buildpack in the list to determine the process type for the application. Having Middleware buildpack as the last buildpack would override the process type preventing your application to run correctly.

In addition, any buildpacks that override /app directory should be before Middleware buildpack in order. For example, if your application requires Node, Middleware and apt buildpacks, the output of heroku buildpacks should be as follows:

1. https://github.com/heroku/heroku-buildpack-apt.git
2. https://github.com/middleware-labs/heroku-buildpack-middleware.git
3. heroku/nodejs

Buildpack and mw-agent Version

Heroku recommends to always use the latest commit of a buildpack.

However, if you wish to use a specific tag of the buildpack instead of the latest git commit, use the command below.

heroku buildpacks:add https://github.com/middleware-labs/heroku-buildpack-middleware.git#<TAG_NAME>

Use the appropriate value for <TAG_NAME> from here.

By default, the buildpack pins the latest version of the Middleware Agent at the time of release. You can pin the Agent to an earlier version by setting the MW_AGENT_VERSION environment variable.

heroku config:add MW_AGENT_VERSION=1.7.7

You will have to rebuild your application slug when you change the buildpack release and/or change the mw-agent verison.

cd <HEROKU_APPLICATION_DIRECTORY>

# Rebuild your slug with the new Agent version:
git commit --allow-empty -m "Rebuild slug"
git push heroku main

Configuration

Below is the list of all environment variables that you can set in your Heroku application to change the behavior of the Middleware buildpack and mw-agent.

Environment Variables for the buildpack

Dyno Hostnames

mw-agent installed through the buildpack uses operating system APIs to detect the hostname of the Dyno it is running on. By default, the Dyno ID is used as a hostname by the operating system.

Dynos can move to different host machines when there is a new deployment, configuration changes or resource needs/availability change. This causes Dyno ID to change resulting a new host being reported to Middleware. This makes it harder to monitor Dyno metrics and performance.

When MW_DYNO_HOSTNAME is set to true in the buildpack configuration, mw-agent reports the hostname in appname.dyno format (e.g. myapp.web.1, myapp.run.2). This allows for continuity in Dyno metric monitoring and an accurate picture of your Heroku application performance.

For MW_DYNO_HOSTNAME to work, Dyno metadata needs to be enabled in Heroku application configuration.

Dyno metadata can be enabled using command below:

heroku labs:enable runtime-dyno-metadata -a <app name>

Heroku buildpack for Middleware relies on HEROKU_APP_NAME environment variable obtained through Dyno metadata to update the Dyno hostname.

Preexec Script

The buildback behavior can be customized by including a preexec script which runs just before the mw-agent is started. The preexec script can be passed to the buildpack using MW_DYNO_PREEXEC_SCRIPT environment variable.

Below is an example of how you can set MW_DYNO_PREEXEC_SCRIPT:

heroku config:add MW_DYNO_PREEXEC_SCRIPT=/app/preexec.sh

You will have to provide preexec script along with your application.

The preexec script can be used to modify the configuration variables mentioned above and enable Middleware integrations.

One common use case for the preexec script is to disable mw-agent for selected Dyno process types (e.g. run) and to provision integration configuration if any.

Below is an example of a preexec script that will disable mw-agent for run process type and provide MySQL configuration to enable Middleware MySQL integration.

#!/bin/bash

# Extract the process type from the DYNO environment variable
DYNO_PROC_TYPE="${DYNO%%.*}"

# Check if the process type is 'run' and set MW_DYNO_DISABLE_AGENT to true if it is
if [ "$DYNO_PROC_TYPE" == "run" ]; then
  export MW_DYNO_DISABLE_AGENT=true
else
  # Create MySQL credentials file so that mw-agent can collect metrics

Metrics

By default, the buildpack collects system metrics for the host machine running your Dyno. To disable host system metrics collection, set the MW_AGENT_FEATURES_METRIC_COLLECTION environment variable to false in your Heroku application configuration.

Traces

Applications running on Dyno can use language specific Middleware APM packages to submit traces to the Middleware platform. Middleware APM packages seamlessly integrate with the buildpack and allow submission of custom metrics in addition to application traces and spans.

Logs

Logs for applications running on Heroku Dynos can be collected using Middleware's Heroku Log Drains integration.

Troubleshooting

How do I know if mw-agent is running in the background ?

If you are not seeing data in your Middleware account after installing the buildpack, you can check mw-agent log files by connecting to your Dyno.

First ensure that mw-agent is running inside the Dyno.

heroku ps:exec -a <your-app-name>

# Establishing credentials... done
# Connecting to web.1 on ⬢ go-heroku-middleware

~ $ ps aux | grep mw-agent
u14900         4  1.3  0.1 1362616 87804 ?       Sl   23:54   0:00 /app/mw-agent/opt/mw-agent/bin/mw-agent start --otel-config-file=/app/mw-agent/etc/mw-agent/otel-config.yaml

You can also check for any error messages in the mw-agent log file.

~ $ cat /app/mw-agent/mw-agent.log
{"level":"INFO","time":"2024-09-20T23:54:21.682Z","caller":"host-agent/main.go:308","message":"starting host agent","agent location":"/app/mw-agent/opt/mw-agent/bin/mw-agent","hostname":"396f01fe-e4b7-4b94-99e4-89aa79706e74"}