Linux

The Middleware Infrastructure Agent collects telemetry from Linux hosts so you can monitor resource usage and investigate host performance. Configured log collection and application instrumentation add context from logs and traces.

Use this integration to monitor Linux hosts, including resource utilization, system activity, logs, and application telemetry.

Prerequisites#

  1. System Requirements
    • CPU architectures: x86_64 and ARM
    • Memory: ≥ 1 GB (agent uses ~200 MB)
    • Required Tools: curl or wget installed on the host
  2. Supported Distributions
    • DEB-based: Ubuntu, Debian
    • RPM-based: Red Hat, CentOS, Fedora, SUSE, Rocky Linux, Oracle Linux, Amazon Linux
    • Other DEB/RPM distros may work but aren’t officially tested.

Note:

  • If the host connects to the internet through a proxy, configure the proxy environment variables before running the installation script:, set HTTP_PROXY , HTTPS_PROXY (and lowercase variants)
  • Make sure the proxy allows outbound connections to Middleware.

Installation#

1 Run the Installer:#

bash
MW_API_KEY="<MW_API_KEY>" \
MW_TARGET="https://<MW_UID>.middleware.io:443" \
bash -c "$(curl -L https://install.middleware.io/scripts/deb-install.sh)"

2 (Optional) Add host tags:#

bash
MW_HOST_TAGS="env:prod,role:web" \
MW_API_KEY="<MW_API_KEY>" \
MW_TARGET="https://<MW_UID>.middleware.io:443" \
bash -c "$(curl -L https://install.middleware.io/scripts/deb-install.sh)"

3 Verify installation#

bash
sudo systemctl status mw-agent
sudo journalctl -u mw-agent -f    # stream live logs
Linux Agent

1 Run the Installer:#

bash
MW_API_KEY="<MW_API_KEY>" \
MW_TARGET="https://<MW_UID>.middleware.io:443" \
bash -c "$(curl -L https://install.middleware.io/scripts/rpm-install.sh)"

2 (Optional) Add host tags:#

bash
MW_HOST_TAGS="env:prod,role:web" \
MW_API_KEY="<MW_API_KEY>" \
MW_TARGET="https://<MW_UID>.middleware.io:443" \
bash -c "$(curl -L https://install.middleware.io/scripts/rpm-install.sh)"

3 Verify installation#

bash
sudo systemctl status mw-agent
sudo journalctl -u mw-agent -f    # stream live logs

Cloud Network Monitoring and eBPF APM#

From version 2.0.1, the agent includes eBPF features that you turn on with flags at install time:

FeatureEnvironment variableagent-config.yaml keyDefault
Cloud Network MonitoringMW_AGENT_FEATURES_CLOUD_NETWORK_MONITORINGagent-features.cloud-network-monitoringfalse
eBPF APMMW_AGENT_FEATURES_EBPF_APMagent-features.ebpf-apmfalse

Prerequisites#

  • Middleware Agent 2.0.1 or later
  • Linux kernel 5.8 or later, or 4.18 on RHEL-based distributions (Red Hat, CentOS, Rocky Linux, AlmaLinux)
  • BTF enabled in the kernel (/sys/kernel/btf/vmlinux exists)
  • x86_64 or aarch64
  • Kernel lockdown not in confidentiality mode
  • About 65 MB of extra memory for the agent with both features on

Check a host with:

bash
uname -r                             # 5.8 or later (4.18 on RHEL-based)
ls /sys/kernel/btf/vmlinux           # must exist
uname -m                             # x86_64 or aarch64
cat /sys/kernel/security/lockdown    # [confidentiality] must not be selected

The installer runs the same checks and logs eBPF preflight passed when the host can run these features. If a check fails, it logs a warning and installs the agent anyway.

Turn these features on only on hosts that pass the checks. On a host that fails them, the agent's collector doesn't start while a feature is on, and its other data stops too.

Install#

bash
MW_API_KEY="<MW_API_KEY>" \
MW_TARGET="https://<MW_UID>.middleware.io:443" \
MW_AGENT_FEATURES_CLOUD_NETWORK_MONITORING=true \
MW_AGENT_FEATURES_EBPF_APM=true \
MW_ENABLE_INJECTOR=false \
bash -c "$(curl -L https://install.middleware.io/scripts/deb-install.sh)"
bash
MW_API_KEY="<MW_API_KEY>" \
MW_TARGET="https://<MW_UID>.middleware.io:443" \
MW_AGENT_FEATURES_CLOUD_NETWORK_MONITORING=true \
MW_AGENT_FEATURES_EBPF_APM=true \
MW_ENABLE_INJECTOR=false \
bash -c "$(curl -L https://install.middleware.io/scripts/rpm-install.sh)"

Leave out the flag of a feature you don't want. To change a flag on an installed agent, run the installer again with the new value.

MW_ENABLE_INJECTOR=false skips the OTel Injector, which Cloud Network Monitoring and eBPF APM don't use. Leave it out if you also want to instrument systemd services with the injector.

Network data collected#

With Cloud Network Monitoring on, the agent sends these metrics every 30 seconds:

MetricWhat it measures
obi.stat.bytesBytes sent and received
obi.stat.packetsPackets sent and received
obi.stat.tcp.rttTCP round-trip time
obi.stat.tcp.retransmitsTCP retransmitted segments
obi.stat.tcp.successful.connectionsTCP connections established
obi.stat.tcp.failed.connectionsTCP connection attempts that failed
obi.stat.tcp.closed.connectionsTCP connections closed
obi.stat.tcp.flow_control.eventsTCP zero-window events (the receiver's buffer is full)
obi.stat.tcp.congestion.eventsTCP retransmit timeouts and fast recoveries
obi.stat.tcp.ecn.ce.packetsPackets the network marked as congested (ECN)
obi.stat.tcp.reordering.eventsTCP packet reordering events
obi.stat.tcp.out_of_order.packetsTCP packets received out of order
obi.stat.dns.lookup.durationDNS lookup time, with the queried name and the error, if any

Each metric describes both ends of the traffic with these attributes:

  • IP addresses: src.address, dst.address, and the address before NAT when the traffic was translated
  • Names: src.name, dst.name, from reverse DNS or the host name
  • Service port: server.port, the port of the server side, such as 443 or 5432
  • Process and container: the process name, container name and image of each local end
  • Kubernetes: the namespace and owner (deployment, daemonset, …) of each pod, on Kubernetes nodes

Source and destination port numbers aren't kept, so connections from random client ports don't each create a series.

Every remote IP address that talks to the host creates its own series. On a public-facing server, such as a web server, load balancer or API gateway that receives traffic from many client IPs, the number of series can be high. Keep this in mind before you turn on Cloud Network Monitoring on these hosts.

Trace services with eBPF APM#

eBPF APM traces HTTP, gRPC and database calls of the services you select, with no code change and no restart of the service.

1 Turn on eBPF APM#

Install the agent with MW_AGENT_FEATURES_EBPF_APM=true, as shown above.

2 Select the services#

In Middleware, open the Linux agent installation page and go to the Configuration tab. It lists the services the agent found on the host, under Systemd, Docker-Container and Other.

Select the services to trace. In the Mode column, choose OBI. Systemd services in Java, Python and Node.js can also use OTel Injector (see Instrumenting systemd services); every other service supports OBI only.

3 Save#

Click Save Changes. OBI services need no command: within a few minutes, the agent fetches its new configuration, restarts its collector and starts tracing them. To stop tracing a service, clear it and save again.

4 Check the traces#

Traces appear in APM → Services a few minutes after the next request to the service.

Notes:

  • The agent finds a selected service by its listening TCP ports, or, for Java, by its jar file or main class. A service that has neither, such as a worker that doesn't listen on a port, can't be traced with eBPF APM.
  • eBPF APM adds a traceparent header to the service's outgoing HTTP requests, so a trace continues across services.
  • Use one method per service. Don't select OBI for a service that is already instrumented with the OTel Injector or a language APM agent, or its requests are traced twice.

Remove the standalone obi-agent#

Before 2.0.1, eBPF ran as a separate obi-agent service, installed with MW_ENABLE_OBI. From 2.0.1, the installer doesn't install it and MW_ENABLE_OBI has no effect. If a host still has the old service, the installer warns about it. Remove it so data isn't sent twice:

bash
sudo bash -c "$(curl -fsSL https://install.middleware.io/scripts/install-obi.sh)" -- --uninstall

Supported Environments#

The Middleware Linux agent is distributed as standard DEB and RPM packages and runs as a regular systemd service. It does not depend on any cloud provider's APIs or services, so it works the same way on any Linux host that runs a supported distribution and can reach your Middleware account over HTTPS, including:

  • Public cloud virtual machines and bare metal: Amazon EC2, Microsoft Azure Virtual Machines (including Virtual Machine Scale Sets), Google Compute Engine, Oracle Cloud Infrastructure (OCI) Compute, DigitalOcean Droplets and other cloud providers.
  • Private cloud and on-premises: VMware vSphere, Proxmox, KVM, Hyper-V and physical servers.

Use the installer that matches the host's distribution:

Distribution familyDistributionsInstaller
DEB-basedUbuntu, DebianDEB
RPM-basedRed Hat Enterprise Linux, CentOS, Rocky Linux, Oracle Linux, Amazon Linux, Fedora, SUSERPM

Both x86_64 and Arm64 architectures are supported, including Arm-based cloud instances such as AWS Graviton, Azure Cobalt, Google Axion and OCI Ampere.

Network requirements#

The agent sends data to your Middleware account over HTTPS (port 443). Make sure the host's firewall rules, security groups or network security lists allow outbound traffic to https://<YOUR_UID>.middleware.io. Hosts without direct internet access need a NAT gateway or an HTTP proxy (see Prerequisites).

Adding environment metadata#

To filter and group hosts by cloud provider, region or environment in Middleware, add them as host tags using MW_HOST_TAGS when you run the installer, for example:

bash
MW_HOST_TAGS="cloud.provider:oci,cloud.region:sa-santiago-1,env:production"

Using the OpenTelemetry Collector#

If you prefer to run the OpenTelemetry Collector instead of the Middleware agent, the OpenTelemetry Collector Contrib is also supported on any Linux host and can send host metrics, logs and traces directly to Middleware. For managed Kubernetes services such as Amazon EKS, Azure AKS, Google GKE and Oracle OKE, use the Middleware Kubernetes agent.

Zero-Code Instrumentation with OBI#

The Linux agent automatically discovers the applications running on the host, whether they run as host processes, systemd services or Docker containers, and detects the programming language of each one. You can then instrument any discovered application with OpenTelemetry eBPF Instrumentation (OBI).

OBI uses eBPF to capture traces and RED metrics (request rate, errors and duration) for HTTP, gRPC and database calls directly from the Linux kernel. No code changes, application restarts, added packages or image rebuilds are required, which makes it well suited to containerized applications whose images you cannot modify.

From agent 2.0.1, you don't need the steps in this section. OBI runs inside the agent: turn on eBPF APM and select the services in Middleware. The mw-agent instrument --type obi and uninstrument --type obi commands below need the standalone obi-agent service, so they work only with agents older than 2.0.1. mw-agent services list still works on every version.

Requirements#

  • Linux kernel 5.8 or later with BTF enabled, or kernel 4.18 or later on RHEL-family distributions (Red Hat, CentOS, Rocky Linux, Oracle Linux).
  • An agent older than 2.0.1 with the standalone OBI agent, which the DEB and RPM installers install automatically on supported kernels for those versions. To skip it, set MW_ENABLE_OBI=false when running the installer.

Step 1: Discover applications#

List the applications the agent has discovered on the host, along with their detected language, type (for example, systemd or docker) and listening ports:

bash
sudo mw-agent services list
text
SERVICE ID         SERVICE NAME               LANGUAGE   TYPE      PORTS   INSTANCES   INSTRUMENTED
e6f38b1db677d920   book-service-java          java       systemd   8086    1           -
9fff4fcb924d5413   book-service-java-docker   java       docker    8086    1           -
8d3392312658de2e   orders-api                 python     docker    8000    1           -

Use --language to filter by language (java, python, node, go, rust, php, ruby) and --all to show each running instance individually.

Step 2: Instrument an application#

Instrument an application with OBI by its service name or service ID:

bash
sudo mw-agent instrument --type obi --language python orders-api

All running instances (replicas) of the same application share a service ID, so they are instrumented together. Traces appear in Middleware under APM → Services within a few minutes.

Remove instrumentation#

bash
sudo mw-agent uninstrument --type obi orders-api

Instrumenting systemd services#

For Java, Python and Node.js applications that run as systemd services, you can also instrument them with the full OpenTelemetry language agent instead of OBI, still without code changes. This works on every agent version, including 2.0.1 and later. The DEB and RPM installers install the required OpenTelemetry Injector by default (set MW_ENABLE_INJECTOR=false to skip it).

On the Configuration tab of the Linux agent installation page, this is the OTel Injector mode, which is the default for these services. After you save, the page shows the command to run on the host.

bash
sudo mw-agent instrument --type systemd --language java book-service-java

This adds a systemd drop-in that loads the language agent into the service, and restarts the service to apply it. To remove it:

bash
sudo mw-agent uninstrument --type systemd book-service-java

Don't use the OTel Injector and OBI on the same service: its requests would be traced twice. To move a service from one to the other, remove the first one (uninstrument, or clear it on the Configuration tab) before you turn on the second.

OBI captures traces at the protocol level (HTTP, gRPC and database calls). For code-level detail such as custom spans, method-level traces or continuous profiling, use the language-specific APM agents.

Manage the agent (Commands)#

Middleware agent runs as a systemd service in Linux based systems, usually on Linux systems, the Middleware Agent runs as a systemd service. The service definition is typically located under:

/usr/lib/systemd/system/

Use the following commands to start, stop, restart, and check the status of the Middleware Agent.

CommandDescription
sudo systemctl start mw-agentStart the agent
sudo systemctl stop mw-agentStop the agent
sudo systemctl restart mw-agentRestart the agent
sudo systemctl status mw-agentCheck agent status
sudo journalctl -u mw-agent -fStream live agent logs

Verify data collection#

After installing the Agent, allow a few moments for the host to start reporting telemetry.

Go to Infrastructure Monitoring → Hosts and verify that your Linux host appears and is reporting metrics.

After data collection starts, you can:

Uninstall#

bash
bash -c "$(curl -L https://install.middleware.io/scripts/deb-uninstall.sh)"
bash
bash -c "$(curl -L https://install.middleware.io/scripts/rpm-uninstall.sh)"

Troubleshooting#

It can take a few minutes for a new host to appear. Check Infrastructure Monitoring → Hosts first. If the host doesn't appear or data stops, work through these checks in order.

1. Check that the agent is running#

bash
sudo systemctl status mw-agent

The service should be active (running). If it's restarting or failed, the logs in the next step show why.

2. Read the agent logs#

The agent logs to the systemd journal:

bash
sudo journalctl -u mw-agent -n 200 --no-pager
sudo journalctl -u mw-agent --since "30 min ago" | grep -i -E "error|warn|fatal"

If you set logfile in /etc/mw-agent/agent-config.yaml, the agent writes to that file instead.

3. Check the API key and target#

The installer writes both values to /etc/mw-agent/agent-config.yaml:

bash
sudo grep -E "^(api-key|target):" /etc/mw-agent/agent-config.yaml
  • target must be your account URL, https://<YOUR_UID>.middleware.io:443.
  • api-key must be your project's ingestion key, not a Public API key. See API Keys.

After changing either value, restart the agent with sudo systemctl restart mw-agent.

4. Check network connectivity#

The host needs outbound HTTPS to your account URL:

bash
curl -sS -o /dev/null -w "%{http_code}\n" https://<YOUR_UID>.middleware.io:443

Any HTTP status code means the host can reach Middleware. A timeout or a DNS or TLS error points to a firewall, security group, DNS or proxy problem.

If the host is behind a proxy, check that the running agent has the proxy variables and that the proxy allows *.middleware.io:

bash
sudo cat /proc/$(pgrep -o -x mw-agent)/environ | tr '\0' '\n' | grep -i proxy

See Prerequisites for how to set them.

5. Check for more than one agent#

Run only one Middleware agent per host. A second agent, for example a containerized one, can cause duplicate or missing data:

bash
pgrep -a mw-agent
sudo docker ps --filter "name=mw-agent" 2>/dev/null

6. Check the APM ports#

Applications in host-agent mode send traces to the agent on port 9319 (OTLP/gRPC) or 9320 (OTLP/HTTP). If APM data is missing but host metrics arrive, check that the agent is listening:

bash
sudo ss -ltnp | grep -E ":(9319|9320)\b"

Another process on these ports stops the agent from receiving APM data.

Still stuck?#

Collect the output of the checks above, without the API key, and contact Middleware support through the chat at the bottom right of this page or at support{at}middleware{dot}io.

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us or join our Slack channel.

What did you think of this content?