Ingesting logs from Fluentd

Middleware supports the ingestion of logs from Fluentd. Fluentd sends logs over the Fluent Forward protocol, which is received by a Middleware Agent or an OpenTelemetry Collector and exported to your Middleware account.

1Fluentd ──(forward protocol)──► Middleware Agent / OTel Collector ──(OTLP)──► Middleware

You can use either of the following options:

  • Option 1: Middleware Agent (recommended). The agent already listens for Fluent Forward traffic on port 8006.
  • Option 2: OpenTelemetry Collector. Use your own collector with the fluentforward receiver.

Prerequisites#

  • Fluentd (or td-agent / fluent-package) installed and running.
  • Your Middleware UID (https://<uid>.middleware.io) and API key (MW_API_KEY).

Option 1: Middleware Agent (recommended)#

1 Install the Middleware Agent#

Install the Middleware Agent on the host where Fluentd runs, or on a host Fluentd can reach. The agent listens for Fluent Forward traffic on port 8006 by default. To change the port, set fluent-port in /etc/mw-agent/agent-config.yaml (Linux) or the MW_AGENT_FLUENT_PORT environment variable, then restart the agent.

2 Configure the Fluentd output#

Add a forward output to your Fluentd configuration (usually /etc/fluent/fluentd.conf or /etc/td-agent/td-agent.conf):

1<match <directive>>
2  @type forward
3  send_timeout 60s
4  recover_wait 10s
5  hard_timeout 60s
6
7  <server>
8    name middleware-agent
9    host localhost
10    port 8006
11  </server>
12</match>
  • Replace <directive> with the tag pattern of the logs you want to send (for example, app.**, or ** for all logs).
  • If the Middleware Agent runs on a different host, replace localhost with that host's address.

3 Restart Fluentd#

1sudo systemctl restart fluentd

Logs appear in the Logs section of your Middleware account. The forward output flushes its buffer every 60 seconds by default, so the first logs can take up to a minute to appear. The Fluentd tag is available as the fluent.tag attribute and is used as the log source.

Option 2: OpenTelemetry Collector#

Use this option if you already run an OpenTelemetry Collector (contrib distribution) and do not want to install the Middleware Agent.

1 Add the Fluent Forward receiver#

Add the fluentforward receiver to your collector config.yaml:

1receivers:
2  fluentforward:
3    endpoint: 0.0.0.0:24224

24224 is the default Fluent Forward port. You can change it if required. See the receiver documentation for all options.

2 Add the Middleware exporter and logs pipeline#

1processors:
2  batch:
3  resourcedetection:
4    detectors: [env, system]
5    system:
6      hostname_sources: [os]
7  resource:
8    attributes:
9      - key: mw.account_key
10        value: <MW_API_KEY>
11        action: upsert
12      - key: host.id
13        from_attribute: host.name
14        action: upsert
15
16exporters:
17  otlp/middleware:
18    endpoint: https://<MW_UID>.middleware.io:443
19    headers:
20      authorization: <MW_API_KEY>
21
22service:
23  pipelines:
24    logs:
25      receivers: [otlp, fluentforward]
26      processors: [resourcedetection, resource, batch]
27      exporters: [otlp/middleware]

Replace:

  • <uid> with your Middleware project UID.
  • <MW_API_KEY> with your Middleware API key.

Keep the resourcedetection processor and the host.id attribute. Logs received over the Fluent Forward protocol carry no resource attributes, and Middleware rejects logs that do not identify a resource (such as host.id or service.name).

Recent collector versions log deprecation warnings for the fluentforward and otlp names. Both still work. To remove the warnings, rename them to fluent_forward and otlp_grpc.

3 Configure the Fluentd output#

1<match <directive>>
2  @type forward
3  send_timeout 60s
4  recover_wait 10s
5  hard_timeout 60s
6
7  <server>
8    name otel-collector
9    host <otel-collector-host>
10    port 24224
11  </server>
12</match>

Replace <directive> with your tag pattern and <otel-collector-host> with the collector's address (localhost if it runs on the same host).

4 Restart Fluentd and the collector#

Restart both services. Logs appear in the Logs section of your Middleware account.

Test with sample logs#

To check the setup without touching your application logs, add a sample source that generates test logs:

1<source>
2  @type sample
3  sample [{"message": "my log data", "source": "myhost"}]
4  tag sample
5  rate 10
6</source>
7
8<match sample>
9  @type forward
10  <server>
11    host localhost
12    port 8006
13  </server>
14</match>

Use port 24224 instead of 8006 if you followed Option 2. Remove the sample source after you see the logs in Middleware.

Troubleshooting#

  • No logs in Middleware: check the Fluentd log (/var/log/fluent/fluentd.log) for connection errors to the forward target.
  • Connection refused: make sure the agent or collector is running and that port 8006 (or 24224) is open between Fluentd and the receiver.
  • Wrong logs are sent: check that the <match> pattern matches the tags of the logs you want to forward.

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us or join our Slack channel.