Ingesting logs from Fluent Bit

Middleware supports the ingestion of logs from Fluent Bit. Fluent Bit sends logs with its forward output plugin over the Fluent Forward protocol, which is received by a Middleware Agent or an OpenTelemetry Collector and exported to your Middleware account.

1Fluent Bit ──(forward protocol)──► Middleware Agent / OTel Collector ──(OTLP)──► Middleware

You can use either of the following options:

  • Option 1: Middleware Agent (recommended). The agent already listens for Fluent Forward traffic on port 8006.
  • Option 2: OpenTelemetry Collector. Use your own collector with the fluentforward receiver.

Running Fluent Bit on AWS ECS with FireLens? Follow the AWS ECS guide instead.

Prerequisites#

  • Fluent Bit installed and running.
  • Your Middleware UID (https://<uid>.middleware.io) and API key (MW_API_KEY).

Option 1: Middleware Agent (recommended)#

1 Install the Middleware Agent#

Install the Middleware Agent on the host where Fluent Bit runs, or on a host Fluent Bit can reach. The agent listens for Fluent Forward traffic on port 8006 by default. To change the port, set fluent-port in /etc/mw-agent/agent-config.yaml (Linux) or the MW_AGENT_FLUENT_PORT environment variable, then restart the agent.

2 Configure the Fluent Bit output#

Add a forward output to your Fluent Bit configuration (usually /etc/fluent-bit/fluent-bit.conf):

1[OUTPUT]
2    Name   forward
3    Match  *
4    Host   localhost
5    Port   8006
6    Retain_Metadata_In_Forward_Mode  false
1pipeline:
2  outputs:
3    - name: forward
4      match: '*'
5      host: localhost
6      port: 8006
7      retain_metadata_in_forward_mode: false
  • Retain_Metadata_In_Forward_Mode false is required for Fluent Bit v5.0.4 and later. From v5.0.4, Fluent Bit sends extended metadata by default, and the receiver silently drops those records. Older versions accept the setting too.
  • Match * sends all logs. Use a tag pattern (for example, app.*) to send only some logs.
  • If the Middleware Agent runs on a different host, replace localhost with that host's address.

3 Restart Fluent Bit#

1sudo systemctl restart fluent-bit

Logs appear in the Logs section of your Middleware account. The Fluent Bit tag is available as the fluent.tag attribute and is used as the log source.

Option 2: OpenTelemetry Collector#

Use this option if you already run an OpenTelemetry Collector (contrib distribution) and do not want to install the Middleware Agent.

1 Add the Fluent Forward receiver#

Add the fluentforward receiver to your collector config.yaml:

1receivers:
2  fluentforward:
3    endpoint: 0.0.0.0:24224

24224 is the default Fluent Forward port. You can change it if required. See the receiver documentation for all options.

2 Add the Middleware exporter and logs pipeline#

1processors:
2  batch:
3  resourcedetection:
4    detectors: [env, system]
5    system:
6      hostname_sources: [os]
7  resource:
8    attributes:
9      - key: mw.account_key
10        value: <MW_API_KEY>
11        action: upsert
12      - key: host.id
13        from_attribute: host.name
14        action: upsert
15
16exporters:
17  otlp/middleware:
18    endpoint: https://<MW_UID>.middleware.io:443
19    headers:
20      authorization: <MW_API_KEY>
21
22service:
23  pipelines:
24    logs:
25      receivers: [otlp, fluentforward]
26      processors: [resourcedetection, resource, batch]
27      exporters: [otlp/middleware]

Replace:

  • <uid> with your Middleware project UID.
  • <MW_API_KEY> with your Middleware API key.

Keep the resourcedetection processor and the host.id attribute. Logs received over the Fluent Forward protocol carry no resource attributes, and Middleware rejects logs that do not identify a resource (such as host.id or service.name).

Recent collector versions log deprecation warnings for the fluentforward and otlp names. Both still work. To remove the warnings, rename them to fluent_forward and otlp_grpc.

3 Configure the Fluent Bit output#

1[OUTPUT]
2    Name   forward
3    Match  *
4    Host   <otel-collector-host>
5    Port   24224
6    Retain_Metadata_In_Forward_Mode  false

Replace <otel-collector-host> with the collector's address (localhost if it runs on the same host).

4 Restart Fluent Bit and the collector#

Restart both services. Logs appear in the Logs section of your Middleware account.

Test with sample logs#

To check the setup without touching your application logs, add a dummy input that generates test logs:

1[INPUT]
2    Name   dummy
3    Tag    sample
4    Dummy  {"message": "my log data", "source": "myhost"}
5
6[OUTPUT]
7    Name   forward
8    Match  sample
9    Host   localhost
10    Port   8006
11    Retain_Metadata_In_Forward_Mode  false

Use port 24224 instead of 8006 if you followed Option 2. Remove the dummy input after you see the logs in Middleware.

Troubleshooting#

  • No logs in Middleware: run Fluent Bit in the foreground (fluent-bit -c /etc/fluent-bit/fluent-bit.conf) and check for connection errors from the forward output.
  • Fluent Bit reports successful flushes but no logs arrive: make sure Retain_Metadata_In_Forward_Mode false is set on the forward output.
  • Connection refused: make sure the agent or collector is running and that port 8006 (or 24224) is open between Fluent Bit and the receiver.
  • Wrong logs are sent: check that the Match pattern matches the tags of the logs you want to forward.

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us or join our Slack channel.