Ingesting logs from Fluent Bit

Middleware supports the ingestion of logs from Fluent Bit. Fluent Bit sends logs with its forward output plugin over the Fluent Forward protocol, which is received by a Middleware Agent or an OpenTelemetry Collector and exported to your Middleware account.

text

Fluent Bit ──(forward protocol)──► Middleware Agent / OTel Collector ──(OTLP)──► Middleware

You can use either of the following options:

Option 1: Middleware Agent (recommended). The agent already listens for Fluent Forward traffic on port 8006.
Option 2: OpenTelemetry Collector. Use your own collector with the fluentforward receiver.

Running Fluent Bit on AWS ECS with FireLens? Follow the AWS ECS guide instead.

Prerequisites#

Fluent Bit installed and running.
Your Middleware UID (https://<uid>.middleware.io) and API key (MW_API_KEY).

Option 1: Middleware Agent (recommended)#

1 Install the Middleware Agent#

Install the Middleware Agent on the host where Fluent Bit runs, or on a host Fluent Bit can reach. The agent listens for Fluent Forward traffic on port 8006 by default. To change the port, set fluent-port in /etc/mw-agent/agent-config.yaml (Linux) or the MW_AGENT_FLUENT_PORT environment variable, then restart the agent.

2 Configure the Fluent Bit output#

Add a forward output to your Fluent Bit configuration (usually /etc/fluent-bit/fluent-bit.conf):

ini

[OUTPUT]
    Name   forward
    Match  *
    Host   localhost
    Port   8006
    Retain_Metadata_In_Forward_Mode  false

yaml

pipeline:
  outputs:
    - name: forward
      match: '*'
      host: localhost
      port: 8006
      retain_metadata_in_forward_mode: false

Retain_Metadata_In_Forward_Mode false is required for Fluent Bit v5.0.4 and later. From v5.0.4, Fluent Bit sends extended metadata by default, and the receiver silently drops those records. Older versions accept the setting too.
Match * sends all logs. Use a tag pattern (for example, app.*) to send only some logs.
If the Middleware Agent runs on a different host, replace localhost with that host's address.

3 Restart Fluent Bit#

bash

sudo systemctl restart fluent-bit

Logs appear in the Logs section of your Middleware account. The Fluent Bit tag is available as the fluent.tag attribute and is used as the log source.

Option 2: OpenTelemetry Collector#

Use this option if you already run an OpenTelemetry Collector (contrib distribution) and do not want to install the Middleware Agent.

1 Add the Fluent Forward receiver#

Add the fluentforward receiver to your collector config.yaml:

yaml

receivers:
  fluentforward:
    endpoint: 0.0.0.0:24224

24224 is the default Fluent Forward port. You can change it if required. See the receiver documentation for all options.

2 Add the Middleware exporter and logs pipeline#

yaml

processors:
  batch:
  resourcedetection:
    detectors: [env, system]
    system:
      hostname_sources: [os]
  resource:
    attributes:
      - key: mw.account_key
        value: <MW_API_KEY>
        action: upsert
      - key: host.id
        from_attribute: host.name
        action: upsert

exporters:
  otlp/middleware:
    endpoint: https://<MW_UID>.middleware.io:443
    headers:
      authorization: <MW_API_KEY>

service:
  pipelines:
    logs:
      receivers: [otlp, fluentforward]
      processors: [resourcedetection, resource, batch]
      exporters: [otlp/middleware]

Replace:

<uid> with your Middleware project UID.
<MW_API_KEY> with your Middleware API key.

Keep the resourcedetection processor and the host.id attribute. Logs received over the Fluent Forward protocol carry no resource attributes, and Middleware rejects logs that do not identify a resource (such as host.id or service.name).

Recent collector versions log deprecation warnings for the fluentforward and otlp names. Both still work. To remove the warnings, rename them to fluent_forward and otlp_grpc.

3 Configure the Fluent Bit output#

ini

[OUTPUT]
    Name   forward
    Match  *
    Host   <otel-collector-host>
    Port   24224
    Retain_Metadata_In_Forward_Mode  false

Replace <otel-collector-host> with the collector's address (localhost if it runs on the same host).

4 Restart Fluent Bit and the collector#

Restart both services. Logs appear in the Logs section of your Middleware account.

Test with sample logs#

To check the setup without touching your application logs, add a dummy input that generates test logs:

ini

[INPUT]
    Name   dummy
    Tag    sample
    Dummy  {"message": "my log data", "source": "myhost"}

[OUTPUT]
    Name   forward
    Match  sample
    Host   localhost
    Port   8006
    Retain_Metadata_In_Forward_Mode  false

Use port 24224 instead of 8006 if you followed Option 2. Remove the dummy input after you see the logs in Middleware.

Troubleshooting#

No logs in Middleware: run Fluent Bit in the foreground (fluent-bit -c /etc/fluent-bit/fluent-bit.conf) and check for connection errors from the forward output.
Fluent Bit reports successful flushes but no logs arrive: make sure Retain_Metadata_In_Forward_Mode false is set on the forward output.
Connection refused: make sure the agent or collector is running and that port 8006 (or 24224) is open between Fluent Bit and the receiver.
Wrong logs are sent: check that the Match pattern matches the tags of the logs you want to forward.