--- title: "Ingesting logs from Fluent Bit" description: "Forward logs from Fluent Bit to Middleware using the Fluent Forward protocol through the Middleware Agent or an OpenTelemetry Collector." section: "Third Party Agents" breadcrumb: "Third Party Agents > Fluent Bit" canonical: "https://docs.middleware.io/get-started/agent/fluent-bit" --- # Ingesting logs from Fluent Bit Middleware supports the ingestion of logs from [Fluent Bit](https://fluentbit.io/). Fluent Bit sends logs with its `forward` output plugin over the [Fluent Forward protocol](https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1), which is received by a Middleware Agent or an OpenTelemetry Collector and exported to your Middleware account. ```text Fluent Bit ──(forward protocol)──► Middleware Agent / OTel Collector ──(OTLP)──► Middleware ``` You can use either of the following options: - **Option 1: Middleware Agent (recommended).** The agent already listens for Fluent Forward traffic on port `8006`. - **Option 2: OpenTelemetry Collector.** Use your own collector with the `fluentforward` receiver. > **Note:** Running Fluent Bit on AWS ECS with FireLens? Follow the [AWS ECS guide](/get-started/agent/amazon-ecs) instead. ## Prerequisites - Fluent Bit installed and running. - Your Middleware **UID** (`https://.middleware.io`) and **API key** (`MW_API_KEY`). ## Option 1: Middleware Agent (recommended) #### 1 Install the Middleware Agent Install the [Middleware Agent](/get-started/agent) on the host where Fluent Bit runs, or on a host Fluent Bit can reach. The agent listens for Fluent Forward traffic on port `8006` by default. To change the port, set `fluent-port` in `/etc/mw-agent/agent-config.yaml` (Linux) or the `MW_AGENT_FLUENT_PORT` environment variable, then restart the agent. #### 2 Configure the Fluent Bit output Add a `forward` output to your Fluent Bit configuration (usually `/etc/fluent-bit/fluent-bit.conf`): #### Classic (.conf) ```ini [OUTPUT] Name forward Match * Host localhost Port 8006 Retain_Metadata_In_Forward_Mode false ``` #### YAML (.yaml) ```yaml pipeline: outputs: - name: forward match: '*' host: localhost port: 8006 retain_metadata_in_forward_mode: false ``` - `Retain_Metadata_In_Forward_Mode false` is required for Fluent Bit v5.0.4 and later. From v5.0.4, Fluent Bit sends extended metadata by default, and the receiver silently drops those records. Older versions accept the setting too. - `Match *` sends all logs. Use a tag pattern (for example, `app.*`) to send only some logs. - If the Middleware Agent runs on a different host, replace `localhost` with that host's address. #### 3 Restart Fluent Bit ```bash sudo systemctl restart fluent-bit ``` Logs appear in the [Logs](/features/logs) section of your Middleware account. The Fluent Bit tag is available as the `fluent.tag` attribute and is used as the log `source`. ## Option 2: OpenTelemetry Collector Use this option if you already run an [OpenTelemetry Collector](/get-started/agent/otel-contrib) (contrib distribution) and do not want to install the Middleware Agent. #### 1 Add the Fluent Forward receiver Add the `fluentforward` receiver to your collector `config.yaml`: ```yaml receivers: fluentforward: endpoint: 0.0.0.0:24224 ``` `24224` is the default Fluent Forward port. You can change it if required. See the [receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/fluentforwardreceiver) for all options. #### 2 Add the Middleware exporter and logs pipeline ```yaml processors: batch: resourcedetection: detectors: [env, system] system: hostname_sources: [os] resource: attributes: - key: mw.account_key value: action: upsert - key: host.id from_attribute: host.name action: upsert exporters: otlp/middleware: endpoint: https://.middleware.io:443 headers: authorization: service: pipelines: logs: receivers: [otlp, fluentforward] processors: [resourcedetection, resource, batch] exporters: [otlp/middleware] ``` Replace: - `` with your Middleware project UID. - `` with your Middleware API key. > **Warning:** Keep the `resourcedetection` processor and the `host.id` attribute. Logs received over the Fluent Forward protocol carry no resource attributes, and Middleware rejects logs that do not identify a resource (such as `host.id` or `service.name`). Recent collector versions log deprecation warnings for the `fluentforward` and `otlp` names. Both still work. To remove the warnings, rename them to `fluent_forward` and `otlp_grpc`. #### 3 Configure the Fluent Bit output ```ini [OUTPUT] Name forward Match * Host Port 24224 Retain_Metadata_In_Forward_Mode false ``` Replace `` with the collector's address (`localhost` if it runs on the same host). #### 4 Restart Fluent Bit and the collector Restart both services. Logs appear in the Logs section of your Middleware account. ## Test with sample logs To check the setup without touching your application logs, add a `dummy` input that generates test logs: ```ini [INPUT] Name dummy Tag sample Dummy {"message": "my log data", "source": "myhost"} [OUTPUT] Name forward Match sample Host localhost Port 8006 Retain_Metadata_In_Forward_Mode false ``` Use port `24224` instead of `8006` if you followed Option 2. Remove the `dummy` input after you see the logs in Middleware. ## Troubleshooting - **No logs in Middleware:** run Fluent Bit in the foreground (`fluent-bit -c /etc/fluent-bit/fluent-bit.conf`) and check for connection errors from the `forward` output. - **Fluent Bit reports successful flushes but no logs arrive:** make sure `Retain_Metadata_In_Forward_Mode false` is set on the `forward` output. - **Connection refused:** make sure the agent or collector is running and that port `8006` (or `24224`) is open between Fluent Bit and the receiver. - **Wrong logs are sent:** check that the `Match` pattern matches the tags of the logs you want to forward. Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us page or join our Slack channel.