FluentBit Performance

[brunooliveiramac]

Hi everyone :)
I'm receiving logs from 2 persistent TCP connections to fluentbit. This is my config.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [INPUT]
        Name         tcp
        Listen       0.0.0.0
        Port         24224
        Format       none

    [SERVICE]
        Flush        1 
        Daemon       Off
        Log_Level    info
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        Health_Check On
        Mem_Buf_Limit 2GB 

    [INPUT]
        Name         tcp
        Listen       0.0.0.0
        Port         514
        Format       none
        Buffer_Size  1024KB   
        Chunk_Size   512KB
        Threaded      on
        Tag          syslogs 

    [INPUT]
        Name         udp
        Listen       0.0.0.0
        Port         514
        Format       none
        Buffer_Size  1024KB   
        Chunk_Size   512KB
        Threaded      on
        Tag          syslogs 
    
    [OUTPUT]
        Name         kafka
        Match        syslogs
        Brokers      kafka:9092
        Topics       syslogs
        Retry_Limit  False
        Format       json
        rdkafka.queue.buffering.max.ms 100  
        rdkafka.request.required.acks 1     
        rdkafka.message.send.max.retries 5  
        rdkafka.retry.backoff.ms 100
        Workers      10

I would need to handle:
~ 70MB per sec
~ 700 bytes per message
110.000 messages per sec

Not being able to reach full capacity

I don't have issues with forwarding, and my Kafka is working fine.

The point is, I still have CPU and memory available. I would expect Fluentbit to be able to handle more throughput.
If anyone know what can I do to improve its performance?

[Sasha-mina]

The symptom you're describing — CPU and memory available but throughput capped — is a classic single-threaded I/O bottleneck, not a resource bottleneck. A few things to check:

1. Your TCP input is missing Threaded on
You have Threaded on on port 514 but not on port 24224. The port 24224 input is running on the main thread, which means it's competing with the rest of the pipeline for CPU time. Add Threaded on to both TCP inputs:

[INPUT]
    Name tcp
    Listen 0.0.0.0
    Port 24224
    Format none
    Threaded on

2. Your Kafka output workers may be the ceiling
With Workers 10, you have 10 threads writing to Kafka. At 700 bytes/message and 110k messages/sec, that's ~77MB/sec — right at your target. But rdkafka.queue.buffering.max.ms 100 means each worker flushes every 100ms, capping batch sizes. Try reducing this to 10 or even 5 to increase flush frequency and reduce per-message latency overhead.

3. Check if the bottleneck is in parsing
You're using Format none which is raw — good, no parsing overhead. But confirm with vector top equivalent: Fluent Bit's HTTP monitoring endpoint at :2020/api/v1/metrics will show you per-plugin throughput. If the input plugin rate matches your target but the output is lagging, the bottleneck is in the Kafka write path.

4. Mem_Buf_Limit 2GB is set at the service level
This is a global limit. If your two TCP inputs are competing for that buffer under burst conditions, one will start dropping. Move Mem_Buf_Limit to the individual input level and set it per input.

For what it's worth, 70 MiB/sec is a threshold where Fluent Bit's C-based single-process model starts to show its limits for aggregator-style deployments (as opposed to lightweight daemonset agents). We ran into similar ceilings when building WarpParse — ended up parallelizing the ingestion and Kafka write paths across a shared thread pool rather than per-plugin workers. Happy to share more detail on what we found if useful.

[patrick-stephens]

Super helpful answer @Sasha-mina