Bug Report
Hi,
We have noticed that, after a few days of processing Kubernetes events, fluentbit just hangs. If debug not turned on it does not log any event to stdout.
The config is pretty basic:
containers:
- image: cr.fluentbit.io/fluent/fluent-bit:5.0.7@sha256:c96ee743cba9b1d5a38654931f411700af80bb7652697afbe67daad46cae237b
args:
- -vvvv
- --http
- -i
- kubernetes_events
- -o
- stdout
- -p
- format=json_lines
- -p
- json_date_key=false with debug turned on we see this output:
[last processed event]
2026-06-22T14:53:06.497Z {"kind":"Event","apiVersion":"v1","metadata": [.... redacted ....]
2026-06-22T14:53:06.497Z [2026/06/22 14:53:06.497] [debug] [task] destroy task=0x7fa4b282e780 (task_id=0)
2026-06-22T14:53:06.997Z [2026/06/22 14:53:06.996] [debug] [input coro] destroy coro_id=3003
2026-06-22T14:53:07.497Z [2026/06/22 14:53:07.496] [debug] [input coro] destroy coro_id=3004
2026-06-22T14:53:07.997Z [2026/06/22 14:53:07.996] [debug] [input coro] destroy coro_id=3005
[many more of these for days....]
2026-06-22T14:53:15.497Z [2026/06/22 14:53:15.496] [debug] [input coro] destroy coro_id=7893
at this broken state the health endpoint seems fine:
➜ ~ curl localhost:2020
{"fluent-bit":{"version":"5.0.7","edition":"Community","flags":["FLB_HAVE_SYS_WAIT_H","FLB_HAVE_SIMD","FLB_HAVE_IN_STORAGE_BACKLOG","FLB_HAVE_CHUNK_TRACE","FLB_HAVE_PARSER","FLB_HAVE_RECORD_ACCESSOR","FLB_HAVE_STREAM_PROCESSOR","FLB_HAVE_YYJSON","FLB_HAVE_UNICODE_ENCODER","FLB_EVENT_LOOP_AUTO_DISCOVERY","FLB_HAVE_TLS","FLB_HAVE_OPENSSL","FLB_HAVE_METRICS","FLB_HAVE_PROFILES","FLB_HAVE_WASM","FLB_HAVE_RIPSER","FLB_HAVE_KAFKA_SASL","FLB_HAVE_KAFKA_OAUTHBEARER","FLB_HAVE_AWS_MSK_IAM","FLB_HAVE_AWS","FLB_HAVE_AWS_CREDENTIAL_PROCESS","FLB_HAVE_SIGNV4","FLB_HAVE_SQLDB","FLB_LOG_NO_CONTROL_CHARS","FLB_HAVE_METRICS","FLB_HAVE_HTTP_SERVER","FLB_HAVE_SYSTEMD","FLB_HAVE_SYSTEMD_SDBUS","FLB_HAVE_FORK","FLB_HAVE_GMTOFF","FLB_HAVE_TIME_ZONE","FLB_HAVE_UNIX_SOCKET","FLB_HAVE_LITTLE_ENDIAN_SYSTEM","FLB_HAVE_LIBYAML","FLB_HAVE_ATTRIBUTE_ALLOC_SIZE","FLB_HAVE_PROXY_GO","FLB_HAVE_JEMALLOC","FLB_HAVE_LIBBACKTRACE","FLB_HAVE_REGEX","FLB_HAVE_UTF8_ENCODER","FLB_HAVE_LUAJIT","FLB_HAVE_ACCEPT4","FLB_HAVE_INOTIFY","FLB_HAVE_GETENTROPY","FLB_HAVE_GETENTROPY_SYS_RANDOM"]}}
➜ ~ curl localhost:2020/api/v2/health
{"status":"ok","errors":0,"retries_failed":0,"error_limit":5,"retry_failure_limit":5,"period_limit":60}
I also tried collecting a coredump. Maybe it is useful:
Thread 6 (LWP 834239):
#0 0x00007fa4b4afa9ee in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa4b4aef668 in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa4b4aef6ad in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa4b4b70aad in epoll_wait () from ./lib/x86_64-linux-gnu/libc.so.6
#4 0x00005593b9b34c9e in _mk_event_wait_2 (loop=0x7fa4af60a000, timeout=250) at /src/fluent-bit/lib/monkey/mk_core/mk_event_epoll.c:441
#5 0x00005593b9b350ea in mk_event_wait_2 (loop=0x7fa4af60a000, timeout=250) at /src/fluent-bit/lib/monkey/mk_core/mk_event.c:217
#6 0x00005593b945d18a in flb_http_server_worker_thread (data=0x7fa4b2870600) at /src/fluent-bit/src/http_server/flb_http_server.c:645
#7 0x00007fa4b4af2b7b in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#8 0x00007fa4b4b70630 in clone () from ./lib/x86_64-linux-gnu/libc.so.6
Thread 5 (LWP 834238):
#0 0x00007fa4b4afa9ee in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa4b4aef668 in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa4b4aef6ad in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa4b4b70aad in epoll_wait () from ./lib/x86_64-linux-gnu/libc.so.6
#4 0x00005593b9b34c9e in _mk_event_wait_2 (loop=0x7fa4b280af78, timeout=-1) at /src/fluent-bit/lib/monkey/mk_core/mk_event_epoll.c:441
#5 0x00005593b9b350c8 in mk_event_wait (loop=0x7fa4b280af78) at /src/fluent-bit/lib/monkey/mk_core/mk_event.c:207
#6 0x00005593b93eabd9 in output_thread (data=0x7fa4b286d000) at /src/fluent-bit/src/flb_output_thread.c:257
#7 0x00005593b940cf0c in step_callback (data=0x7fa4b2846240) at /src/fluent-bit/src/flb_worker.c:43
#8 0x00007fa4b4af2b7b in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#9 0x00007fa4b4b70630 in clone () from ./lib/x86_64-linux-gnu/libc.so.6
Thread 4 (LWP 834237):
#0 0x00007fa4b4afa9ee in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa4b4aef668 in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa4b4aef6ad in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa4b4b70aad in epoll_wait () from ./lib/x86_64-linux-gnu/libc.so.6
#4 0x00005593b9b34c9e in _mk_event_wait_2 (loop=0x7fa4b280aa50, timeout=-1) at /src/fluent-bit/lib/monkey/mk_core/mk_event_epoll.c:441
#5 0x00005593b9b350c8 in mk_event_wait (loop=0x7fa4b280aa50) at /src/fluent-bit/lib/monkey/mk_core/mk_event.c:207
#6 0x00005593b93e490c in input_thread (data=0x7fa4b28151c0) at /src/fluent-bit/src/flb_input_thread.c:417
#7 0x00005593b940cf0c in step_callback (data=0x7fa4b2846180) at /src/fluent-bit/src/flb_worker.c:43
#8 0x00007fa4b4af2b7b in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#9 0x00007fa4b4b70630 in clone () from ./lib/x86_64-linux-gnu/libc.so.6
Thread 3 (LWP 834236):
#0 0x00007fa4b4afa9ee in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa4b4aef668 in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa4b4aef6ad in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa4b4b70aad in epoll_wait () from ./lib/x86_64-linux-gnu/libc.so.6
#4 0x00005593b9b34c9e in _mk_event_wait_2 (loop=0x7fa4b280a018, timeout=-1) at /src/fluent-bit/lib/monkey/mk_core/mk_event_epoll.c:441
#5 0x00005593b9b350c8 in mk_event_wait (loop=0x7fa4b280a018) at /src/fluent-bit/lib/monkey/mk_core/mk_event.c:207
#6 0x00005593b93cd5c5 in log_worker_collector (data=0x7fa4b2815000) at /src/fluent-bit/src/flb_log.c:534
#7 0x00005593b940cf0c in step_callback (data=0x7fa4b28460c0) at /src/fluent-bit/src/flb_worker.c:43
#8 0x00007fa4b4af2b7b in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#9 0x00007fa4b4b70630 in clone () from ./lib/x86_64-linux-gnu/libc.so.6
Thread 2 (LWP 834235):
#0 0x00007fa4b4afa9ee in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
#1 0x00007fa4b4aef668 in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa4b4aef6ad in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa4b4b70aad in epoll_wait () from ./lib/x86_64-linux-gnu/libc.so.6
#4 0x00005593b9b34c9e in _mk_event_wait_2 (loop=0x7fa4b280a000, timeout=-1) at /src/fluent-bit/lib/monkey/mk_core/mk_event_epoll.c:441
#5 0x00005593b9b350c8 in mk_event_wait (loop=0x7fa4b280a000) at /src/fluent-bit/lib/monkey/mk_core/mk_event.c:207
#6 0x00005593b93fd525 in flb_engine_start (config=config@entry=0x7fa4b3a5a000) at /src/fluent-bit/src/flb_engine.c:1140
#7 0x00005593b93cbe84 in flb_lib_worker (data=0x7fa4b3a16440) at /src/fluent-bit/src/flb_lib.c:909
#8 0x00007fa4b4af2b7b in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#9 0x00007fa4b4b70630 in clone () from ./lib/x86_64-linux-gnu/libc.so.6
Thread 1 (LWP 834223):
#0 0x00007fa4b4afa9ee in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa4b4aef668 in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa4b4b3bfba in clock_nanosleep () from ./lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa4b4b473d3 in nanosleep () from ./lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fa4b4b58da8 in sleep () from ./lib/x86_64-linux-gnu/libc.so.6
#5 0x00005593b9336504 in flb_main_run (argc=<optimized out>, argv=<optimized out>) at /src/fluent-bit/src/fluent-bit.c:1488
#6 0x00007fa4b4a89ca8 in ?? () from ./lib/x86_64-linux-gnu/libc.so.6
#7 0x00007fa4b4a89d65 in __libc_start_main () from ./lib/x86_64-linux-gnu/libc.so.6
#8 0x00005593b93341a1 in _start ()
If needed, I can share the coredump via email, as I am unsure whether it contains any private data.
Expected behavior
Not to silently drop events. If something bad happens at least crash and let the supervisor/schedular/init-system restart fluentbit or return 5xx statuscode on healthcheck endpoint.
Your Environment
- Version used: 5.0.7
- Configuration: See above
- Environment name and version (e.g. Kubernetes? What version?): Kubernetes 1.35.6
- Server type and version: Kubernetes Worker on Debian 12
- Operating System and version: offical fluentbit image
Bug Report
Hi,
We have noticed that, after a few days of processing Kubernetes events, fluentbit just hangs. If debug not turned on it does not log any event to stdout.
The config is pretty basic:
with debug turned on we see this output:
at this broken state the health endpoint seems fine:
I also tried collecting a coredump. Maybe it is useful:
If needed, I can share the coredump via email, as I am unsure whether it contains any private data.
Expected behavior
Not to silently drop events. If something bad happens at least crash and let the supervisor/schedular/init-system restart fluentbit or return 5xx statuscode on healthcheck endpoint.
Your Environment