Bug Report
What happens
With out_forward using tls on + Require_ack_response true + keepalive (default on), if an upstream — or a TLS-terminating proxy in front of it — sends a TLS close_notify on an idle pooled connection, that connection is returned to the keepalive pool instead of being destroyed. On the next reuse, SSL_read for the ack returns SSL_ERROR_ZERO_RETURN immediately from cached session state (no network read), so the flush logs cannot get ack and fails. The same poisoned connection keeps being handed out, so every retry fails identically until Retry_Limit is exhausted → dropped_records (silent log loss).
Root cause (two parts)
tls_net_read (src/tls/openssl.c) does not flag the closed session. On SSL_read returning SSL_ERROR_ZERO_RETURN (and SSL_ERROR_SSL) it returns -1 without setting connection->net_error; only SSL_ERROR_SYSCALL sets it. flb_upstream_conn_release() only refuses to recycle a connection when net_error is set, so a TLS-half-closed connection is recycled back into the keepalive pool.flb_upstream_conn_get() cannot detect the half-close. It validates a pooled connection only with getsockopt(SO_ERROR), which is 0 for a socket whose peer sent close_notify (the TCP layer may still be ESTABLISHED/CLOSE_WAIT). There is no recv(MSG_PEEK)/EOF probe, so the half-closed connection passes the health check and is reused.
Net effect: a "poisoned" connection is reused indefinitely, producing repeated instant cannot get ack.
Reproduction (observed)
- Sender: Fluent Bit v3.2.3,
out_forward with tls on, tls.verify off, Require_ack_response true, keepalive on, net.io_timeout unset.
- Path: sender → (TLS) istio/envoy gateway that terminates TLS → (plaintext) fluentd/fluent-bit aggregator. When the aggregator pod is churned, envoy relays a
close_notify onto the sender's pooled TLS connection while the downstream TCP stays up.
- Result: bursts of
[output:forward:...] cannot get ack; no per-failure TCP FIN/RST; the connection stays reused. strace shows the failing flushes do app-data writes but no network read for the ack (ZERO_RETURN from cached session state). After Retry_Limit is exhausted, dropped_records climbs.
- Config-only mitigations that work while keeping keepalive on:
net.keepalive_max_recycle N forces the poisoned connection out of the pool after N reuses so it self-heals; net.keepalive off also avoids it. net.io_timeout does not help — the failure is instant (a cached ZERO_RETURN), not a hang.
Impact
Silent, sustained log loss on any TLS forward path sitting behind a peer/proxy that half-closes: graceful upstream shutdown, LB/proxy connection recycling, or proxy upstream failover.
Versions
Reproduced on 3.2.3. The tls_net_read / flb_upstream_conn_get code paths are unchanged on current master.
Suggested fix
- Primary (most fundamental, TLS transport layer): in
tls_net_read (src/tls/openssl.c), set connection->net_error on SSL_ERROR_ZERO_RETURN (and SSL_ERROR_SSL) so flb_upstream_conn_release() destroys the half-closed connection instead of recycling it. ZERO_RETURN means the peer closed the TLS session — such a connection must never be returned to the pool. Fixing it here covers every consumer of the upstream pool, not just out_forward.
- Complementary: add an EOF /
recv(MSG_PEEK) liveness probe in flb_upstream_conn_get() so a connection closed while idle in the pool is caught before reuse (removes even the single first-reuse failure).
#12046 force-closes on any non-FLB_OK flush result in out_forward, which would incidentally cover this case (a failed ack read returns non-OK). However, its enumerated triggers are partial-write / handshake / WOULDBLOCK, not the TLS-read ZERO_RETURN path, and it compensates at the plugin level rather than fixing the net_error handling at the TLS layer. The primary fix above addresses the root and benefits all pool consumers.
Bug Report
What happens
With
out_forwardusingtls on+Require_ack_response true+ keepalive (default on), if an upstream — or a TLS-terminating proxy in front of it — sends a TLSclose_notifyon an idle pooled connection, that connection is returned to the keepalive pool instead of being destroyed. On the next reuse,SSL_readfor the ack returnsSSL_ERROR_ZERO_RETURNimmediately from cached session state (no network read), so the flush logscannot get ackand fails. The same poisoned connection keeps being handed out, so every retry fails identically untilRetry_Limitis exhausted →dropped_records(silent log loss).Root cause (two parts)
tls_net_read(src/tls/openssl.c) does not flag the closed session. OnSSL_readreturningSSL_ERROR_ZERO_RETURN(andSSL_ERROR_SSL) it returns-1without settingconnection->net_error; onlySSL_ERROR_SYSCALLsets it.flb_upstream_conn_release()only refuses to recycle a connection whennet_erroris set, so a TLS-half-closed connection is recycled back into the keepalive pool.flb_upstream_conn_get()cannot detect the half-close. It validates a pooled connection only withgetsockopt(SO_ERROR), which is0for a socket whose peer sentclose_notify(the TCP layer may still be ESTABLISHED/CLOSE_WAIT). There is norecv(MSG_PEEK)/EOF probe, so the half-closed connection passes the health check and is reused.Net effect: a "poisoned" connection is reused indefinitely, producing repeated instant
cannot get ack.Reproduction (observed)
out_forwardwithtls on,tls.verify off,Require_ack_response true, keepalive on,net.io_timeoutunset.close_notifyonto the sender's pooled TLS connection while the downstream TCP stays up.[output:forward:...] cannot get ack; no per-failure TCP FIN/RST; the connection stays reused.straceshows the failing flushes do app-data writes but no network read for the ack (ZERO_RETURNfrom cached session state). AfterRetry_Limitis exhausted,dropped_recordsclimbs.net.keepalive_max_recycle Nforces the poisoned connection out of the pool after N reuses so it self-heals;net.keepalive offalso avoids it.net.io_timeoutdoes not help — the failure is instant (a cachedZERO_RETURN), not a hang.Impact
Silent, sustained log loss on any TLS forward path sitting behind a peer/proxy that half-closes: graceful upstream shutdown, LB/proxy connection recycling, or proxy upstream failover.
Versions
Reproduced on 3.2.3. The
tls_net_read/flb_upstream_conn_getcode paths are unchanged on currentmaster.Suggested fix
tls_net_read(src/tls/openssl.c), setconnection->net_erroronSSL_ERROR_ZERO_RETURN(andSSL_ERROR_SSL) soflb_upstream_conn_release()destroys the half-closed connection instead of recycling it.ZERO_RETURNmeans the peer closed the TLS session — such a connection must never be returned to the pool. Fixing it here covers every consumer of the upstream pool, not justout_forward.recv(MSG_PEEK)liveness probe inflb_upstream_conn_get()so a connection closed while idle in the pool is caught before reuse (removes even the single first-reuse failure).Relation to #12046
#12046 force-closes on any non-
FLB_OKflush result inout_forward, which would incidentally cover this case (a failed ack read returns non-OK). However, its enumerated triggers are partial-write / handshake /WOULDBLOCK, not the TLS-readZERO_RETURNpath, and it compensates at the plugin level rather than fixing thenet_errorhandling at the TLS layer. The primary fix above addresses the root and benefits all pool consumers.