out_s3: $INDEX sequence index resets on restart because the metadata stream is deleted on shutdown (causes object overwrites)

[dacort]

Bug Report

Fluent Bit version: v4.2.2 (also present in v5.0.7 and current master)
Plugin: out_s3
Environment: Linux x86_64; Fluent Bit runs as a sidecar that tails rolled log files and uploads them to S3, with store_dir on a persistent volume that survives process/container restarts.

One caveat is that I'm unsure if this is intentional behavior or not? For context, I'm trying to mirror Spark event logs to S3 similar to this blog post

Configuration (minimal repro)

[INPUT]
    name             tail
    path             /var/log/app/segment_*
    db               /data/flb/tail.db
    read_from_head   on

[OUTPUT]
    name             s3
    match            *
    bucket           my-bucket
    region           us-east-1
    total_file_size  50M
    store_dir        /data/flb/s3
    s3_key_format    /logs/myapp/segment_$INDEX

store_dir (and the tail db) are on a volume that survives restarts.

Expected behavior

Per the docs:

$INDEX ... is saved in the store_dir. If you restart Fluent Bit with the same disk, it can continue incrementing the index from its last value in the previous run.

So after a restart on the same store_dir, the next upload should continue from the last value (e.g. segment_5), and previously uploaded objects should never be overwritten.

Actual behavior

After a graceful restart (SIGTERM) with the same persistent store_dir, $INDEX resets to 0. The next uploads recreate segment_0, segment_1, ... overwriting objects from the previous run. There is no Successfully recovered index. Continuing at index=N line on startup.

Root cause

$INDEX state is stored as a plain file at <store_dir>/<bucket>/sequence/index_metadata/seq_index_<id> (written with fopen/fprintf in init_seq_index() / write_seq_index()), inside an fstore stream named sequence. This file is not a Chunk I/O chunk, and Chunk I/O's scan only registers regular files located directly under a stream directory — cio_scan_stream_files() skips the nested index_metadata/ subdirectory because it is not DT_REG. As a result the sequence fstore stream has zero registered files.

On shutdown, cb_s3_exit() → s3_store_exit() → flb_fstore_destroy() iterates all streams and deletes any stream with files == 0 via flb_fstore_stream_destroy(stream, FLB_TRUE) → cio_stream_delete(), which recursively removes the stream directory. That deletes sequence/index_metadata/seq_index_<id>. On the next start, init_seq_index()'s access(seq_index_file, F_OK) misses and resets $INDEX to 0.

Relevant code (master):

• plugins/out_s3/s3.c — init_seq_index() stores $INDEX as a raw file; access()-or-reset logic.
• lib/chunkio/src/cio_scan.c — cio_scan_stream_files() only handles DT_REG entries, so the nested index_metadata/ dir is ignored and the stream has 0 files.
• src/flb_fstore.c — flb_fstore_destroy() deletes streams with 0 files (if (files == 0) delete = FLB_TRUE;).
• plugins/out_s3/s3_store.c — s3_store_exit() calls flb_fstore_destroy().

Additional notes

• Not multipart-specific: both the PutObject and multipart upload paths depend on the same init_seq_index() state file.
• Confirmed with a marker + inode test: a marker file placed inside sequence/ is gone after a graceful restart, while a marker placed one level above it (still on the same persistent volume) survives — i.e. the volume is healthy; Fluent Bit deletes its own metadata stream. The sequence/ directory comes back with a fresh inode and mtime after the restart.
• A hard crash (SIGKILL) does not delete it, because cb_s3_exit() / flb_fstore_destroy() only run on graceful shutdown — but typical restarts use SIGTERM, which does.

Proposed fix

Detach the metadata stream reference in s3_store_exit() before flb_fstore_destroy() runs, so the empty-stream cleanup skips it and the persisted $INDEX file survives:

if (ctx->stream_metadata != NULL) {
    flb_fstore_stream_destroy(ctx->stream_metadata, FLB_FALSE);
    ctx->stream_metadata = NULL;
}

I have a fix ready and will open a PR shortly.

[dacort]

This may be because I had two out configs share the same store_dir - verifying.

[dacort]

Having a different store_dir per output did not solve the issue without this patch included. But it does seem like there's still an opportunity for a more complete fix.

• s3_store_init() creates stream_upload (multipart_upload_metadata) unconditionally, for every output (s3_store.c:315-322) — which is why multipart metadata survives across all co-located outputs.
• stream_metadata (sequence) is created only in init_seq_index(), i.e. only for the $INDEX-owning output.

So with a shared store_dir (and there's literally a comment at s3.c:674 — "to support multiple instances of this plugin using the same buffer dir"), every output's cio_load registers sequence/ as a 0-file stream, but only s3.0 (events) knows it's special. My detach patch guards if (ctx->stream_metadata != NULL), which is false for s3.1 (appstatus) → appstatus's flb_fstore_destroy nukes sequence/.

AI-generated root cause follows

Follow-up after end-to-end validation of the initial fix: guarding only the graceful-exit path is incomplete.

The root issue is that the S3 sequence stream stores $INDEX state as a raw nested file under sequence/index_metadata/seq_index_<id>. Chunk I/O only counts valid chunk files directly under the stream directory, so sequence is still seen as an empty stream unless it contains a real chunk. Any fstore cleanup path that destroys empty streams can therefore remove the whole sequence/ directory and reset $INDEX on the next start.

The fix should make sequence non-empty to fstore/chunkio, not just detach it during s3_store_exit(). The patch I am testing now keeps the existing raw index_metadata/seq_index_* format for compatibility, but also creates a valid marker chunk directly under sequence/ when $INDEX is enabled. It also skips the sequence stream when mapping old S3 data files and when checking for old buffered data, so the marker is not treated as upload backlog.

Relevant call paths:

• restore decision: cb_s3_init() -> s3_store_init() -> init_seq_index() -> access(seq_index_file) / read_seq_index()
• empty-stream deletion: flb_fstore_destroy() -> flb_fstore_stream_destroy(..., FLB_TRUE) -> cio_stream_delete() -> recursive delete
• reason sequence looks empty: cio_scan_stream_files() only loads regular files directly under the stream dir; nested index_metadata/ is ignored

This means the final fix needs to cover both startup/load cleanup and exit cleanup by preventing the sequence stream from ever being empty once $INDEX is in use.

[dacort]

In addition to the PR referenced above, I also tried this patch which seems to work better as well but might just fix it for s3 and not address the underlying root cause, which AI seems to think is "persisting the sequence index as a first-class Chunk I/O chunk, not a raw fopen file in a chunkio-invisible subdir".

diff --git a/plugins/out_s3/s3_store.c b/plugins/out_s3/s3_store.c
index fbaf175..de60c45 100644
--- a/plugins/out_s3/s3_store.c
+++ b/plugins/out_s3/s3_store.c
@@ -337,6 +337,22 @@ int s3_store_init(struct flb_s3 *ctx)
     }
     ctx->stream_quarantine = fs_stream;
 
+    /*
+     * Metadata ("sequence") stream that holds the $INDEX state. It is created
+     * for every instance -- like the multipart upload stream above -- so that
+     * when multiple S3 outputs share a store_dir, an output that does not use
+     * $INDEX does not delete the $INDEX state written by the output that does
+     * (the deletion is avoided in s3_store_exit()).
+     */
+    fs_stream = flb_fstore_stream_create(ctx->fs, "sequence");
+    if (!fs_stream) {
+        flb_plg_error(ctx->ins, "could not initialize sequence metadata stream");
+        flb_fstore_destroy(fs);
+        ctx->fs = NULL;
+        return -1;
+    }
+    ctx->stream_metadata = fs_stream;
+
     set_files_context(ctx);
     return 0;
 }
@@ -353,6 +369,25 @@ int s3_store_exit(struct flb_s3 *ctx)
         return 0;
     }
 
+    /*
+     * Detach the metadata ("sequence") stream so flb_fstore_destroy() does not
+     * delete it as an empty stream.
+     *
+     * The $INDEX value is persisted as a plain file inside this stream's
+     * directory (sequence/index_metadata/seq_index_<id>); it is not a Chunk I/O
+     * chunk, so the stream has zero registered fstore files. Because
+     * flb_fstore_destroy() recursively deletes any stream that has no files, the
+     * metadata stream (and the persisted $INDEX) would be removed on every
+     * graceful shutdown and reset to 0 on the next start. Since the metadata
+     * stream is created for every instance (see s3_store_init()), detaching it
+     * here -- without deleting the directory -- also prevents a co-located
+     * output that shares the store_dir but does not use $INDEX from deleting it.
+     */
+    if (ctx->stream_metadata != NULL) {
+        flb_fstore_stream_destroy(ctx->stream_metadata, FLB_FALSE);
+        ctx->stream_metadata = NULL;
+    }
+
     /* release local context on non-multi upload files */
     mk_list_foreach(head, &ctx->fs->streams) {
         fs_stream = mk_list_entry(head, struct flb_fstore_stream, _head);