Can Vector drop old files from the checkpointing?

[tzinckgraf]

Question

We have a directory structure of log files that we need to monitor. This is from a system that schedules batch jobs, so each batch job drops a file. This ends up being about 3k files per day. The ignore_older_secs seems to only work on startup to ignore files in the checkpointer.

Is there an option or pattern to drop old files? We are on an older version, but I went through the code and don't see how it would drop older files. Without that, I would expect the file handles to continually grow until it runs out.

If this doesn't exist, would this be a useful feature? I'm happy to contribute if others would use the same feature.

Vector Config

No response

Vector Logs

No response

[mg1986jp]

The ignore_older_secs only applies at startup like you noticed. For ongoing file cleanup in the checkpointer, I believe there's max_line_bytes and max_read_bytes but nothing specifically for dropping old files from the checkpoint state during runtime.

With 3k files per day, the checkpointer state will keep growing unless you prune it. One approach that works: set ignore_older_secs to something reasonable (e.g. 86400 for 1 day), and then periodically restart Vector. On restart, it'll ignore all the old files and start fresh with only recent ones in the checkpointer.

If restarts aren't acceptable, you could also set up an external cron job to delete old log files before Vector sees them, combined with glob_minimum_cooldown_ms to reduce how often Vector scans for new files. That way the old files are gone before they can accumulate in the checkpoint.

[s3onghyun]

The file source has a built-in option for this: remove_after_secs (alias remove_after). It deletes each file a configurable number of seconds after Vector has fully read it and the data has been acknowledged. That frees the OS file handle and lets Vector prune the file's checkpoint entry, since checkpoints for files that no longer exist are dropped. So it bounds both the handle count and the checkpoint growth you are worried about, without any restarts.

sources:
  my_files:
    type: file
    include: ["/var/log/batch/*.log"]
    remove_after_secs: 86400

Vector must be allowed to delete the files for this to work. If the files have to be retained, then there is no runtime pruning and you would pair an external cleanup job with ignore_older_secs. But if Vector can own the files, remove_after_secs is the mechanism you are after.