Could not flush page cache between epochs warning

[rasar07]

Flush page cache command times out during a training run. Our system requires more than 30 seconds to flush caches.

Output from an mlpstorage closed training retinanet run file command:

[OUTPUT] 2026-06-20T13:34:43.831067 Worker pre-warm complete (8 workers spawned)
[WARNING] Could not flush page cache between epochs: Command '['sudo', '-n', 'sh', '-c', 'echo 3 > /proc/sys/vm/drop_caches']' timed out after 30 seconds. Per-epoch reads may be served from the OS buffer cache, inflating throughput numbers. To enable, configure passwordless sudo for sh -c 'echo 3 > /proc/sys/vm/drop_caches'.
[OUTPUT] 2026-06-20T13:35:13.865508 Starting epoch 1: 32641 steps expected

[FileSystemGuy]

Triage: thanks for the report. The 30s timeout lives in DLIO (dlio_benchmark/main.py:687–693), not in mlpstorage — DLIO PR #23 introduced it to bound #391, which hung for ~16 hours waiting on an interactive sudo prompt. The current except Exception block doesn't distinguish "the kernel needs longer than 30s" from "sudo refuses," so on your host it's getting silently disabled and reads are being served from the page cache.

We're addressing this with both fixes you'd want:

1. Distinguish timeout from auth failure — split the except into subprocess.TimeoutExpired (slow-kernel: warn once, don't disable, retry next epoch) vs other (sudo refused: keep the existing warn-and-disable behavior).
2. Expose the timeout — DLIO_DROP_CACHES_TIMEOUT env var, default 30, minimum 1. Operators on large-RAM hosts can raise the ceiling without an upstream change.

DLIO patch posted at mlcommons/DLIO_local_changes#28 with unit tests for the env-var parsing. Once that lands, we'll bump the dlio-benchmark pin in this repo and add an mlpstorage CLI knob (--drop-caches-timeout-seconds) that plumbs through to the env var, so you'll be able to set it either way.

For your immediate run, after the bump lands you'll be able to do something like DLIO_DROP_CACHES_TIMEOUT=300 mlpstorage closed training retinanet run file ... (or pass --drop-caches-timeout-seconds 300 once the CLI knob is in).

[rasar07]

Is the fix included in the latest mlpstorage version (3.0.16?) It doesn't work for us yet (as of this morning.)