KV Cache benchmark hangs after results.json is written

[LouisDDN]

KV Cache benchmark hangs after results.json is written

environment

code: https://github.com/hazemawadalla/storage/tree/modular-refactor
Branch Tested: modular_refactor
Issue Type: Hang / Deadlock

Environment
OS: Ubuntu 24.04.4 LTS
Python: Python 3.12.3
MPI: Open MPI 4.1.6
Hardware:
CPU-only execution (no GPU)
GPU memory explicitly set to 0
Host memory limited to 4 GB
Execution Mode: Standalone Python CLI (kv-cache.py), not mlpstorage kvcache run

Steps to Reproduce

Check out the KV cache branch and install dependencies:
Run the KV cache benchmark using the standalone CLI:

   python3 kv-cache.py      --config config.yaml      --model llama3.1-8b      --num-users 50      --duration 120      --gpu-mem-gb 0      --cpu-mem-gb 4      --generation-mode realistic      --cache-dir /exafs/400NVX2/scratch/kvcache      --seed 42      --output results.json

Observe console output and process behavior after completion message.

Expected Behavior

The benchmark should:
Complete execution
Flush logs
Exit cleanly after writing results.json
Control should return to the shell prompt without manual interruption.

Actual Behavior

The process hangs indefinitely after reporting successful completion.
results.json is created successfully, indicating the benchmark logic finishes.
The process does not exit and remains stuck until manually interrupted (Ctrl+C).
Console output:

================================================================================
NOTES:
  - End-to-end latency includes simulated GPU inference
================================================================================
2026-03-12 16:50:48 - kv_cache.cli - INFO - Results saved to results.json
ls
^C^C^C^C^C^C

Misc

pytest test_kv_cache.py -v
================== 206 passed, 23 skipped in 81.18s (0:01:21) ==================

[LouisDDN]

@hazemawadalla I wasn't able to create an issue from your repo so I put it there.

[LouisDDN]

Reproductible on two different setup. It's unclear yet what's happening. It might be related to the storage I'm using.

[LouisDDN]

works fine w/ ramfs

[LouisDDN]

fixed on the test system that didn't work.