Clarify explanation of latencies in KV results

[dslik]

We should consider adding explanatory text to the top of the stdout text when the KV benchmark is run that indicates that measured latencies are for entire KV blocks, which can be quite large (e.g. 1 GB, with exact sizes and the distribution of sizes depending on the simulated KV cache workload).

This will reduce confusion about the results.

(Issue raised and discussed on the 2026-03-10 KV Cache TF call)

[dslik]

Output from Hazem:

venv) sped@sped:~/test-dedup-fix/storage/kv_cache_benchmark$ kv-cache --config config.yaml --model llama3.1-8b --num-users 100 --duration 60 --gpu-mem-gb 0 --cpu-mem-gb 0 --max-concurrent-allocs 4 --generation-mode none --performance-profile throughput --cache-dir /mnt/dc3000me --seed 42 
2026-03-17 01:26:54 - kv_cache.config - INFO - Loaded configuration from config.yaml
2026-03-17 01:26:54 - kv_cache.cli - INFO - Loaded configuration from config.yaml
2026-03-17 01:26:54 - kv_cache.cli - INFO - Using random seed: 42
2026-03-17 01:26:54 - kv_cache.cache - WARNING - Could not initialize GPU backend: No GPU available for PyTorch backend
2026-03-17 01:26:54 - kv_cache.cache - INFO - Pre-generating 256 MB noise buffer...

Integrated Multi-User KV Cache Benchmark - MLPerf Edition
Model: Llama 3.1 8B
Users: 100
Duration: 60s
Seed: 42
Generation Mode: none (0.0ms/token)
Features:
  - Phase-Aware Processing: Enabled
  - Multi-turn Conversations: Enabled
  - Prefix Caching: Enabled
  - RAG Workload: Disabled
  - Autoscaling: Disabled
  - QoS Support: Enabled (Interactive/Responsive/Batch)
  - Trace-Driven (BurstGPT): Disabled
  - ShareGPT Dataset: Disabled
  - Max Concurrent Allocations: 4 (bounds RAM usage)
================================================================================

User Context Length Distribution:
  Min: 598 tokens (74.75 MB)
  Max: 22617 tokens (2827.12 MB)
  Mean: 8383 tokens (1047.88 MB)

QoS Distribution:
  interactive: 14 users
  responsive: 35 users
  batch: 51 users

Starting benchmark... Mode: standard (prefill+decode)
--------------------------------------------------------------------------------

================================================================================
BENCHMARK RESULTS - MLPerf KV Cache Storage Benchmark
Generation Mode: none (0.0ms/token)
================================================================================

### STORAGE PERFORMANCE ASSESSMENT: PASS [OK] ###
  Criteria Passed: 2/2
  [OK] Storage KV Read Bandwidth: 8.83GB/s (target: >0GB/s)
  [OK] Storage KV Write Bandwidth: 1.74GB/s (target: >0GB/s)

### OVERALL PERFORMANCE ###
Requests Completed: 198
Total Tokens Generated: 30950
Throughput (wall-clock): 511.47 tokens/sec
Throughput (storage I/O): 10.57 tokens/sec
Requests/sec: 3.27

### END-TO-END LATENCY (Queue Wait + Storage I/O + Generation) ###
  Mean: 54884.58 ms
  P50:  51370.87 ms
  P95:  127452.62 ms
  P99:  134642.86 ms

### PER-REQUEST STORAGE LATENCY (All I/O ops for one request) ###
  Mean: 14781.81 ms
  P50:  5319.00 ms
  P95:  65252.13 ms
  P99:  94629.08 ms
  (= 1 prefill write + N decode reads per request)

### STORAGE PERFORMANCE ###
  Cache Hit Rate: 88.2%
  Total Read: 534.36 GB
  Total Write: 105.24 GB
  Read/Write Ratio: 5.08
  Storage KV Read Operations/sec: 18.90
  Storage KV Write Operations/sec: 3.45

### CACHE TIER DISTRIBUTION ###
  GPU Entries: 0 (0.00 GB)
  CPU Entries: 0 (0.00 GB)
  Storage Entries: 207

### TIER-SPECIFIC KV BYTES ###
  Storage KV Bytes Written: 105.24 GB
  Storage KV Bytes Read: 534.36 GB

### STORAGE KV BANDWIDTH ###
  Storage KV Read Bandwidth: 8.83 GB/s
  Storage KV Write Bandwidth: 1.74 GB/s

### TIER-SPECIFIC LATENCIES (Total = Host + Device) ###
  Storage Read P95 (Total): 8516.94 ms
  Storage Write P95 (Total): 1930.33 ms

### STORAGE TIER LATENCY BREAKDOWN (Device = Disk I/O, Host = Serialization) ###
  Storage Read:
    - Device P95 (Disk I/O): 6366.33 ms
    - Host P95 (Serialization): 2184.70 ms
    - Total P95: 8516.94 ms
  Storage Write:
    - Device P95 (Disk I/O): 332.87 ms
    - Host P95 (Serialization): 1662.55 ms
    - Total P95: 1930.33 ms

### CACHE TYPE BREAKDOWNS ###
  System Prompt Hits: 273
  Common Phrase Hits: 0
  User Cache Hits: 849
  Multi-turn Hits: 12

### PREFIX CACHING ###
  Prefix Hits: 44
  Prefix Misses: 163
  System Prompt Reuse: 44
  Bytes Saved: 0.04 GB

### MULTI-TURN CONVERSATIONS ###
  Multi-turn Cache Hits: 12
  Multi-turn Cache Misses: 108
  Multi-turn Hit Rate: 10.0%

===

[dslik]

Will review in main MLPerf Storage meeting, and close if above resolves the concerns.