feat: integrate dlio_benchmark v3.0.0-beta with multi-library S3 readers and parquet support#282

Closed

russfellows wants to merge 69 commits intomlcommons:mainfrom

russfellows:feature/parquet-readers-and-iterable-s3_2026-0319

Commits on Nov 25, 2025

Merge pull request #214 from hazemawadalla/TF_KVCache
FileSystemGuy
authored

Commits on Dec 11, 2025

vdb_benchmark commit with unit tests
idevasena
committed

Commits on Dec 12, 2025

Merge pull request #220 from idevasena/TF_VDBBench
idevasena
authored

Commits on Dec 22, 2025

feat: Replace legacy spillover logic with Waterfall LRU architecture (#219)
hazemawadalla
authored

Commits on Jan 27, 2026

Commits on Jan 30, 2026

Updated README.md
ram-sangle
authored

Commits on Feb 12, 2026

Merge pull request #230 from ram-sangle/vdb
FileSystemGuy
authored

Commits on Feb 18, 2026

Commits on Feb 19, 2026

Merge pull request #245 from idevasena/TF_VDB_Recall
dslik
authored
feat: Add s3dlio integration for MLPerf Storage with s3torchconnector compatibility

Eva Luator
authored and
Russ Fellows
committed
feat: Add multi-library S3 storage support (s3torchconnector, minio, s3dlio)

Eva Luator
authored and
Russ Fellows
committed
refactor: Organize integration tests into tests/integration/

Eva Luator
authored and
Russ Fellows
committed
docs: Add branch strategy and PR management infrastructure

Eva Luator
authored and
Russ Fellows
committed
feat: Integrate dgen-py for 155x faster checkpoint data generation

Eva Luator
authored and
Russ Fellows
committed
feat: Add StreamingCheckpointing implementation for producer-consumer checkpoint I/O
Russ Fellows
committed
feat: Add multi-library streaming checkpoint support
Russ Fellows
committed
test: Add comprehensive streaming checkpoint tests and demos
Russ Fellows
committed
docs: Consolidate and enhance documentation
Russ Fellows
committed
security: Remove hardcoded credentials and internal IPs from test files
Russ Fellows
committed
docs: Remove unnecessary TWO_PR_WORKFLOW.md
Russ Fellows
committed
Point to russfellows/dlio_benchmark fork for integrated setup
Russ Fellows
committed
docs: Clean up outdated documentation and remove azstoragetorch references
Russ Fellows
committed
refactor: Remove all azstoragetorch references from codebase
Russ Fellows
committed
deps: Update s3dlio requirement to version 0.9.50
Russ Fellows
committed
refactor: Remove dlio_benchmark from git tracking
Russ Fellows
committed
Add required dependencies and remove native Azure backend
Russ Fellows
committed

Commits on Feb 25, 2026

Merged TF_KVCache into main, accepting all incoming changes for kv_cache_benchmark
Russ Fellows
committed

Commits on Feb 27, 2026

Commits on Mar 3, 2026

Commits on Mar 4, 2026

Merge branch 'mlcommons:main' into main
russfellows
authored

Commits on Mar 10, 2026

enhancements to benchmark script for GT comparisons for recall from Alessandro and statistics updates + Updated README
idevasena
committed

Commits on Mar 12, 2026

Merge pull request #10 from russfellows/TF_VDBBench
idevasena
authored

Commits on Mar 13, 2026

Commits on Mar 18, 2026

Merge upstream mlcommons/storage main (156 commits) into local main
Russ Fellows
committed
Merge mlc-fork/main (VDBBench PRs and bench-fill-comparison)
Russ Fellows
committed
feat: restore --io-trace-log, --num-gpus, --tensor-parallel from ac5970c
Russ Fellows
committed
chore: remove dgen-py integration files (preserved on feature/dgen-py-integration branch)
Russ Fellows
committed
chore: add .env and env-fast to .gitignore (contain S3 credentials)
Russ Fellows
committed
Merge pull request #12 from russfellows/feature/io-trace-log-tensor-parallel
russfellows
authored

Commits on Mar 19, 2026

feat: object-store S3 integration, storage readers/writers, ban boto3
Russ Fellows
committed
chore: update submodule branch tracking to main (feature branch merged)
Russ Fellows
committed
chore: update dlio_benchmark submodule pointer to main (post-merge a207bcb)
Russ Fellows
committed
chore: bump version to 3.0.0; derive VERSION from package metadata
Russ Fellows
committed
feat: integrate dlio_benchmark v3.0.0-beta with multi-library S3 readers and parquet support
Russ Fellows
committed