Fix issues #449 #450 #451 #464 #466 #472 #475: large-scale listing, S3 object storage, epoch-2+ AU, TFRecord via s3dlio#27

Merged

FileSystemGuy merged 8 commits into

mlcommons:mainfrom

russfellows:russfellows/issue472-skip-listing

Jun 19, 2026

Commits on Jun 19, 2026

feat(listing): add skip_listing for deterministic file URI generation (issue #472)
russfellows
committed
feat(listing): add sampling validation for skip_listing (issue #472)
russfellows
committed
feat(listing): add progress output during skip_listing validation
russfellows
committed
fix(storage): fix BytesView incompatibility and S3 epoch-2+ cache bypass
russfellows
committed
fix(mpi): allreduce_min/alltoall safe in child-process and single-rank contexts
russfellows
committed
fix(mpi): repair CHILD_INITIALIZED state in initialize(); fix TFRecord datagen validation
russfellows
committed
feat(tfrecord): full generate+read via s3dlio; fix routing and read_index
russfellows
committed
docs: add skip-listing.md — operational guide for skip_listing feature
russfellows
committed