feat: add UnifiedLoader with config-driven loader dispatch

[ABNER-1]

Summary

Implement #65

Add a config-driven UnifiedLoader that automatically dispatches to the appropriate loader class (SafeTensorsFileLoader or ThreeFSLoader) based on a YAML/JSON configuration file.

Configuration

• docs/configuration.md (new): Comprehensive configuration guide with 5 usage examples
• README.md: Add Configuration section linking to docs

Usage

from fastsafetensors import UnifiedLoader

loader = UnifiedLoader(pg, hf_weights_files, device="cuda:0")
for key, tensor in loader.iterate_weights():
    process(key, tensor)
loader.close()

Configuration via fastsafetensors.yaml or FASTSAFETENSORS_CONFIG env var.

Default Config (GDS)

When no config file is present, UnifiedLoader uses the following defaults:

loader: "base"
framework: "pytorch"
debug_log: false
set_numa: true
disable_cache: true

parallel:
  use_pipeline: false

The default call chain:

UnifiedLoader(pg, files, device)
  ├─ load_config() → no config file → LoaderConfig()
  ├─ _resolve_loader_class("base") → SafeTensorsFileLoader
  ├─ get_extension_config("base") → {}
  │   └─ process_extension_config({}) → {nogds: False}
  ├─ SafeTensorsFileLoader(pg, device, nogds=False, ...)
  ├─ create_parallel_kwargs() → {queue_size: -1}  (use_pipeline=False)
  └─ PipelineParallel(pg, loader, files, queue_size=-1)
       └─ serial: copy_files → broadcast → copy_files → ...

Note: The base loader defaults to copier_type: "gds" (GPU Direct Storage), matching the behavior of calling SafeTensorsFileLoader directly with nogds=False.

3FS Config Example

To use the 3FS high-performance I/O loader, create a fastsafetensors.yaml:

loader: "3fs"
framework: "pytorch"

"3fs":
  mount_point: "/mnt/3fs"

parallel:
  use_pipeline: true
  max_concurrent_producers: 1
  queue_size: 0
  use_tqdm_on_load: true

The 3FS call chain:

UnifiedLoader(pg, files, device)
  ├─ load_config() → fastsafetensors.yaml → LoaderConfig(loader="3fs")
  ├─ _resolve_loader_class("3fs") → ThreeFSLoader
  ├─ get_extension_config("3fs") → {mount_point: "/mnt/3fs"}
  │   └─ process_extension_config({mount_point: "/mnt/3fs"}) → {mount_point: "/mnt/3fs"}
  ├─ ThreeFSLoader(pg, device, mount_point="/mnt/3fs", ...)
  ├─ create_parallel_kwargs() → {max_concurrent_producers: 1, queue_size: 0, ...}
  └─ PipelineParallel(pg, loader, files, max_concurrent_producers=1, queue_size=0)
       └─ pipeline: producer threads copy files concurrently

Note: If mount_point is omitted, ThreeFSLoader.process_extension_config will auto-infer it from file paths via fastsafetensor_3fs_reader.extract_mount_point().

[takeshi-yoshimura]

please resolve my comments. thanks!

[takeshi-yoshimura]

I want to avoid new dependencies. Can we only use JSON?

[takeshi-yoshimura]

I am not sure if UnifiedLoader is correct naming from user's perspective. AutoLoader?

[takeshi-yoshimura]

This section is too detailed for users. Maybe you can just delete this section.

[takeshi-yoshimura]

gds?

[takeshi-yoshimura]

Please use logger instead of print in this file.

[takeshi-yoshimura]

3fs?