diff --git a/PUBLIC_API.md b/PUBLIC_API.md index b0f275e..54e5b9c 100644 --- a/PUBLIC_API.md +++ b/PUBLIC_API.md @@ -392,7 +392,7 @@ Common errors when using streaming APIs: ## Storage Prerequisites -**Ensure storage exists before constructing a dataset or reader.** +**Ensure storage exists before constructing a dataset, volume, or reader.** Lode does not create storage infrastructure (directories, buckets). This is intentional: diff --git a/README.md b/README.md index 1fd2bb2..02969d0 100644 --- a/README.md +++ b/README.md @@ -202,8 +202,7 @@ Common pitfalls when using Lode: - **Metadata must be non-nil** — Pass `lode.Metadata{}` for empty metadata, not `nil`. - **Raw mode expects `[]byte`** — Without a codec, `Write` expects exactly one `[]byte` element. -- **Single-writer only** — Concurrent writers to the same dataset may corrupt history. -- **Large uploads are now atomic** — Uploads >5GB (S3) use conditional completion for atomic no-overwrite. +- **Single-writer only** — Concurrent writers to the same dataset or volume may corrupt history. - **Cleanup is best-effort** — Failed streams may leave partial objects in storage. - **StreamWriteRecords requires streaming codec** — Not all codecs support streaming. @@ -237,7 +236,7 @@ Reads always target a snapshot explicitly. ## Storage Prerequisites -**Ensure storage exists before constructing a dataset or reader.** +**Ensure storage exists before constructing a dataset, volume, or reader.** Lode does not create storage infrastructure. This is intentional: - No hidden side effects in constructors diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md index 2f7b763..53574eb 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/IMPLEMENTATION_PLAN.md @@ -337,40 +337,40 @@ Additional resolved items: - Future roadmap: CAS (multi-process) + parallel staging (single-process) ### PR1 — Contract Finalization and Design Decisions -- [ ] Finalize `CONTRACT_VOLUME.md` as authoritative (remove draft, apply DD-1–DD-8) -- [ ] Add concurrency matrices to `CONTRACT_WRITE_API.md` and `CONTRACT_VOLUME.md` -- [ ] Add `ErrOverlappingBlocks` to `CONTRACT_ERRORS.md` -- [ ] Add Layout scope note to `CONTRACT_LAYOUT.md` (Dataset-specific) -- [ ] Update `PUBLIC_API.md` Volume section with finalized API surface -- [ ] Update `IMPLEMENTATION_PLAN.md` with resolved decisions +- [x] Finalize `CONTRACT_VOLUME.md` as authoritative (remove draft, apply DD-1–DD-8) +- [x] Add concurrency matrices to `CONTRACT_WRITE_API.md` and `CONTRACT_VOLUME.md` +- [x] Add `ErrOverlappingBlocks` to `CONTRACT_ERRORS.md` +- [x] Add Layout scope note to `CONTRACT_LAYOUT.md` (Dataset-specific) +- [x] Update `PUBLIC_API.md` Volume section with finalized API surface +- [x] Update `IMPLEMENTATION_PLAN.md` with resolved decisions ### PR2 — Volume Types + Dataset Rename -- [ ] Rename `Snapshot` → `DatasetSnapshot`, `SnapshotID` → `DatasetSnapshotID` in `api.go` -- [ ] Rename `Reader` → `DatasetReader`, `NewReader` → `NewDatasetReader` -- [ ] Rename `SegmentRef` → `ManifestRef`, `ListSegments` → `ListManifests` in DatasetReader API -- [ ] Add Volume types: `VolumeID`, `VolumeSnapshotID`, `BlockRef`, `VolumeManifest`, `VolumeSnapshot` -- [ ] Add sentinels: `ErrRangeMissing`, `ErrOverlappingBlocks` -- [ ] Add `VolumeOption` type with `WithVolumeChecksum` -- [ ] Cascade renames through all code, tests, and examples +- [x] Rename `Snapshot` → `DatasetSnapshot`, `SnapshotID` → `DatasetSnapshotID` in `api.go` +- [x] Rename `Reader` → `DatasetReader`, `NewReader` → `NewDatasetReader` +- [x] Rename `SegmentRef` → `ManifestRef`, `ListSegments` → `ListManifests` in DatasetReader API +- [x] Add Volume types: `VolumeID`, `VolumeSnapshotID`, `BlockRef`, `VolumeManifest`, `VolumeSnapshot` +- [x] Add sentinels: `ErrRangeMissing`, `ErrOverlappingBlocks` +- [x] Add `VolumeOption` type with `WithVolumeChecksum` +- [x] Cascade renames through all code, tests, and examples ### PR3 — Core Volume Implementation (Stage + Commit + ReadAt) -- [ ] `NewVolume` constructor with validation -- [ ] `StageWriteAt`: write data to final path, return `BlockRef` -- [ ] `Commit`: validate, build cumulative manifest, write to store -- [ ] `ReadAt`: resolve range from manifest, read from store -- [ ] `Latest` / `Snapshots` / `Snapshot` for history access -- [ ] Fixed layout: `volumes//snapshots//manifest.json`, `volumes//data/-.bin` -- [ ] Overlap validation (sort + sweep on full cumulative block set) -- [ ] Works with fsStore and memoryStore +- [x] `NewVolume` constructor with validation +- [x] `StageWriteAt`: write data to final path, return `BlockRef` +- [x] `Commit`: validate, build cumulative manifest, write to store +- [x] `ReadAt`: resolve range from manifest, read from store +- [x] `Latest` / `Snapshots` / `Snapshot` for history access +- [x] Fixed layout: `volumes//snapshots//manifest.json`, `volumes//data/-.bin` +- [x] Overlap validation (sort + sweep on full cumulative block set) +- [x] Works with fsStore and memoryStore ### PR4 — Comprehensive Test Suite -- [ ] Sparse write patterns, multi-snapshot progression -- [ ] Resume pattern: new Volume instance, load latest, continue staging -- [ ] Edge cases: zero-length read, boundary reads, offset validation -- [ ] Missing range errors, overlap rejection -- [ ] Checksum validation when configured -- [ ] FS and memory store coverage -- [ ] Manifest round-trip (serialize → deserialize → validate) +- [x] Sparse write patterns, multi-snapshot progression +- [x] Resume pattern: new Volume instance, load latest, continue staging +- [x] Edge cases: zero-length read, boundary reads, offset validation +- [x] Missing range errors, overlap rejection +- [x] Checksum validation when configured +- [x] FS and memory store coverage +- [x] Manifest round-trip (serialize → deserialize → validate) ### PR5 — Example: Sparse Volume Ranges - [x] Runnable `examples/volume_sparse/` demonstrating stage → commit → read