Fix sharded arrays losing their shard config on axis changes - #1077
Merged
TomNicholas merged 3 commits intoAug 6, 2026
Conversation
A chunk manifest entry locates a whole shard, but several operations in `manifests/` read `ArrayV3Metadata.chunks` for that shape - which zarr defines as the *inner* chunk shape when a sharding codec is present. The visible symptom was that concatenating sharded virtual arrays along a new dimension raised "The shard's `chunk_shape` and array's `shape` need to have the same number of dimensions.", because `stack` grew the array's ndim without touching the shard config. But the same conflation also let `stack`/`broadcast_to` write an inner chunk shape back as the outer chunk grid, and stopped a no-op slice of a sharded array short-circuiting. Adds `manifest_chunk_shape` for "the region one manifest entry locates" and uses it at those sites, and realigns a shard's inner `chunk_shape` whenever the outer one gains or loses length-1 axes - which happens in both directions, since integer indexing drops an axis. Doing so is byte-safe: it leaves the inner chunk grid, the C-order shard index layout, and every inner chunk's bytes untouched. Closes zarr-developers#1076
TomNicholas
temporarily deployed
to
test-release
August 6, 2026 15:11 — with
GitHub Actions
Inactive
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1077 +/- ##
==========================================
+ Coverage 89.60% 89.64% +0.03%
==========================================
Files 41 41
Lines 2664 2674 +10
==========================================
+ Hits 2387 2397 +10
Misses 277 277
🚀 New features to boost your workflow:
|
Most of the sharding metadata tests called copy_and_replace_metadata directly, even though the same behaviour is reachable via expand_dims, broadcast_to and integer indexing. Move them to TestSharded so they exercise the path users hit rather than pinning an internal signature. Only the guard against a chunk shape that isn't an axis change stays a direct call - no array operation can produce one, so there is no public route to that branch.
The three shapes involved in realigning a shard's inner chunk shape are easy to confuse, so spell out for each which axes it is on and how it relates to the others - including that _realign_sharding_codecs' old_chunks is the enclosing shard's inner chunk shape when recursing, not the array's outer one. Also fills in copy_and_replace_metadata, whose contract this branch changed and which had no parameter docs, covering the "default" sentinel on new_dimension_names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1076.
A chunk manifest entry locates a whole shard, but several operations in
manifests/readArrayV3Metadata.chunksfor that shape — which zarr defines as the inner chunk shape when a sharding codec is present (the outer chunk grid is.shards). The parsers and the icechunk writer already get this right; the array-op layer did not.The visible symptom was that concatenating sharded virtual arrays along a new dimension raised:
because
stackgrew the array's ndim without touching the shard config. But the same conflation had quieter consequences:stack/broadcast_towrote an inner chunk shape back as the outer chunk grid, and a no-op slice of a sharded array failed to short-circuit because it compared a chunk-grid selector derived from inner chunks against the shard-shaped manifest grid.What's in here
manifest_chunk_shape(metadata)— one accessor for "the region one manifest entry locates", used instack,broadcast_to,concatenate's partial-chunk check,_missing_element_mask,apply_selection, the reindex path,check_combinable_zarr_arrays,ManifestArray.empty_like, and the oversized-chunk warning.copy_and_replace_metadatarealigns anysharding_indexedcodec's innerchunk_shapewhen the outer chunk shape gains or loses length-1 axes. Both directions matter:stack/expand_dims/broadcast_toadd an axis, integer indexing drops one (only legal where the chunk length is already 1). Nested shards recurse.Realigning length-1 axes is byte-safe — it leaves the inner chunk grid, the C-order shard index layout, and every inner chunk's bytes untouched — so existing references stay valid rather than needing a rewrite.
Tests
Written first, all red before the fix:
TestCopyAndReplaceMetadataSharding— prepended/middle/multiple added axes, dropped axis, nested shards, the ambiguous-alignment case whereold_chunksalready contains 1s, unsharded codecs left alone, and rejection of a chunk shape that isn't an axis change.TestShardedintest_manifests/test_array.py—expand_dims,stack,broadcast_to,concatenate, plus the indexing paths (no-op full slice, integer index dropping an axis, shard-aligned slice).test_concat_sharded_arrays_along_new_dim_roundtrip_icechunk— the end-to-end payoff: two sharded Zarr stores, virtualized, concatenated along a new dim, written to Icechunk, and read back element-for-element equal toxr.concatof the originals. That's what makes the byte-safety claim concrete rather than asserted.Full suite: 642 passed, no new mypy errors.
Why it came up
Building an Icechunk datacube from an archive of per-forecast-hour
.zarr.zipfiles (via the newZippedZarrParser), where every variable is a single(720, 1440)shard of(45, 45)chunks — so every variable tripped this the moment the files were concatenated alonginit_time/forecast_hour.