Skip to content

Fix sharded arrays losing their shard config on axis changes - #1077

Merged
TomNicholas merged 3 commits into
zarr-developers:mainfrom
TomNicholas:fix-sharding-expand-dims
Aug 6, 2026
Merged

Fix sharded arrays losing their shard config on axis changes#1077
TomNicholas merged 3 commits into
zarr-developers:mainfrom
TomNicholas:fix-sharding-expand-dims

Conversation

@TomNicholas

@TomNicholas TomNicholas commented Aug 6, 2026

Copy link
Copy Markdown
Member

Closes #1076.

A chunk manifest entry locates a whole shard, but several operations in manifests/ read ArrayV3Metadata.chunks for that shape — which zarr defines as the inner chunk shape when a sharding codec is present (the outer chunk grid is .shards). The parsers and the icechunk writer already get this right; the array-op layer did not.

The visible symptom was that concatenating sharded virtual arrays along a new dimension raised:

ValueError: The shard's `chunk_shape` and array's `shape` need to have the same number of dimensions.

because stack grew the array's ndim without touching the shard config. But the same conflation had quieter consequences: stack/broadcast_to wrote an inner chunk shape back as the outer chunk grid, and a no-op slice of a sharded array failed to short-circuit because it compared a chunk-grid selector derived from inner chunks against the shard-shaped manifest grid.

What's in here

  • manifest_chunk_shape(metadata) — one accessor for "the region one manifest entry locates", used in stack, broadcast_to, concatenate's partial-chunk check, _missing_element_mask, apply_selection, the reindex path, check_combinable_zarr_arrays, ManifestArray.empty_like, and the oversized-chunk warning.
  • copy_and_replace_metadata realigns any sharding_indexed codec's inner chunk_shape when the outer chunk shape gains or loses length-1 axes. Both directions matter: stack/expand_dims/broadcast_to add an axis, integer indexing drops one (only legal where the chunk length is already 1). Nested shards recurse.

Realigning length-1 axes is byte-safe — it leaves the inner chunk grid, the C-order shard index layout, and every inner chunk's bytes untouched — so existing references stay valid rather than needing a rewrite.

Tests

Written first, all red before the fix:

  • TestCopyAndReplaceMetadataSharding — prepended/middle/multiple added axes, dropped axis, nested shards, the ambiguous-alignment case where old_chunks already contains 1s, unsharded codecs left alone, and rejection of a chunk shape that isn't an axis change.
  • TestSharded in test_manifests/test_array.pyexpand_dims, stack, broadcast_to, concatenate, plus the indexing paths (no-op full slice, integer index dropping an axis, shard-aligned slice).
  • test_concat_sharded_arrays_along_new_dim_roundtrip_icechunk — the end-to-end payoff: two sharded Zarr stores, virtualized, concatenated along a new dim, written to Icechunk, and read back element-for-element equal to xr.concat of the originals. That's what makes the byte-safety claim concrete rather than asserted.

Full suite: 642 passed, no new mypy errors.

Why it came up

Building an Icechunk datacube from an archive of per-forecast-hour .zarr.zip files (via the new ZippedZarrParser), where every variable is a single (720, 1440) shard of (45, 45) chunks — so every variable tripped this the moment the files were concatenated along init_time/forecast_hour.

A chunk manifest entry locates a whole shard, but several operations in
`manifests/` read `ArrayV3Metadata.chunks` for that shape - which zarr
defines as the *inner* chunk shape when a sharding codec is present.

The visible symptom was that concatenating sharded virtual arrays along a
new dimension raised "The shard's `chunk_shape` and array's `shape` need
to have the same number of dimensions.", because `stack` grew the array's
ndim without touching the shard config. But the same conflation also let
`stack`/`broadcast_to` write an inner chunk shape back as the outer chunk
grid, and stopped a no-op slice of a sharded array short-circuiting.

Adds `manifest_chunk_shape` for "the region one manifest entry locates"
and uses it at those sites, and realigns a shard's inner `chunk_shape`
whenever the outer one gains or loses length-1 axes - which happens in
both directions, since integer indexing drops an axis. Doing so is
byte-safe: it leaves the inner chunk grid, the C-order shard index
layout, and every inner chunk's bytes untouched.

Closes zarr-developers#1076
@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.64%. Comparing base (1ba8470) to head (e8c947c).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1077      +/-   ##
==========================================
+ Coverage   89.60%   89.64%   +0.03%     
==========================================
  Files          41       41              
  Lines        2664     2674      +10     
==========================================
+ Hits         2387     2397      +10     
  Misses        277      277              
Files with missing lines Coverage Δ
virtualizarr/manifests/array.py 86.79% <ø> (ø)
virtualizarr/manifests/array_api.py 98.59% <100.00%> (ø)
virtualizarr/manifests/indexing.py 92.42% <100.00%> (ø)
virtualizarr/manifests/store.py 90.38% <100.00%> (ø)
virtualizarr/manifests/utils.py 93.24% <100.00%> (+1.05%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Most of the sharding metadata tests called copy_and_replace_metadata
directly, even though the same behaviour is reachable via expand_dims,
broadcast_to and integer indexing. Move them to TestSharded so they
exercise the path users hit rather than pinning an internal signature.

Only the guard against a chunk shape that isn't an axis change stays a
direct call - no array operation can produce one, so there is no public
route to that branch.
The three shapes involved in realigning a shard's inner chunk shape are
easy to confuse, so spell out for each which axes it is on and how it
relates to the others - including that _realign_sharding_codecs'
old_chunks is the enclosing shard's inner chunk shape when recursing,
not the array's outer one.

Also fills in copy_and_replace_metadata, whose contract this branch
changed and which had no parameter docs, covering the "default" sentinel
on new_dimension_names.
@TomNicholas
TomNicholas merged commit 18b2569 into zarr-developers:main Aug 6, 2026
2 checks passed
@TomNicholas
TomNicholas deleted the fix-sharding-expand-dims branch August 6, 2026 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

expand_dims/stack fails on arrays using the sharding_indexed codec

1 participant