Czi reading fix#15
Draft
mrariden wants to merge 2 commits into
Draft
Conversation
_read_czi_plane was calling czi.open_czi() for every single plane, which parses the full subblock directory on each open. For a 107K-plane CZI on NFS this cost ~14s per open, so reading one 1024-Z-plane shard consumed the entire 4-hour LSF budget before writing a single chunk. Add a process-level reader cache (_czi_reader_cache) so the file is opened once; a per-file threading.Lock serialises concurrent reads since czidoc.read() is not thread-safe. Also add result.copy() to avoid aliasing the shared buffer across concurrent callers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…verhead The base DaskReader._read_fn path goes through a 107K-task dask graph on every virtual_chunked read call. Even though only one task needs to run, dask's graph traversal adds ~100-500ms of overhead per plane. For a shard spanning 1024 Z-planes this accumulates to 2-3 seconds per plane observed (vs ~50ms expected for raw NFS IO), making one shard take 40+ minutes. Override _read_fn in CZIReader to call czidoc.read() directly via the process-level cached reader (_get_czidoc), bypassing dask entirely. Keeps the existing LRU plane cache (keyed by non-spatial coords) so adjacent X/Y shards sharing the same Z-planes hit cache on subsequent reads. The per-file threading.Lock from the previous fix still serialises concurrent czidoc.read() calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, loading large .czi files timed out and blocks never saved. Entirely vibe coded...