Fix false disk-full read-only during concurrent snapshot directory de…#18114
Merged
jt2594838 merged 1 commit intoJul 6, 2026
Merged
Conversation
…letion When receiving snapshot fragments, getOccupiedSpace walks the shared snapshot root while other regions may delete their staging subdirectories concurrently. NoSuchFileException was misinterpreted as folder full, triggering node read-only and failed region migration during datanode removal.
jt2594838
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem
During datanode removal, multiple DataRegions may concurrently migrate snapshots to the same target node. All snapshot staging directories share one root (
data/datanode/data/snapshot/). While one region is receiving snapshots,getNextFolder()walks this shared root to compute occupied space; if another region deletes its staging subdirectory at the same time,Files.walkthrowsNoSuchFileException. This was misinterpreted as "folder full" → node switched to read-only (DISK_FULL) → snapshot transmission failed → region migration / datanode removal blocked.Fix
JVMCommonUtils.getOccupiedSpace: ReplaceFiles.walkwithwalkFileTree; returnCONTINUEinvisitFileFailedto ignore entries deleted during traversal.MinFolderOccupiedSpaceFirstStrategy: On refresh failure, keep the previous cached occupied space instead of settingLong.MAX_VALUE.FolderManager: Only switch to read-only when no healthy folder has available disk space (hasSpace()), not merely when folder selection fails.Tests
JVMCommonUtilsTest.getOccupiedSpaceIgnoresConcurrentlyDeletedEntriesMinFolderOccupiedSpaceFirstStrategyRealFsTest.getNextFolderDoesNotEnterReadOnlyWhenDiskHasSpaceThis PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
org.apache.iotdb.commons.utils.JVMCommonUtilsorg.apache.iotdb.commons.disk.strategy.MinFolderOccupiedSpaceFirstStrategyorg.apache.iotdb.commons.disk.FolderManagerorg.apache.iotdb.commons.i18n.UtilMessages