Skip to content

Fix false disk-full read-only during concurrent snapshot directory de…#18114

Merged
jt2594838 merged 1 commit into
apache:masterfrom
hongzhi-gao:fix/snapshot-folder-walk-toctou-hongzhi
Jul 6, 2026
Merged

Fix false disk-full read-only during concurrent snapshot directory de…#18114
jt2594838 merged 1 commit into
apache:masterfrom
hongzhi-gao:fix/snapshot-folder-walk-toctou-hongzhi

Conversation

@hongzhi-gao

Copy link
Copy Markdown
Contributor

Description

Problem

During datanode removal, multiple DataRegions may concurrently migrate snapshots to the same target node. All snapshot staging directories share one root (data/datanode/data/snapshot/). While one region is receiving snapshots, getNextFolder() walks this shared root to compute occupied space; if another region deletes its staging subdirectory at the same time, Files.walk throws NoSuchFileException. This was misinterpreted as "folder full" → node switched to read-only (DISK_FULL) → snapshot transmission failed → region migration / datanode removal blocked.

Fix

  1. JVMCommonUtils.getOccupiedSpace: Replace Files.walk with walkFileTree; return CONTINUE in visitFileFailed to ignore entries deleted during traversal.
  2. MinFolderOccupiedSpaceFirstStrategy: On refresh failure, keep the previous cached occupied space instead of setting Long.MAX_VALUE.
  3. FolderManager: Only switch to read-only when no healthy folder has available disk space (hasSpace()), not merely when folder selection fails.

Tests

  • JVMCommonUtilsTest.getOccupiedSpaceIgnoresConcurrentlyDeletedEntries
  • MinFolderOccupiedSpaceFirstStrategyRealFsTest.getNextFolderDoesNotEnterReadOnlyWhenDiskHasSpace

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.commons.utils.JVMCommonUtils
  • org.apache.iotdb.commons.disk.strategy.MinFolderOccupiedSpaceFirstStrategy
  • org.apache.iotdb.commons.disk.FolderManager
  • org.apache.iotdb.commons.i18n.UtilMessages

…letion

When receiving snapshot fragments, getOccupiedSpace walks the shared snapshot root while other regions may delete their staging subdirectories concurrently. NoSuchFileException was misinterpreted as folder full, triggering node read-only and failed region migration during datanode removal.
@jt2594838 jt2594838 merged commit 9eb39be into apache:master Jul 6, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants