Skip to content

Fix pipe history LoadTsFile after deleting aligned measurements#18116

Merged
jt2594838 merged 1 commit into
masterfrom
fix-pipe-history-load-tsfile-deleted-measurements
Jul 6, 2026
Merged

Fix pipe history LoadTsFile after deleting aligned measurements#18116
jt2594838 merged 1 commit into
masterfrom
fix-pipe-history-load-tsfile-deleted-measurements

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Summary

  • Skip inactive aligned measurements when flushing AlignedWritableMemChunk to TsFile.
  • Keep active schema columns mapped to their underlying TVList column indexes during both aligned flush paths.
  • Add a minimal Pipe history IT for receiver enable_auto_create_schema=false with unflushed aligned delete timeseries before after-batch history sync.

Root cause

delete timeseries on an aligned measurement removes it from the active measurement map and marks its TVList column deleted, but the flush writer still used the full historical schemaList. A history Pipe created before the memtable was flushed could therefore produce a TsFile whose metadata still contained the deleted measurement, without a corresponding mod file. The receiver then tried to verify/create a schema that should no longer exist, and LoadTsFile failed or retried forever when auto-create schema was disabled.

Tests

  • mvn spotless:check -pl iotdb-core/datanode,integration-test -P with-integration-tests
  • git diff --check
  • mvn -pl integration-test -am -P with-integration-tests -DskipUTs -Dit.test=IoTDBPipeReceiverAutoCreateDisabledIT#testAutoSplitHistoryTsFileWithUnflushedAlignedDeletionWhenReceiverAutoCreateSchemaDisabled -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false verify (blocked before the IT by existing upstream local compile failure in iotdb-thrift: generated thrift sources import missing javax.annotation on JDK 17)

@Caideyipi Caideyipi force-pushed the fix-pipe-history-load-tsfile-deleted-measurements branch from 271906d to 57e4d5f Compare July 6, 2026 07:11
@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 42.10526% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.84%. Comparing base (8940ba0) to head (959f153).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
...e/dataregion/memtable/AlignedWritableMemChunk.java 42.10% 22 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             master   #18116   +/-   ##
=========================================
  Coverage     41.84%   41.84%           
  Complexity      318      318           
=========================================
  Files          5296     5296           
  Lines        373593   373633   +40     
  Branches      48234    48243    +9     
=========================================
+ Hits         156313   156346   +33     
- Misses       217280   217287    +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi Caideyipi force-pushed the fix-pipe-history-load-tsfile-deleted-measurements branch from 57e4d5f to 959f153 Compare July 6, 2026 08:23
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Follow-up for the table-model CI failures:

The failed IoTDBWindowFunction2IT / IoTDBWindowTVF2IT cases expose a table-model edge case: a table device can have rows whose value columns are all null, but the device/id row still needs to be persisted in TsFile metadata. My previous early return skipped encoding when the active aligned value schema list was empty, which is valid for tree model (ignoreAllNullRows=true) but invalid for table model (ignoreAllNullRows=false).

I updated the guard to return only when activeSchemaList.isEmpty() && ignoreAllNullRows, so tree model still drops all-null/deleted aligned devices while table model still writes the time/device metadata for all-null value rows.

Local validation:

  • mvn spotless:check -pl iotdb-core/datanode,integration-test -P with-integration-tests
  • git diff --check

I also tried to run the affected ITs locally, but the local build is still blocked before tests by the existing iotdb-thrift generated-source compile issue: missing javax.annotation under JDK 17.

@sonarqubecloud

sonarqubecloud Bot commented Jul 6, 2026

Copy link
Copy Markdown

@jt2594838 jt2594838 merged commit eec9637 into master Jul 6, 2026
42 of 44 checks passed
@jt2594838 jt2594838 deleted the fix-pipe-history-load-tsfile-deleted-measurements branch July 6, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants