Pipe: Retry history LoadTsFile while waiting for schema#18031
Merged
jt2594838 merged 3 commits intoJun 26, 2026
Conversation
09eff22 to
7a56499
Compare
7a56499 to
fdc6b14
Compare
fdc6b14 to
dca8085
Compare
jt2594838
reviewed
Jun 25, 2026
Comment on lines
77
to
80
| || status.getCode() == TSStatusCode.LOAD_TEMPORARY_UNAVAILABLE_EXCEPTION.getStatusCode() | ||
| || status.getCode() == TSStatusCode.LOAD_FILE_ERROR.getStatusCode() | ||
| && status.getMessage() != null | ||
| && status.getMessage().contains("memory")) { |
Contributor
There was a problem hiding this comment.
Will the message one day be involved in i18n?
If so, the contains() call may become dangerous.
Collaborator
Author
There was a problem hiding this comment.
Updated in 2b62bd9: LoadTsFile schema-wait detection now uses a dedicated LoadAnalyzeMissingSchemaException and LOAD_TEMPORARY_UNAVAILABLE_EXCEPTION, and the pipe status visitor no longer classifies by message text such as contains(memory).
Comment on lines
+86
to
+89
| private static final String MISSING_SCHEMA_MESSAGE = | ||
| "does not exist in IoTDB and can not be created"; | ||
| private static final String AUTO_CREATE_SCHEMA_HINT_MESSAGE = | ||
| "Please check weather auto-create-schema is enabled"; |
Collaborator
Author
There was a problem hiding this comment.
Updated in 2b62bd9: moved the missing device/measurement schema messages and the pipe schema-wait message into DataNodeQueryMessages for en/zh i18n.
jt2594838
approved these changes
Jun 26, 2026
MileaRobertStefan
pushed a commit
to MileaRobertStefan/iotdb
that referenced
this pull request
Jun 26, 2026
* Fix pipe history LoadTsFile schema retry * Address load tsfile schema retry review * Fix load tsfile tests with real temp files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a full pipe is auto-split, historical data TsFiles can be delivered by the history/data side before the schema metadata side finishes replaying the current schema. If the receiver has
enable_auto_create_schema=false, LoadTsFile schema verification may see a missing device or measurement and fail the history TsFile withLOAD_FILE_ERROR.That failure is too strong for Pipe: in
inclusion=all, the missing schema can still arrive from the schema stream. Treating the file as a permanent load failure can drop the history TsFile and produce partial data loss on the receiver.Behavior after this change
This patch does not bypass the receiver's
enable_auto_create_schema=falsesetting and does not force Pipe-generated LoadTsFile to auto-create database or timeseries schema from the TsFile.Instead, for Pipe-generated LoadTsFile only, the receiver now classifies the failure as temporary when all of these are true:
In that case LoadTsFile returns
LOAD_TEMPORARY_UNAVAILABLE_EXCEPTION. The Pipe statement status visitor maps it toPIPE_RECEIVER_TEMPORARY_UNAVAILABLE_EXCEPTION, so the sender keeps retrying instead of treating the history TsFile as a permanent failed file.Other LoadTsFile failures, such as corrupted files, type conflicts, non-Pipe loads, and errors without the missing-schema signal, still follow the existing permanent failure handling.
Why this is better for Pipe
This preserves the semantic contract of
inclusion=all: schema should be transferred by the metadata stream, even if it arrives after a historical data TsFile. The data side waits for that schema instead of creating receiver schema from historical TsFile metadata.It also avoids resurrecting schema that was deleted before pipe creation. Historical TsFiles may still contain old metadata, while the schema snapshot represents the current source schema. Retrying until the schema side catches up is safer than overriding the receiver's auto-create setting and potentially recreating deleted timeseries.
For async active LoadTsFile, this also works with the existing active-load retry policy because
ActiveLoadFailedMessageHandleralready retriesLOAD_TEMPORARY_UNAVAILABLE_EXCEPTION.Tests
LOAD_TEMPORARY_UNAVAILABLE_EXCEPTIONto Pipe receiver temporary unavailable status.Local checks run:
mvn -Ddevelocity.off=true -pl iotdb-core/datanode spotless:checkmvn spotless:apply -pl integration-test -P with-integration-testsgit diff --checkTargeted unit test command was attempted, but this local workspace currently fails during unrelated datanode main compilation before tests run, with missing generated/parser symbols and unrelated Pipe constants.