Skip to content

Pipe: Implemented schema batch for meta transfer#17665

Merged
jt2594838 merged 9 commits into
masterfrom
batch
Jun 30, 2026
Merged

Pipe: Implemented schema batch for meta transfer#17665
jt2594838 merged 9 commits into
masterfrom
batch

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

As the title said.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
11.1% Duplication on New Code (required ≤ 5%)

See analysis details on SonarQube Cloud

@codecov

codecov Bot commented May 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 80 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.49%. Comparing base (f971de2) to head (bebe100).
⚠️ Report is 33 commits behind head on master.

Files with missing lines Patch % Lines
...nk/protocol/thrift/sync/IoTDBSchemaRegionSink.java 67.10% 25 Missing ⚠️
...ble/batch/PipeSchemaRegionWritePlanEventBatch.java 91.44% 23 Missing ⚠️
...k/protocol/airgap/IoTDBSchemaRegionAirGapSink.java 66.66% 23 Missing ⚠️
.../pipe/agent/task/subtask/sink/PipeSinkSubtask.java 0.00% 7 Missing ⚠️
...he/iotdb/commons/pipe/sink/protocol/IoTDBSink.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17665      +/-   ##
============================================
+ Coverage     41.21%   41.49%   +0.27%     
  Complexity      318      318              
============================================
  Files          5258     5287      +29     
  Lines        366348   370001    +3653     
  Branches      47389    47858     +469     
============================================
+ Hits         151004   153514    +2510     
- Misses       215344   216487    +1143     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

luoluoyuyu

This comment was marked as outdated.

@luoluoyuyu luoluoyuyu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Adds PipeSchemaRegionWritePlanEventBatch to batch schema-region write plan events before sink transfer, with histogram metrics for batch size and cost.

Please fix CI and address the metric setter naming inline before merge.

Comment on lines +91 to +101
return BytesUtils.concatByteArray(
byteArrayOutputStream.toByteArray(), planNode.serializeToByteBuffer().array());
byteArrayOutputStream.toByteArray(), getBytes(serializedPlanNode));
}
}

private static byte[] getBytes(final ByteBuffer byteBuffer) {
final ByteBuffer duplicateBuffer = byteBuffer.duplicate();
final byte[] bytes = new byte[duplicateBuffer.remaining()];
duplicateBuffer.get(bytes);
return bytes;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to avoid copying by extending BytesUtils.concatByteArra?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I avoided the extra intermediate copy on the IoTDB side by filling the final byte[] directly from the PublicBAOS backing buffer and the duplicated ByteBuffer's remaining bytes. Extending BytesUtils itself would require a tsfile-side API change, so I kept the change local to this PR.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
7.2% Duplication on New Code (required ≤ 5%)

See analysis details on SonarQube Cloud

@jt2594838 jt2594838 merged commit 837711d into master Jun 30, 2026
42 of 43 checks passed
@jt2594838 jt2594838 deleted the batch branch June 30, 2026 02:07
jt2594838 pushed a commit that referenced this pull request Jun 30, 2026
* Schema

* fix

* fix

* Update PipeSchemaRegionSinkMetricsTest.java

* spt

* Address schema batch review comments

* Avoid extra copy in plan node airgap transfer

* Fix pipe plan node byte buffer test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants