Skip to content

Optimize relational tablet batch inserts#18132

Open
jt2594838 wants to merge 9 commits into
apache:masterfrom
Caideyipi:table_insert_tablets
Open

Optimize relational tablet batch inserts#18132
jt2594838 wants to merge 9 commits into
apache:masterfrom
Caideyipi:table_insert_tablets

Conversation

@jt2594838

@jt2594838 jt2594838 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Description

This PR optimizes relational insertTablets handling in the Java Session client.

  • Add a merge switch for relational tablets in Session builders and pools.
  • Reuse column index maps when checking and merging relational tablets.
  • Avoid unnecessary bitmap creation during merge.
  • Copy tablet data column-by-column with array copy.
  • Auto-disable merge after consecutive misses or when merge overhead is too high.
  • Add an ignored integration test for manually comparing insertTablet and insertTablets performance as tablet count increases on the same table.

Tests

  • mvn test -pl iotdb-client/session -am "-Dtest=SessionTest#testMergeRelationalTabletsWithHighDuplicatedColumns+testMergeRelationalTabletsSkipLowDuplicatedColumnsAndDifferentTables+testMergeRelationalTabletsStopAfterConsecutiveMisses+testMergeRelationalTabletsResetConsecutiveMissesAfterMerge+testMergeTabletsDisabledWhenMergeCostExceedsHalfOfInsertCost+testMergeTabletsKeepsEnabledWhenMergeCostIsNotTooHigh+testEnableMergeTabletsUsingBuilder" "-DfailIfNoTests=false" "-Dsurefire.failIfNoSpecifiedTests=false"
  • mvn test-compile -pl integration-test -am -P with-integration-tests -DskipTests "-Drat.skip=true"
  • mvn verify "-DskipUTs" "-Dit.test=IoTDBSessionRelationalIT#compareInsertTabletAndInsertTabletsPerformanceWithIncreasingTabletCount" "-DfailIfNoTests=false" "-Dfailsafe.failIfNoSpecifiedTests=false" "-Drat.skip=true" -pl integration-test -am -PTableSimpleIT -P with-integration-tests

@Caideyipi

Copy link
Copy Markdown
Collaborator

I found a few issues that should be addressed before merging:

  1. High: Relational insertTablets cannot auto-create a new table.

In InsertTablets.validateTableSchema, the call to validateTableSchema passes allowCreateTable=false. This makes ITableSession.insertTablets(List) fail when inserting into a new table, while the existing single-tablet insert path can auto-create the table. The current integration tests all create the tables first, so this behavior difference is not covered. Please either align this with the single-tablet path or clearly document and test that batch tablet insertion requires the table to exist.

  1. Medium: The new insertRelationalTablets path bypasses table-model redirection.

Session.insertRelationalTablets always sends the request through getDefaultSessionConnection().insertTablets(request) and ignores RedirectException. The existing insertRelationalTablet path uses the table-model leader cache and groups rows/tablets by endpoint when redirection is enabled. In a cluster, this new path can keep sending requests to the default node and will not refresh tableModelDeviceIdToEndpoint, which may offset the intended performance gain. Please consider reusing or extending the existing relational tablet redirection logic for the batch path.

  1. Low: Merge cost is recorded even when the insert fails.

recordMergeTabletsCost is called from finally, so failed RPCs or execution failures still contribute to the auto-disable decision. This can permanently disable merge due to failure-path latency rather than actual merge overhead. It would be safer to record the cost only after a successful insert, or explicitly exclude failed attempts from the heuristic.

Question: Is insertTablets intentionally not supposed to support auto table creation? If yes, the API documentation and tests should make that clear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants