Add mutual SSL support for pipe sinks by Caideyipi · Pull Request #18080 · apache/iotdb

Caideyipi · 2026-07-01T06:25:43Z

Summary

This PR extends the pipe thrift SSL sink path to support mutual SSL authentication, following the general thrift client mTLS support added in #18026.

Changes

Add pipe sink/connector SSL key-store parameters:
- sink.ssl.key-store-path
- sink.ssl.key-store-pwd
- connector.ssl.key-store-path
- connector.ssl.key-store-pwd
Let iotdb-thrift-ssl-sink / iotdb-thrift-ssl-connector read both sink and connector SSL aliases for:
- SSL enable flag
- trust store path/password
- key store path/password
Pass the optional key store path/password through:
- IoTDBSslSyncSink
- IoTDBSyncClientManager
- IoTDBSyncClient
- DataNode sync sink/client manager
- ConfigNode sync sink/client manager
Preserve existing one-way SSL behavior: key-store parameters are optional, but if either key-store path or password is specified, both must be specified.
Keep async thrift sink behavior explicit by rejecting all SSL attributes there, including the new key-store attributes.
Extend legacy pipe sink to accept connector/sink SSL aliases and to pass key-store settings into both its thrift client and SessionPool.
Hide ssl.key-store-pwd in pipe parameter value masking.

Tests

Added UT coverage for:
- ssl.key-store-pwd value hiding.
- DataRegion sync SSL sink accepting mutual SSL parameters.
- DataRegion sync SSL connector accepting connector-prefixed SSL aliases.
- DataRegion sync SSL sink rejecting incomplete key-store parameters.
- Async thrift sink rejecting SSL key-store parameters.
- Legacy pipe sink accepting connector-prefixed mutual SSL aliases.
- ConfigRegion sink accepting mutual SSL parameters.
- ConfigRegion sink rejecting incomplete key-store parameters.
Added IT coverage:
- IoTDBPipeMutualSSLIT starts a receiver with thrift SSL client auth enabled, creates an iotdb-thrift-ssl-sink pipe with trust/key stores, and verifies pipe data transfer through a mutual-SSL receiver.

Local Verification

mvn -Ddevelocity.off=true spotless:apply -pl iotdb-api/pipe-api,iotdb-core/datanode,iotdb-core/confignode,integration-test -P with-integration-tests
git diff --check

I also attempted targeted UT execution with mvn -Ddevelocity.off=true test -pl iotdb-api/pipe-api -Dtest=PipeParametersTest ..., but the local Windows environment failed to start/continue the JVM due to native memory/pagefile exhaustion (There is insufficient memory for the Java Runtime Environment to continue). The same environment issue also affected broader compile/test attempts.

Copilot

Pull request overview

This PR extends Apache IoTDB Pipe Thrift SSL sink/connector configuration to support mutual TLS (mTLS) by adding optional client key-store parameters, propagating them through the sync sink/client stack (DataNode + ConfigNode), and adding UT/IT coverage to verify acceptance/rejection behavior and end-to-end mTLS transfer.

Changes:

Add connector.ssl.* aliases (enable/trust-store/key-store) and new *.ssl.key-store-* parameters, and wire them into sync thrift pipe sinks/connectors and legacy pipe sink.
Propagate optional key-store path/password through IoTDBSslSyncSink -> IoTDBSyncClientManager -> IoTDBSyncClient and node-specific managers/sinks.
Add UTs for parameter validation/masking and an IT (IoTDBPipeMutualSSLIT) that validates mutual-SSL pipe transfer.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/sink/protocol/IoTDBSslSyncSink.java	Adds connector/sink SSL alias handling + passes key-store settings into sync client construction.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/sink/client/IoTDBSyncClientManager.java	Stores and forwards key-store settings when creating sync clients.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/sink/client/IoTDBSyncClient.java	Adds constructor overload + forwards key-store to SSL transport creation.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/config/constant/PipeSinkConstant.java	Introduces `connector.ssl.*` constants and new key-store constants.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/pipe/sink/PipeSinkTest.java	Adds UTs for accepting/rejecting mTLS-related pipe parameters (sync/async/legacy).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/thrift/sync/IoTDBDataNodeSyncSink.java	Propagates key-store settings with DataNode home-dir path resolution.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/thrift/async/IoTDBDataRegionAsyncSink.java	Explicitly rejects any SSL attributes (including new key-store ones) for async thrift sink.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/legacy/IoTDBLegacyPipeSink.java	Adds connector/sink alias support and passes key-store into both thrift client and SessionPool.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/client/IoTDBDataNodeSyncClientManager.java	Threads key-store parameters through DataNode sync client manager constructor.
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/manager/pipe/sink/IoTDBConfigRegionSinkTest.java	Adds UTs for ConfigNode sink accepting/rejecting mTLS parameters.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/sink/protocol/IoTDBConfigRegionSink.java	Propagates key-store settings with ConfigNode home-dir path resolution.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/sink/client/IoTDBConfigNodeSyncClientManager.java	Threads key-store parameters through ConfigNode sync client manager constructor.
iotdb-client/jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBConnection.java	Uses parsed connection params for username (minor robustness improvement).
iotdb-api/pipe-api/src/test/java/org/apache/iotdb/pipe/api/customizer/parameter/PipeParametersTest.java	Adds UT ensuring `ssl.key-store-pwd` is masked by `ValueHider`.
iotdb-api/pipe-api/src/main/java/org/apache/iotdb/pipe/api/customizer/parameter/PipeParameters.java	Adds `ssl.key-store-pwd` to masked keys set.
integration-test/src/test/java/org/apache/iotdb/pipe/it/dual/treemodel/auto/basic/IoTDBPipeMutualSSLIT.java	New IT covering end-to-end pipe transfer through a receiver requiring Thrift mTLS client auth.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Caideyipi · 2026-07-03T04:25:58Z

+    validator
+        .validate(
+            args -> !((boolean) args[0]) || ((boolean) args[1] && (boolean) args[2]),
+            String.format(
+                "When ssl transport is enabled, %s and %s must be specified",
+                SINK_IOTDB_SSL_TRUST_STORE_PATH_KEY, SINK_IOTDB_SSL_TRUST_STORE_PWD_KEY),
+            IOTDB_THRIFT_SSL_CONNECTOR.getPipePluginName().equals(userSpecifiedConnectorName)
+                || IOTDB_THRIFT_SSL_SINK.getPipePluginName().equals(userSpecifiedConnectorName)
+                || parameters.getBooleanOrDefault(
+                    Arrays.asList(CONNECTOR_IOTDB_SSL_ENABLE_KEY, SINK_IOTDB_SSL_ENABLE_KEY),
+                    false),
+            parameters.hasAnyAttributes(
+                CONNECTOR_IOTDB_SSL_TRUST_STORE_PATH_KEY, SINK_IOTDB_SSL_TRUST_STORE_PATH_KEY),
+            parameters.hasAnyAttributes(
+                CONNECTOR_IOTDB_SSL_TRUST_STORE_PWD_KEY, SINK_IOTDB_SSL_TRUST_STORE_PWD_KEY))
+        .validate(
+            args -> (boolean) args[0] == (boolean) args[1],
+            String.format(
+                "%s and %s must be specified together",
+                SINK_IOTDB_SSL_KEY_STORE_PATH_KEY, SINK_IOTDB_SSL_KEY_STORE_PWD_KEY),
+            parameters.hasAnyAttributes(
+                CONNECTOR_IOTDB_SSL_KEY_STORE_PATH_KEY, SINK_IOTDB_SSL_KEY_STORE_PATH_KEY),
+            parameters.hasAnyAttributes(
+                CONNECTOR_IOTDB_SSL_KEY_STORE_PWD_KEY, SINK_IOTDB_SSL_KEY_STORE_PWD_KEY));


Applied in 4127178: validation now requires trust-store path/password to be complete within the same connector., sink., or unprefixed alias, and rejects half key-store pairs per alias. Added UT coverage for cross-alias trust/key-store rejection.

Caideyipi · 2026-07-03T04:26:13Z

+            parameters.getBooleanOrDefault(
+                Arrays.asList(CONNECTOR_IOTDB_SSL_ENABLE_KEY, SINK_IOTDB_SSL_ENABLE_KEY), false),
+            parameters.hasAnyAttributes(
+                CONNECTOR_IOTDB_SSL_TRUST_STORE_PATH_KEY, SINK_IOTDB_SSL_TRUST_STORE_PATH_KEY),
+            parameters.hasAnyAttributes(
+                CONNECTOR_IOTDB_SSL_TRUST_STORE_PWD_KEY, SINK_IOTDB_SSL_TRUST_STORE_PWD_KEY))


Applied in 4127178 as well: the legacy sink now uses the same per-alias trust-store/key-store pair validation, and I added UT coverage for the cross-alias SSL case.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Caideyipi · 2026-07-03T06:18:24Z

+            String.format(
+                "When ssl transport is enabled, %s and %s must be specified",
+                SINK_IOTDB_SSL_TRUST_STORE_PATH_KEY, SINK_IOTDB_SSL_TRUST_STORE_PWD_KEY),


Applied in ca064a0: the trust-store validation error now mentions connector., sink., and unprefixed aliases.

Caideyipi · 2026-07-03T06:18:25Z

+            String.format(
+                "%s and %s must be specified together",
+                SINK_IOTDB_SSL_KEY_STORE_PATH_KEY, SINK_IOTDB_SSL_KEY_STORE_PWD_KEY),


Applied in ca064a0: the key-store pairing error now lists the accepted connector., sink., and unprefixed aliases.

Caideyipi · 2026-07-03T06:18:25Z

@@ -151,9 +160,74 @@
                SINK_IOTDB_SSL_ENABLE_KEY,
                SINK_IOTDB_SSL_TRUST_STORE_PATH_KEY,
                SINK_IOTDB_SSL_TRUST_STORE_PWD_KEY),


Applied in ca064a0: the legacy sink SSL-enable error now reflects connector.ssl.enable / sink.ssl.enable and the accepted trust-store aliases.

Caideyipi · 2026-07-03T06:18:25Z

+            String.format(
+                "%s and %s must be specified together",
+                SINK_IOTDB_SSL_KEY_STORE_PATH_KEY, SINK_IOTDB_SSL_KEY_STORE_PWD_KEY),
+            true,


Applied in ca064a0: the legacy key-store pairing error now mentions connector., sink., and unprefixed aliases.

Caideyipi added 3 commits July 1, 2026 14:20

Add mutual SSL support for pipe sinks

3819d71

Fix JDBC default user handling

85fad38

Update IoTDBPipeMutualSSLIT.java

eb63781

HTHou requested a review from Copilot July 2, 2026 02:21

Copilot started reviewing on behalf of HTHou July 2, 2026 02:22 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Validate SSL store aliases consistently

4127178

HTHou requested a review from Copilot July 3, 2026 04:37

Copilot started reviewing on behalf of HTHou July 3, 2026 04:38 View session

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Clarify SSL alias validation messages

ca064a0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add mutual SSL support for pipe sinks#18080

Add mutual SSL support for pipe sinks#18080
Caideyipi wants to merge 5 commits into
apache:masterfrom
Caideyipi:codex/pipe-mutual-ssl

Caideyipi commented Jul 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Caideyipi Jul 3, 2026

Uh oh!

Caideyipi Jul 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Caideyipi Jul 3, 2026

Uh oh!

Caideyipi Jul 3, 2026

Uh oh!

Caideyipi Jul 3, 2026

Uh oh!

Caideyipi Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Caideyipi commented Jul 1, 2026

Summary

Changes

Tests

Local Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Caideyipi Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Caideyipi Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Caideyipi Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Caideyipi Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Caideyipi Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Caideyipi Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants