From a6ce7d376710957d37e50dd022b5f710a5fbb63f Mon Sep 17 00:00:00 2001 From: Leto_b Date: Tue, 20 Jan 2026 11:45:26 +0800 Subject: [PATCH] supplement param mods same as mods.enable to data sync --- .../Table/User-Manual/Data-Sync_apache.md | 11 ++++--- .../Table/User-Manual/Data-Sync_timecho.md | 13 ++++---- .../Tree/User-Manual/Data-Sync_apache.md | 22 ++++++------- .../Tree/User-Manual/Data-Sync_timecho.md | 26 +++++++-------- .../User-Manual/Data-Sync_apache.md | 1 + .../User-Manual/Data-Sync_timecho.md | 1 + .../latest/User-Manual/Data-Sync_apache.md | 17 +++++----- .../latest/User-Manual/Data-Sync_timecho.md | 25 ++++++++------- .../Table/User-Manual/Data-Sync_apache.md | 27 ++++++++-------- .../Table/User-Manual/Data-Sync_timecho.md | 32 ++++++++++--------- .../Tree/User-Manual/Data-Sync_apache.md | 9 +++--- .../Tree/User-Manual/Data-Sync_timecho.md | 12 +++---- .../User-Manual/Data-Sync_apache.md | 25 ++++++++------- .../User-Manual/Data-Sync_timecho.md | 27 ++++++++-------- .../latest/User-Manual/Data-Sync_apache.md | 5 +-- .../latest/User-Manual/Data-Sync_timecho.md | 7 ++-- 16 files changed, 136 insertions(+), 124 deletions(-) diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index 8f3b77baf..ea7a19f63 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -349,7 +349,7 @@ This example demonstrates cascading data transmission from IoTDB A to IoTDB B an ![](/img/sync_en_04.png) - + SQL Example: On IoTDB A: ```SQL @@ -485,6 +485,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | | start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -499,7 +500,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 |:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | | sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | | node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | | password | Password for the username. | String | No | root | | batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | | batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | @@ -508,7 +509,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | | compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | | rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | | format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | #### iotdb-thrift-ssl-sink @@ -518,7 +519,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 |:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | | sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | | node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | | password | Password for the username. | String | No | root | | batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | | batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | @@ -527,7 +528,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | | compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | | rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | | ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | | ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | | format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | \ No newline at end of file diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index e3d98f7f1..73594769c 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -394,7 +394,7 @@ This example demonstrates cascading data transmission from IoTDB A to IoTDB B an ![](/img/sync_en_04.png) - + SQL Example: On IoTDB A: ```SQL @@ -570,6 +570,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -580,24 +581,22 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink - | **Parameter** | **Description** | Value Range | Required | Default Value | |:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| :------- | :------------ | | sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | | node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | | password | Password for the username. | String | No | root | | batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | | batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | | batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | | compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | | compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | | rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | | format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | -| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes). (Available since v2.0.5) | Boolean: true / false. True: can synchronize; False: cannot synchronize; | Optional | False | - +| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes). | Boolean: true / false. True: can synchronize; False: cannot synchronize; | Optional | False | #### iotdb-air-gap-sink diff --git a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md index 212c3be0a..0219d2a55 100644 --- a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md @@ -483,18 +483,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter -| key | value | value range | required or not | default value | -| :------------------------------ | :----------------------------------------------------------- | :------------------------------------- | :------- | :------------- | -| source | iotdb-source | String: iotdb-source | Required | - | +| key | value | value range | required or not | default value | +| :------------------------------ | :----------------------------------------------------------- |:-----------------------------------------------------------------------| :------- | :------------- | +| source | iotdb-source | String: iotdb-source | Required | - | | inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | | inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | 否 | true | +| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | Optional | true | | mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | | mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | -| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | -| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | -| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | -| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true | Optional | true | +| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | +| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | +| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | +| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true /false | Optional | true | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -514,7 +515,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | Password for the username. | String | No | root | | batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | | batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | | batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | | compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | | compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | @@ -524,7 +525,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-ssl-sink - | **Parameter** | **Description** | Value Range | Required | Default Value | |:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | | sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | @@ -533,7 +533,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | Password for the username. | String | No | root | | batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | | batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | | batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | | compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | | compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | diff --git a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md index ae0813daa..3d492ed28 100644 --- a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md @@ -569,18 +569,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter（V1.3.3） -| key | value | value range | required or not | default value | -| :------------------------------ | :----------------------------------------------------------- | :------------------------------------- | :------- | :------------- | -| source | iotdb-source | String: iotdb-source | Required | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | 否 | true | -| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | -| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | -| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | -| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | -| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | -| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true | Optional | true | +| key | value | value range | required or not | default value | +|:-------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------|:-----------------|:--------------| +| source | iotdb-source | String: iotdb-source | Required | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | +| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | Optional | true | +| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | +| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | +| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | +| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | +| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | +| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true / false | Optional | true | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -607,7 +608,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | | format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | - #### iotdb-air-gap-sink | key | value | value Range | required or not | Default Value | diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index 16a2ef9ac..ea7a19f63 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -485,6 +485,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | | start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index 8ad66fa0a..73594769c 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -570,6 +570,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > diff --git a/src/UserGuide/latest/User-Manual/Data-Sync_apache.md b/src/UserGuide/latest/User-Manual/Data-Sync_apache.md index 66f196575..39db1203d 100644 --- a/src/UserGuide/latest/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/latest/User-Manual/Data-Sync_apache.md @@ -483,18 +483,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter -| key | value | value range | required or not | default value | -| :------------------------------ | :----------------------------------------------------------- | :------------------------------------- | :------- | :------------- | -| source | iotdb-source | String: iotdb-source | Required | - | +| key | value | value range | required or not | default value | +| :------------------------------ | :----------------------------------------------------------- |:-----------------------------------------------------------------------| :------- | :------------- | +| source | iotdb-source | String: iotdb-source | Required | - | | inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | | inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | 否 | true | +| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | Optional | true | | mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | | mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | -| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | -| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | -| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | -| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true | Optional | true | +| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | +| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | +| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | +| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true /false | Optional | true | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > diff --git a/src/UserGuide/latest/User-Manual/Data-Sync_timecho.md b/src/UserGuide/latest/User-Manual/Data-Sync_timecho.md index 83282a7e1..5bc5852ca 100644 --- a/src/UserGuide/latest/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/latest/User-Manual/Data-Sync_timecho.md @@ -569,18 +569,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter（V1.3.3） -| key | value | value range | required or not | default value | -| :------------------------------ | :----------------------------------------------------------- | :------------------------------------- | :------- | :------------- | -| source | iotdb-source | String: iotdb-source | Required | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | 否 | true | -| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | -| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | -| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | -| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | -| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | -| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true | Optional | true | +| key | value | value range | required or not | default value | +|:-------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------|:-----------------|:--------------| +| source | iotdb-source | String: iotdb-source | Required | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | +| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false |Optional | true | +| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | +| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | +| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | +| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | +| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | +| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true / false | Optional | true | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index 068e946be..1a25595b9 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -464,18 +464,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### source 参数 -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | -| source | iotdb-source | String: iotdb-source | 必填 | - | -| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | data.insert | -| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | -| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | -| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | -| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | -| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | -| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +| ----------------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | +| source | iotdb-source | String: iotdb-source | 必填 | - | +| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | data.insert | +| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | +| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | +| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | +| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | +| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | +| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | +| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | +| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 @@ -493,7 +494,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | | batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | | batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms） (V2.0.5及以后版本支持） | Integer | 选填 | 1 | | batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | | compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | | compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index 865553a54..8d145a69a 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -392,6 +392,7 @@ with sink ( ![](/img/1706698610134.jpg) + 在 A IoTDB 上执行下列语句，将 A 中数据同步至 B： ```SQL @@ -550,19 +551,20 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### source 参数 -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | -| source | iotdb-source | String: iotdb-source | 必填 | - | -| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl),auth | 选填 | data.insert | -| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | -| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | -| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | -| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | -| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | -| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | -| mode.double-living | 是否开启全量双活模式，开启后将忽略`-sql_dialect`连接方式，树表模型数据均会被捕获，且不会转发由另一pipe同步而来的数据。 | Boolean: true / false | 否 | false | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | +| source | iotdb-source | String: iotdb-source | 必填 | - | +| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl),auth | 选填 | data.insert | +| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | +| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | +| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | +| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | +| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | +| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | +| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | +| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| mode.double-living | 是否开启全量双活模式，开启后将忽略`-sql_dialect`连接方式，树表模型数据均会被捕获，且不会转发由另一pipe同步而来的数据。 | Boolean: true / false | 否 | false | +| mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 @@ -581,7 +583,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | | batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | | batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | | batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | | compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | | compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | @@ -616,7 +618,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | | batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | | batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | | batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | | compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | | compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | diff --git a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md index 97100bb68..3263023ad 100644 --- a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md @@ -479,8 +479,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source 参数 -| 参数 | 描述 | value 取值范围 | 是否必填 | 默认取值 | -| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| -------- | -------------- | +| 参数 | 描述 | value 取值范围 | 是否必填 | 默认取值 | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| -------- | -------------- | | source | iotdb-source | String: iotdb-source | 必填 | - | | inclusion | 用于指定数据同步任务中需要同步范围，分为数据、元数据和权限 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | data.insert | | inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | 空字符串 | @@ -491,6 +491,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | start-time | 同步所有数据的开始 event time，包含 start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | 选填 | Long.MIN_VALUE | | end-time | 同步所有数据的结束 event time，包含 end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | 选填 | Long.MAX_VALUE | | forwarding-pipe-requests | 是否转发由其他 Pipe （通常是数据同步）写入的数据 | Boolean: true, false | 选填 | true | +| mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 @@ -509,7 +510,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | | batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | | batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | | batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | | compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | | compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | @@ -527,7 +528,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | | batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | | batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | | batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | | compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | | compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | diff --git a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md index cf6547635..ad7b7524b 100644 --- a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md @@ -564,18 +564,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source 参数 -| 参数 | 描述 | value 取值范围 | 是否必填 | 默认取值 | -| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| -------- | -------------- | +| 参数 | 描述 | value 取值范围 | 是否必填 | 默认取值 | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| -------- | -------------- | | source | iotdb-source | String: iotdb-source | 必填 | - | | inclusion | 用于指定数据同步任务中需要同步范围，分为数据、元数据和权限 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | data.insert | | inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | 空字符串 | | mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | | mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | | mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| path | 当用户连接指定的sql_dialect为tree时可以指定。对于升级上来的用户pipe,默认sql_dialect为tree。此参数决定时序数据的捕获范围,影响 inclusion中的data数据,以及部分序列相关的元数据。当数据的树模型路径能够被path匹配时，数据会被筛选出来进入流处理pipe。 | String:IoTDB标准的树路径模式,可以带通配符 | 选填 | root.** | +| path | 当用户连接指定的sql_dialect为tree时可以指定。对于升级上来的用户pipe,默认sql_dialect为tree。此参数决定时序数据的捕获范围,影响 inclusion中的data数据,以及部分序列相关的元数据。当数据的树模型路径能够被path匹配时，数据会被筛选出来进入流处理pipe。 | String:IoTDB标准的树路径模式,可以带通配符 | 选填 | root.** | | start-time | 同步所有数据的开始 event time，包含 start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | 选填 | Long.MIN_VALUE | | end-time | 同步所有数据的结束 event time，包含 end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | 选填 | Long.MAX_VALUE | | forwarding-pipe-requests | 是否转发由其他 Pipe （通常是数据同步）写入的数据 | Boolean: true, false | 选填 | true | +| mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 @@ -594,7 +595,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | | batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | | batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | | batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | | compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | | compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | @@ -602,7 +603,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | | format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | - #### iotdb-air-gap-sink | key | value | value 取值范围 | 是否必填 | 默认取值 | @@ -627,7 +627,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | | batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | | batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | | batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | | compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | | compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index be4ea57fd..1a25595b9 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -464,18 +464,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### source 参数 -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | -| source | iotdb-source | String: iotdb-source | 必填 | - | -| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | data.insert | -| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | -| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | -| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | -| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | -| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | -| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +| ----------------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | +| source | iotdb-source | String: iotdb-source | 必填 | - | +| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | data.insert | +| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | +| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | +| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | +| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | +| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | +| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | +| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | +| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index 9e444918e..8d145a69a 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -551,19 +551,20 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### source 参数 -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | -| source | iotdb-source | String: iotdb-source | 必填 | - | -| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl),auth | 选填 | data.insert | -| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | -| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | -| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | -| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | -| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | -| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | -| mode.double-living | 是否开启全量双活模式，开启后将忽略`-sql_dialect`连接方式，树表模型数据均会被捕获，且不会转发由另一pipe同步而来的数据。 | Boolean: true / false | 否 | false | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| ------------ | ------------------------------- | +| source | iotdb-source | String: iotdb-source | 必填 | - | +| inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl),auth | 选填 | data.insert | +| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | +| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | +| mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | +| mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | +| database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | +| table-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名，也可以是 Java 风格正则表达式来匹配多个表。默认情况下，匹配所有的表。 | String：数据表名或数据表正则模式串，可以是未创建的、不存在的表 | 否 | ".*" | +| start-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 大于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE， Long.MAX_VALUE] （unix 裸时间戳）或 String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE（unix 裸时间戳） | +| end-time | 此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。当数据的 event time 小于等于该参数时，数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]（unix 裸时间戳）或String：IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE（unix 裸时间戳） | +| mode.double-living | 是否开启全量双活模式，开启后将忽略`-sql_dialect`连接方式，树表模型数据均会被捕获，且不会转发由另一pipe同步而来的数据。 | Boolean: true / false | 否 | false | +| mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 diff --git a/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md index e029b66b7..3263023ad 100644 --- a/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md @@ -479,8 +479,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source 参数 -| 参数 | 描述 | value 取值范围 | 是否必填 | 默认取值 | -| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| -------- | -------------- | +| 参数 | 描述 | value 取值范围 | 是否必填 | 默认取值 | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| -------- | -------------- | | source | iotdb-source | String: iotdb-source | 必填 | - | | inclusion | 用于指定数据同步任务中需要同步范围，分为数据、元数据和权限 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | data.insert | | inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | 空字符串 | @@ -491,6 +491,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | start-time | 同步所有数据的开始 event time，包含 start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | 选填 | Long.MIN_VALUE | | end-time | 同步所有数据的结束 event time，包含 end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | 选填 | Long.MAX_VALUE | | forwarding-pipe-requests | 是否转发由其他 Pipe （通常是数据同步）写入的数据 | Boolean: true, false | 选填 | true | +| mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 diff --git a/src/zh/UserGuide/latest/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/latest/User-Manual/Data-Sync_timecho.md index c0a869aa5..ad7b7524b 100644 --- a/src/zh/UserGuide/latest/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/latest/User-Manual/Data-Sync_timecho.md @@ -564,18 +564,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source 参数 -| 参数 | 描述 | value 取值范围 | 是否必填 | 默认取值 | -| ------------------------ |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| -------- | -------------- | +| 参数 | 描述 | value 取值范围 | 是否必填 | 默认取值 | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| -------- | -------------- | | source | iotdb-source | String: iotdb-source | 必填 | - | | inclusion | 用于指定数据同步任务中需要同步范围，分为数据、元数据和权限 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | data.insert | | inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | 空字符串 | | mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | | mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | | mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | -| path | 当用户连接指定的sql_dialect为tree时可以指定。对于升级上来的用户pipe,默认sql_dialect为tree。此参数决定时序数据的捕获范围,影响 inclusion中的data数据,以及部分序列相关的元数据。当数据的树模型路径能够被path匹配时，数据会被筛选出来进入流处理pipe。 | String:IoTDB标准的树路径模式,可以带通配符 | 选填 | root.** | +| path | 当用户连接指定的sql_dialect为tree时可以指定。对于升级上来的用户pipe,默认sql_dialect为tree。此参数决定时序数据的捕获范围,影响 inclusion中的data数据,以及部分序列相关的元数据。当数据的树模型路径能够被path匹配时，数据会被筛选出来进入流处理pipe。 | String:IoTDB标准的树路径模式,可以带通配符 | 选填 | root.** | | start-time | 同步所有数据的开始 event time，包含 start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | 选填 | Long.MIN_VALUE | | end-time | 同步所有数据的结束 event time，包含 end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | 选填 | Long.MAX_VALUE | | forwarding-pipe-requests | 是否转发由其他 Pipe （通常是数据同步）写入的数据 | Boolean: true, false | 选填 | true | +| mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | > 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** > - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐