Skip to content

supplement param mods same as mods.enable to data sync #971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leto-b wants to merge 1 commit into
base: main
Choose a base branch
from
+136 −124
Open

supplement param mods same as mods.enable to data sync #971

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop

Large diffs are not rendered by default.

13 changes: 6 additions & 7 deletions src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
View file
Open in desktop

Large diffs are not rendered by default.

22 changes: 11 additions & 11 deletions src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md
View file
Open in desktop

Large diffs are not rendered by default.

26 changes: 13 additions & 13 deletions src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -485,6 +485,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" |
| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) |
| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) |
| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false |

> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming`
>
Expand Down
1 change: 1 addition & 0 deletions src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -570,6 +570,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) |
| end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) |
| mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false |
| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false |

> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming`
>
Expand Down
17 changes: 9 additions & 8 deletions src/UserGuide/latest/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,18 +483,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1

### 5.1 source parameter

| key | value | value range | required or not | default value |
| :------------------------------ | :----------------------------------------------------------- | :------------------------------------- | :------- | :------------- |
| source | iotdb-source | String: iotdb-source | Required | - |
| key | value | value range | required or not | default value |
| :------------------------------ | :----------------------------------------------------------- |:-----------------------------------------------------------------------| :------- | :------------- |
| source | iotdb-source | String: iotdb-source | Required | - |
| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert |
| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - |
| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | 否 | true |
| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | Optional | true |
| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true |
| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false |
| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** |
| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE |
| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE |
| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true | Optional | true |
| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** |
| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE |
| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE |
| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true /false | Optional | true |
| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false |

> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming`
>
Expand Down
Loading