Skip to content

feat: implement filter in cdc and incremental#756

Merged
hash-data merged 77 commits intostagingfrom
feat/filter-in-cdc&incremental
Mar 23, 2026
Merged

feat: implement filter in cdc and incremental#756
hash-data merged 77 commits intostagingfrom
feat/filter-in-cdc&incremental

Conversation

@vikaxsh
Copy link
Copy Markdown
Collaborator

@vikaxsh vikaxsh commented Jan 14, 2026

Description

This PR adds filtering capability for CDC and incremental sync modes, allowing users to selectively sync records based on configurable filter conditions.

Fixes # (issue)

Type of change

  • Multiple Conditions: Support for multiple filter conditions with AND/OR logical operators
  • Operators: Supports =, !=, <, >, <=, >=
  • Type-Aware Filtering: Automatic type conversion and validation based on schema
  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Comprehensive test coverage in destination/filter_test.go including:

  • Legacy filter support

  • Multiple operators and data types

  • AND/OR logic

  • Null value handling

  • Type coercion

  • Edge cases

  • Scenario A

  • Scenario B

Screenshots or Recordings

Documentation

Related PR's (If Any):

New Filter Format

Filters can be specified in the stream configuration using the new FilterInput structure:

{
  "selected_streams": {
    "namespace": [
      {
        "stream_name": "table_1",
        "filter_input": {
          "logical_operator": "AND",
          "conditions": [
            {
              "column": "id",
              "operator": ">",
              "value": 100
            },
            {
              "column": "status",
              "operator": "=",
              "value": "active"
            }
          ]
        }
      }
    ]
  }
}

Legacy Filter Format (Still Supported)

The legacy string-based filter format continues to work:

{
  "filter": "id > 100 AND status = \"active\""
}

@vikaxsh vikaxsh force-pushed the feat/filter-in-cdc&incremental branch from b75a508 to ab0848f Compare January 20, 2026 10:26
@vikaxsh vikaxsh marked this pull request as ready for review January 21, 2026 08:21
@vikaxsh vikaxsh had a problem deploying to integration_tests March 12, 2026 11:28 — with GitHub Actions Failure
@vikaxsh vikaxsh temporarily deployed to integration_tests March 12, 2026 12:04 — with GitHub Actions Inactive
@vikaxsh vikaxsh temporarily deployed to integration_tests March 16, 2026 05:53 — with GitHub Actions Inactive
vishalm0509
vishalm0509 previously approved these changes Mar 16, 2026
@vikaxsh vikaxsh requested a deployment to integration_tests March 16, 2026 12:29 — with GitHub Actions Waiting
@vikaxsh vikaxsh requested a deployment to integration_tests March 16, 2026 12:48 — with GitHub Actions Waiting
@vikaxsh vikaxsh temporarily deployed to integration_tests March 16, 2026 13:13 — with GitHub Actions Inactive
@shubham19may shubham19may temporarily deployed to integration_tests March 18, 2026 17:12 — with GitHub Actions Inactive
@hash-data hash-data temporarily deployed to integration_tests March 20, 2026 12:08 — with GitHub Actions Inactive
@hash-data hash-data merged commit 8208b6b into staging Mar 23, 2026
10 checks passed
@hash-data hash-data deleted the feat/filter-in-cdc&incremental branch March 23, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants