fix(connector_sdk) : Adding IBM db2 log based replication example by fivetran-JenasVimal · Pull Request #546 · fivetran/fivetran_connector_sdk

fivetran-JenasVimal · 2026-03-23T15:22:23Z

Jira ticket

Description of Change

Adding a new ibm db2 log based replication example
IBM Db2 Log-Based Replication Connector

This connector syncs data from IBM Db2 to your destination using log-based replication — instead of repeatedly querying the source table, it watches Db2's transaction log for changes and
syncs only what changed.

How it works

First sync: Reads all rows from the source table and loads them into the destination.
Every sync after that: Picks up only the rows that were inserted, updated, or deleted since the last sync — no full scans.
Progress is saved every 500 rows, so if a sync is interrupted, it resumes where it left off instead of starting over.

Testing

`Fivetran debug`

incremental sync : Leo was added

Duckdb warehouse

Checklist

Some tips and links to help validate your PR:

Tested the connector with fivetran debug command.
Added/Updated example-specific README.md file, see the README template for the required structure and guidelines.
Followed Python Coding Standards, refer here

github-actions · 2026-03-23T15:22:45Z

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.

Copilot

Pull request overview

Adds a new Connector SDK example demonstrating IBM Db2 log-based replication (CDC) using the ASN Capture / Change Data (CD) table approach.

Changes:

Introduces a new ibm_db2_log_based_replication connector implementation that performs an initial load and then applies changes from ASN.IBMSNAP_EMPCD.
Adds example documentation (README.md) describing the setup and how the CDC pipeline works.
Adds requirements.txt and configuration.json for the example connector.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 17 comments.

File	Description
connectors/ibm_db2_log_based_replication/connector.py	New connector implementation for initial load + CD-table-driven incremental sync
connectors/ibm_db2_log_based_replication/configuration.json	Example configuration added for running the connector
connectors/ibm_db2_log_based_replication/requirements.txt	Adds `ibm_db` dependency pin
connectors/ibm_db2_log_based_replication/README.md	Documentation for setup, configuration, and behavior

Copilot · 2026-03-23T15:27:35Z

+# ASN schema and CD table name as created by setup_cdc.sh
+ASN_SCHEMA = "ASN"
+CD_TABLE = "IBMSNAP_EMPCD"
+
+# How many CD rows to process before writing an intermediate checkpoint
+CHECKPOINT_INTERVAL = 500


Module constants don’t follow the repo’s convention of private, double-underscore, upper snake case for connector constants (e.g., __CHECKPOINT_INTERVAL). Rename these constants accordingly to align with the Connector SDK Python guidelines used in this repo.

Copilot · 2026-03-23T15:27:36Z

+        # Save the final state so the next sync knows where to continue from.
+        # Learn more about checkpointing:
+        # https://fivetran.com/docs/connectors/connector-sdk/best-practices#largedatasetrecommendation


This op.checkpoint() call is also missing the required standard checkpoint comment block immediately above it (the repo expects the full checkpoint explanation before every checkpoint operation).

Suggested change

# Save the final state so the next sync knows where to continue from.

# Learn more about checkpointing:

# https://fivetran.com/docs/connectors/connector-sdk/best-practices#largedatasetrecommendation

# Save the progress by checkpointing the state. This is important for ensuring that the sync process can resume

# from the correct position in case of next sync or interruptions.

# You should checkpoint even if you are not using incremental sync, as it tells Fivetran it is safe to write to destination.

# For large datasets, checkpoint regularly (e.g., every N records) not only at the end.

# Learn more about how and where to checkpoint by reading our best practices documentation

# (https://fivetran.com/docs/connector-sdk/best-practices#optimizingperformancewhenhandlinglargedatasets).

Copilot · 2026-03-23T15:27:37Z

+
+        if last_log_marker is None:
+            # ── First sync: full initial load ──────────────────────────────────
+            # Reads directly from the source EMPLOYEE table once to populate the
+            # destination. After this, all changes come from the ASN CD table.
+            log.info("No previous state found. Starting initial full load.")


If the connector crashes during perform_initial_load(), the intermediate checkpoints set initial_load_complete=False, but update() only checks for last_log_marker and will skip the initial load on the next run. This can leave the destination partially loaded forever. Use initial_load_complete to decide whether to (re)run/continue the initial load, or store a resumable cursor for the initial scan (e.g., last processed PK) and only set initial_load_complete=True after the scan finishes successfully.

Suggested change

if last_log_marker is None:

# ── First sync: full initial load ──────────────────────────────────

# Reads directly from the source EMPLOYEE table once to populate the

# destination. After this, all changes come from the ASN CD table.

log.info("No previous state found. Starting initial full load.")

initial_load_complete = state.get("initial_load_complete")

if last_log_marker is None or not initial_load_complete:

# ── First sync or incomplete initial load: full initial load ────────

# Reads directly from the source EMPLOYEE table once to populate the

# destination. After this, all changes come from the ASN CD table.

if last_log_marker is None:

log.info("No previous state found. Starting initial full load.")

else:

log.warning(

"Previous state indicates the initial load did not complete successfully. "

"Restarting initial full load."

)

Copilot · 2026-03-23T15:27:39Z

+## Getting started
+
+### 1. Start the Db2 Docker container
+
+```bash
+docker-compose up -d
+```


The ## Getting started section should include the standard Setup Guide sentence from the README template, and headings should not include numbers (e.g., ### 1. ...). Also, the README is missing the required ## Features section from the example README structure.

Copilot · 2026-03-23T15:27:39Z

+    """
+    Define the schema function which lets you configure the schema your connector delivers.
+    See the technical reference documentation for more details on the schema function:
+    https://fivetran.com/docs/connectors/connector-sdk/technical-reference#schema


The schema() docstring doesn’t match the required template (notably the documentation link path). In this repo, the schema docstring is expected to match the template connector’s wording/link exactly for consistency.

Suggested change

https://fivetran.com/docs/connectors/connector-sdk/technical-reference#schema

https://fivetran.com/docs/connector-sdk/technical-reference/connector-sdk-code/connector-sdk-methods#schema

…d CDC connector Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… naming conventions - Rename abbreviated variables: conn→connection, stmt→statement, sql→query, row→database_row, conn_str→connection_string, current_seq→current_commit_sequence - Rename connect_to_db→connect_to_database, standardize_row→normalize_row, get_current_log_marker→get_current_commit_sequence - Replace LOGMARKER cursor with IBMSNAP_COMMITSEQ hex cursor for correctness - Fix CD table reference: ASN.IBMSNAP_EMPCD→DB2INST1.CDEMPLOYEE - State key renamed: last_log_marker→last_commit_sequence - Add required checkpoint comment block before every op.checkpoint() call - Add required upsert comment block before every op.upsert() call - Add template-compliant module docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ow also checks the datatype of the conifgurations

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

fivetran-chinmayichandrasekar

Left a couple of suggestions

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.

fivetran-chinmayichandrasekar

LGTM

Lets hope

704f3d3

Copilot AI review requested due to automatic review settings March 23, 2026 15:22

fivetran-JenasVimal requested review from a team as code owners March 23, 2026 15:22

github-actions Bot added size/L PR size: Large ai-assisted/unknown risk/unknown labels Mar 23, 2026

fivetran-JenasVimal marked this pull request as draft March 23, 2026 15:22

Copilot started reviewing on behalf of fivetran-JenasVimal March 23, 2026 15:22 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

fivetran-JenasVimal added risk/low ai-assisted/no ai-assisted/yes and removed risk/unknown ai-assisted/unknown ai-assisted/no labels Mar 23, 2026

fivetran-JenasVimal self-assigned this Mar 23, 2026

fivetran-JenasVimal marked this pull request as ready for review March 23, 2026 18:54

github-actions Bot added size/XL PR size: extra large and removed size/L PR size: Large labels Mar 23, 2026

fivetran-JenasVimal force-pushed the ibm_db2_log_based_replication branch from 36e7a8e to 704f3d3 Compare March 23, 2026 18:59

github-actions Bot added size/L PR size: Large and removed size/XL PR size: extra large labels Mar 23, 2026

fivetran-JenasVimal marked this pull request as draft March 23, 2026 18:59

fivetran-JenasVimal and others added 3 commits March 24, 2026 00:30

feature(ibm_db2_log_based_replication): Add README entry for log-base…

2a17318

…d CDC connector Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(ibm_db2_log_based_replication): Remove unused datetime imports

4d23c1d

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions Bot added size/XL PR size: extra large and removed size/L PR size: Large labels Mar 24, 2026

fivetran-JenasVimal added 3 commits March 24, 2026 13:33

modified configuration.json

8ed6dd1

modified configuration.json and the validate_configuration function n…

1742d09

…ow also checks the datatype of the conifgurations

made changes to README.md and configuration.json

582d855

github-actions Bot added size/L PR size: Large and removed size/XL PR size: extra large labels Mar 24, 2026

fivetran-JenasVimal marked this pull request as ready for review March 24, 2026 08:37

fivetran-JenasVimal requested a review from Copilot March 24, 2026 16:48

Copilot started reviewing on behalf of fivetran-JenasVimal March 24, 2026 16:49 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

fivetran-JenasVimal requested review from fivetran-chinmayichandrasekar and fivetran-sahilkhirwal March 25, 2026 07:14

fivetran-chinmayichandrasekar reviewed Mar 25, 2026

View reviewed changes

Comment thread connectors/ibm_db2_log_based_replication/README.md Outdated

Comment thread connectors/ibm_db2_log_based_replication/README.md Outdated

fivetran-JenasVimal and others added 2 commits March 25, 2026 20:52

Made changes to README.md acc to tech writter comments

2a758f8

Update connectors/ibm_db2_log_based_replication/README.md

ef062aa

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fivetran-JenasVimal requested a review from fivetran-chinmayichandrasekar March 25, 2026 15:38

fivetran-sahilkhirwal requested a review from Copilot March 25, 2026 16:46

Copilot started reviewing on behalf of fivetran-sahilkhirwal March 25, 2026 16:47 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

The code works , and is now more robust and better

fa9a609

github-actions Bot added size/XL PR size: extra large and removed size/L PR size: Large labels Mar 26, 2026

fivetran-JenasVimal requested a review from Copilot March 26, 2026 10:54

Copilot started reviewing on behalf of fivetran-JenasVimal March 26, 2026 10:55 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

mde new changes

f13bbed

fivetran-chinmayichandrasekar approved these changes Mar 26, 2026

View reviewed changes

fivetran-rishabhghosh removed the request for review from a team April 13, 2026 18:26

fivetran-sahilkhirwal approved these changes Apr 14, 2026

View reviewed changes

fivetran-JenasVimal merged commit a7b8234 into main Apr 14, 2026
4 checks passed

fivetran-JenasVimal deleted the ibm_db2_log_based_replication branch April 14, 2026 10:06

-        # Save the final state so the next sync knows where to continue from.
-        # Learn more about checkpointing:
-        # https://fivetran.com/docs/connectors/connector-sdk/best-practices#largedatasetrecommendation
+        # Save the progress by checkpointing the state. This is important for ensuring that the sync process can resume
+        # from the correct position in case of next sync or interruptions.
+        # You should checkpoint even if you are not using incremental sync, as it tells Fivetran it is safe to write to destination.
+        # For large datasets, checkpoint regularly (e.g., every N records) not only at the end.
+        # Learn more about how and where to checkpoint by reading our best practices documentation
+        # (https://fivetran.com/docs/connector-sdk/best-practices#optimizingperformancewhenhandlinglargedatasets).

	https://fivetran.com/docs/connectors/connector-sdk/technical-reference#schema
	https://fivetran.com/docs/connector-sdk/technical-reference/connector-sdk-code/connector-sdk-methods#schema

Conversation

fivetran-JenasVimal commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira ticket

Description of Change

Testing

Fivetran debug

incremental sync : Leo was added

Duckdb warehouse

Checklist

Uh oh!

github-actions Bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧹 Python Code Quality Check

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fivetran-chinmayichandrasekar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fivetran-chinmayichandrasekar left a comment

Choose a reason for hiding this comment

Uh oh!

fivetran-JenasVimal commented Mar 23, 2026 •

edited

Loading

`Fivetran debug`

github-actions Bot commented Mar 23, 2026 •

edited

Loading