Skip to content

feature(ehi): Adding high volume EHI example#540

Open
fivetran-sahilkhirwal wants to merge 11 commits intomainfrom
sahil/ehi_at_scale
Open

feature(ehi): Adding high volume EHI example#540
fivetran-sahilkhirwal wants to merge 11 commits intomainfrom
sahil/ehi_at_scale

Conversation

@fivetran-sahilkhirwal
Copy link
Copy Markdown
Contributor

@fivetran-sahilkhirwal fivetran-sahilkhirwal commented Mar 17, 2026

Jira ticket

Closes https://fivetran.atlassian.net/browse/RD-1172756

Description of Change

  • Adding high volume EHI example

Testing

Screenshot 2026-03-25 at 00 42 13 Screenshot 2026-03-25 at 00 52 25 Screenshot 2026-04-15 at 20 58 24

Checklist

Some tips and links to help validate your PR:

  • Tested the connector with fivetran debug command.
  • Added/Updated example-specific README.md file, see the README template for the required structure and guidelines.
  • Followed Python Coding Standards, refer here

@fivetran-sahilkhirwal fivetran-sahilkhirwal self-assigned this Mar 17, 2026
@fivetran-sahilkhirwal fivetran-sahilkhirwal requested a review from a team as a code owner March 17, 2026 07:38
@fivetran-sahilkhirwal fivetran-sahilkhirwal requested a review from a team as a code owner March 17, 2026 07:38
Copilot AI review requested due to automatic review settings March 17, 2026 07:38
@github-actions github-actions bot added the size/XXL PR size: extra extra large label Mar 17, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 17, 2026

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds two new EHI (Microsoft SQL Server / Caboodle-style) Connector SDK examples—one “simple” connector and one “high volume” connector—and updates the repository’s top-level connector index to reference both variants.

Changes:

  • Update the top-level README.md connector list to point to two EHI example variants.
  • Add a simple_ehi connector example using python-tds with basic schema discovery, batch fetching, and incremental sync based on a modified-date column.
  • Add a high_volume_ehi connector example using pyodbc with parallel schema discovery, keyset/offset pagination readers, and threaded table syncing.

Reviewed changes

Copilot reviewed 9 out of 13 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
README.md Splits the EHI entry into simple_ehi and high_volume_ehi references.
connectors/ehi/simple_ehi/requirements.txt Adds the python-tds dependency for the simple connector.
connectors/ehi/simple_ehi/README.md Adds documentation for the simple EHI connector example.
connectors/ehi/simple_ehi/connector.py Implements the simple EHI connector (schema discovery + sync loop + state/checkpointing).
connectors/ehi/simple_ehi/configuration.json Adds placeholder configuration for the simple connector.
connectors/ehi/high_volume_ehi/requirements.txt Adds dependencies for the high-volume connector (currently includes requests).
connectors/ehi/high_volume_ehi/README.md Adds documentation for the high-volume EHI connector example.
connectors/ehi/high_volume_ehi/readers.py Adds keyset/offset/incremental reader implementations.
connectors/ehi/high_volume_ehi/models.py Adds schema detection + replication key detection logic.
connectors/ehi/high_volume_ehi/constants.py Adds performance/retry/tuning constants.
connectors/ehi/high_volume_ehi/connector.py Implements the high-volume connector orchestration and threaded table syncing.
connectors/ehi/high_volume_ehi/configuration.json Adds placeholder configuration for the high-volume connector.
connectors/ehi/high_volume_ehi/client.py Adds a pyodbc connection wrapper + pooled acquire/retry behavior.

Comment thread README.md
Comment thread connectors/ehi/high_volume_ehi/requirements.txt Outdated
Comment thread connectors/ehi/high_volume_ehi/README.md Outdated
Comment thread connectors/ehi/high_volume_ehi/README.md
Comment thread connectors/ehi/high_volume_ehi/README.md
Comment thread connectors/ehi/high_volume_ehi/connector.py
Comment thread connectors/ehi/high_volume_ehi/readers.py Outdated
Comment thread connectors/ehi/high_volume_ehi/readers.py Outdated
Comment thread connectors/ehi/high_volume_ehi/readers.py
Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
@fivetran-satvikpatil
Copy link
Copy Markdown
Collaborator

@codex review

Comment thread connectors/ehi/high_volume_ehi/client.py
Comment thread connectors/ehi/high_volume_ehi/client.py
Comment thread connectors/ehi/high_volume_ehi/models.py
Comment thread connectors/ehi/high_volume_ehi/models.py Outdated
Comment thread connectors/ehi/high_volume_ehi/models.py Outdated
Comment thread connectors/ehi/high_volume_ehi/readers.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py
Comment thread connectors/ehi/high_volume_ehi/client.py
Comment thread connectors/ehi/high_volume_ehi/client.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py
Comment thread connectors/ehi/high_volume_ehi/constants.py
Comment thread connectors/ehi/high_volume_ehi/models.py Outdated
Comment thread connectors/ehi/high_volume_ehi/models.py
Comment thread connectors/ehi/high_volume_ehi/readers.py
Comment thread connectors/ehi/high_volume_ehi/readers.py Outdated
Comment thread connectors/ehi/high_volume_ehi/readers.py Outdated
Comment thread connectors/ehi/high_volume_ehi/readers.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Comment thread connectors/ehi/high_volume_ehi/constants.py
Comment thread connectors/ehi/high_volume_ehi/readers.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py
Comment thread connectors/ehi/high_volume_ehi/connector.py
Copy link
Copy Markdown
Contributor

@fivetran-JenasVimal fivetran-JenasVimal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the comments

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6c6bc312b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Comment thread connectors/ehi/high_volume_ehi/readers.py Outdated
Comment thread connectors/ehi/high_volume_ehi/constants.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py
Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Comment thread connectors/ehi/high_volume_ehi/connector.py
Comment thread connectors/ehi/high_volume_ehi/connector.py
Copy link
Copy Markdown
Collaborator

@fivetran-satvikpatil fivetran-satvikpatil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address one comment, else LGTM. Also please add testing details again after the changes.

Comment thread connectors/ehi/high_volume_ehi/connector.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants