Skip to content

feat(Spanner): integrate SourceConfigParser to centralize shard configuration loading for SourceDbToSpanner pipelines. - #3854

Open
pratickchokhani wants to merge 16 commits into
GoogleCloudPlatform:mainfrom
pratickchokhani:shard-config-bulk
Open

feat(Spanner): integrate SourceConfigParser to centralize shard configuration loading for SourceDbToSpanner pipelines.#3854
pratickchokhani wants to merge 16 commits into
GoogleCloudPlatform:mainfrom
pratickchokhani:shard-config-bulk

Conversation

@pratickchokhani

@pratickchokhani pratickchokhani commented May 25, 2026

Copy link
Copy Markdown
Contributor
  1. Implemented centralize shard configuration loading for SourceDbToSpanner
  2. Added integration tests for Astra DB
  3. Updated integration tests across the template.

@pratickchokhani pratickchokhani added the addition New feature or request label May 25, 2026
@codecov

codecov Bot commented May 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.41667% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.17%. Comparing base (fdf3c92) to head (2d3b2a3).

Files with missing lines Patch % Lines
...loud/teleport/v2/templates/PipelineController.java 0.00% 7 Missing ⚠️
...ource/jdbc/AbstractJdbcSrcToSpSourceConnector.java 0.00% 4 Missing ⚠️
...cloud/teleport/v2/templates/SourceDbToSpanner.java 83.33% 1 Missing and 1 partial ⚠️
...rce/cassandra/CassandraSrcToSpSourceConnector.java 50.00% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##               main    #3854       +/-   ##
=============================================
+ Coverage     38.08%   56.17%   +18.08%     
- Complexity      511     7336     +6825     
=============================================
  Files           195     1126      +931     
  Lines         11764    68711    +56947     
  Branches       1179     7778     +6599     
=============================================
+ Hits           4480    38596    +34116     
- Misses         6919    27619    +20700     
- Partials        365     2496     +2131     
Components Coverage Δ
spanner-templates 87.59% <83.72%> (∅)
spanner-import-export 68.91% <ø> (∅)
spanner-live-forward-migration 89.47% <ø> (∅)
spanner-live-reverse-replication 83.62% <ø> (∅)
spanner-bulk-migration 92.24% <83.72%> (∅)
gcs-spanner-dv 88.79% <ø> (∅)
Files with missing lines Coverage Δ
...eam/it/gcp/cloudsql/CloudSqlShardOrchestrator.java 85.83% <100.00%> (-0.52%) ⬇️
...ud/teleport/v2/options/OptionsToConfigBuilder.java 92.94% <100.00%> (ø)
...ud/teleport/v2/source/ISrcToSpSourceConnector.java 100.00% <ø> (ø)
...cassandra/iowrapper/CassandraIOWrapperFactory.java 100.00% <100.00%> (ø)
.../cassandra/iowrapper/CassandraIOWrapperHelper.java 100.00% <100.00%> (ø)
...der/io/cassandra/iowrapper/CassandraIoWrapper.java 100.00% <ø> (ø)
...t/v2/source/jdbc/ShardedJdbcDbConfigContainer.java 100.00% <100.00%> (ø)
...urce/jdbc/SingleInstanceJdbcDbConfigContainer.java 100.00% <100.00%> (ø)
...t/v2/source/mysql/MySqlSrcToSpSourceConnector.java 98.85% <100.00%> (ø)
...ource/postgres/PostgresSrcToSpSourceConnector.java 99.00% <100.00%> (ø)
... and 4 more

... and 943 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pratickchokhani
pratickchokhani force-pushed the shard-config-bulk branch 6 times, most recently from a19cf9e to 581ffd3 Compare June 29, 2026 07:25
@pratickchokhani
pratickchokhani force-pushed the shard-config-bulk branch 4 times, most recently from 0da7475 to e13bdd2 Compare June 30, 2026 14:13
@pull-request-size pull-request-size Bot added size/XL and removed size/L labels Jul 1, 2026
@pratickchokhani
pratickchokhani force-pushed the shard-config-bulk branch 3 times, most recently from 41f2355 to 461938b Compare July 2, 2026 10:46
@pratickchokhani
pratickchokhani marked this pull request as ready for review July 2, 2026 10:55
@pratickchokhani
pratickchokhani requested a review from a team as a code owner July 2, 2026 10:55
@pratickchokhani
pratickchokhani requested review from jsuhani-2026 and sm745052 and removed request for sm745052 July 2, 2026 10:55
@pratickchokhani
pratickchokhani force-pushed the shard-config-bulk branch 3 times, most recently from c7f9560 to 2649e0a Compare July 28, 2026 08:38

@bharadwaj-aditya bharadwaj-aditya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

"additionalExperiments", Collections.singletonList("disable_runner_v2"));

if (!multiSharded) {
builder.addParameter("jdbcDriverClassName", "com.mysql.jdbc.Driver");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the updated flow, the sharded and non-sharded flows are basically the same. Why do we need this special handling ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

addition New feature or request size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants