feat(mongodb-to-mongodb]: Read/Write Performance and Reliability Improvements - #4079

Open

michaeltle-goog wants to merge 20 commits into

GoogleCloudPlatform:mainfrom

michaeltle-goog:monogo-to-mongo-improvements

Commits on Jul 29, 2026

Improve MongoDB to MongoDB template throughput by removing GroupIntoBatches shuffle barrier
michaeltle-goog
committed
4516199
Add parallel index-slice reader to MongoDbToMongoDb template for high-throughput reads without splitVector
michaeltle-goog
committed
6799cf4
Add comprehensive operational and debugging logs across MongoDB template stages with UriSanitizer
michaeltle-goog
committed
4a53f4c
Use official FindQuery in MongoDbIO.read().withQueryFn to fix IllegalArgumentException
michaeltle-goog
committed
e29199f
Apply spotless formatting and checkstyle validation
michaeltle-goog
committed
fef5af7
Group parallel read slices under ReadSlices(collection) in Dataflow UI
michaeltle-goog
committed
ce844f2
Add parameterized linear write rate ramp-up with Guava RateLimiter
michaeltle-goog
committed
e63787e
Add data-driven $sample quantile split discovery and automatic key-type detection
michaeltle-goog
committed
6f51b96
Remove useDataDrivenReadSplits flag and automatically attempt data-driven read splits when numReadPrefixSplits > 1
michaeltle-goog
committed
9ede4d0
Rename numReadPrefixSplits parameter to numReadSplits and remove unused useBucketAuto and numSplits options
michaeltle-goog
committed
644b257
fix: implement Type-Aware Data-Driven Splitting to handle mixed BSON types
michaeltle-goog
committed
4731f2d
test: add mockito tests for Type-Aware splits
michaeltle-goog
committed
b4cff74
fix(mongodb): convert verbose batch flush logging to debug level
michaeltle-goog
committed
0387807
docs(mongodb): update template parameter descriptions and group parameter order numbers
michaeltle-goog
committed
cca114d

Navigation Menu

Uh oh!

feat(mongodb-to-mongodb]: Read/Write Performance and Reliability Improvements - #4079

feat(mongodb-to-mongodb]: Read/Write Performance and Reliability Improvements#4079
michaeltle-goog wants to merge 20 commits into
GoogleCloudPlatform:mainfrom
michaeltle-goog:monogo-to-mongo-improvements

Commits on Jul 29, 2026

Improve MongoDB to MongoDB template throughput by removing GroupIntoBatches shuffle barrier

Add parallel index-slice reader to MongoDbToMongoDb template for high-throughput reads without splitVector

Add comprehensive operational and debugging logs across MongoDB template stages with UriSanitizer

Use official FindQuery in MongoDbIO.read().withQueryFn to fix IllegalArgumentException

Apply spotless formatting and checkstyle validation

Group parallel read slices under ReadSlices(collection) in Dataflow UI

Add parameterized linear write rate ramp-up with Guava RateLimiter

Add data-driven $sample quantile split discovery and automatic key-type detection

Remove useDataDrivenReadSplits flag and automatically attempt data-driven read splits when numReadPrefixSplits > 1

Rename numReadPrefixSplits parameter to numReadSplits and remove unused useBucketAuto and numSplits options

fix: implement Type-Aware Data-Driven Splitting to handle mixed BSON types

test: add mockito tests for Type-Aware splits

fix(mongodb): convert verbose batch flush logging to debug level

docs(mongodb): update template parameter descriptions and group parameter order numbers

Commits on Jul 30, 2026

style(mongodb): apply spotless java formatting rules to ReadSplitGenerator and tests

style(mongodb): replace inline fully qualified imports in tests and transforms

Commits on Jul 31, 2026

feat(mongodb): log explicit DLQ directories and inspection command at startup

feat(mongodb): rate-limit DLQ reason logging to every 30s per worker thread

Commits on Aug 7, 2026

Merge remote-tracking branch 'origin/main' into monogo-to-mongo-improvements

feat(mongodb): add standard UUID representation support and update DLQ writers