Skip to content

support default primary key detection for mysql for int/temporal types#4440

Open
jgao54 wants to merge 1 commit into
mainfrom
mysql-parallel-snapshot
Open

support default primary key detection for mysql for int/temporal types#4440
jgao54 wants to merge 1 commit into
mainfrom
mysql-parallel-snapshot

Conversation

@jgao54

@jgao54 jgao54 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

When custom partition key is not specified, determine default partition key based on table's primary key. If the primary key(s) starts with an integer or temporal type, perform parallel snapshot instead of full table snapshot.

Testing:

  • add unit & e2e tests
  • validated parallel snapshotting locally

@jgao54 jgao54 force-pushed the mysql-parallel-snapshot branch from ea632d8 to f1df47e Compare June 17, 2026 19:00
@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
2304 2 2302 197
View the top 2 failed test(s) by shortest run time
github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg
Stack Traces | 0.01s run time
=== RUN   TestApiPg
=== PAUSE TestApiPg
=== CONT  TestApiPg
--- FAIL: TestApiPg (0.01s)
2026/06/17 20:01:12 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/06/17 20:01:12 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/06/17 20:01:12 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_64ugcxyz.test_time
2026/06/17 20:01:12 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_aldueesm.test_update_pkey_enabled
github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
Stack Traces | 4.12s run time
=== RUN   TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
=== PAUSE TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
=== CONT  TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
2026/06/17 19:59:36 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/06/17 19:59:36 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/06/17 19:59:36 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_k7lxhkpi.\"table1\" ORDER BY id"
2026/06/17 19:59:36 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_k7lxhkpi.\"table1\" ORDER BY id"
2026/06/17 19:59:36 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_5027000992335193366 CURSOR FOR SELECT id,val FROM e2e_test_api_k7lxhkpi.\"table1\" ORDER BY id" args=[]
2026/06/17 19:59:36 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_k7lxhkpi.\"table1\" ORDER BY id" channelLen=0
2026/06/17 19:59:36 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5027000992335193366
2026/06/17 19:59:36 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5027000992335193366 records=2 bytes=19 channelLen=1
2026/06/17 19:59:36 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_k7lxhkpi.\"table1\" ORDER BY id" rows=2 bytes=19 channelLen=1
2026/06/17 19:59:36 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5027000992335193366
2026/06/17 19:59:36 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5027000992335193366 records=0 bytes=0 channelLen=0
2026/06/17 19:59:36 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_k7lxhkpi.\"table1\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/06/17 19:59:36 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/06/17 19:59:36 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_k7lxhkpi.\"table1\" ORDER BY id" rows=2 bytes=19 channelLen=0
    api_test.go:3479: WaitFor wait for flow to be running 2026-06-17 19:59:40.258755513 +0000 UTC m=+405.352641579
    api_test.go:3490: 
        	Error Trace:	.../flow/e2e/api_test.go:3490
        	            				.../hostedtoolcache/go/1.26.4.../src/runtime/asm_amd64.s:1771
        	Error:      	"1" is not greater than "1"
        	Test:       	TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
        	Messages:   	Should have multiple executions (continue-as-new happened)
    api_test.go:51: begin tearing down postgres schema api_azlkcktl
--- FAIL: TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew (4.12s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@github-actions

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: The PR's own new test Test_MySQL_Default_Partition_Key_Parallel_Snapshot deterministically times out during snapshot (STATUS_SNAPSHOT) in multiple matrix legs, indicating a real bug in the new MySQL default-partition-key parallel-snapshot feature rather than flakiness.
Confidence: 0.8

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

@jgao54 jgao54 force-pushed the mysql-parallel-snapshot branch from f1df47e to 57e75a7 Compare June 17, 2026 19:47
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: The test failed on a non-retried assertion (require.Greater, "1 is not greater than 1") that races against Temporal's eventually-consistent ListWorkflow visibility indexing, and failed in only one of several parallel matrix jobs—both hallmarks of a flaky timing race rather than a real bug.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

@jgao54 jgao54 marked this pull request as ready for review June 17, 2026 20:11
@jgao54 jgao54 requested a review from a team as a code owner June 17, 2026 20:11
@claude

claude Bot commented Jun 17, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant