Tracking Kafka enhancement and migrate to franz-go

[3AceShowHand]

This issue tracks the complete Kafka Sink migration from Sarama to franz-go. It is the roadmap and dependency index; detailed implementation and acceptance criteria belong to the linked task issues.

Goal and invariants

• Plan and implement from master; the existing franz-go POC is reference material, not a branch to merge directly.
• Replace only the Kafka client layer. Preserve Kafka Sink configuration, routing, message, error, retry, callback, checkpoint, and resource-lifecycle behavior unless a separately reviewed change says otherwise.
• Define TiCDC behavior through client-independent tests over TiCDC-owned interfaces. Sarama and franz-go adapter tests may verify client-specific mapping and cleanup, but cannot be the only definition of Kafka Sink behavior.
• Complete the migration in four milestones: establish the test baseline, introduce franz-go and make it the default with a Sarama fallback, validate production readiness, then remove Sarama.

Execution order

1. Complete all Milestone 0 baseline tasks.
2. Merge and validate the franz-go implementation in Milestone 1.
3. Complete staged production validation in Milestone 2.
4. Remove Sarama and migration-only code in Milestone 3.

The dependency chain is:

client-independent baseline → franz-go + Sarama fallback → production validation → Sarama removal

Milestone 0: establish the client-independent test baseline

Task issues

• tests: decouple Kafka Sink tests from Sarama #5884 — decouple Kafka Sink tests from Sarama.
• kafka: decouple configuration and authentication tests from Sarama #5887 — closed as not planned after its configuration and authentication scope was absorbed by tests: decouple Kafka Sink tests from Sarama #5884 and kafka: decouple security SASL types from Sarama #5892; no separate work remains under this issue.
• kafka: remove Sarama broker mocks from sink tests #5888 — closed as not planned after its broker-mock cleanup scope was absorbed by tests: decouple Kafka Sink tests from Sarama #5884; no separate work remains under this issue.
• kafka: decouple security SASL types from Sarama #5892 — decouple security SASL types from Sarama.
• kafka: decouple admin tests from Sarama #5885 — decouple Admin and Topic Manager tests from Sarama.
• kafka: decouple producer tests from Sarama #5886 — define client-independent producer behavior contracts.
• tests: decouple Kafka topic helper from Sarama #5882 — decouple the integration-test Kafka topic helper from Sarama.
• tests: prevent Sarama dependencies in Kafka Sink tests #5889 — enforce the common-test/adapter-test boundary after tests: decouple Kafka topic helper from Sarama #5882, kafka: decouple admin tests from Sarama #5885, and kafka: decouple producer tests from Sarama #5886 are complete.

Closed or superseded task issues remain in this list so that the tracker preserves the complete scope and disposition history.

Work that still needs task issues

Create focused task issues before implementing each area, then link those issues here:

• Complete Factory, construction, partial-cleanup, shutdown-order, context-cancellation, goroutine-leak, and metric-series lifecycle tests.
• Complete Verify and Sink creation tests for invalid configuration, missing Topic, Topic validation, encoder and Schema Registry failures, and partial-construction cleanup.
• Record Kafka package coverage before and after the work; critical correctness, error, and lifecycle branches require behavior assertions.
• Establish a reusable Kafka regression baseline covering initial and incremental replication, protocols, dispatchers, large messages, Topic management, Schema Registry, multiple Changefeeds, scaling, and HA.
• Establish a failure-recovery baseline covering broker restart, controller failover, network interruption, metadata refresh, timeout, retry, idle connection, and broken pipe. Recovery must be determined by controller/broker state, checkpoint progress, and final data consistency rather than fixed sleeps.

Exit criteria

• The current Sarama implementation passes the complete common behavior and regression baseline.
• Common tests contain no Sarama or franz-go types, errors, mocks, configuration objects, or client-specific call-order assumptions.
• Client-specific tests are explicitly adapter-only and do not replace common acceptance tests.
• Every failure-recovery case verifies automatic recovery, catch-up, and no data loss.

Milestone 1: introduce franz-go and make it the default

Task issue

• kafka client is not robust #832 — replace the fragile Kafka client path with a franz-go implementation while retaining Sarama as a per-Changefeed fallback during migration.
• Migrate the Kafka consumer from confluentic-kafka-go to franz-go

Required validation

• Run the same Admin, sync producer, async producer, configuration, error, and lifecycle contract tests against both clients without duplicating testcase logic.
• Verify that Verify and runtime Sink creation select the same client and that different Changefeeds can use different clients in one process.
• Validate supported Kafka versions, TLS, mTLS, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI, OAUTHBEARER, invalid credentials, and minimum ACLs.
• Run the reusable regression and failure-recovery suites with franz-go as the default and Sarama as the explicit fallback.
• Validate message-size boundaries, retry and ordering guarantees, callback/checkpoint correctness, bounded producer memory, resource cleanup, logs, metrics, and Grafana dashboards.
• Compare franz-go and Sarama throughput, latency, CPU, memory, GC, goroutines, and retry behavior in equivalent environments.

Exit criteria

• franz-go is the default Kafka client for PR, Nightly, and Release validation.
• Sarama remains selectable per Changefeed as a tested fallback.
• No release-blocking correctness, compatibility, authentication, stability, resource, performance, or observability issue remains.

Milestone 2: staged production validation

Create and link a dedicated rollout issue before this milestone begins.

Required work

• Deploy franz-go progressively to representative production workloads and Kafka environments.
• Validate correctness, stability, recovery, performance, resource usage, authentication, and observability over an agreed observation window.
• Confirm that production failures are diagnosable through logs, metrics, and dashboards.
• Exercise the per-Changefeed Sarama fallback and version rollback procedure.

Exit criteria

• franz-go has operated as the default implementation in key production scenarios without data-correctness failures or unexplained stability/performance regressions.
• Monitoring and rollback procedures have been validated.
• The team has an explicit decision to remove the Sarama fallback.

Milestone 3: remove Sarama and migration-only code

• kafka: remove legacy Sarama implementation and tests #5890 — starts only after Milestone 2 approval and removes the Sarama implementation, fallback selector, adapter-only tests, generated mocks, dependencies, metrics, and migration-only compatibility code.

Exit criteria

• Kafka Sink production code, tests, test tooling, dependencies, and monitoring no longer reference Sarama.
• franz-go retains equivalent user-visible configuration, error, lifecycle, and regression coverage.
• Repository checks, race-enabled unit tests, Kafka integration tests, and relevant test-infra plans pass.