eventservice: scan window throttles bursty TPC-C workloads

[asddongmen]

Problem

With a bursty TPC-C-like workload (hot and cold table spans, redo enabled, MySQL downstream), enabling scan window can reduce downstream throughput instead of improving fairness.

The direct v8.5.8 A/B comparison shows that scan window applies a changefeed-local global fence based on the slowest local dispatcher's sentResolvedTs. A hot/slow span can therefore pin otherwise independent spans. Fast spans stop scanning at the fence, event-service concurrency collapses, and the MySQL sink is underfed even though it still has capacity.

Sink-aware batch-by-bytes prevents the severe lag in the tested build and shows no observable regression relative to v8.5.7. However, it only keeps the slowest path draining; it does not remove the global head-of-line behavior in scan window itself.

Related issues: #4172, #5041.

Test matrix

All executions use the same redo_apply_multiple_times workload. The links below are PingCAP-internal TCMS artifacts.

Build / configuration	Execution	Scan window	Batch-by-bytes	Result
master daily	8224444	on	on	success
v8.5.7 baseline A	8178415	unavailable/off	off	success
v8.5.7 baseline B	8200949	unavailable/off	off	success
original v8.5.8	8225352	on	off	failure / OOM
v8.5.8 scan-window A/B	8178387	off	off	success
v8.5.8 batch-by-bytes backport	8178422	on	on	success

The relevant event-service, EventCollector, dynstream, and MySQL sink code is identical between the original v8.5.8 run and the scan-window-disabled A/B run. Their code difference is only in unrelated Kafka files.

Direct comparison: scan window enabled vs disabled

The following Clinic metrics are aligned to minutes 5-20 after changefeed creation. Batch-by-bytes is disabled in both runs.

Metric	Enabled: 8225352	Disabled: 8178387	Enabled / disabled
EventStore input	155,393 KV/s	138,614 KV/s	1.12x
event-service normal send	14,612 KV/s	88,699 KV/s	0.16x
event-service redo send	18,317 KV/s	77,316 KV/s	0.24x
MySQL sink rows	12,875 rows/s	86,332 rows/s	0.15x
scan tasks	10.4/s	113.1/s	0.09x
average events per scan	3,155	1,469	2.15x
approximate scanned events	32.9k/s	166.1k/s	0.20x
TiCDC process CPU	3.22 cores	3.42 cores	0.94x
dispatcher-quota skips	6.08/s	6.89/s	0.88x

This is not explained by a smaller input workload or CPU saturation. Scan window creates larger individual scans, but issues about 10.8x fewer tasks; the larger scans do not compensate.

MySQL metrics also indicate starvation rather than downstream saturation:

Metric	Enabled	Disabled
average SQL batch rows	254.7	253.6
flush calls	50.5/s	340.4/s
actual single-flush duration	18.4 ms	10.3 ms

The SQL batch size is effectively unchanged, and flush latency is nowhere near enough to explain the 6.7x row-rate gap. The sink spends much more time waiting for work.

Evidence for global head-of-line blocking

Each changefeedStatus tracks local normal and redo dispatchers together. Every second it calculates the minimum eligible sentResolvedTs, and getScanTaskDataRange caps every dispatcher at:

scanMaxTs = minDispatcherSentResolvedTs + scanInterval

An exact log snapshot about two minutes after changefeed creation shows:

• scan interval: 12.20703125 s;
• the hot workload.stock dispatcher is the minimum;
• workload.district.sentResolvedTs is exactly stock.sentResolvedTs + 12.20703125s;
• district has already received another 76.893 s of resolved progress, but is not allowed to scan it.

With scan window disabled, stock can remain behind while district advances to its own received resolved TS.

During enabled minutes 5-20:

• median receivedResolvedTs - sentResolvedTs: 573.6 s;
• median sentResolvedTs - checkpointTs: 57.7 s.

With scan window disabled, the median event-service gap is only 1.0 s. Most of the enabled-run lag is therefore created before the sink.

The adaptive controller uses EventCollector pending-memory feedback. When the global fence itself underfeeds EventCollector, pending memory looks healthy, so the controller treats the condition as permission to widen the interval. In this run the interval grows from 5 s to 12.2 s, then to about 80 s, and eventually to 30 min. When the accumulated backlog is released, path releases/resets and a large normal/redo checkpoint gap can follow.

Batch-by-bytes is an effective mitigation, with no observed v8.5.7 regression

Execution 8178422 keeps scan window enabled but bounds EventCollector work by both count and downstream byte capability. It reaches the same 30-minute scan interval, yet has no EventCollector path releases or out-of-order resets, catches up, and stays around 8-13 s checkpoint lag.

Aligned initial 0-20-minute results:

Execution	EventStore input	Normal send	Redo send	MySQL sink	Peak checkpoint lag	First sustained <=20 s
v8.5.7 8178415	158.7k/s	19.75k/s	40.49k/s	16.24k/s	3269.6 s	57.44 min
v8.5.7 8200949	159.9k/s	66.88k/s	83.26k/s	63.26k/s	1205.3 s	23.50 min
v8.5.8 + batch-by-bytes 8178422	161.7k/s	66.98k/s	68.06k/s	65.36k/s	804.8 s	23.64 min

The workload-matched comparison is 8200949 vs 8178422:

• initial MySQL sink throughput is 3.3% higher;
• catch-up time differs by only 9 seconds;
• peak checkpoint lag is 33.2% lower;
• stable phase-2/phase-3 sink throughput, checkpoint lag, CPU, and RSS are effectively identical.

Stable phase	Metric	v8.5.7 8200949	Batch-by-bytes 8178422
phase 2	sink rows	29.32k/s	30.01k/s
phase 2	lag p50	9.06 s	8.76 s
phase 2	CPU	3.16 cores	3.22 cores
phase 2	combined RSS	3.61 GiB	3.62 GiB
phase 3	sink rows	27.48k/s	27.66k/s
phase 3	lag p50	9.65 s	8.91 s
phase 3	CPU	3.02 cores	3.03 cores
phase 3	combined RSS	3.60 GiB	3.60 GiB

There is one slower first-resume transient, but it is not repeatable: the backport receives 11% more input in that window, and the second resume is effectively identical. The current evidence does not show an end-to-end regression relative to v8.5.7.

Why batching does not close this issue

Batch-by-bytes changes the work unit:

byte-bounded work
  -> shorter and less variable per-path await
  -> fairer sink/conflict-detector scheduling
  -> EventCollector drains and returns quota
  -> slowest sentResolvedTs advances
  -> global scan fence advances

This makes the tested workload recover, but scan window still uses a single slow dispatcher as the changefeed-wide base and still lacks direct downstream checkpoint-lag/in-flight-byte feedback. A workload or downstream condition that prevents the slowest path from draining can reproduce the global stall.

Expected behavior / optimization directions

Enabling scan window should not starve a healthy downstream when one hot span lags behind independent spans.

Possible directions:

1. Include downstream checkpoint lag and in-flight bytes in congestion feedback, instead of relying mainly on EventCollector pending memory.
2. Add an emergency brake or hard bound when receivedResolvedTs - sentResolvedTs or checkpoint lag grows.
3. Avoid a single changefeed-wide head-of-line fence; consider per-dispatcher/per-mode windows or explicit fairness scheduling.
4. Treat path release/reset as high-pressure/instability signals rather than relief.
5. Add a regression test with TPC-C-like hot/cold spans, redo enabled, and batch-by-bytes both enabled and disabled.

Summary

• Scan window itself causes a reproducible throughput problem for this TPC-C-like workload and needs optimization.
• Batch-by-bytes is a strong mitigation and has no observed regression versus v8.5.7.
• The successful batching result should not be interpreted as proving the scan-window design is safe.