State Production: Parallel Backfill Worker Pool

[aristidesstaffieri]

Summary

Add parallel batch processing with streaming ordered commit to both migration subcommands.

Scope

• Worker pool that processes ledger batches concurrently (each worker: isolated LedgerBackend, isolated BatchBuffer, processes ledgers sequentially within batch)
• Ordered commit buffer: committer goroutine writes completed batches to DB in order (maintains nextToCommit counter, buffers out-of-order results until their turn)
• Each batch commit: CAS-advance respective cursor + write batch data in atomic transaction. CAS fail at any point → handoff detected, stop.
• Different parallelization modes:
  • History migration: batches are truly independent (state changes for N don't depend on N-1), can be more aggressively parallelized
  • Current-state migration: must commit in order (additive state), but processing is still parallel
• Crash recovery: on restart, resume from cursor + 1 (no work lost)

Dependencies

• State Production: protocol-migrate history Command #515 (protocol-migrate history Command)
• State Production: protocol-migrate current-state Command #516 (protocol-migrate current-state Command)

Reference

docs/feature-design/data-migrations.md (State Production section, line 619+)

[aristidesstaffieri]

@aditya1702 has done some benchmarking and learned that the bottleneck for our ingestion patterns is DB writes and these kinds of optimizations introduce a lot of complexity for little gain in real throughput. I am going to defer this one until we decide we need this, if at all.