RocketMQ version
develop at a06836dd564e5e43115493f775626cf98d51d10e
Describe the bug
ConsumeQueue.truncateDirtyLogicFiles updates the ConsumeQueue positions but skips truncating the associated ConsumeQueueExt during every normal partial-truncation path.
The method tracks the last retained extension address in maxExtAddr, and its tail calls:
consumeQueueExt.truncateByMaxAddress(maxExtAddr);
However, scanning returns directly when it reaches the first dirty CQ entry, the end of a full valid CQ file, or an unwritten slot. Those returns bypass the extension truncation. The tail is only reached after all CQ files have been deleted or no CQ file exists; in that case maxExtAddr remains 1, which is not an extension address, so truncateByMaxAddress is a no-op.
Deterministic reproduction
- Create a file-backed
ConsumeQueue with enableConsumeQueueExt=true.
- Write four CQ entries with physical offsets
0, 100, 200, and 300, using distinct tag codes so that four CQExt units are created.
- Record the raw extension addresses stored by CQ entries 2 and 3.
- Call
truncateDirtyLogicFiles(200), retaining entries 0 and 1.
- Write replacement entries at CQ offsets 2 and 3 with different tag codes.
- Flush, load a second
ConsumeQueue from the same directory, and call recover().
- Read the extension unit at the original address of entry 2.
Expected:
- CQExt is truncated together with CQ.
- Replacement entries reuse the truncated extension addresses.
- After reload, the original address of entry 2 contains the replacement tag code.
Actual:
- The old extension units remain readable after CQ truncation.
- Replacement extension units are appended after the orphaned units.
- Recovery retains the orphaned units because the newly appended CQ entries now reference later extension addresses.
The controlled regression test is named:
ConsumeQueueTest#testTruncateDirtyLogicFilesTruncatesConsumeQueueExtAndSurvivesReload
It uses no sleeps, randomized scheduling, or external services. On the latest unmodified develop baseline under JDK 8, it failed at the same business assertion in 5/5 isolated Maven/JUnit processes: expected replacement tag code 202, but the original orphan still returned 102.
Impact
After CommitLog/CQ rollback followed by rebuilding, unreachable ConsumeQueueExt units remain between retained and replacement units. This leaks persistent store space and cannot be repaired by a later restart, because recovery truncates only after the newest referenced extension address.
Suggested fix
- Route every normal scan exit through the CQExt truncation tail instead of returning early.
- Continue deleting wholly dirty tail CQ files until the last retained entry is found.
- Treat an empty first slot as an empty tail file and continue scanning the preceding retained file.
- If a retained CQ entry references CQExt, truncate after its last retained extension address; if no retained CQ entry references CQExt, physically clear the extension files so stale bytes cannot reappear during reload/recovery.
- Make the full-clear path recoverable when an extension mapped file is still referenced: retain failed mappings, block address reuse while cleanup is pending, and allow a later retry after the reader releases the buffer.
- When recovery finds no CQ file, clear any loaded CQExt files so a crash between CQ deletion and extension cleanup self-heals on restart.
- Preserve the non-destructive
deleteFile=false overload by not physically truncating CQExt while detached CQ files remain on disk.
- Add partial-file, empty/full-tail-file, full-truncation, cleanup-failure/retry, replacement-write, and reload/recovery regression coverage.
Related work checked
Immediate pre-publication searches across open and closed issues and open, closed, and merged pull requests for ConsumeQueueExt, truncateDirtyLogicFiles, truncateByMaxAddress, extension truncation, and orphaned extension data found no equivalent report, claimed fix, or assignee.
RocketMQ version
developata06836dd564e5e43115493f775626cf98d51d10eDescribe the bug
ConsumeQueue.truncateDirtyLogicFilesupdates the ConsumeQueue positions but skips truncating the associatedConsumeQueueExtduring every normal partial-truncation path.The method tracks the last retained extension address in
maxExtAddr, and its tail calls:However, scanning returns directly when it reaches the first dirty CQ entry, the end of a full valid CQ file, or an unwritten slot. Those returns bypass the extension truncation. The tail is only reached after all CQ files have been deleted or no CQ file exists; in that case
maxExtAddrremains1, which is not an extension address, sotruncateByMaxAddressis a no-op.Deterministic reproduction
ConsumeQueuewithenableConsumeQueueExt=true.0,100,200, and300, using distinct tag codes so that four CQExt units are created.truncateDirtyLogicFiles(200), retaining entries 0 and 1.ConsumeQueuefrom the same directory, and callrecover().Expected:
Actual:
The controlled regression test is named:
It uses no sleeps, randomized scheduling, or external services. On the latest unmodified
developbaseline under JDK 8, it failed at the same business assertion in 5/5 isolated Maven/JUnit processes: expected replacement tag code202, but the original orphan still returned102.Impact
After CommitLog/CQ rollback followed by rebuilding, unreachable ConsumeQueueExt units remain between retained and replacement units. This leaks persistent store space and cannot be repaired by a later restart, because recovery truncates only after the newest referenced extension address.
Suggested fix
deleteFile=falseoverload by not physically truncating CQExt while detached CQ files remain on disk.Related work checked
truncateDirtyLogicFiles; they do not truncate CQExt.correctMinOffset, a different method and postcondition.ConsumeQueue(estimateMessageCount) and does not change truncation.Immediate pre-publication searches across open and closed issues and open, closed, and merged pull requests for
ConsumeQueueExt,truncateDirtyLogicFiles,truncateByMaxAddress, extension truncation, and orphaned extension data found no equivalent report, claimed fix, or assignee.