Skip to content

[fix][broker] Skip backlog-quota eviction on fenced/closing topics#25684

Open
lhotari wants to merge 1 commit intoapache:masterfrom
lhotari:lh-fix-backlogquota-skip-fenced
Open

[fix][broker] Skip backlog-quota eviction on fenced/closing topics#25684
lhotari wants to merge 1 commit intoapache:masterfrom
lhotari:lh-fix-backlogquota-skip-fenced

Conversation

@lhotari
Copy link
Copy Markdown
Member

@lhotari lhotari commented May 5, 2026

Motivation

BrokerService.monitorBacklogQuota runs every backlogQuotaCheckIntervalInSeconds and iterates every persistent topic on the broker via forEachPersistentTopic (BrokerService.java:2476). For each topic it calls BacklogQuotaManager.handleExceededBacklogQuota, which mutates managed-ledger cursor state via slowestConsumer.skipEntries and markDeletePositionMoveForward (see BacklogQuotaManager.java:140-213).

Today this runs unconditionally — including on topics that have already been fenced (e.g. transient ledger-interceptor exception) or are closing/deleting (fenceTopicToCloseOrDelete() sets isClosingOrDeleting = true and isFenced = true).

When a namespace is being force-deleted under quota pressure, the checker concurrently mutates cursor state on a topic the delete path is trying to tear down. The two compete for the same managed-ledger / cursor, slowing the deletion. This was observed as intermittent Awaitility.deleteNamespace timeouts in BacklogQuotaManagerTest — see https://github.com/apache/pulsar/actions/runs/25387367872/job/74455105884 (PR #25676 attempt 1).

A test-side workaround for the symptom is in #25680; this PR addresses the underlying broker behavior.

Modifications

Early-return at the top of BacklogQuotaManager.handleExceededBacklogQuota if the topic reports isFenced() or isClosingOrDeleting():

if (persistentTopic.isFenced() || persistentTopic.isClosingOrDeleting()) {
    // entries are about to be discarded — running eviction is wasted work
    // and contends with the close/delete path
    return;
}

Both methods are auto-generated by Lombok @Getter on the existing fields (AbstractTopic.isFenced, PersistentTopic.isClosingOrDeleting). No new accessors needed.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as BacklogQuotaManagerTest.testConsumerBacklogEvictionSizeQuotaCleansPendingAcks and BacklogQuotaManagerTest.testConsumerBacklogEvictionTimeQuotaNotPreciseCleansPendingAcks (the normal-path eviction is exercised; both pass locally with this change applied).

The skipped path (fenced/closing topic) is a correctness no-op: the entries to be evicted are about to be discarded anyway by the close/delete pipeline, so the only observable effect is the absence of redundant cursor mutations during teardown.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

`BrokerService.monitorBacklogQuota` runs every
`backlogQuotaCheckIntervalInSeconds` (default 60s, but reduced to 2s in
some tests) and iterates every persistent topic on the broker. For each
topic it calls `BacklogQuotaManager.handleExceededBacklogQuota`, which
ultimately mutates managed-ledger cursor state via
`slowestConsumer.skipEntries` and `markDeletePositionMoveForward`.

Today this runs unconditionally, even on topics that have already been
fenced (transient interceptor exception) or are being torn down
(`isClosingOrDeleting`). When a namespace is being force-deleted under
quota pressure, the checker contends with the delete path on the same
managed ledger / cursor — slowing the deletion and causing intermittent
`Awaitility.deleteNamespace` timeouts in tests like
`BacklogQuotaManagerTest`.

Fix: early-return at the top of `handleExceededBacklogQuota` if the
topic reports `isFenced()` or `isClosingOrDeleting()`. The entries are
about to be discarded by the close/delete path — running eviction is
both wasted work and a source of contention.

`isFenced` is auto-generated by Lombok `@Getter` on
`AbstractTopic.isFenced`; `isClosingOrDeleting` is auto-generated on
`PersistentTopic.isClosingOrDeleting`. No new accessors needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants