Back off when a batch fails for a reason no queue item can be charged for

[dpage]

Closes #52.

Summary

#51 bounded the spin for failures attributable to a particular queue item: the item is charged, next_retry_at holds it out of the claim, and max_attempts retires it. Failures belonging to the batch rather than to any one row are deliberately not charged, and rightly so, since billing a blameless item for a misconfigured provider would work through the queue retiring one innocent row per max_attempts cycles.

Those failures therefore still spun. The same rows were reclaimed and failed identically on the very next poll, indefinitely, filling the log at whatever rate worker_poll_interval allowed. An unset or misspelled pgedge_vectorizer.provider is enough to provoke it, as is a failed provider->init() or a dimension probe failure, so a single mistyped GUC could do it.

Measured at the 200ms poll the new test uses: 100 failure cycles in a twenty second window, against three with this change.

Fix

Nothing in the queue can help here, because all of it is keyed on attempts moving and there is no item to move it against, so the wait itself is lengthened instead. Five seconds after the first such failure, doubling to a five minute ceiling, cleared by any batch that gets through.

That is deliberately a third instance of a shape already in this file rather than a new idea: it mirrors the extension-not-installed retry a few lines above it and the launcher's failed-start backoff. queue_item_record_failure() now returns whether it found an item to charge, which is what tells the two kinds of failure apart, and the backoff is floored at the poll interval so a configuration with a long poll is never made to poll faster by failing.

A reload clears the backoff too, since a reload is how a misconfigured provider gets corrected and making the operator wait out a backoff their fix has already invalidated would be obtuse.

One thing the test caught

The reload reset did not work as first written. A SIGHUP arriving during the wait was only handled at the top of the following iteration, so the worker ran one more batch against the configuration just corrected, doubled the backoff again, and logged the stale interval before resetting. A reload during a 20 second backoff reported waiting 40s. Handling the reload before any further work fixes it, and the test asserts the reset lands on the floor rather than merely that a retry happens, which is what made the difference visible.

Test plan

test/t/005_batch_failure_backoff.pl injects a provider that does not exist, which needs no network and no API key and is the likeliest way a real deployment lands here. The queue row points at a chunk table that genuinely exists, so the dense-embedding probe succeeds and the batch reaches the provider lookup; were it missing, the failure would be charged to the item and the queue's own backoff would cover it, which is the case this test is specifically not about.

• 19 regression tests and all five TAP suites green on PostgreSQL 18.4, 28 assertions in total
• Six of the eight new assertions fail against unfixed main, including the spin bound, which fails at 100 cycles against a limit of 8
• The backoff sequence is asserted, not just its existence: the floor is 5s and each successive failure waits longer
• The reload path is asserted to reset to the floor, which is what caught the ordering defect above

[coderabbitai]

📝 Walkthrough

Walkthrough

The worker now distinguishes item-specific failures from batch-wide failures. Batch-wide failures use exponential backoff from 5 seconds to 5 minutes. Successful batches and configuration reloads clear the backoff. SIGHUP handling prevents processing with stale configuration. A new integration test verifies bounded retries, increasing delays, and backoff reset after reload.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the worker-level backoff added for batch failures that cannot be charged to a queue item.
Description check	✅ Passed	The description directly explains the batch-level retry problem, the backoff design, reload behavior, and regression tests.
Linked Issues check	✅ Passed	The changes implement exponential worker backoff, preserve item-level charging, reset on success or reload, and add regression coverage for issue [#52].
Out of Scope Changes check	✅ Passed	The implementation and integration test are directly related to the linked issue and stated objectives; no unrelated changes are identified.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/issue-52-batch-failure-backoff

---

_{Comment @coderabbitai help to get the list of available commands.}

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity · 0 duplication

Metric	Results
Complexity	0
Duplication	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

test/t/005_batch_failure_backoff.pl (1)

42-42: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add coverage for poll-interval flooring.

worker_poll_interval is 200 ms, which is below BATCH_RETRY_MIN. This test cannot detect a regression when the poll interval exceeds the 5-second batch backoff. Add a case with a poll interval above 5000 ms and assert that the effective wait uses that interval.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/t/005_batch_failure_backoff.pl` at line 42, Add a test case in the batch
failure backoff coverage around the worker_poll_interval setting that uses a
value above 5000 ms and verifies the effective retry wait matches that poll
interval, while retaining the existing below-BATCH_RETRY_MIN coverage.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/worker.c`:
- Around line 1353-1357: Update the pgedge_vectorizer worker failure log to
report the effective backoff duration in milliseconds without truncating
fractional seconds. In test/t/005_batch_failure_backoff.pl lines 113-120 and
130-136, parse the revised millisecond unit and assert the first backoff and
reload-reset values respectively.

---

Nitpick comments:
In `@test/t/005_batch_failure_backoff.pl`:
- Line 42: Add a test case in the batch failure backoff coverage around the
worker_poll_interval setting that uses a value above 5000 ms and verifies the
effective retry wait matches that poll interval, while retaining the existing
below-BATCH_RETRY_MIN coverage.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 61287ac0-e64f-4c9d-ae18-59e55b275106

📥 Commits

Reviewing files that changed from the base of the PR and between 24d0611 and 64b5f8c.

📒 Files selected for processing (2)

• src/worker.c
• test/t/005_batch_failure_backoff.pl