v4.8.1: resume checkpoints can silently skip changed documents not yet processed

[grishkovei]

Summary

The smart-reindex checkpoint records every key currently present in self._indexed_docs, not only documents successfully processed by the current run. On resume, every ID in that checkpoint is skipped before the changed-file check.

As a result, an existing document that changed before the reindex, but had not yet been reached when the process stopped, can be silently skipped after resume and keep stale chunks/vectors.

Affected version: knowledge-rag==4.8.1 / tag v4.8.1 (da167a58fdc00af78d9c5b7cda6895316b5a6030).

Relevant code

• Checkpoint writes list(self._indexed_docs.keys())
• Resume converts the stored IDs into resume_doc_ids
• Membership is checked before mtime/size change detection

Reproduction scenario

1. Metadata already contains documents A, B, and C from a previous completed index.
2. B changes on disk.
3. A smart reindex processes A and writes a checkpoint before reaching B.
4. The checkpoint contains A, B, and C because it serializes all existing metadata keys.
5. Stop the process and resume.
6. B is skipped solely because its ID is in resume_doc_ids; its changed mtime/size is never checked.

Exact-tag execution of this scenario:

CHECKPOINT_IDS ['already_processed', 'changed_but_not_reached', 'unchanged']
UNPROCESSED_CHANGED_DOC_SKIPPED True

Without resume_state, the same changed document proceeds to the stale-document eviction/update path.

Expected behavior

A checkpoint should contain only documents committed by the current run, or store a reliable cursor plus the completed IDs for that run. Existing metadata IDs must not be interpreted as proof that those documents were processed before the interruption.

Impact

Resume can finish successfully, remove its checkpoint, and leave stale search results without any error or warning.

[lyonzin]

Thanks for the audit — the CHECKPOINT_IDS reproduction (['already_processed', 'changed_but_not_reached', 'unchanged'] + UNPROCESSED_CHANGED_DOC_SKIPPED True) was decisive.

Fixed in v4.8.3 (merged via #169, released as v4.8.3).

The fix follows your first proposed approach: _init_reindex_tracking now creates a per-run tracking["committed_this_run"] set. _commit_indexed_doc adds doc.id to that set only on successful processing. _maybe_persist_checkpoint serializes list(tracking["committed_this_run"]) instead of list(self._indexed_docs.keys()).

Resume preserves prior-run committed IDs by folding resume_state["doc_ids"] into the initial committed_this_run set, so multiple interrupted resumes still converge without losing what previous runs actually processed.

Regression coverage in tests/test_v483_hotfix.py::test_gh162_tracking_only_records_committed_this_run — asserts the set contains only IDs the run committed, not the pre-existing metadata keys.