Annotation reply jobs can overlap per app and remain stuck in waiting

[csurong]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

main (7aba539, 2026-08-12)

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

1. Run the API and Celery worker from current main.
2. For the same app, call AppAnnotationService.enable_app_annotation(...) twice before the first task finishes.
3. Observe that both requests receive different job IDs and both enqueue enable_annotation_reply_task.
4. While an enable task is in progress, call disable_app_annotation(...) for the same app. Enable and disable use separate app-level keys, so the two operations are not mutually exclusive.
5. Let a queued task run after the app was deleted, or disable annotation reply when its setting is already absent. These early-return paths do not write a terminal job status.

The service checks enable_app_annotation_{app_id} and disable_app_annotation_{app_id}, but it never writes either key when enqueueing a task. It only creates the per-job status key with setnx, without an expiration. The tasks delete the app-level keys even though those keys were never acquired.

Related: #37646 reported the missing app-level key write and was closed as stale after its proposed fixes were not merged. The same code path is still present on current main; the shared enable/disable exclusion and terminal-status paths are also affected.

✔️ Expected Behavior

Annotation reply mutations should have one application-scoped job lifecycle:

• only one enable or disable job may mutate an app at a time;
• repeated requests for the same action should reuse the in-flight job;
• a conflicting action should not start concurrently;
• enqueue failures and all worker exit paths should release the application reservation;
• every accepted job should transition from waiting to processing and then to completed or error.

❌ Actual Behavior

Multiple jobs can be queued for one app and can concurrently rebuild/delete the same annotation vector data while updating or deleting the same annotation setting. Some early-return paths leave the job status at waiting indefinitely, so the caller can continue polling a job that will never finish.