Fix waiting-file jobs in Cloudflare Tunnel self-hosted deployments / 修复Cloudflare Tunnel自托管部署中等待文件的任务

[thinktraveller]

Summary

This PR fixes self-hosted LocalStack SNS subscription confirmation when the public
S3 endpoint differs from the internal LocalStack endpoint.

In Cloudflare Tunnel deployments, S3_ENDPOINT_URL may point to a public HTTPS
domain such as https://storage-knowhere.example.com, while the app should still
trust LocalStack's internal confirmation URL such as http://localstack:4566.
Previously, the app only used S3_ENDPOINT_URL to validate SNS subscription
confirmation URLs, causing LocalStack SNS subscriptions to remain in
PendingConfirmation and uploaded files to stay stuck in waiting-file.

Changes

• Add a patched subscription_service.py that validates LocalStack SNS
  confirmation URLs against internal self-hosted endpoints.
• Prefer SELF_HOSTED_AWS_ENDPOINT_URL and S3_PRIVATE_DOMAIN before falling back
  to S3_ENDPOINT_URL.
• Mount the patched service file into the app container through compose.yaml.

Root Cause

When using a public storage domain for uploads, the app rejects LocalStack SNS
subscription confirmation URLs because their host is localstack, while
S3_ENDPOINT_URL points to the public Cloudflare domain.

As a result:

• SNS subscription remains PendingConfirmation=true
• S3 upload events are not delivered to the API service
• parse jobs remain stuck in waiting-file
• jobs may eventually fail with JOB_EXPIRED

Verification

Verified on a self-hosted deployment with Cloudflare Tunnel:

• API service starts successfully
• LocalStack SNS subscription is confirmed
• SNS subscription shows PendingConfirmation=false
• test S3 upload event reaches /v1/internal/s3-events
• reuploaded document is parsed successfully
• job status changes to done
• result object is generated under results/<job_id>.zip

Notes

This is currently implemented as a deployment-level patch mounted through Docker
Compose. A future upstream-native fix could move the endpoint selection logic
directly into the application source instead of using an overlay patch.

摘要

本PR修复了当公共S3端点与内部LocalStack端点不同时，自托管LocalStack SNS订阅确认的问题。

在Cloudflare Tunnel部署中，S3_ENDPOINT_URL可能指向公共HTTPS域名（如https://storage-knowhere.example.com），而应用仍应信任LocalStack的内部确认URL（如http://localstack:4566）。此前，应用仅使用S3_ENDPOINT_URL验证SNS订阅确认URL，导致LocalStack SNS订阅停留在PendingConfirmation状态，上传文件也卡在waiting-file状态。

变更内容

• 新增修补版subscription_service.py，使LocalStack SNS确认URL能针对内部自托管端点进行验证
• 优先使用SELF_HOSTED_AWS_ENDPOINT_URL和S3_PRIVATE_DOMAIN，再回退到S3_ENDPOINT_URL
• 通过compose.yaml将修补后的服务文件挂载到应用容器中

根本原因

当使用公共存储域名进行上传时，应用会拒绝LocalStack SNS订阅确认URL，因为其主机名为localstack，而S3_ENDPOINT_URL指向公共Cloudflare域名。

结果导致：

• SNS订阅保持PendingConfirmation=true状态
• S3上传事件无法传递给API服务
• 解析任务卡在waiting-file状态
• 任务最终可能因JOB_EXPIRED而失败

验证结果

已在使用Cloudflare Tunnel的自托管部署中验证：

• API服务成功启动
• LocalStack SNS订阅已确认
• SNS订阅显示PendingConfirmation=false
• 测试S3上传事件成功到达/v1/internal/s3-events
• 重新上传的文档解析成功
• 任务状态变更为done
• 在results/<job_id>.zip路径下生成结果对象

备注

当前方案通过Docker Compose挂载的部署级补丁实现。未来的上游原生修复可将端点选择逻辑直接移入应用程序源码，而非使用覆盖补丁。

[Copilot]

Pull request overview

This PR introduces a deployment-level patch to unblock self-hosted LocalStack SNS subscription confirmation in Cloudflare Tunnel setups where the public S3 endpoint differs from the internal LocalStack endpoint, preventing jobs from getting stuck in waiting-file.

Changes:

• Add a patched subscription_service.py that rewrites/validates LocalStack SNS confirmation URLs using internal self-hosted endpoint settings.
• Prefer SELF_HOSTED_AWS_ENDPOINT_URL and S3_PRIVATE_DOMAIN ahead of S3_ENDPOINT_URL when determining the trusted LocalStack endpoint.
• Mount the patched service into the app container via compose.yaml.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
patches/knowhere/subscription_service.py	Adds LocalStack-aware URL rewrite/validation and a custom resolution path for SNS subscription confirmation.
compose.yaml	Mounts the patched service file into the Knowhere container to override the upstream implementation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.