feat(jobs): filter job listing by namespace and add job deletion

[SusannaShu]

Why

A multi-tenant proxy (our keyboard-backend) fans every end user's uploads into a single Knowhere account, separated only by namespace. Two gaps made that proxy unable to own its own view of a user's files:

• GET /v1/jobs had no way to scope results to one namespace, so the proxy had to keep its own job-id list and issue one GET /v1/jobs/{id} per file.
• There was no way to remove a job at all. DELETE /v1/jobs/{id} returned 500 Method not allowed, so callers had no delete path and had to hide jobs client-side, which meant they reappeared on the next sync.

What changed

Namespace filtering

GET /v1/jobs, GET /v1/jobs/page, and the v2 equivalents accept a namespace query parameter, matched against job_metadata ->> 'namespace'.

Three behaviors worth calling out:

• The filter is normalized the same way creation normalizes it. Job creation runs the namespace through normalize_retrieval_namespace, so an unnormalized filter value would silently match zero rows. ?namespace= and ?namespace=%20%20 now mean the default namespace.
• Omitting the parameter still lists every namespace. Absent means "no filter", not "default".
• Jobs predating the metadata key are treated as default. Rows whose job_metadata has no namespace key are matched by the default filter rather than being invisible to every query.

This also fixes a latent break in the same code path: JobReadModel.list_jobs_for_user did not accept namespace, so read_service forwarding it raised TypeError on every list call. That method now accepts and forwards it.

Job deletion

DELETE /v1/jobs/{job_id} and DELETE /v2/jobs/{job_id}.

Deletion is soft, via a new jobs.deleted_at column:

• The row is retained, so in-flight workers, billing records, and state audit history are untouched. Deleting a running job is safe — the worker finishes normally, the job is simply no longer listed.
• Soft-deleted jobs are invisible to GET /jobs, GET /jobs/{id}, and confirm-upload. check_job_permission raises NotFound after the ownership check, so deletion never leaks whether a job exists.
• Deleting twice returns 404 the second time, same as an unknown job.

The linked document is archived by default so the content leaves retrieval, with ?archive_document=false to opt out. Archiving is skipped when another live job still targets the same document, so re-parsing a document and then deleting the older job does not wipe the current content.

Response:

{
  "job_id": "job_abc",
  "deleted": true,
  "document_id": "doc_xyz",
  "document_archived": true
}

Indexes

Two partial indexes for the new query shapes, both WHERE deleted_at IS NULL:

• idx_job_user_active_created_at on (user_id, created_at) — the default listing.
• idx_job_user_namespace on (user_id, (job_metadata ->> 'namespace')) — namespace-scoped listing.

Migration

fce1d2e3f4a5, on top of fbe1c2d3e4f5. Adds the nullable deleted_at column and the two indexes. No backfill needed — existing rows are active by definition.

The repo already had two alembic heads before this change (f0d85d209e68 and fbe1c2d3e4f5); this branch does not change that count, and tests/deploys use upgrade heads. Worth a separate cleanup, out of scope here.

Tests

• tests/contract/test_job_delete_contract.py — delete hides the job from GET and list, other jobs survive, double delete is 404, unknown job is 404, archive_document=false is honored.
• tests/contract/test_job_read_contract.py — namespace filter isolates tenants, omitting it lists everything, blank means default.
• tests/migrations/test_schema_contract.py — deleted_at and both indexes exist after upgrade heads.

Verification status

make lint and make typecheck both pass clean. Route registration, the generated namespace SQL, and the alembic revision graph were verified directly.

The contract tests were not executed locally — this machine has no libpq or Postgres server binaries, so pytest_postgresql cannot start, and that blocks collection of the whole suite (pre-existing, not introduced here). They need a CI run to confirm.

The delete service's control flow was exercised against stubbed repository and document services, covering all four paths: archive, skip-archive-when-shared, already-deleted → NotFound, and wrong-owner → PermissionDenied.

Compatibility

Additive. New optional query parameter, new endpoint, new nullable column. No existing response shape or default behavior changes.

🤖 Generated with Claude Code