Skip to content

fix: record terminal job state with a fresh context so timed-out jobs don't hang in processing - #33

Open
rajarshidattapy wants to merge 1 commit into
Anakin-Inc:masterfrom
rajarshidattapy:fix/persist-terminal-state-after-timeout
Open

fix: record terminal job state with a fresh context so timed-out jobs don't hang in processing#33
rajarshidattapy wants to merge 1 commit into
Anakin-Inc:masterfrom
rajarshidattapy:fix/persist-terminal-state-after-timeout

Conversation

@rajarshidattapy

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #32.

Processor.handleFailure wrote the failure using the same context that had just expired.
database/sql rejects any query whose context is already done — it never reaches the
driver — so with the PostgreSQL store the row kept status = 'processing' forever:
pollers on GET /v1/url-scraper/:id never terminated, POST /v1/scrape burned its whole
wait window for a job that was already dead, and one timed-out child pinned an entire
batch. The same thing happened to every in-flight job on shutdown, since bgCancel()
cancels their parent context before pool.Drain().

Adds persistCtx, which drops the job's deadline via context.WithoutCancel and applies
its own 5s bound, and uses it at the three sites that record a job's final state:

  • handleFailureStoreResult, UpdateStatus, UpdateParentBatchStatus
  • the success path in processScrapeJobStoreResult, UpdateCompleted (same hazard
    when a scrape finishes just as the deadline lands)
  • the parent-batch update in ProcessJob

The processing write at job start still uses the job context — it is not terminal state,
and a job whose context is already dead has nothing to report yet.

The startup reaper the issue mentions as a "would also" is deliberately left out. Marking
every processing row failed at boot is wrong as soon as two servers share a database; it
needs row ownership or a heartbeat to be correct, which is a larger design than this fix.
Worth a follow-up for rows stranded by a hard crash — the shutdown path is covered here.

How to test

  1. cd server && go test ./internal/processor/ — two new tests cover the report's
    scenarios: a scrape that outlives JOB_TIMEOUT, and a job cancelled mid-flight the way
    shutdown cancels it. They use a store stub that rejects done contexts the way
    database/sql does; MemoryStore ignores ctx entirely, which is why this bug only
    ever appeared with DATABASE_URL set.
  2. Confirm they are real regression tests — check out processor.go from master and
    re-run:
    --- FAIL: TestProcessJob_RecordsFailureAfterJobTimeout
    job status = "processing", want "failed"
    expected the failure reason to be persisted
    --- FAIL: TestProcessJob_RecordsFailureAfterCancellation
    job status = "processing", want "failed"
  3. End to end against Postgres: JOB_TIMEOUT=5 make up, submit an async job for a URL
    that takes ~45s, wait, then curl http://localhost:8080/v1/url-scraper/<JOB_ID>
    status is now failed with error: "all handlers failed: ... context deadline exceeded" instead of processing forever.
    SELECT status, error, completed_at FROM scrape_requests WHERE id = '<JOB_ID>';
    shows the row updated.
  4. Shutdown path: submit a long-running job and docker compose restart server while it
    is in flight — the row comes back failed, not processing.

Checklist

  • Tests pass (cd server && go test ./...)
  • No breaking changes to existing API endpoints
  • Documentation updated (if applicable) — none needed; failed is already the
    documented terminal status, jobs just never reached it

…cenarios

- Introduced a new context management for persisting job states after timeouts or cancellations.
- Added `ctxStrictStore` to simulate strict context behavior in tests.
- Created tests to verify job failure recording after timeout and cancellation events.
- Updated `Processor` methods to utilize the new context for database operations related to job status and results.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Jobs are stuck in processing forever when they hit JOB_TIMEOUT — the failure is written with the already-expired job context

1 participant