fix: record terminal job state with a fresh context so timed-out jobs don't hang in processing - #33
Open
rajarshidattapy wants to merge 1 commit into
Conversation
…cenarios - Introduced a new context management for persisting job states after timeouts or cancellations. - Added `ctxStrictStore` to simulate strict context behavior in tests. - Created tests to verify job failure recording after timeout and cancellation events. - Updated `Processor` methods to utilize the new context for database operations related to job status and results.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #32.
Processor.handleFailurewrote the failure using the same context that had just expired.database/sqlrejects any query whose context is already done — it never reaches thedriver — so with the PostgreSQL store the row kept
status = 'processing'forever:pollers on
GET /v1/url-scraper/:idnever terminated,POST /v1/scrapeburned its wholewait window for a job that was already dead, and one timed-out child pinned an entire
batch. The same thing happened to every in-flight job on shutdown, since
bgCancel()cancels their parent context before
pool.Drain().Adds
persistCtx, which drops the job's deadline viacontext.WithoutCanceland appliesits own 5s bound, and uses it at the three sites that record a job's final state:
handleFailure—StoreResult,UpdateStatus,UpdateParentBatchStatusprocessScrapeJob—StoreResult,UpdateCompleted(same hazardwhen a scrape finishes just as the deadline lands)
ProcessJobThe
processingwrite at job start still uses the job context — it is not terminal state,and a job whose context is already dead has nothing to report yet.
The startup reaper the issue mentions as a "would also" is deliberately left out. Marking
every
processingrow failed at boot is wrong as soon as two servers share a database; itneeds row ownership or a heartbeat to be correct, which is a larger design than this fix.
Worth a follow-up for rows stranded by a hard crash — the shutdown path is covered here.
How to test
cd server && go test ./internal/processor/— two new tests cover the report'sscenarios: a scrape that outlives
JOB_TIMEOUT, and a job cancelled mid-flight the wayshutdown cancels it. They use a store stub that rejects done contexts the way
database/sqldoes;MemoryStoreignoresctxentirely, which is why this bug onlyever appeared with
DATABASE_URLset.processor.gofrommasterandre-run:
--- FAIL: TestProcessJob_RecordsFailureAfterJobTimeout
job status = "processing", want "failed"
expected the failure reason to be persisted
--- FAIL: TestProcessJob_RecordsFailureAfterCancellation
job status = "processing", want "failed"
JOB_TIMEOUT=5 make up, submit an async job for a URLthat takes ~45s, wait, then
curl http://localhost:8080/v1/url-scraper/<JOB_ID>—status is now
failedwitherror: "all handlers failed: ... context deadline exceeded"instead ofprocessingforever.SELECT status, error, completed_at FROM scrape_requests WHERE id = '<JOB_ID>';shows the row updated.
docker compose restart serverwhile itis in flight — the row comes back
failed, notprocessing.Checklist
cd server && go test ./...)failedis already thedocumented terminal status, jobs just never reached it