fix: return batch children in creation order, and start jobs pending, in MemoryStore - #63
Open
GautamSharma99 wants to merge 2 commits into
Open
Conversation
GetChildJobs ranged over the job map and returned whatever order Go's randomised map iteration produced. PostgresStore orders by created_at, so the two implementations behind one JobStore interface disagreed. GetBatchJob turns that order into each result's index, so in the DB-less mode the README leads with, polling the same batch twice returned the results — and the response's own urls list — in different orders, with index identifying nothing. MemoryStore now records an insertion sequence and sorts on it. The sequence rather than CreatedAt, because time.Now().UTC() strips the monotonic clock reading: the stored timestamp is wall-clock only, so it is subject to NTP adjustment and, on platforms with coarse resolution, ties outright for batch children created in a tight loop. The sequence is assigned inside the same critical section as the insert, so it is exactly creation order. Storing it needs a wrapper around the record, which is why the other methods change shape; none of their behaviour does. internal/store had no tests. Adds the regression test — eight children created back to back with no delay, order asserted over 50 calls — plus coverage for copy-on-read and concurrent access. The regression test fails on the previous implementation at the first attempt.
…contract suite
The two JobStore implementations had no shared tests, which is how the child
ordering divergence survived. This adds a contract suite that runs the same
assertions against both: MemoryStore always, PostgresStore when
TEST_DATABASE_URL points at a database with the scripts/init-db.sql schema.
The suite immediately found a second divergence. PostgresStore.CreateJob
inserts the literal 'pending' for status; MemoryStore copied the caller's
JobRecord, and no caller sets Status, so jobs were stored with "".
That is not cosmetic. UpdateParentBatchStatus counts children by status, and ""
matches neither the pending nor the processing branch — so `pending == total` is
false, `pending > 0 || processing > 0` is false, and the parent falls through to
the completed branch with CompletedAt stamped. GetBatchJob derives the response
status the same way. In DB-less mode a batch therefore reported completed the
moment it was submitted, before any child had run.
MemoryStore now sets pending on create, matching Postgres, which hardcodes it
and ignores the caller's value for the same reason.
Verified against both backends:
TEST_DATABASE_URL=... go test -race ./internal/store/
ok .../internal/store 1.650s
Coverage: internal/store 0% -> 50.9%.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #56.
Two commits. The first is the reported bug; the second is a second divergence the new contract suite found, kept separate so it can be dropped independently if you'd rather see it on its own.
1. Child ordering (
4429aa4)MemoryStore.GetChildJobsranged over the job map and returned whatever order Go's randomised iteration produced, whilePostgresStoreorders bycreated_at.GetBatchJobturns that order into each result'sindex, so in the DB-less mode the README leads with, polling the same batch twice returned the results — and the response's ownurlslist — in different orders.MemoryStore now records an insertion sequence and sorts on it.
Why the sequence rather than
CreatedAt, which the issue suggested:time.Now().UTC()strips the monotonic clock reading, so the stored timestamp is wall-clock only. That makes it subject to NTP adjustment, and on platforms with coarse resolution it ties outright for batch children created in a tight loop — which is exactly how they're created (scraper.go). The sequence is assigned inside the same critical section as the insert, so it is creation order, with no tie-break needed.Storing it needs a small wrapper around the record, which is why the other methods change shape. None of their behaviour does.
2. Jobs now start
pending(860b25c)internal/storehad no tests at all, which is how the ordering divergence survived. This adds the shared contract suite I offered in the issue — the same assertions run against both implementations.It immediately found a second one.
PostgresStore.CreateJobinserts the literal'pending';MemoryStorecopied the caller'sJobRecord, and no caller setsStatus, so jobs were stored with"".That is not cosmetic:
UpdateParentBatchStatuscounts children by status, and""matches neither the pending nor the processing branchpending == totalis false,pending > 0 || processing > 0is false, and the parent falls through to the completed branch withCompletedAtstampedGetBatchJobderives the response status the same wayIn DB-less mode a batch therefore reported
completedthe moment it was submitted, before any child had run.GET /v1/url-scraper/:idalso returned"status": ""for a queued job.MemoryStore now sets
pendingon create, matching Postgres — which hardcodes it and ignores the caller's value, so the contract is "a new job starts pending" in both.Verification
Both new tests fail on the previous implementation:
The contract suite was run against a real database, not just the memory backend:
Without
TEST_DATABASE_URLthe Postgres half skips with a logged note, sogo test ./...stays green on a machine with no database and CI is unaffected. If you'd like it wired into CI, adding apostgres:16-alpineservice to.github/workflows/ci.ymland setting the env var is a couple of lines — happy to add that here or separately.Coverage:
internal/store0% → 50.9%.Full CI parity locally —
gofmt -lclean,go build ./...,go vet ./...,go test -race ./...all pass.Notes for the reviewer
## UnreleasedCHANGELOG heading. Happy to rebase whichever lands last.MemoryStorenever evicts jobs — the default zero-config mode grows unbounded until OOM #57 (MemoryStore never evicts) is still open and lands in the same file. The contract suite here would give that change a safety net; say the word and I'll follow up.