Skip to content

fix: return batch children in creation order, and start jobs pending, in MemoryStore - #63

Open
GautamSharma99 wants to merge 2 commits into
Anakin-Inc:masterfrom
GautamSharma99:fix/memorystore-child-order
Open

fix: return batch children in creation order, and start jobs pending, in MemoryStore#63
GautamSharma99 wants to merge 2 commits into
Anakin-Inc:masterfrom
GautamSharma99:fix/memorystore-child-order

Conversation

@GautamSharma99

Copy link
Copy Markdown

Fixes #56.

Two commits. The first is the reported bug; the second is a second divergence the new contract suite found, kept separate so it can be dropped independently if you'd rather see it on its own.

1. Child ordering (4429aa4)

MemoryStore.GetChildJobs ranged over the job map and returned whatever order Go's randomised iteration produced, while PostgresStore orders by created_at. GetBatchJob turns that order into each result's index, so in the DB-less mode the README leads with, polling the same batch twice returned the results — and the response's own urls list — in different orders.

MemoryStore now records an insertion sequence and sorts on it.

Why the sequence rather than CreatedAt, which the issue suggested: time.Now().UTC() strips the monotonic clock reading, so the stored timestamp is wall-clock only. That makes it subject to NTP adjustment, and on platforms with coarse resolution it ties outright for batch children created in a tight loop — which is exactly how they're created (scraper.go). The sequence is assigned inside the same critical section as the insert, so it is creation order, with no tie-break needed.

Storing it needs a small wrapper around the record, which is why the other methods change shape. None of their behaviour does.

2. Jobs now start pending (860b25c)

internal/store had no tests at all, which is how the ordering divergence survived. This adds the shared contract suite I offered in the issue — the same assertions run against both implementations.

It immediately found a second one. PostgresStore.CreateJob inserts the literal 'pending'; MemoryStore copied the caller's JobRecord, and no caller sets Status, so jobs were stored with "".

That is not cosmetic:

  • UpdateParentBatchStatus counts children by status, and "" matches neither the pending nor the processing branch
  • so pending == total is false, pending > 0 || processing > 0 is false, and the parent falls through to the completed branch with CompletedAt stamped
  • GetBatchJob derives the response status the same way

In DB-less mode a batch therefore reported completed the moment it was submitted, before any child had run. GET /v1/url-scraper/:id also returned "status": "" for a queued job.

MemoryStore now sets pending on create, matching Postgres — which hardcodes it and ignores the caller's value, so the contract is "a new job starts pending" in both.

Verification

Both new tests fail on the previous implementation:

--- FAIL: TestMemoryStore_ChildOrderIsStableWithoutTimestampGaps
    memory_test.go:54: attempt 0: children[1].URL = "https://example.com/2", want "https://example.com/1"
--- FAIL: TestContract_GetChildJobsReturnsCreationOrder
    store_contract_test.go:161: attempt 3: children[0].URL = "https://example.com/2", want "https://example.com/1" — order is not creation order

The contract suite was run against a real database, not just the memory backend:

$ docker compose up postgres -d
$ TEST_DATABASE_URL="postgres://postgres:postgres@localhost:5432/anakinscraper?sslmode=disable" \
    go test -race -v -run TestContract ./internal/store/

--- PASS: TestContract_CreateAndGetJob
    --- PASS: TestContract_CreateAndGetJob/memory
    --- PASS: TestContract_CreateAndGetJob/postgres
--- PASS: TestContract_GetChildJobsReturnsCreationOrder
    --- PASS: TestContract_GetChildJobsReturnsCreationOrder/memory
    --- PASS: TestContract_GetChildJobsReturnsCreationOrder/postgres
... 9 contract tests, both backends
ok      .../internal/store  1.650s

Without TEST_DATABASE_URL the Postgres half skips with a logged note, so go test ./... stays green on a machine with no database and CI is unaffected. If you'd like it wired into CI, adding a postgres:16-alpine service to .github/workflows/ci.yml and setting the env var is a couple of lines — happy to add that here or separately.

Coverage: internal/store 0% → 50.9%.

Full CI parity locally — gofmt -l clean, go build ./..., go vet ./..., go test -race ./... all pass.

Notes for the reviewer

GetChildJobs ranged over the job map and returned whatever order Go's
randomised map iteration produced. PostgresStore orders by created_at, so the
two implementations behind one JobStore interface disagreed.

GetBatchJob turns that order into each result's index, so in the DB-less mode
the README leads with, polling the same batch twice returned the results — and
the response's own urls list — in different orders, with index identifying
nothing.

MemoryStore now records an insertion sequence and sorts on it. The sequence
rather than CreatedAt, because time.Now().UTC() strips the monotonic clock
reading: the stored timestamp is wall-clock only, so it is subject to NTP
adjustment and, on platforms with coarse resolution, ties outright for batch
children created in a tight loop. The sequence is assigned inside the same
critical section as the insert, so it is exactly creation order.

Storing it needs a wrapper around the record, which is why the other methods
change shape; none of their behaviour does.

internal/store had no tests. Adds the regression test — eight children created
back to back with no delay, order asserted over 50 calls — plus coverage for
copy-on-read and concurrent access. The regression test fails on the previous
implementation at the first attempt.
…contract suite

The two JobStore implementations had no shared tests, which is how the child
ordering divergence survived. This adds a contract suite that runs the same
assertions against both: MemoryStore always, PostgresStore when
TEST_DATABASE_URL points at a database with the scripts/init-db.sql schema.

The suite immediately found a second divergence. PostgresStore.CreateJob
inserts the literal 'pending' for status; MemoryStore copied the caller's
JobRecord, and no caller sets Status, so jobs were stored with "".

That is not cosmetic. UpdateParentBatchStatus counts children by status, and ""
matches neither the pending nor the processing branch — so `pending == total` is
false, `pending > 0 || processing > 0` is false, and the parent falls through to
the completed branch with CompletedAt stamped. GetBatchJob derives the response
status the same way. In DB-less mode a batch therefore reported completed the
moment it was submitted, before any child had run.

MemoryStore now sets pending on create, matching Postgres, which hardcodes it
and ignores the caller's value for the same reason.

Verified against both backends:

    TEST_DATABASE_URL=... go test -race ./internal/store/
    ok  .../internal/store  1.650s

Coverage: internal/store 0% -> 50.9%.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch results are returned in random order with meaningless index values when running without DATABASE_URL

1 participant