fix: bound MemoryStore retention so DB-less mode stops growing until OOM

[GautamSharma99]

Fixes #57.

Stacked on #63. This branch is based on fix/memorystore-child-order, because both change internal/store/memory.go and building on master would have meant a hand-merge. The first two commits below belong to #63; the change to review here is 363de45. Once #63 merges this diff collapses to that single commit automatically. Happy to rebase onto master instead if you'd rather take them in the other order.

The problem

MemoryStore never removed anything — no eviction, no TTL, no cap, and no Delete on the interface. What accumulates is not small: each completed job retains the serialised result, carrying the raw HTML (up to the 10 MB read limit in the HTTP handler), the cleaned HTML and the markdown, so roughly two to three copies of every page body. A server scraping 500 KB pages at one job/second accumulates on the order of a gigabyte an hour, with no backpressure and no log line before the OOM kill.

This is the backend selected whenever DATABASE_URL is unset — the zero-config mode the README leads with.

The fix

A retention cap: MEMORY_STORE_MAX_JOBS, default 500, evicting oldest-first once exceeded. 0 or less restores the previous unbounded behaviour for anyone relying on it.

Eviction shrinks to 90% of the cap rather than exactly to it, so it runs once per ~10% of capacity instead of on every insert once the store is full.

I implemented the count cap (option 1 in the issue) and not the TTL sweep (option 2), even though I'd said I leaned toward both. The cap alone gives the hard bound that fixes the bug; a TTL only changes when memory is released while already under the cap, and it needs a background goroutine plus shutdown plumbing through main.go. Easy to add later against this structure if you want idle servers to release sooner. Option 3 (dropping HTML from the retained payload) changes what GET /v1/url-scraper/:id returns, so it felt like a product decision rather than a bug fix.

Two properties the policy has to preserve

Both are tested, and both are the "care" the issue asked for:

• Batch families are evicted as a unit. Dropping a parent while its children remain would strand them; dropping children while the parent remains would silently shrink the batch response.
• In-flight jobs are never evicted. A worker is still writing to them and a sync request may be polling, so eviction would surface as not found. The cap is therefore a target, not a hard ceiling — the overshoot is bounded by WORKER_POOL_SIZE + JOB_BUFFER_SIZE, and that tradeoff is documented at the function.

Families are ordered by their most recently inserted member, so the newest work is evicted last. That's what keeps a just-completed job from being dropped before its sync caller reads it.

Verification

The three eviction tests fail on the previous implementation:

--- FAIL: TestMemoryStore_EvictsOldestOnceOverLimit
    retained 500 jobs, want at most the 20 limit
    oldest job survived; eviction is not oldest-first
--- FAIL: TestMemoryStore_EvictionReleasesStoredResults
    retained 13107200 bytes of results, want at most 1310720
--- FAIL: TestMemoryStore_NeverEvictsInFlightJobs
    retained 70 jobs after the in-flight ones completed, want at most 10

The payload test is the one that matters — it asserts the bug is fixed in bytes, not just in entry count.

Because the real risk here is evicting a job out from under a live sync request, I also checked it end to end rather than only in unit tests. Server in memory mode with the cap set absurdly low, against a local origin:

$ MEMORY_STORE_MAX_JOBS=5 TELEMETRY=off ./server
$ # 60 consecutive POST /v1/scrape
200s: 60   non-200: 0
eviction log lines: 28
{"msg":"memory store evicted oldest jobs","evicted":2,"retained":4,"max":5}

60/60 succeeded across 28 eviction cycles at a cap of 5, so the ordering holds under far more churn than any real deployment.

Coverage: internal/store 50.9% → 59.7%, internal/config stays at 96.4% with the new var covered.

Full CI parity locally — gofmt -l clean, go build ./..., go vet ./..., go test -race ./... all pass.

Notes for the reviewer

• NewMemoryStore() keeps its signature and now defaults to DefaultMaxJobs; NewMemoryStoreWithLimit(n) is the explicit form. Adding a parameter to the existing constructor would have broken the call site in fix: preserve target HTTP status so blocked proxies take the severe penalty #59.
• internal/config now imports internal/store for DefaultMaxJobs, so the default lives in one place. No cycle — store imports nothing from config.
• Documented in .env.example and the README config table.
• Expected trivial conflict with fix: preserve target HTTP status so blocked proxies take the severe penalty #59/fix: test the shipping sync-timeout clamp, and fix an overflow it hid #60/fix: apply domain config isEnabled, priority and requestTimeoutMs #61/fix: copy proxy scores under the lock instead of sharing live pointers #62 on the ## Unreleased CHANGELOG heading.