Skip to content

feat(player-info): wire dispatch buffer execution gate - #141

Merged
ChechiDev merged 2 commits into
mainfrom
feat/player-info-buffered-mode-integration
Aug 2, 2026
Merged

feat(player-info): wire dispatch buffer execution gate#141
ChechiDev merged 2 commits into
mainfrom
feat/player-info-buffered-mode-integration

Conversation

@ChechiDev

@ChechiDev ChechiDev commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Scope

PR 6 of the player_info buffered-mode series — partial PR6. Integrates the `RateLimitGate` execution gate and `CandidateProducer` / `BoundedCandidateBuffer` dispatch pipeline into the buffered player_info worker. Adds controlled cooldown + readiness-probe recovery, gate-after-claim race handling, dead-engine teardown safety, and explicit producer pause when gate is closed.

`WarmBrowserPool` is NOT integrated in this PR. See Warm pool status section below.

Branch

`feat/player-info-buffered-mode-integration` → `main`
Head: `055c285`

Relationship to merged PRs

Changed files

File Change
`infrastructure/browser/gate.py` NEW — `RateLimitGate`: cooldown+probe recovery, `mark_engine_ready()`, `cancel_recovery()`, `shutdown()`, `_MAX_PROBE_BACKOFF_SECS` constant
`infrastructure/browser/dispatch.py` +`wait_fn` param to `CandidateProducer` — explicit gate pause
`config/settings.py` +3 gate settings fields
`infrastructure/persistence/repositories/player_info_queue.py` +`release_to_pending()` with `execution_options(synchronize_session=False)`
`scripts/scrape_player_info.py` Gate integration, mode logging, `mark_engine_ready` after warmup, `cancel_recovery` in all teardown paths, `probe_fn` for `WarmableEngine`, `wait_fn` wired to producer, `shutdown()` in finally
`tests/unit/infrastructure/browser/test_rate_limit_gate.py` NEW — 19 unit tests
`tests/unit/infrastructure/player_info/test_mode_selection.py` NEW — 10 unit tests
`tests/unit/infrastructure/player_info/init.py` NEW — package init
`tests/unit/infrastructure/browser/test_dispatch_buffer.py` Replaced 2 placeholder tests; added producer wait_fn test
`tests/integration/test_player_info_queue_release.py` NEW — 6 integration tests (including DONE/FAILED no-op cases)
`docs/operations/player_info_buffered_mode_soak.md` NEW — 5-stage manual soak procedure

Feature flag behavior

`SCRAPING__PLAYER_INFO_DISPATCH_BUFFER_ENABLED=false` (default) → direct mode, no gate, no producer, no buffer created.

`SCRAPING__PLAYER_INFO_DISPATCH_BUFFER_ENABLED=true` → buffered mode, `RateLimitGate` created from settings, shared across all workers, `CandidateProducer` started with `wait_fn=gate.wait_if_closed`.

`SCRAPING__PLAYER_INFO_WARM_POOL_ENABLED` flag exists in settings (default `false`) but is not wired into the buffered execution path in this PR.

Mutual exclusion

Strict `if/else` in `main()` — only one mode runs per process.

Dispatch buffer integration

Buffered mode uses `CandidateProducer` + `BoundedCandidateBuffer`. Producer has an explicit `wait_fn=gate.wait_if_closed` that pauses it when the gate is closed. Workers consume from the buffer via `buffer.get()` and claim via `claim_by_id()` at handoff — no direct `claim_next()` DB poll.

Gate safety invariants

  • `cancel_recovery()` does NOT reopen the gate. It only cancels the stale recovery task. Gate stays CLOSED.
  • `mark_engine_ready()` is the only path that reopens the gate. Called after `on_browser_ready()`/`warmup()` succeeds.
  • `cancel_recovery()` is called in all three engine teardown paths: outer engine-startup failure, inner unexpected error from `on_browser_ready`/`_run_buffered_loop`, and `CooldownRequired` path.
  • `shutdown()` cancels any in-flight recovery task on clean process exit without changing gate state.

Gate-after-claim race handling

If gate closes between `claim_by_id()` and navigation, `release_to_pending()` transitions IN_PROGRESS → PENDING without incrementing `retry_count`. Only player_info IN_PROGRESS rows are affected; DONE, FAILED, and PENDING rows are untouched (verified by integration tests).

Warm pool status — NOT INTEGRATED

`WarmBrowserPool` (PR #138 scaffold) is not used in the buffered execution path. Workers in buffered mode still call `self._build_engine()` independently — each worker owns its own `PydollEngine` instance.

Why not integrated in this PR: Integration requires replacing `run_buffered()`'s entire outer lifecycle loop with `WorkerSlot.run()`, adapting `CooldownRequired` semantics (no equivalent in `WorkerSlot`), threading `on_browser_ready`/`mark_engine_ready` into the slot post-warmup step, rewiring gate `cancel_recovery()` calls into the slot's exception path, and resolving a protocol mismatch (`BrowserSlotEngine` exposes only `start/warmup/close`; `_run_buffered_loop` requires `navigate/wait_for_challenge`). Estimated 100–200 lines of structural change. This is architectural work beyond the gate-integration scope of this PR.

This PR is a partial PR6. It delivers: dispatch buffer wiring + rate-limit gate + cooldown + readiness reopening + producer pause + all gate safety invariants + soak docs. It does NOT deliver: warm browser pool wiring.

Security notes

  • No cookies, CDP tokens, WebSocket URLs, session data, or credentials are logged
  • Probe exceptions log only `type(_exc).name` — no stack trace
  • `gate.py` has zero `infrastructure.persistence` or `config` imports (verified by lint-imports)

PostgreSQL impact

`release_to_pending()` issues one `UPDATE ... RETURNING id` per gate-after-claim race event. No new indexes or migrations required.

Test evidence

tests/unit/infrastructure/browser/test_rate_limit_gate.py     19 passed
tests/unit/infrastructure/player_info/test_mode_selection.py  10 passed
tests/unit/infrastructure/browser/test_dispatch_buffer.py     29 passed
tests/integration/test_player_info_queue_release.py            6 passed
Full suite: 801 passed, coverage 82.81%

Quality gates

ruff check .   CLEAN
flake8 .       CLEAN
mypy --strict  Success: 188 source files, 0 errors
lint-imports   4 contracts KEPT, 0 broken
pytest         801 passed, 82.81% (threshold 80%)
pip-audit      No known vulnerabilities

Soak procedure

`docs/operations/player_info_buffered_mode_soak.md` — 5-stage manual soak with stop criteria, SQL snapshots, and rollback instructions.

Unresolved risks

  • WarmBrowserPool not wired — buffered mode workers each own their own engine; warm pool deferred to a future PR
  • `recover_failed()` (pre-existing) missing `locked_at = NULL` in recovery SQL — not in scope
  • `ScrapeStatus.value` vs Postgres enum label mismatch (pre-existing) — not in scope

PR 7 status

PR 7 has not been started.

Merge status

This PR has not been merged and has not been configured for auto-merge.

- Add RateLimitGate (infrastructure/browser/gate.py): cooldown+probe
  recovery, idempotent signal_rate_limit(), cancel_recovery() for
  dead-engine teardown paths
- Add release_to_pending() to PlayerInfoQueueRepository: IN_PROGRESS →
  PENDING without retry penalty (gate-after-claim race handler)
- Wire gate into _run_buffered_loop: wait_if_closed() before navigation,
  gate-after-claim race release, signal on RateLimitError with probe_fn,
  cancel_recovery() in both engine teardown paths
- Add 3 gate config fields to ScrapingSettings (cooldown, probe timeout,
  max probe attempts)
- Add direct/buffered mode logging in main()
- Add 16 unit tests for RateLimitGate, 10 for mode selection, 4
  integration tests for release_to_pending()
- Add manual soak procedure (docs/operations/player_info_buffered_mode_soak.md)
- cancel_recovery() no longer blindly reopens gate; gate stays CLOSED
  until new engine proves readiness via mark_engine_ready()
- Add mark_engine_ready(): called after on_browser_ready()/warmup()
  succeeds; also cancels any residual stale recovery task
- Add shutdown(): cancels recovery task on clean process exit without
  changing gate state; called in main() finally block
- Add _MAX_PROBE_BACKOFF_SECS = 300.0 module-level constant
- Add cancel_recovery() to inner except block in run_buffered() so stale
  probes are cancelled when on_browser_ready/loop throws unexpectedly
- Fix probe_fn for non-WarmableEngine: pass probe_fn=None instead of
  returning True blindly (gate reopens after cooldown, not immediately)
- Pass wait_fn=_gate.wait_if_closed to CandidateProducer so producer
  explicitly pauses while gate is closed (not just via backpressure)
- Add execution_options(synchronize_session=False) to release_to_pending
  UPDATE to prevent identity-map stale-read trap
- Add test: wait_if_closed blocks while gate is closed and unblocks on
  mark_engine_ready
- Add tests: producer respects wait_fn; cancel_recovery leaves gate
  closed; mark_engine_ready opens gate; shutdown is idempotent
- Add integration tests: release_to_pending is noop for DONE and FAILED
  rows
@ChechiDev ChechiDev changed the title feat(player-info): integrate buffered mode execution gate feat(player-info): wire dispatch buffer execution gate Aug 2, 2026
@ChechiDev
ChechiDev merged commit 0adc551 into main Aug 2, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant