Port follow-up: migrate every worker into the new BFF/service architecture; BFF scalability

[Xore]

Follow-up to #1608 (per Xore). After the frontend/BFF/Rust-service port reaches parity:

Worker migration

Port every worker into the new architecture — inventory includes the dashboard's background loops (webhook alerts notifyLoop, scheduled PDFs reportScheduleLoop), es-results-importer, attacker-identity-worker, agent-intrusion-worker, and the payload/yara pipeline hooks. For each worker the target tier is chosen by the same serving-quality decision matrix used on #1608:

• Rust service tier for CPU/IO-heavy correlation, importing, scanning-adjacent work;
• BFF (Node) tier where the work is session/UI-adjacent (notifications surfaced to operators, report orchestration).

Each migration ships with parity tests against the current worker's observable outputs before the old one retires.

BFF scalability (hard requirement)

The BFF must scale properly and never get overwhelmed — it carries no compose resource limit:

• Node cluster mode / worker_threads for CPU-bound composition; the event loop stays free
• Backpressure end-to-end: bounded queues for fan-out to the Rust tier, 429/shed-load before collapse, per-route concurrency caps
• Streaming responses (no buffering whole payloads), keep-alive pooling to the Rust service
• Horizontal-ready: no in-process state that prevents running N BFF instances later (session in redis, prefetch cache shareable)
• Load-test gate: sustained SSE fan-out + burst navigation without p99 regression

🤖 Generated with Claude Code

[Xore]

Addition (per Xore): BFF workers must be movable to other docker hosts when needed — distributed horizontal scaling, not just more processes on one machine. Design consequences: workers are stateless containers configured purely by env; all coordination goes through network-reachable primitives (redis streams/queues over the honeynet/WireGuard network, never localhost sockets or shared host volumes); work items carry everything needed to process them; the BFF discovers workers via URLs/DNS, and any worker container can run on any host that reaches redis + the Rust service.

🤖 Generated with Claude Code

[Xore]

Embedded-worker inventory for the port (discovered while porting #1608)

The Go dashboard binary hosts these background producers, which must move into the new architecture (Rust service tier or standalone workers) before cutover — the UI/read tier is fully ported without them, but killing the Go process would stop these flows:

1. payloadInventoryLoop — scans /dionaea-lib + /cowrie-downloads + script payloads, writes dashboard-payload-inventory-v1 + dashboard-payload-bytes-v1. Real producer; new captures stop being inventoried without it.
2. notifyLoop (alertManager.observe) — scans campaign scores / YARA hits / pipeline problems, upserts dashboard-alert-state-v1, fires ALERT_WEBHOOK_URL with cooldown. The ported UI reads+acks this store; nothing would write it.
3. reportScheduleLoop — scheduled PDF generation into dashboard-generated-reports-v1 (report_pdf.go composer). Ported UI lists/streams the PDFs.
4. workbench orchestrator (workbench_orchestrator.go) — runs analysis recipes over the sandbox/ghidra/github-analysis request/results spool dirs; submissions UI depends on it.
5. canarytokens manager polling (fired-event handling stays with the adapter; mint/download already ported to the Rust tier).
6. in-memory refresh loops (es.refresh, ML/LLM/agent/auth caches) — obsolete by design: the Rust tier queries live with track_total_hits; nothing to port.
7. filebeat/log-tail health (source-health's host-local half) + services-adapter control socket — host-local observability/control, tier per matrix.

Cross-host scalability requirements (from Xore, recorded on #1608): BFF unlimited (no compose limit, horizontal-ready, SERVE_MODE=all|frontend|bff + BFF_INTERNAL_URL shipped), workers movable to other docker hosts (stateless env-configured, redis-only coordination), redis-backed shared payload cache replacing the in-process 15s TTL.

[Xore]

Progress update (branch port-foundation): the worker framework is in — WORKER_LOOPS env selects background roles on the apiary-backend image (stateless, ES-only state, movable across docker hosts), with alert-notifier ported and verified live, plus the redis-shared BFF payload cache (replicas share prefetch warmth via the oidc-sessions valkey). Compose has a backend-worker service behind the next profile.

Remaining here, in order: mounted worker role (submissions/spools/orchestrator + payload bytes mirror), reports scheduler + PDF composer, host-local notifier signals, then the standalone-worker decision matrix (payload-inventory / correlator / ip-enrichment / attacker-identity / agent-intrusion / ml / llm / auth-events workers) and a BFF load test + replica validation. Full current handoff on #1608. Test harness for new loops: port-tests/worker-notifier.sh is the template.

[Xore]

Progress update from #1612 (merged via #1613): the two dashboard-embedded worker loops explicitly named in this issue's inventory are now ported to the Rust tier —

• webhook alerts notifyLoop — the full signal set (campaigns, stale sensors, activity spikes, dead-letters, YARA matches, sandbox risk, LLM-analysis flags, OT command alerts, log-stream size, sandbox/ghidra/cape/github-analysis spool health, filebeat ingestion stats) is in backend-service/src/worker.rs's alert-notifier loop.
• scheduled PDFs reportScheduleLoop — now backend-service/src/worker.rs's reports-scheduler loop, driving the same generate pipeline as the manual API.

Not part of #1612's scope and still outstanding per this issue's own inventory: es-results-importer, attacker-identity-worker, agent-intrusion-worker, and the payload/yara pipeline hooks, plus the BFF scalability hard requirements (cluster mode/worker_threads, end-to-end backpressure, streaming responses, horizontal-readiness, the sustained-SSE-fan-out load-test gate) — none of that was touched by this pass. Leaving this open.

[Xore]

Further progress: all 4 explicitly-named workers in this issue's inventory are now ported and merged via #1614 (on top of #1613):

• es-results-importer → new backend-worker-importer compose service (needed root+DAC_READ_SEARCH, kept separate from the stateless backend-worker)
• attacker-identity-worker → backend-worker's attacker-identity loop (added real PIT/search_after pagination to es.rs along the way — a real gap, this crate had nothing past ES's default 10k-result window before)
• agent-intrusion-worker → backend-worker's agent-intrusion loop (all 12 detection rules + the bounded decoder + union-find correlation, with unit test coverage)
• payload/yara pipeline hooks (payload-inventory-worker) → backend-service's payload-inventory loop, reusing code already built in Port backend remainder: submissions, orchestrator, reports generation, host-local signals, settings admin APIs #1612

Also surfaced two more Go workers in the same family that weren't in this issue's named list but clearly belong to the same migration: ip-enrichment-worker (2,174 lines — produces the canonical_* fields several of the above already depend on) and correlator-worker (953 lines — campaigns-v1/attacker-clusters-v1). Starting those next.

Still open per this issue's own text: the BFF scalability hard requirements (cluster mode/worker_threads, backpressure, streaming, horizontal-readiness, load-test gate) — none of that touched yet.