Port #1610's remaining named workers to the Rust tier - #1614
Merged
Conversation
Ports analysis/es-results-importer/importer.py (591 lines Python) into the Rust backend-worker tier as a new src/es_importer.rs module and a new "es-results-importer" WORKER_LOOPS entry, faithfully preserving the full SOURCES table (14 explicit + 9 generated chunked sandbox-export sources), mtime-based dedup state, sha256(path)%SHARD_COUNT horizontal sharding, and the correctness-critical advance_state_after_bulk all-or-nothing-per-key semantics (unit tested). Added a bulk_index primitive to es.rs (elasticsearch crate's BulkOperation API), returning per-item failed ids rather than a single pass/fail flag, since a chunked/aggregate-sample file's multiple bulk operations sharing one dedup key need per-operation success to decide whether that key's mtime advances. Tier decision: a new compose service (backend-worker-importer), not folded into backend-worker or backend-service-mounted. This worker needs root + DAC_READ_SEARCH to read root-owned host result directories (same requirement the Python service's own compose block already has) and a persistent local dedup-state file, both of which conflict with backend-worker's stateless/unprivileged-by-design posture. The existing Python es-results-importer service stays running unchanged — this is a parity port behind the `next` profile, not a cutover, per #1610's own text ("each migration ships with parity tests... before the old one retires"). Note: src/main.rs and src/worker.rs also carry a small, independent `attacker-identity` WORKER_LOOPS registration from a concurrently landed port of a different #1610 worker (src/attacker_identity.rs, not part of this commit) — both additions are non-overlapping, non-conflicting one-liners/match-arms in the same shared files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports honeypot-attacker-identity-worker (1668 lines Go across main.go/fetch.go/identity.go/verdicts.go/es.go) into the Rust backend-service tier as src/attacker_identity.rs, wired as a new "attacker-identity" WORKER_LOOPS entry. Faithfully preserves the deterministic entity-merge algorithm: IPs merge into durable attackers-v1 entities only on >=2 shared signal categories (fingerprint/payload-sha256/credential-pair), never on one alone; entities are never deleted for going quiet; same-cycle transitive multi-way merges are folded together; deterministic sorted-IP iteration order for reproducible merge results. 7 unit tests port identity_test.go's coverage (single/double signal-count merge behavior, transitive same-cycle merges, absorbing a pre-existing entity, untouched entities aren't rewritten, credential-pair validation, signal intersection). Also ports the verdicts.go join (ghidra/sandbox/github-analysis/revdeck lookups per payload hash). Added a real correctness fix along the way, not just a straight port: the Go worker's PIT+search_after pagination (openPointInTime/docScrollAll) was missing from this crate entirely — every existing query here does a single bounded search, which would hard-fail past Elasticsearch's default 10,000 index.max_result_window once the event window or attackers-v1 exceeds it. Added Es::search_paginated (PIT-based, verified against the elasticsearch crate's actual OpenPointInTime/ClosePointInTime API) and used it for both the event fetch and the existing-entity load. Tier decision: this worker is pure ES (no host mounts, no local state), so it runs on the existing backend-worker service (already stateless-by-design) rather than a new one — unlike es-results-importer (concurrently landed), which needed its own service for root+DAC_READ_SEARCH and a persistent dedup state file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nt field Was sorting the "agent-campaigns" store entry (agent-intrusion-campaigns index) by "last_seen" -- that field doesn't exist in this index's document shape (see build_campaign_verdict in the Python worker: @timestamp/ campaign_id/start/end/severity/matched_categories/correlation_identifiers/ event_count/events, no last_seen). dashboard/agent_campaigns.go's own refreshAgentCampaigns sorts @timestamp:asc; matched that. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports honeypot-agent-intrusion-worker's Python pipeline (worker.py/ campaign_correlator.py/criticality_rules.py/decode_correlate.py) into the Rust backend-service tier as a new `agent-intrusion` WORKER_LOOPS entry on the existing backend-worker service (pure ES, no host mounts, no local state -- same shape as attacker-identity). - decode_correlate.rs: bounded, non-executing recursive decoder (base64 -> gzip/zlib -> single-byte-XOR-then-gzip) with a provenance chain, plus candidate-blob extraction and chunk-message parsing. ChunkCorrelator itself is not ported -- confirmed by grep that nothing in the live per-event rule pipeline instantiates it, only the free parse_chunk_message function is used live. - campaign_correlator.rs: union-find campaign correlation over shared identifiers (session/src_ip/host/channel), gated by a 72h window. - criticality_rules.rs: all 12 deterministic detection rules from ALL_RULES, campaign_severity scoring, and the two-stage campaign_breadcrumb_followed check. - agent_intrusion.rs: fetch/normalize/correlate/score/write orchestration against real ES data, using es.rs's search_paginated (added by the sibling attacker-identity-worker port) for the fetch past ES's default 10k result window. Every event's `raw` field is the whole raw sensor sub-document (source.honeypot / source.suricata.eve / bare _source) -- not this crate's usual flattened honeypot.canonical_* convention -- since the correlator and every rule read sensor-native field names directly, ported faithfully from the Python corpus/production fixtures. New dependencies: flate2 (gzip/zlib decompression, no std equivalent), data-encoding (base32, for the DNS-label exfil detection path). 89 unit tests total (up from 68), ported from test_campaign_correlator.py, test_criticality_rules.py (fixture-based cases; corpus.jsonl-dependent cases deliberately not ported), and test_decode_correlate.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Last of #1610's worker-migration inventory. Reuses payload_kind::classify_payload and payload_paths::payload_dirs (both already ported for submission/workbench use in #1612 phase 3a) rather than re-deriving either — the "yara" half of #1610's "payload/yara pipeline hooks" item was already covered by es-results-importer's aggregate_samples source, so this worker is the whole remaining scope. Every SCAN_INTERVAL (default 5m), walks PAYLOAD_DIRS, classifies each hash-named file, hex-dumps a preview, cross-directory-dedups by hash, and writes dashboard-payload-inventory-v1 (merge-onto-existing, never a blind overwrite -- dashboard-added GitHubAnalysisURL/GitHubAnalysisLabel fields must survive an unrelated rescan) plus dashboard-payload-bytes-v1 (the proactive counterpart to payload_bytes.rs's on-demand self-heal mirror). New WORKER_LOOPS entry "payload-inventory" lands on backend-service, not a new service -- it already mounts PAYLOAD_DIRS read-only for the bytes self-heal path, so this is a second consumer of an existing mount, not a new trust boundary. MIME detection is a coarse approximation from classify_payload's own category/magic-byte checks, not a full port of Go's http.DetectContentType (a WHATWG-sniffing-spec implementation this crate has no dependency for) -- informational-only field, not a security or routing decision. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dependency ReviewThe following issues were found:
License Issuesarcane/home/honeypot-dashboard/backend-service/Cargo.toml
OpenSSF Scorecard
Scanned Files
|
This was referenced Aug 18, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the four workers explicitly named in #1610's inventory (beyond
notifyLoop/reportScheduleLoop, already done via #1612/#1613) into the Rustbackend-servicetier, on top ofport-foundation(post-#1613).es-results-importer(Python → Rust) — mirrors ~14 host-side JSON/binary result sources (Ghidra/sandbox/GitHub-analysis/revdeck/CAPE results, Ghidra report/callgraph artifacts, sandbox export PCAPs/diagnostics, cowrie ttylogs, reporter metrics, YARA aggregate results) into ES via bulk indexing, with mtime-based dedup state and horizontal sharding preserved. Newbackend-worker-importercompose service — kept separate frombackend-service-mounted/backend-workerbecause it genuinely needsroot+DAC_READ_SEARCH(root-owned Ghidra/GitHub-analysis result dirs, a real requirement copied from the existing Python service's own compose block) plus a persistent local dedup-state file, both of which conflict withbackend-worker's stateless/unprivileged design.attacker-identity-worker(Go → Rust) — the ≥2-of-3-signal (fingerprint/payload-sha256/credential-pair) durable entity-merge algorithm, plus the 4-index verdict join (ghidra/sandbox/github-analysis/revdeck). Pure ES, no host mounts — added to the existingbackend-worker. Added a real point-in-time +search_afterpagination primitive toes.rs(Elasticsearch's default 10k-result window was a genuine gap — every existing query in this crate was a single bounded search).agent-intrusion-worker(Python → Rust) — union-find campaign correlation + 12 deterministic security-detection rules + a bounded non-executing recursive decoder (base64/gzip/zlib/single-byte-XOR with provenance chain) + severity scoring. Pure ES, no host mounts — added tobackend-worker.payload-inventory-worker(Go → Rust) — payload directory discovery/classification/inventory, reusingpayload_kind::classify_payloadandpayload_pathsalready built in Port backend remainder: submissions, orchestrator, reports generation, host-local signals, settings admin APIs #1612 rather than re-porting them. Slots directly intobackend-service's existingWORKER_LOOPS+ already-mounted payload directories — no new infrastructure needed.Also fixed one real pre-existing bug found while researching:
stores.rs's genericagent-campaignspassthrough sorted on alast_seenfield that doesn't exist in that document shape; now sorts@timestamp, matchingdashboard/agent_campaigns.go's own query.Not done (per #1610's own remaining scope, not touched by this PR): the BFF scalability hard requirements (Node cluster mode/worker_threads, end-to-end backpressure, streaming responses, horizontal-readiness, sustained-SSE-fan-out load-test gate). The old Python/Go workers stay running unchanged — this is a parity port, not a cutover, per #1610's own text ("each migration ships with parity tests... before the old one retires").
Test plan
cargo buildclean on every commit (no new warnings beyond one pre-existing unrelatedcanarytokens.rswarning)cargo test— 94/94 passing across all four workers' ported test suites (deterministic algorithms — union-find merge, entity resolution, all 12 detection rules, the bounded decoder, merge-not-overwrite inventory logic — got real unit test coverage, not just a compile check)docker compose --profile next configresolves all new/changed services cleanly🤖 Generated with Claude Code