Full-stack ES store coverage audit — implement into the Rust/BFF tier (#1611) - #1617
Merged
Conversation
Ports dashboard/classify.go's ~1040-line per-sensor detail renderer (all 21 sensors: cowrie, dionaea, multipot, every conpot persona, endlessh, http-honeypot/api-honeypot, dicompot, sentrypeer, rdp-honeypot, cisco-asa-honeypot, hellpot, beelzebub, dnp3, dns-honeypot, citrix-honeypot, mailoney, canarytokens, tanner, galah, elasticpot, wordpot) into events.rs::row_from_source, shared by the list endpoint and the SSE live stream. Also adds brand-new suricata-v2-* detail (alert/anomaly matching classify.go's proven logic exactly; http/tls/ssh/smtp/dns/fileinfo are new — classify.go's legacy renderer skipped every suricata event_type except alert/anomaly outright). flow/netflow/stats stay excluded from the default events view (both list() and live.rs's stream), matching that same legacy posture -- 5700+/hour on a real deployment vs 50 alert/anomaly in the same window. Also populates EventRow.proto/port from honeypot.proto/honeypot.port/ honeypot.dst_port when network.protocol/destination.port are empty -- multipot/conpot/dnp3 only ever carry them under honeypot.*. Two corrections found during implementation, not blind translation: - The issue's own audit table describes beelzebub's honeypot.event as an object, but classify.go actually reads flat lowercase fields (protocol/username/password/command/path) -- confirmed against ip_enrichment/sensors.rs's own beelzebub promotion (ported earlier this session), which writes exactly those flat keys. Read the flat fields, matching the proven-live legacy code, not the raw upstream shape ip-enrichment-worker already normalizes away. - suricata's http.status is a JSON number in real eve.json, not a string -- caught by a unit test that would have silently rendered an empty status on every http detail line otherwise. 21 unit tests using literal fixtures shaped like the issue's own live- sampled field examples. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…1611 workstream C) honeypot.event ∈ {login, auth_attempt} missed most real auth activity per #1611's own live-cluster audit -- cowrie logs cowrie.login.success/failed on honeypot.eventid, never honeypot.event (1.26M docs, the largest auth source, entirely uncounted before this), and rdp-honeypot's auth rides on a plain "connect" event with a non-empty honeypot.username, no dedicated login/auth_attempt kind of its own. Added es::logins_filter(), a shared bool.should query fragment (any of: the original login/auth_attempt terms, cowrie's own eventid terms, the rdp-honeypot+username combination, or honeypot.canonical_user's mere existence -- a sensor-agnostic catch-all that improves automatically as ip-enrichment-worker's canonical_user promotion coverage grows) and used it in the three places that were duplicating the narrower filter: aggregates.rs (sources page), dashboard.rs (overview KPIs), reports_data.rs (report summaries). Verified search.rs's "Credentials (usernames)" honeypot.username aggregation does NOT need the same fix: cowrie's raw JSON only ever carries a username key on login events in the first place, so a plain terms-agg on that field is already correctly scoped. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…stream D) attck.go's own header deliberately deferred T1190 (web exploit) and the ICS pair (T0886/T1692.001) to "the ES-coverage round" -- this is that round. promote_attck_technique_fields now also reads raw per-persona fields directly (path, proto, app_function, request, event-kind), not just canonical_* fields, since conpot/dnp3 have no canonical-field promotion case at all and several bespoke sensors only promote a subset. - T0886/T1692.001: every conpot persona and dnp3 tag T0886 outright; T1692.001 escalates on a write/command (dnp3 app_function present, conpot non-empty request, or an explicit command/write event kind). - T1190: a request path combined with a query string, path-traversal/ wp-/php marker, or an explicit exploit-tagged event (CVE payload captures, cisco's method_pri, elasticpot's attack event, a real tanner detection past its "index" default). - T1110 gap: also fires on a login *attempt* regardless of whether credentials ended up non-empty (cowrie.login.failed with an empty password is the dominant case) -- matches the legacy dashboard's IsLogin || HasCredential gate, not credential-presence alone. Wired into all 7 enrich functions (the generic path plus every bespoke per-sensor one -- dionaea-incident/beelzebub/hellpot/galah/sentrypeer/ wordpot), not just the ones that already had canonical-field promotion. kill_chain.rs's three query-time supplemental filter aggregations (ics/ics_write/web_exploit -- narrower approximations forced by honeypot.*'s flattened mapping) are removed now that a real promoted tag covers them; technique_counts is a plain terms aggregation again. Historical documents predating this change won't carry the new tags until reindexed/rolled over -- not backfilled here. 10 new unit tests covering every new branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.eml bytes only ever lived on disk; the ES event stream carried
size/body_path but nothing queryable. es_importer.rs's new
mailoney_mail source mirrors .eml files into mailoney-mail-v1 keyed
by sha256(filename) (mailoney's own filenames carry no content-hash
guarantee, unlike cowrie's ttylog convention). New GET
/api/v1/mail/{session_id} does the two-step join: honeypot-v2-*'s
mail-body event for body_path, then mailoney-mail-v1 for the bytes,
parsed via the mail-parser crate rather than hand-rolled MIME/RFC5322
parsing since the input is attacker-controlled. Bodies surface as
plain text only and attachments as metadata (name/type/size/sha256)
without bytes, so the endpoint can't become an HTML-render sink or a
malware distribution point.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the audit's 10 concrete gaps:
- E.2: portbridge-v2-* OS/first-last-seen/ports-touched joined into
/api/v1/investigate/ip/{ip} (p0f's the only ground truth for which
ports a tunneled-sensor IP actually knocked on).
- E.3: document that dionaea-incidents-v1-* needs no separate endpoint
(same incidents already render via honeypot-v2-*, Workstream A).
- E.4: document that network.community_id (the Arkime pivot key) is
already in the record pane's full-doc clone.
- E.5: new GET /api/v1/ml-health — last retrain outcome per model.
- E.6: new GET /api/v1/gpu-queue — read-only queue visibility (the
audit's own example: 2 stuck queued jobs, previously invisible).
- E.7: document that /api/v1/reporter-stats already satisfies this.
- E.8: new GET /api/v1/revdeck/{sha} — was entirely missing, so a
revdeck error state rendered as a blank page.
- E.9: document that the auth-events store passthrough already
carries every field a frontend column addition would need.
- E.10: document worker-state indices as intentionally unsurfaced.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Telnet attackers embed NUL/control bytes in usernames/passwords; these were only ever cleaned display-side (dashboard.rs, investigate.rs), so ES terms aggregations, attackers-v1's credentials list, and search grouping all saw "root" and "root\0\0\0" as distinct buckets. Strip both raw NUL bytes and the literal "\x00" text form (same two shapes dashboard.rs's clean() already defends against) in ip_enrichment/canonical.rs's cowrie login promotion, before the enriched line is written and before canonical_user/canonical_pass are derived from it. Display-side cleanup stays in place for documents written before this fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
honeypot.* is mapped `flattened` for good reason (heterogeneous per-sensor shapes) but that means no stats/range aggs and no wildcard queries on its leaves. The geoip-honeypot ingest pipeline already copies a few hot fields into real typed ECS fields at ingest time (user.name, process.command_line, url.path); this extends that same mechanism to the two fields the audit called out: - held_ms -> a new top-level `held_ms` (long): the endlessh held-time chart no longer bucket-counts up to 10k raw hits client-side, it runs a real ES `range` aggregation. - password -> user.password (keyword), alongside the existing user.name, for future prefix/wildcard credential hunting. Proof of the wildcard side: /api/v1/search's "HTTP paths" group moved off flattened honeypot.path (prefix-only) onto url.path (wildcard, already pipeline-populated from honeypot.path/url) — genuine substring path hunting, not just prefix matching. Template/pipeline changes only affect newly-indexed documents; the datastream rolls daily so coverage is close to immediate. No backfill, same precedent as workstream D's technique promotion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements all 7 workstreams from #1611's live-cluster audit directly into
backend-service(the Rust/BFF tier), in the suggested order (A → C → B → D → E → G → F):event_detail.rs,detail_for()), wired into both/api/v1/eventsand the SSE live stream; suricata alert/http/tls/ssh/smtp/dns/fileinfo rows now render richly instead of empty, and flow/netflow/stats no longer swamp the default view.logins_filter()now covers cowrie'scowrie.login.{success,failed}(1.26M previously-uncounted docs) and rdp-honeypot's connect+username auth, alongside the existinglogin/auth_attemptvocabulary..emlfiles intomailoney-mail-v1(sha256(filename)-keyed); newGET /api/v1/mail/{session_id}joins session_id → body_path → bytes and parses via themail-parsercrate (bodies as plain text, attachments as metadata only — no auto-render, no bytes).kill_chain.rs's query-time supplemental filter aggregations are removed./api/v1/investigate/ip/{ip}, new/api/v1/ml-healthand/api/v1/gpu-queueendpoints, new/api/v1/revdeck/{sha}(previously missing entirely), plus documentation for items that turned out to already be covered (dionaea incidents, Arkime community_id, reporter-stats, auth-events columns, worker-state indices).ip_enrichment/canonical.rs), not just display-side, so ES terms aggs stop splitting identical creds into separate buckets.held_ms→ a reallongfield andpassword→user.password(keyword). Proof: the endlessh histogram is now a genuine ESrangeaggregation (was client-side bucketing of up to 10k raw hits), and/api/v1/search's "HTTP paths" group does real substring wildcard matching againsturl.path(was prefix-only on the flattened field).Test plan
cargo buildandcargo build --releaseclean (only a pre-existing unrelated warning)cargo test— 162 passed, 1 ignored (a scratch/manual-inspection test), 0 failedelasticsearch-setup.sh:bash -nsyntax check clean; both the ingest pipeline body and thehoneypot-events-v2template extracted and validated as well-formed JSON; Painless script brace/paren balance verified around both edit sitesdocker compose configvalidates cleanly with the newMAILONEY_MAIL_DIRmount/env onbackend-worker-importer🤖 Generated with Claude Code