diff --git a/TODO.md b/TODO.md index 9a191cc6..a090be20 100644 --- a/TODO.md +++ b/TODO.md @@ -75,7 +75,7 @@ Replaced manual per-emitter field coordination with SecurityEvent intermediate r - [x] **P0** Loop 3 blind-review follow-up / up-to-10 assessment loop — completed the requested up-to-10 loop run. Final loop eval passed at 94/100 and rendered probes found zero DNS response-accounting contradictions, zero Sysmon parent-create-after-visible-parent-termination cases, and zero eCAR post-termination references. Blind reviewers still scored synthetic across all four roles (avg synthetic confidence 79.5/100, avg realism 71/100). Top remaining root-cause finding: Sysmon Event 5 can carry an EventData `UtcTime` earlier than later Event 7 module-load telemetry for the same ProcessGuid, even when XML System `TimeCreated` ordering looks normalized. Next fixes should also address all-zero Sysmon `TerminalSessionId`, SYSTEM subject-domain rendering, IP CN-only public-CA X.509 records, DNS tunnel/web scan regularity, and source-specific collection imperfection profiles. - [x] Harden web scan preset `max_effective_rate` validation to prevent overlay-driven generation crashes or hangs. - [x] Aardvark beacon timing vulnerability fix — verified HEAD still routed generic beacons through DNS-tunnel pacing, restored exact periodic beacon timing, added regression coverage, and ran targeted tests plus Ruff checks. -- [ ] **IN PROGRESS** **P0** Loop 10 continuation / second up-to-10 assessment run — ignore "source coverage too perfect" style findings for now because source-specific missingness/coverage variance is already deferred to the imperfect-observation/profile TODO items. Active fixes should prioritize concrete source-native contradictions from loop 10: Sysmon Event 5 EventData `UtcTime` before later same-ProcessGuid telemetry, all-zero Sysmon `TerminalSessionId`, SYSTEM subject-domain rendering, public-CA IP CN-only X.509 records, DNS PID disagreement, DNS tunnel regularity, and web scan regularity. Continue looped assessment until no P0/P1/P2 findings remain after excluding deferred source-coverage findings, or until scores appear to show true regressions, plateau, or diminishing returns. +- [x] **SUPERSEDED** **P0** Loop 10 continuation / second up-to-10 assessment run — superseded by later assessment loops through the mid/high 90s and the current Post-Loop 95 roadmap. The remaining active issues from this era are now tracked as narrower web/session, Zeek timing, eCAR variance, imperfect observation, and statistical-polish TODOs. Loop 20 fix pass: stabilized repeated Zeek certificate hashes, ordered Zeek certificate file rows after SSL analyzer rows, suppressed static ASA NAT xlate churn, ordered same-second systemd lifecycle syslog rows, and added SCP receiver-side Linux/file artifacts. Verified with full `uv run pytest -v`, Ruff, and `eforge validate-config`. Loop 21 fix pass: preserved caller-pinned successful TLS handshakes, carried explicit POST body sizes through proxy egress, emitted visible early DHCP renewals for storyline-DHCP hosts, aligned DNS FLOW PID inference with the DNS Client service, localized SCP receiver-side eCAR actor IDs, and normalized remote-thread target image paths. Local loop scenario NAT now includes the DMZ so proxy egress renders with mapped public sources. Verified with full `uv run pytest -v`, Ruff, and `eforge validate-config`. @@ -87,15 +87,15 @@ Replaced manual per-emitter field coordination with SecurityEvent intermediate r - [x] Security hardening: bounded `workstation_lock.min_unlock_gap_seconds` with schema upper-bound validation and runtime clamping to prevent `timedelta` overflow from malicious local overlays. - [x] **P1** Security fix: prevent quadratic behavior in Linux `systemd-logind` session ID allocation for warm-up/pre-boot and same-second event bursts. -- [ ] **IN PROGRESS** **P0** Comprehensive correlated-event timing audit — after the current 78% synthetic blind-review fixes, perform a full audit similar to the emitter field-provenance audit, but focused on timing relationships between correlated events. Inventory all generated event clusters that are expected to correlate across Security/Sysmon/eCAR/Zeek/proxy/ASA/syslog/baseline/storyline outputs; identify where timestamps are source-native exact, realistically offset, impossible, or accidentally reordered; verify same-source ordering invariants such as process-create before process follow-on artifacts; verify cross-source offsets such as DNS before TCP, proxy client leg before proxy egress, firewall deny before absent downstream evidence, process create before WFP/Sysmon network evidence, auth before process, module/file/registry after process, and teardown after build/start; then implement root-cause fixes with tests and generated-output probes. -- [ ] **P0** Timing-audit baseline blind review follow-up — broad data-only baseline review of `/private/tmp/eforge-timing-baseline-output/data` scored **92% synthetic**. Critical findings: visible 4634 logoff followed by later same-host/same-LogonID process/lock/unlock activity; Sysmon Event 3/7/follow-on records preceding a later visible Event 1 for the same ProcessGuid. High findings: uniform 4624 `ElevatedToken=%%1842`; anonymous Type 3 logons use unrealistic domain/source/elevation fields. Medium findings: exact cross-source network timestamp reuse between Windows 5156, Zeek conn, and eCAR FLOW; proxy inspected HTTP paths remain domain-class inconsistent for update/vendor hosts. +- [x] **P0** Comprehensive correlated-event timing audit — completed through the timing-audit loop and subsequent iteration-test loops. The concrete findings were fixed or split into narrower follow-ups; remaining lower-return work now lives in the post-timing statistical polish, well-synced Zeek sensor timing, web/session realism, source-observation profile, and process-lifecycle architecture TODOs. +- [x] **P0** Timing-audit baseline blind review follow-up — completed or superseded by the same-session lifecycle timestamp guard, process follow-on timestamp guard, source-native Windows auth timing profiles, cross-source network timing profiles, proxy domain-class/path profiles, ElevatedToken variance, anonymous logon source-field fixes, and DC/auth semantic fixes. - [x] **P0** Same-session lifecycle ordering guard — fixed baseline scheduling so planned logoffs are known before user activity, user activity/lock-unlock events skip inactive sessions, activity updates session last-use time, and Windows Security rendering has a narrow 4634-after-dependent backstop. Generated-output probe on `/private/tmp/eforge-timing-loop3-output/data` found zero same-host/same-LogonID 4688/4801 events after visible 4634 logoff. - [x] **P0** Process follow-on timestamp guard — fixed process-dependent generation for module loads, registry noise, process access, and remote-thread evidence to clamp after process start and carry process start metadata where needed; added a Sysmon render-time ProcessGuid ordering backstop. Generated-output probe on `/private/tmp/eforge-timing-loop3-output/data` found zero Sysmon follow-on records before their Event 1 for the same ProcessGuid. - [x] **P1** Cross-source network timestamp profile — Windows Security 5156, Sysmon Event 3, and eCAR FLOW now use data-driven `source.*` timing profiles so host audit/EDR telemetry renders after the canonical wire event instead of tying Zeek conn timestamps exactly. Generated-output probe on `/private/tmp/eforge-timing-loop12-output/data` found 20,892 Zeek/eCAR common tuples with zero exact or millisecond timestamp matches. - [x] **P2** Proxy domain-class path/content profile completion — inspected proxy GET rows can still pair vendor/update user agents and hosts with generic browser paths such as `/login` or `/favicon.ico`. Current domain-class path selection and non-browser site-map exclusions cover the generated timing scenario; probe on `/private/tmp/eforge-timing-loop17-output/data` found zero infra/update/cert proxy GET rows with browser-generic paths, favicons, CSS, or webp assets. -- [ ] **P3** Time-window-aware blind-eval prompt/library — keep the bounded-window guidance in every reviewer prompt and codify it in the local eval helper/script once one exists, so reviewers do not treat missing pre-window initiators as impossible while still flagging visible initiators that occur after dependent events. +- [x] **P3** Time-window-aware blind-eval prompt/library — completed for the current workflow in the local `eforge-assess` skill briefing/prompt guidance. There is still no repo-side assessment helper to update, so this is no longer an open repository TODO. - [x] **P0** Sysmon transitive parent-create ordering guard — `_shift_process_creates_after_visible_parent()` now iterates until stable so cascading parent shifts in multi-level ProcessGuid chains cannot leave a child Event 1 before its shifted visible parent Event 1; added focused unit coverage for three-level chains. -- [ ] **P0** Follow-up timing blind review findings — follow-up data-only review of `/private/tmp/eforge-timing-loop3-output/data` scored **96% synthetic**. Critical: visible Sysmon Event 5 process termination followed by later Event 3/Event 7 telemetry for the same ProcessGuid. High: SSH syslog lifecycle entries for the same sshd PID/source tuple sorted as `Accepted` before `Connection from`; Linux `systemd-logind` session IDs mixed huge epoch-derived IDs with small sequential IDs. Medium: some accepted SSH logins lacked nearby visible session-open messages, likely same root as syslog second-level ordering. +- [x] **P0** Follow-up timing blind review findings — completed by the Sysmon Event 5 lifecycle floor, SSH syslog lifecycle/source-port ordering fixes, and systemd-logind session ID ordering fixes. - [x] **P1** Harden IDS DNS template validation/rendering against unsafe format fields — enforce that `dns_query_templates` only allow the `token` replacement field with sane format syntax/width and reject malformed or resource-exhausting templates in both config validation and runtime rendering. - [x] **P0** Harden timing profile overlay parsing in generation path — enforced safe integer parsing and range clamps for `relationships.*` and `windows_event_time.collision_spacing` so malformed `.eforge/config/activity/timing_profiles.yaml` values cannot crash generation or produce pathological timing offsets; added focused unit coverage for invalid-type and extreme-value overlays. - [x] **P0** Sysmon process-termination lifecycle guard — fixed Sysmon rendering so Event 5 process termination cannot appear before later visible telemetry for the same ProcessGuid; added focused unit coverage. @@ -107,11 +107,11 @@ Replaced manual per-emitter field coordination with SecurityEvent intermediate r - [x] **P1** Security/Sysmon logoff source-offset margin — follow-up review of `/private/tmp/eforge-timing-loop6-output/data` found visible Security 4634 logoffs tens of milliseconds before later Sysmon Event 1 process creates for the same LogonID, caused by Security's render-time lifecycle guard ignoring Sysmon source-native collection offsets. Fixed the generator logoff margin after session activity and widened the Windows Security 4634 render-time guard to clear downstream endpoint source offsets. Generated-output probe on `/private/tmp/eforge-timing-loop9-output/data` found zero Sysmon Event 1 records after a same-session visible Security 4634. - [x] **P0** IDS DNS alert/query contradiction — fixed Snort DNS alert/Zeek DNS payload disagreement by making DNS IDS signatures carry data-driven `dns_query_templates`, loading them with overlay support, and building a canonical `DnsContext` from the selected signature during IDS false-positive generation. Generated-output probe on `/private/tmp/eforge-timing-loop10-output/data` found 13 DNS IDS alerts and zero same-tuple Zeek query suffix mismatches. - [x] **P0** Timestamp compression bursts — added overlay-aware `timing_profiles.yaml` for causal/source-latency/teardown timing and Windows/Sysmon tied-timestamp collision spacing. Causal DNS/Kerberos/remote-thread/audit offsets and logoff margins now use the profile, and Windows/Sysmon render-time normalization keeps small tied clusters near-zero while spreading large tied clusters across seconds. Generated-output probe on `/private/tmp/eforge-timing-loop11-output/data` found worst 1ms windows of 7 Security events and 4 Sysmon events, down from earlier 174/106 event spikes. -- [ ] **P2** ASA static NAT teardown cadence — follow-up review of `/private/tmp/eforge-timing-loop6-output/data` found Cisco ASA static NAT translation records mechanically paired with immediate same-second connection teardown. Review ASA connection/NAT lifecycle timing as part of source-native network timing profiles. -- [ ] **P2** Deterministic cross-source offset fingerprints — follow-up review of `/private/tmp/eforge-timing-loop6-output/data` still found deterministic-looking Security/Sysmon/eCAR offsets. Fold this into the cross-source timestamp profile work so offsets are stable enough to correlate but varied enough to avoid source-fingerprint artifacts. +- [x] **P2** ASA static NAT teardown cadence — resolved: static NAT mappings no longer emit per-flow 305011/305012 xlate churn, dynamic NAT teardown uses connection duration and out-of-window suppression, and ASA regression coverage documents the behavior. +- [x] **SUPERSEDED** **P2** Deterministic cross-source offset fingerprints — superseded by source timing profiles and the narrower well-synced Zeek sensor timing TODO. Any remaining source-offset work should be handled there rather than as a duplicate broad item. - [x] **P0** Blind-review time-window context — every blind reviewer prompt should explicitly state that the dataset is an extract for a bounded collection window, so initiating events that occurred before the window can still have in-window echoes. Acceptable: processes, sessions, connections, leases, or logoffs whose creation/start event predates the extract and is therefore absent. Error: a visible initiating event for the same identifier appears later than its dependent event, such as a 4688 before a later same-host 4624 with the same LogonID. Added this guidance to `/eforge evaluate` for blind qualitative reviews and used it in the current blind-eval prompt. - [x] **P0** Source-native timestamp precision/rendering profiles — include rendered precision and per-source formatting in the timing audit. Known example: Windows Security XML now renders EVTX-like 100ns precision, but a blind review caught that the 7th fractional digit was previously always `0`. Audited the current renderers: Windows Security/Sysmon share EVTX-like 100ns formatting with deterministic 7th-digit variation, Zeek renders microsecond epoch seconds, eCAR renders integer milliseconds, and proxy/web/ASA/syslog render source-native second precision. Generated-output probe on `/private/tmp/eforge-timing-loop12-output/data` found 17,698 Windows Security timestamps with 7th-fractional-digit coverage across all digits 0-9. -- [ ] **P0** Windows auth/network timing examples to include in the audit — verify remote auth causality across Zeek/Windows/DC evidence: TCP connection start before 4625/4624, established/reset-after-payload state before any host auth result, successful remote 4624 source port matching the network tuple, 4771/4776 offset from member-host 4625 without sub-microsecond cross-host alignment, and audit/process events such as 1102 following the causative process while preserving source-native EventRecordID reset behavior. +- [x] **P0** Windows auth/network timing examples to include in the audit — completed as part of the Windows auth, DNS, proxy, ASA, and cross-source timing audit fixes. Remaining sensor calibration is tracked separately under the well-synced Zeek timing TODO. - [x] **P0** Loop 12 blind-review timing findings — data-only review of `/private/tmp/eforge-timing-loop12-output/data` scored **92% synthetic**. Fixed the critical Sysmon parent GUID ordering issue by preserving parent start time on `ProcessContext`, filtering future-started parents during parent resolution, and normalizing visible Sysmon parent/child Event 1 ordering. Fixed the high Zeek TLS lifetime issue by data-driving Zeek SSL/X.509 analyzer offsets, enforcing a completed-TLS conn duration floor, and suppressing ssl/x509 analyzer rows for partial TCP handshakes. Generated-output probe on `/private/tmp/eforge-timing-loop17-output/data` found zero Sysmon parent-after-child cases, zero Zeek ssl/x509 rows after conn end, and zero ssl/x509 rows on non-`SF` connections. Low follow-up remains: web access logs can rotate bot/script/browser User-Agents rapidly from the same public IP; track as web-client identity/session realism polish after P0/P1 timing fixes. - [x] **P1** Loop 17 blind-review follow-up — data-only review of `/private/tmp/eforge-timing-loop17-output/data` scored **87% synthetic**. Fixed actionable findings: Linux `systemd-logind` session IDs/PIDs now come from shared state and timestamp-derived per-host sequencing; ASA connection IDs no longer collide across adjacent-second high-volume bursts; generated SSH session `conn.json` rows include `orig_ip_bytes`/`resp_ip_bytes`; RSA-named TLS issuers no longer choose ECDSA certificate metadata and `eforge validate-config` rejects that combination. Prompt-calibration finding: Type 7 unlock 4624 events are not original session initiators, so reviewers should not treat pre-unlock process activity with the same LogonID as impossible when the original Type 2/10 session began before the extract. Generated-output probe on `/private/tmp/eforge-timing-loop19-output/data` found zero logind PID/order issues, zero ASA concurrent connection-ID reuse, zero SSH rows missing IP byte counters, zero RSA/ECDSA x509 mismatches, and zero process-before-later-non-Type7-4624 cases. - [x] **P1** Loop 19 blind-review follow-up — data-only review of `/private/tmp/eforge-timing-loop19-output/data` scored **92% synthetic**. Fixed the critical SSH disconnect timing issue by recording SSH transport close time on session state, making logoff/disconnect evidence wait for the latest session end marker, and reusing the same baseline SSH duration for conn.log and syslog disconnect timing. Fixed DNS TXT conn accounting so response-bearing TXT rows retain originator query payload. Fixed denied explicit-proxy CONNECT accounting so proxy-access rows use proxy denial byte/time scale rather than inherited tunnel byte counts. Quick tests, ruff, `eforge validate-config`, and generated-output probes on `/private/tmp/eforge-timing-loop22-output/data` passed for DNS TXT and denied CONNECT, and found zero matching SSH disconnect-before-conn-close cases. @@ -119,7 +119,7 @@ Replaced manual per-emitter field coordination with SecurityEvent intermediate r - [x] **P1** Loop 23 blind-review follow-up — data-only review of `/private/tmp/eforge-timing-loop29-output/data` scored **82% synthetic**. Fixed the critical SSH same-PID close mismatch by suppressing Linux `sshd` syslog close evidence when the backing SSH transport is stale or self-sourced, leaving bounded-window extracts with unmatched visible opens rather than impossible visible closes. Tightened the data-driven DNS tunnel RTT default from `0.04-1.5s` to `0.04-0.35s` while preserving overlay support and `eforge validate-config` range validation. Generated-output probe on `/private/tmp/eforge-timing-loop30-output/data` found zero SSH same-PID close/Zeek contradictions, zero tuple-bearing disconnect lines, and zero DNS tunnel RTTs above 1s. - [x] **P1** Loop 24 blind-review follow-up — data-only review of `/private/tmp/eforge-timing-loop30-output/data` scored **85% synthetic**. Fixed the critical eCAR FLOW after visible PROCESS/TERMINATE issue by making connection generation update the owning process last-activity marker, dropping stale non-system PID attribution when the process is no longer running, and protecting Windows PID 4/System in the seeded process map. Fixed the medium Zeek identical-timestamp burst with data-driven `source.zeek_conn_start` jitter in `timing_profiles.yaml`. Generated-output probe on `/private/tmp/eforge-timing-loop31-output/data` found zero eCAR FLOW-after-visible-terminate cases and reduced exact Zeek conn timestamp bursts to a max of 2 rows. - [x] **P2** Security hardening: validate `dns_tunnel_rtt` overlay shape/range at load/runtime boundary so malformed overlay values cannot crash generation. -- [ ] **P2** Post-timing-audit statistical polish — same-dataset blind reviews of `/private/tmp/eforge-timing-loop31-output/data` scored **30% synthetic** and **60% synthetic** with no Critical/High findings and no recurrence of the prior eCAR/SSH/DNS/Zeek burst timing defects. Remaining medium/low polish: Zeek `conn.json` still has repeated exact duration constants across unrelated rows (`0.8`, `2.0`, `0.01`), two nonlocal SSH failed-password syslog rows lacked exact matching Zeek SSH tuples, and stale `svc_deploy8` SSH failures occur on a very regular two-hour cadence. +- [x] **P2** Post-timing-audit statistical polish — fixed repeated generator-owned Zeek duration constants by jittering default-derived connection durations while preserving caller-authored values and DNS RTT locks. Added data-driven `auth_noise.yaml` scheduling for stale credential noise, replaced rigid modulo cadence with deterministic irregular intervals/skips/backoff, expanded service-account defaults, and made remote Linux failed-password syslog rows emit matching Zeek SSH tuples with the same source port. Verification passed: `eforge validate-config`, targeted unit coverage, Ruff check/format check, full normal pytest, and full slow-inclusive `uv run pytest -v --include-slow` (`3033 passed, 1 skipped`). - [x] **P1** Explicit proxy origin-error egress regression — in explicit forward-proxy mode, preserve proxy→origin egress emission for canonical origin HTTP 4xx/5xx responses (`cache_result=MISS`) and only short-circuit egress for proxy-generated deny/auth/gateway failure outcomes. - [x] Loop 26 Host/EDR blind authenticity report — analyzed only `scenarios/iteration-test/data` for endpoint evidence realism and saved the persona report under `scenarios/iteration-test/blind-test/loop-26/host-forensics-report.md`. - [x] Loop 30 iteration-test assessment continuation — fixed observation-anchored X.509 validity windows, Linux eCAR parent PID semantics, short-command process lifetimes, Linux-native eCAR failed-logon fields, Linux parent anchors for minimal state, and nmap probe side-effect leakage. Verification passed: focused tests, full normal `uv run pytest -v`, Ruff, `eforge validate-config`, regeneration, quantitative eval, and blind review. Blind scores: Threat Hunter 82, Detection 78, Network 78, Host/EDR 74 synthetic confidence. @@ -133,7 +133,7 @@ Replaced manual per-emitter field coordination with SecurityEvent intermediate r - [x] Loop 38 iteration-test assessment continuation — implemented root-cause fixes for the loop-37 P0/P1 findings before regeneration: Windows `explorer.exe` parent selection is anchored to the logon chain instead of browser/app history; extra syslog program entries now have validated selection weights and the suspicious web/proxy sudo-denial pool is rare and varied; Windows 4672 emission rates are data-driven per account class instead of always-on for every eligible service/machine/admin logon; package/update proxy domains are non-browser and path/content aware; and ASA ICMP faddr/gaddr/laddr rendering now follows inbound/outbound address roles. Verification passed: `eforge validate-config`, focused tests, full normal `uv run pytest -v` (2770 passed, 37 skipped), Ruff check, Ruff format check, regeneration, quantitative eval at 94/100, and blind review. Blind scores: Threat Hunter 76, Detection 78, Network 74, Host/EDR 72 synthetic confidence; average synthetic confidence 75.0. No early exit: reviewers converged on deeper fixable regularity issues rather than evidence that the loop-38 fixes made realism worse. - [x] Loop 39 iteration-test assessment continuation — implemented root-cause fixes for high-agreement loop-38 P1/P2 findings before regeneration: multi-sensor Zeek rendering now applies per-sensor observation variance for timings, durations, byte/packet counters, and HTTP body lengths instead of cloning rows across sensors; bash history treats `shred -u .bash_history` as destructive; Linux SSH/syslog baseline is server-scoped, rarer, and scenario-roster based instead of generic `root/admin/ubuntu` churn everywhere; extra sudo noise is lower-weight and no longer includes repeated root-to-root apt update; Sysmon registry Events 12/13 render `User`; ambient registry noise is lower volume and prefers dynamic key pools instead of repeatedly touching static Office/Winlogon/EventLog keys; and web-scan path selection now shuffles/skips between passes with a lower Nikto rate cap. Verification passed: `eforge validate-config`, focused tests, full normal `uv run pytest -v` (2777 passed, 37 skipped), Ruff check, Ruff format check, regeneration, quantitative eval at 94/100, and blind review. Blind scores: Threat Hunter 78, Detection 74, Network 88, Host/EDR 76 synthetic confidence; average synthetic confidence 79.0. No early exit: the score spike was caused by a concrete new P0 in the loop-39 Zeek observation-variance layer, not by plateau or a broad realism regression. - [x] Loop 40 iteration-test assessment continuation — implemented root-cause fixes for the loop-39 P0/P1/P2 findings before regeneration: Zeek multi-sensor observation variance now preserves `*_ip_bytes >= *_bytes + packet overhead` invariants; ICMP Zeek conn rows render source-native type/code ports and echo request/reply history; Zeek files.log transfer rows are delayed after their referenced connection start and share the referenced connection UID as the multi-sensor timing basis; Office reading-location registry `Datetime` values are materialized before the Event 13 timestamp; eCAR failed logons carry an `attempt_failed` session lifecycle qualifier; Sysmon ProcessGuid timestamp segments no longer expose tiny boot-relative counters; and the data-driven dsquery command pool now varies query targets and limits. Verification passed: `eforge validate-config`, focused regression tests, full normal `uv run pytest -v` (2781 passed, 37 skipped), Ruff check, Ruff format check, regeneration, quantitative eval at 93/100, and blind review. Blind scores: Threat Hunter 44, Detection Engineer 45, Network Forensics 42, Host/EDR 63 synthetic confidence; average synthetic confidence 48.5. Early exit triggered because average synthetic confidence is <=60. New prioritized findings: P1 DHCP lease/syslog timing and DNS/DHCP state conflicts; P1 source-native Windows 4625 subject semantics; P1 high-volume benign Sysmon Event 8 remote-thread noise; P1 eCAR parent/child process ordering; P1 proxy HTTPS app-log semantics exceeding Zeek tunnel visibility; P1 OS/user-agent drift for Linux hosts; P2 eCAR processless flow attribution; P2 same-interface ASA denies on perimeter firewall; P2 Sysmon ImageLoad metadata blanks; P3 Linux SSH session duplication/orphaning; P3 homogeneous TLS scanner cadence; P4 HTTP file metadata inferred from extension on redirects. -- [ ] **IN PROGRESS** Loop 41-60 iteration-test assessment continuation — run up to 20 additional loops from the `loop-40-checkpoint` baseline. The average synthetic confidence <=60 early exit is intentionally disabled for this continuation. Keep the other early exits: stop if no P0/P1/P2 findings remain, if substantial work reaches clear diminishing returns/plateau, or if fixes introduce actual realism regressions rather than merely surfacing deeper fixable issues. Start by fixing loop-40 P1/P2 root causes: DHCP/syslog state consistency, Windows 4625 subject semantics, benign Sysmon Event 8 volume, eCAR parent/child ordering and process attribution, proxy HTTPS inspection semantics, OS-aware user-agent selection, ASA same-interface perimeter leakage, and Sysmon ImageLoad metadata. +- [x] **SUPERSEDED** Loop 41-60 iteration-test assessment continuation — superseded by later completed assessment loops through the mid/high 90s. The durable work from this continuation is represented by the remaining statistical polish, source-observation, web/session, Zeek timing, and eCAR variance TODOs. - [x] Loop 42 realism fixes — aligned Kerberos PKINIT issuer names with scenario AD org, restricted DHCP baseline leases to DHCP-like client systems instead of static infrastructure, made Linux systemd-logind session IDs monotonic under out-of-order generation, and populated eCAR thread IDs where endpoint telemetry can derive a source thread. Verification passed: focused tests, full normal `uv run pytest -v`, Ruff, `eforge validate-config`, regeneration, quantitative eval at 93.4/100, and blind review. Blind scores: Threat Hunter 74 real confidence, Detection Engineer 78 synthetic confidence, Network Forensics 82 synthetic confidence, Host/EDR 72 synthetic confidence; average synthetic-equivalent confidence 64.5. - [x] Loop 43 realism fixes — addressed loop-42 concrete root causes: preserved Zeek cleartext HTTP body facts across multi-sensor observations and clamped conn byte counters to protocol body floors, enriched Zeek DHCP source-native lease fields, kept OCSP responder choice stable for a certificate identity, rendered scheduled-task SYSTEM principals as local authority instead of AD-domain users, and derived Sysmon hashes from host OS build when OS binary metadata differs. Verification passed: focused tests, `eforge validate-config`, full normal `uv run pytest -v`, Ruff check, Ruff format check, regeneration, quantitative eval at 93.3/100, and blind review. Blind scores: Threat Hunter 78, Detection Engineer 64, Network Forensics 76, Host/EDR 78 synthetic confidence; average synthetic confidence 74.0. - [x] Loop 44 realism fixes — addressed loop-43 concrete root causes: kept Windows Security EventRecordID order monotonic with rendered TimeCreated values, made scheduled-task SYSTEM XML use service-account semantics, bounded short-lived Windows utility lifetimes for gpresult/gpupdate/dsquery-like commands, dropped stale one-shot utility PID attribution for later network flows, and preserved Zeek same-flow payload bytes across multi-sensor observations while still varying source-native packet/IP counters. Verification passed: focused tests, `eforge validate-config`, full normal `uv run pytest -v`, Ruff check, Ruff format check, regeneration, quantitative eval at 94.2/100, generated-output probes, and blind review. Blind scores: Threat Hunter 72, Detection Engineer 86, Network Forensics 76, Host/EDR 82 synthetic confidence; average synthetic confidence 79.0. The higher confidence appears to reflect deeper concrete issues surfaced after the prior obvious tells were removed, not an early-exit regression. @@ -177,7 +177,7 @@ Replaced manual per-emitter field coordination with SecurityEvent intermediate r - [x] Loop 65 hard-probe follow-up fixes — fixed regenerated-output blockers before launching blind reviews: Linux UFW/DMZ background flow evidence now uses the public inbound address contract, failed TLS handshakes retain partial handshake byte accounting instead of no-payload service labels, and TLS connection duration is extended to cover delayed Zeek certificate analyzer rows. Verification passed: focused regression tests, full normal `uv run pytest -v` (`2861 passed, 37 skipped`), `uv run ruff check .`, `uv run ruff format --check .`, and `uv run eforge validate-config`. - [x] Loop 65 final regeneration and assessment — regenerated `scenarios/iteration-test/data`, saved verbose/JSON eval artifacts, hard-probe results, four bounded-window blind-review reports, scores, and final report under `blind-test/loop-65`. Automated eval was 93.19 across 52,625 records. Hard probes verified zero parent LogonID mismatches, zero unpublished private ASA inbound builds, zero Zeek files outside parent connection lifetimes, zero no-payload service labels, zero missing S0 responder byte fields, and zero TLS certificate depth-order inversions. Blind scores: Threat Hunter synthetic 74, Detection Engineer synthetic 84, Network Forensics synthetic 72, Host/EDR synthetic 76; average synthetic confidence 76.5. Final reported issues: Zeek S0 responder byte contradiction, eCAR actor-before-process ordering, multi-sensor Zeek flow cloning, missing SMB/RPC evidence for PsExec-style execution, repeated `userinit.exe` parenting of many `explorer.exe` shells, proxy request/UA drift, missing `Compress-Archive` file artifacts, narrative-polish labels, and thread ID distribution realism. - [x] Loop 66 documented issue fixes — fixed loop-65 hard/source-native findings for Zeek S0 responder byte accounting, eCAR actor-before-process source offsets, multi-sensor Zeek observation cloning, PsExec-style SMB/RPC causal evidence, proxy request/User-Agent preservation, interactive `userinit.exe`/`explorer.exe` lifecycle realism, and `Compress-Archive` file artifacts. Verified with focused regression tests, Ruff, and full `uv run pytest -q` (`2903 passed, 15 skipped`). -- [ ] **IN PROGRESS** Loops 67-76 iterative assessment run — commit the Loop 66 fixes, regenerate/evaluate/review the iteration-test data, prioritize the next concrete high-score-impact findings, and continue fix/regenerate/blind-review cycles unless early-exit criteria show true plateau, regression, or low-return subjective-only issues. +- [x] **SUPERSEDED** Loops 67-76 iterative assessment run — superseded by the later completed assessment runs through Loop 96. The durable work from this batch is captured in the remaining statistical polish, source-observation, web/session, Zeek timing, and eCAR variance TODOs. Loop 66 final panel after regenerated hard-probe-clean output scored 93.05 automated eval across 51,785 records, with blind synthetic-confidence scores: Threat Hunter 82, Detection Engineer 72, Network Forensics 73, Host/EDR 82 (average 77.25). Top Loop 67 targets are concrete source-native file-artifact bugs: dangling command-shell quote residue in `Compress-Archive` file telemetry and PSReadLine history artifacts from `cmd.exe`/noninteractive SYSTEM PowerShell; broader exact Security/Sysmon/eCAR process-coverage modeling remains the highest-leverage architectural follow-up. Loop 67 final panel after shell file-artifact fixes scored 93.62 automated eval across 52,248 records, with blind synthetic-confidence scores: Threat Hunter 72, Detection Engineer 72, Network Forensics 74, Host/EDR 78 (average 74.0). Hard probes verified zero dangling quote file paths, zero wrong-shell/noninteractive PSReadLine artifacts, and preserved `Compress-Archive` zip artifacts. Top Loop 68 targets are concrete source-native fingerprints: failed 4625 network logons must not show `IpAddress=-` with a nonzero `IpPort`, and TLS/X.509 certificate serials should preserve deterministic identity without making every serial exactly 32 hex characters. The Event 1102 `EventData` complaint is currently treated as a reviewer false positive because fields render under `UserData/LogFileCleared` by source contract and unit test. Loop 68 fix pass: Windows 4625 failed-logon rendering now suppresses `IpPort` whenever the source address is unavailable, preventing `IpAddress=-`/nonzero-port contradictions; TLS certificate serial generation now uses data-driven deterministic byte-length variation instead of fixed 128-bit serials, with config-schema/validator coverage. Verification passed: focused tests, related unit files (`108 passed`), `uv run eforge validate-config`, Ruff checks, and full normal `uv run pytest -q` (`2909 passed, 15 skipped`). @@ -238,10 +238,10 @@ Replaced manual per-emitter field coordination with SecurityEvent intermediate r - [x] **Slow-inclusive pytest verification for sprint stack** — `uv run pytest -v --include-slow` passed from the current stacked branch with `3017 passed, 1 skipped` in 1610.78s (0:26:50). No failures surfaced, so no code fixes were needed. - [x] **Loop 96 blind reviewer pass after sprint merges** — regenerated and evaluated `scenarios/iteration-test` from the merged sprint stack, then ran a blind expert-panel realism review against a neutral copy of the generated data only. Automated eval passed at 94.74 across 47,433 records; blind synthetic-confidence scores were Threat Hunter 76, Detection Engineer 68, Network Forensics 68, Host/EDR 76 (average 72.0, down from Loop 95's 78.5). The panel did not repeat the prior `systemd-logind` remove-before-new issue, whole-millisecond Zeek analyzer offset issue, or Windows scheduled-task/WinSxS/SearchProtocolHost source-native defects; top new findings were web application static response/session realism, too-complete source coverage/correlation, bounded cross-sensor Zeek skew, curated traffic/attack naming, and endpoint/eCAR uniformity. - [x] **P2** Scenario skill anti-curation guidance follow-up — Revised the dev scenario skill so attacker-controlled domains, service accounts, scheduled tasks, files, and process names blend into ordinary naming conventions without becoming semantic breadcrumbs that reveal the attack narrative. Verification: `uv run pytest tests/unit/test_install_skills.py -q --no-cov` passed (`30 passed`); the same focused test file also passed under the default coverage run, but that command failed the whole-repo coverage threshold because it intentionally ran only one test module. -- [ ] **P1** Web application response/session realism follow-up — Loop 96 found server-side web paths and static assets with implausibly variable response sizes plus weak human browsing session structure. Stabilize static/per-path response characteristics, model page-to-asset fanout and session continuity, and reduce one-off deep-path requests from generic external clients. -- [ ] **P1** Well-synced Zeek sensor timing follow-up — Loop 96 found matching `zeek-core`/`zeek-dmz` flows with a narrow 0.184-0.259s lag that looks modeled. Preserve the environment assumption that security sensors have good time sync: remove broad constant cross-sensor lag, model only tiny stable per-sensor clock error (microseconds to low milliseconds), add protocol/event-specific capture or analyzer variance where source-native, and allow ~200ms delays only for flows that traverse latency-inducing infrastructure such as proxying, WAN/VPN paths, TLS inspection, queueing, or async ingestion rather than uniformly across HTTP, DNS, SSH, SMB, and LDAP. -- [ ] **P2** Endpoint/eCAR baseline variance follow-up — Loop 96 found workstation eCAR category volumes and Linux process lifecycle evidence too uniform and complete. Add host/persona-specific variance, long-lived process state, benign unmatched artifacts, and more realistic endpoint observation gaps where source visibility permits. -- [ ] **Later architectural sprint: imperfect observation and source coverage** — defer the broad "too-complete telemetry" problem until after the sharper defects are gone. Model source-specific drop rates, ingestion delay, audit-policy gaps, endpoint coverage variance, and asymmetric Security/Sysmon/eCAR/Zeek visibility as a coherent observation/profile layer rather than one-off omissions. +- [x] **P1** Web application response/session realism follow-up — Added data-driven inbound `web_server` visitor profiles so human visitors consume `traffic_rates.web` as top-level actions, then fan out into required page assets/API calls through `site_maps.yaml`; crawler, health-check, API-client, and opportunistic-probe traffic now uses source-native configured request/status/User-Agent profiles. Static resource sizes are stable per host/path, human navigation and render fanout timing use `timing_profiles.yaml`, and docs/skill references now explain the budget and config ownership. Verification passed: focused web/timing/baseline tests (`107 passed, 1 skipped`), config-related tests (`64 passed`), `uv run eforge validate-config`, repo-wide Ruff checks/format checks, full normal `uv run pytest -q` (`3012 passed, 15 skipped`), and `git diff --check`. +- [x] **P1** Well-synced network sensor timing follow-up — Replaced hardcoded multi-sensor Zeek +/-400ms skew plus broad path delay with a validated `network_sensor_observation` timing profile. The default `well_synced` profile keeps stable per-sensor clock skew within +/-1.5ms and per-flow capture/path delay within 50-2000us while preserving canonical packet/byte truth unless source-native observation variance is explicitly enabled. Verification passed with focused Zeek/timing tests, `uv run eforge validate-config`, repo-wide Ruff checks/format checks, full normal `uv run pytest -q` (`3012 passed, 15 skipped`), and `git diff --check`. +- [ ] **DEFERRED with observation/source coverage architecture** **P2** Endpoint/eCAR baseline variance follow-up — Loop 96 found workstation eCAR category volumes and Linux process lifecycle evidence too uniform and complete. Defer with the broader observation/profile sprint so host/persona-specific variance, long-lived process state, benign unmatched artifacts, and realistic endpoint observation gaps are modeled coherently rather than as eCAR-only omissions. +- [ ] **Later architectural sprint: imperfect observation and source coverage** — defer the broad "too-complete telemetry" problem until after the sharper defects are gone. Model source-specific drop rates, ingestion delay, audit-policy gaps, endpoint coverage variance, and asymmetric Security/Sysmon/eCAR/Zeek visibility as a coherent observation/profile layer rather than one-off omissions. Bundle the related deferred items into this sprint: endpoint/eCAR baseline variance, source-specific process lifecycle completeness modeling, configurable cross-source evidence disagreement, per-host/source log coverage, and the host/activity profile items for per-entity artifact and volume variance. - [x] Full slow-suite regression cleanup after loop-65 merge — explicit-proxy storyline beacons now preserve authored hostname+destination IP pairs only when the storyline marks that pair as intentional, normal proxy-origin DNS resolution remains intact, and the parallel-generation LogonID assertion treats Type 7 unlock reuse as valid slice-of-time Windows behavior. Verified with targeted proxy/parallel tests, `uv run ruff check .`, `uv run ruff format --check .`, and `uv run pytest -v --include-slow` (`2875 passed, 23 skipped`). Detection Engineer blind review completed for the regenerated Loop 61 dataset at `scenarios/iteration-test/data`; reviewer verdict: Synthetic, 63/100 confidence. Main findings: one PROXY-01 sshd accepted-login lifecycle gap/self-source artifact and Windows 4648 explicit-credential caller PID/image provenance ambiguity around `WS-MCHEN-01`. @@ -268,7 +268,7 @@ Verification is complete: dedicated `tests/unit/test_world_model.py` coverage wa - [x] Windows process/Sysmon/eCAR blind-eval cleanup — fixed approved follow-up findings from the 82% synthetic blind eval: eCAR remote-thread `tgt_tid` now matches Sysmon Event 8 `NewThreadId`, Security 4689 avoids blank `SubjectLogonId` for system-owned process exits, process-create render timestamps have deterministic source offsets across Security/Sysmon/eCAR, eCAR `PROCESS/OPEN` uses explicit target fields instead of overloading `command_line`, eCAR module-load timing no longer exactly ties process creation, and failed logons carry explicit eCAR failure outcome/status fields. Focused tests, full unit tests, full non-slow tests, Ruff, and `eforge validate-config` passed. -- [ ] Windows process/Sysmon/eCAR blind-eval follow-up from 88% synthetic review — remaining review item is remote-thread join ambiguity when repeated source/target PID pairs appear. Process lifecycle joins are deferred to the source-specific telemetry coverage/profile design below. The 5156 PID/image attribution, 4688 PID 4 parent fallback, Sysmon/eCAR module-load correlation, and process-access provenance findings were fixed in the canonical emitter field provenance item. +- [x] **SUPERSEDED** Windows process/Sysmon/eCAR blind-eval follow-up from 88% synthetic review — superseded by the broader source-specific process lifecycle completeness modeling TODO and the newer eCAR baseline variance TODO. Concrete field-provenance/process-path fixes landed during the canonical emitter provenance and process lifecycle work. - [x] Canonical emitter field provenance fixes — implemented the approved emitter audit fixes: Windows 5156 process attribution resolves from canonical process state, Sysmon/eCAR share canonical image-load data, Sysmon Event 10 and eCAR `PROCESS/OPEN` use `ProcessAccessContext`, Sysmon parent GUIDs use parent process start time, user process parentage no longer falls back to PID 4, Zeek dhcp.log receives DHCP option-domain data when available, bash history no longer carries non-native `exit_code`, ASA/proxy context-owned fields are honored, and deferred source-specific process lifecycle completeness modeling is documented below. @@ -276,13 +276,13 @@ Verification is complete: dedicated `tests/unit/test_world_model.py` coverage wa - [x] Canonical emitter field provenance blind-review follow-up — targeted blind review scored the focused dataset 88% synthetic. Fixed confirmed actionable findings: Windows 5156 no longer inherits a storyline process from the wrong host/OS, unresolved non-system WFP process images are suppressed instead of rendering `-`, PID 4 WFP fallback renders as `System`, internal DNS preserves scenario IP→FQDN registrations before generated aliases, and `_ldap._tcp...` NXDOMAIN companion probes use SRV. Regenerated-output probes passed. The proxy CONNECT+GET finding was a prompt artifact because the blind prompt omitted the current TLS-inspection assumption; rerun the blind review with that assumption stated. -- [ ] Canonical emitter field provenance blind-review remaining findings from 78% synthetic review — fix Sysmon intra-log causality where file/registry/module follow-on events can render before Event 1 for the same process GUID/PID; normalize bare storyline executable names (e.g. `powershell.exe`) to OS-appropriate full image paths before process creation so Security/Sysmon/eCAR/WFP all receive complete canonical paths; make proxy baseline HTTP path/content-type selection domain-class aware so OS/update/OCSP/CRL hosts do not receive generic browser paths like `/login`, `/favicon.ico`, CSS, image assets, or `text/html`; tune bash typo injection density for short histories. +- [x] **SUPERSEDED** Canonical emitter field provenance blind-review remaining findings from 78% synthetic review — superseded by later full-path storyline normalization, bash typo/path cleanup, proxy domain-class path/content profiles, and Sysmon follow-on ordering fixes. The still-current related work is now represented by web/session realism, imperfect observation/source coverage, and process lifecycle modeling TODOs. - [ ] Source-specific process lifecycle completeness modeling — deferred design item. Add a configurable telemetry coverage/profile layer that can model realistic Security/Sysmon/eCAR missingness, ingestion delay, audit-policy gaps, and endpoint coverage variance without ad hoc omissions in individual emitters. This should be part of the broader cross-source distribution realism layer, not a Windows-only workaround. - [x] Open PR consolidation into `dev` — re-applied the storyline typing-cadence monotonicity fix from PR #81, folded Dependabot pytest/Pygments updates into the dev workflow, and added Dependabot configuration so future dependency PRs target `dev`. -- [ ] **IN PROGRESS** Windows Security/authentication source review — focused baseline eval is complete; fixing high-signal Windows auth realism findings first (4672/session semantics and sparse 4800/4801 rendering), then rerunning focused generation/eval before moving deeper. +- [x] **SUPERSEDED** Windows Security/authentication source review — superseded by the focused Windows auth timing/source semantics fixes and tests completed during the timing and source-review work. No separate active review thread remains. - [x] TODO.md reality audit — verified high-signal open realism/code-cleanup findings against the current codebase, marked stale items, and identified the generated-output validation pass needed before deeper realism work. Targeted verification: `uv run pytest tests/unit/test_network_realism.py tests/unit/test_activity_helpers.py tests/unit/test_dc_kerberos_logon.py -q --no-cov` passed (25 tests). @@ -473,7 +473,7 @@ Data works but experienced analysts spot tells. Grouped by format for efficient **TLS/SSL:** - [x] TLS/x509 correlation gaps — baseline audit found SSL records without `cert_chain_fuids` and x509 issuer/subject pairings that looked implausible. Added deterministic certificate file UIDs, linked ssl.log to x509.log, and tightened domain-to-CA overrides for common CA-owned/Microsoft domains. - [x] TLSv13 ratio too low for 2024 timeframe — audit output showed TLSv13 at 19,669/56,372 SSL records (~35%). TLS version selection now uses explicit weighted constants with TLSv13 as the modern majority default. -- [ ] TLS version/cipher suite mismatches +- [x] TLS version/cipher suite mismatches — resolved by TLS-version-aware cipher selection and certificate key-type coherence tests. - [ ] Non-intercepting proxy mode — current proxy behavior assumes TLS interception, so HTTPS proxy logs can include CONNECT plus inspected request rows and downstream visibility should follow the inspected transaction. Future config can add tunnel-only/non-intercepting behavior separately because it changes proxy URL visibility, Zeek SSL/x509 certificate chains, HTTP visibility inside CONNECT tunnels, and IDS content inspection semantics. - [x] x509 Let's Encrypt certs show 280+ day validity (should be 90) — tls_issuers.yaml with per-issuer validity (LE=90d, DigiCert=397d, etc.); issuer-aware key type selection - [x] No SSL certificate subject/issuer data in ssl.log — zeek_x509.yaml includes subject/issuer fields; generation uses tls_issuers.yaml @@ -489,13 +489,13 @@ Data works but experienced analysts spot tells. Grouped by format for efficient - [x] ✓ phpsessionclean on non-PHP hosts — only on web_server/forward_proxy role - [x] ✓ Transient process (sudo) gets stable PID — sudo/cron children now get random PIDs - [x] ✓ systemd-logind session IDs random — sequential per-host counter from boot -- [ ] Session IDs appear out-of-order (assigned in generation order, not chronological) -- [ ] NTP server mismatch (Zeek shows NIST, syslog shows Ubuntu pool) +- [x] Session IDs appear out-of-order — resolved by later host-local LUID/session ordering fixes for Windows auth and Linux logind session generation. +- [x] NTP server mismatch — resolved: Zeek NTP and systemd-timesyncd syslog both choose from the same scenario infrastructure NTP pool with the same per-host deterministic source selection. - [x] NTP syslog lifecycle semantics — periodic systemd-timesyncd messages now mix source selection, clock sync, offset adjustments, and timeout messages without repeating initial synchronization after the first host sync. - [ ] No SSH protocol negotiation messages - [x] Logrotate/cron.daily fire too frequently (should be daily, not multiple times per hour) — stale audit finding: `systemd_schedules.yaml` defines logrotate and cron-daily as daily scheduled jobs with per-host jitter, outside the per-hour probability loop. - [x] Centralized syslog timestamps not chronologically sorted — _sort_flat_file = True in syslog.py; sorting in host_base.py -- [ ] Dual SSH syslog entries with mismatched PIDs/ports +- [x] Dual SSH syslog entries with mismatched PIDs/ports — resolved by later SSH syslog lifecycle/source-port correlation fixes. Keep any future SSH duplication finding as a fresh concrete regression. **Windows Events:** - [x] ✓ IpAddress "::ffff:-" malformed — handle "-" string in _ipv6_mapped() @@ -584,7 +584,7 @@ Data works but experienced analysts spot tells. Grouped by format for efficient - [ ] ASA message type diversity limited to 106023/302013-16/305011-12 — missing 111008, 113004, 733100, 106001, 725001, 304001 - [ ] ASA deny baseline burstiness/profile variance — defer to a general per-source activity profile rather than a one-off ASA fix. Current deny events are uniformly spaced (3-7s); real scans should have configurable burst/quiet periods, campaign-level cadence, and source-specific variance. - [ ] ASA deny metadata diversity — defer to a general field-distribution realism layer. Current deny events use `[0x0, 0x0]` hash values uniformly; a later profile should model when hashes remain zero vs vary by platform/message/context. -- [ ] NAT mapped_ip 45.33.32.1 is scanme.nmap.org — recognizable IP used as scenario PAT address +- [ ] Recognizable 45.33.32.x public IPs remain in built-in scan/attacker pools — the original `45.33.32.1` NAT PAT finding is stale, but code still uses `45.33.32.156` in scan/attacker pools. Move these values into data/config or replace them with less recognizable public-looking lab addresses during the broader public-IP/profile cleanup. **eCAR:** - [x] Limited object diversity on Linux — expanded _EDR_FILE_PATHS_LINUX from 5 to 20 entries (logs, caches, config files, /proc, package manager) @@ -652,7 +652,7 @@ Data works but experienced analysts spot tells. Grouped by format for efficient - [x] Harden temporal causal-account exclusion against non-string SubjectUserName/principal values to prevent evaluator exceptions on malformed logs - [x] Signal integrity misses web_scan traces in host-scoped web logs and responder-side Zeek HTTP records — generated evidence exists, but evaluator indexing could not find `web_access.log` records by host directory or inbound Zeek HTTP by destination IP. Parser records now carry source-host metadata, and signal-integrity indexing includes responder IPs. Event Presence improved from 1/9 to 9/9 on the HTTP/proxy eval sample. - [x] Causal Ordering hard failure on generated audit sample — root cause was future same-hour session reuse during non-chronological baseline generation. Session lookup now only reuses sessions whose start time is at or before the activity timestamp. Fresh HTTP/proxy sample eval improved Causal Ordering from 95.53% to 99.94%, and all hard acceptance criteria pass. -- [ ] Storyline Trace Coverage hostname normalization bug (traces exist but bare vs FQDN mismatch) +- [x] Storyline Trace Coverage hostname normalization bug — resolved by later FQDN/bare hostname indexing and storyline trace normalization fixes. - [ ] Ground truth File IOCs section truncated in GROUND_TRUTH.md output ### Cross-Source Correlation (depends on Tier 1 baseline migration) diff --git a/commands/eforge/config.md b/commands/eforge/config.md index 020eccaa..e3248735 100644 --- a/commands/eforge/config.md +++ b/commands/eforge/config.md @@ -57,6 +57,7 @@ When writing to the overlay, files are partial — they contain ONLY the user's | Add proxy URI templates | `proxy_uri_templates.yaml` | `dns_registry.yaml` (validate domain exists); use `domain_class` and `referrer_policy` for certificate/update infrastructure | | Modify proxy User-Agent pools | `proxy_user_agents.yaml` | `dns_registry.yaml` for package/update hostnames | | Add site map entries | `site_maps.yaml` | `dns_registry.yaml` (validate domain exists) | +| Modify inbound web visitor mix | `web_session_profiles.yaml` | `site_maps.yaml`, `traffic_rates.yaml`, `timing_profiles.yaml` | | Modify bash commands | `bash_commands.yaml` | Validate role names match persona names; keep `typo_model` rates/counts realistic | | Modify traffic rate defaults | `traffic_rates.yaml` | (standalone — intensity-based rate table for all system traffic) | | Modify systemd schedules | `systemd_schedules.yaml` | (standalone) | @@ -66,6 +67,7 @@ When writing to the overlay, files are partial — they contain ONLY the user's | Modify ProcessAccess masks | `process_access_patterns.yaml` | (standalone — Event 10 baseline source/target pairs and GrantedAccess masks) | | Modify CreateRemoteThread pairs | `create_remote_thread_patterns.yaml` | (standalone — Event 8 baseline source/target pairs) | | Modify Windows auth realism | `windows_auth_realism.yaml` | (standalone — Security log auth timing and failed-logon profile knobs) | +| Modify baseline auth noise | `auth_noise.yaml` | (standalone — stale scheduled-credential accounts and irregular recurrence timing) | | Modify causal/source timing | `timing_profiles.yaml` | (standalone — causal prerequisite, source latency, teardown, and Windows/Sysmon collision-spacing knobs) | | ~~Format definitions~~ | Not user-customizable | Engine internals — requires code changes | | ~~Evaluation rules~~ | Not user-customizable | Must match format definitions — requires code changes | diff --git a/commands/eforge/references/config-dependency-graph.md b/commands/eforge/references/config-dependency-graph.md index 2317cb53..4f121947 100644 --- a/commands/eforge/references/config-dependency-graph.md +++ b/commands/eforge/references/config-dependency-graph.md @@ -47,13 +47,21 @@ Each row is a file; columns show what it depends on and what depends on it. | Direction | File | Relationship | |-----------|------|-------------| | depends on | nothing | Standalone rate table | -| **depended on by** | Engine (runtime) | Drives all baseline traffic rate calculations (user activity, web, DNS, SMB, Kerberos, LDAP, persona connections) | +| **depended on by** | Engine (runtime) | Drives all baseline traffic rate calculations (user activity, web top-level actions, DNS, SMB, Kerberos, LDAP, persona connections) | + +### web_session_profiles.yaml +| Direction | File | Relationship | +|-----------|------|-------------| +| depends on | `site_maps.yaml` | Human visitor sessions use site maps to expand top-level page loads into assets and same-origin API calls | +| depends on | `traffic_rates.yaml` | `web` rates count top-level visitor actions; subresources are dependent fanout | +| depends on | `timing_profiles.yaml` | Uses web session/navigation and asset/tool fanout timing relationships | +| **depended on by** | Engine (runtime) | Drives inbound `web_server` visitor classes, tool/API request shapes, status codes, and User-Agents | ### timing_profiles.yaml | Direction | File | Relationship | |-----------|------|-------------| | depends on | nothing | Standalone timing relationship profile | -| **depended on by** | Engine (runtime) | Drives causal prerequisite offsets, source-latency offsets, teardown margins, and Windows/Sysmon tied-timestamp collision spacing | +| **depended on by** | Engine (runtime) | Drives causal prerequisite offsets, source-latency offsets, web session/fanout timing, sensor observation timing, teardown margins, and Windows/Sysmon tied-timestamp collision spacing | | validated by | `eforge validate-config` | Enforces valid relationship classes, before/after positions, non-negative timing windows, and coherent min/max bounds | ### kerberos_realism.yaml @@ -137,6 +145,12 @@ Each row is a file; columns show what it depends on and what depends on it. | depends on | nothing | Standalone (uses distro/role filters) | | **depended on by** | Engine (runtime) | Adds diversity to syslog baseline | +### auth_noise.yaml +| Direction | File | Relationship | +|-----------|------|-------------| +| depends on | nothing | Standalone authentication-noise profile data | +| **depended on by** | Engine (runtime) | Drives stale scheduled-credential account pools, recurrence timing, jitter, skips, and backoff | + ### network_params.yaml | Direction | File | Relationship | |-----------|------|-------------| diff --git a/commands/eforge/references/config-dns-network.md b/commands/eforge/references/config-dns-network.md index 6799cfc7..f289c826 100644 --- a/commands/eforge/references/config-dns-network.md +++ b/commands/eforge/references/config-dns-network.md @@ -12,10 +12,11 @@ Schema documentation for the network-related config files. User customizations g 2. [traffic_profiles.yaml](#traffic_profilesyaml) 3. [proxy_uri_templates.yaml](#proxy_uri_templatesyaml) 4. [site_maps.yaml](#site_mapsyaml) -5. [network_params.yaml](#network_paramsyaml) -6. [tls_issuers.yaml](#tls_issuersyaml) -7. [tls_realism.yaml](#tls_realismyaml) -8. [smb_file_transfers.yaml](#smb_file_transfersyaml) +5. [web_session_profiles.yaml](#web_session_profilesyaml) +6. [network_params.yaml](#network_paramsyaml) +7. [tls_issuers.yaml](#tls_issuersyaml) +8. [tls_realism.yaml](#tls_realismyaml) +9. [smb_file_transfers.yaml](#smb_file_transfersyaml) --- @@ -338,6 +339,59 @@ Minimal single-page structure for domains with no curated or tag-based match. --- +## web_session_profiles.yaml + +Visitor-class definitions for inbound `web_server` baseline traffic. Human visitors use `site_maps.yaml` to emit a top-level page request plus required JS/CSS/images/fonts/API fanout. Crawler, health-check, API-client, and opportunistic-probe visitors use configured request lists so tool traffic keeps realistic paths, status codes, referrers, and User-Agents. + +The `traffic_rates.yaml` `web` value counts top-level visitor actions only. Subresources required to render a human page load do not consume that budget. + +### Structure + +```yaml +visitor_classes: + human_browser: + weight: 70 + kind: session # session|requests + external: true + internal: true + browsing_intensity: normal + user_agent_pool: browser_any + user_agent_pool_by_os: + linux: browser_linux + + opportunistic_probe: + weight: 5 + kind: requests + external: true + internal: false + request_count: [1, 5] + user_agent_pool: scanner + referrer_mode: none + requests: + - {path: "/wp-login.php", method: "GET", status: 404, type: "text/html", weight: 22} + +user_agent_pools: + browser_any: + - "Mozilla/5.0 (...) Chrome/120.0.0.0 Safari/537.36" + scanner: + - "python-requests/2.31.0" +``` + +### Field Reference + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `visitor_classes..weight` | number | yes | Relative visitor-class frequency | +| `visitor_classes..kind` | string | yes | `session` for site-map browsing, `requests` for configured tool/API paths | +| `external` / `internal` | bool | no | Whether the class can be used for external or internal clients | +| `browsing_intensity` | string | session | Site-map session depth (`light`, `normal`, `heavy`) | +| `request_count` | `[min, max]` | requests | Number of configured requests per visitor action | +| `requests[].path` / `method` / `status` / `type` | mixed | requests | Source-native HTTP request shape | +| `user_agent_pool` | string | yes | Pool name under `user_agent_pools` | +| `user_agent_pool_by_os` | mapping | no | OS-specific override pools for known internal clients | + +--- + ## network_params.yaml MAC OUI (vendor) prefixes, public NTP server defaults, and DNS tunnel transaction timing. Scenario-defined internal/domain NTP servers are preferred at generation time; `public_ntp_servers` is the fallback pool for non-domain environments and for upstream refids on internal NTP servers. @@ -480,7 +534,7 @@ Three top-level keys (`low`, `medium`, `high`), each containing the same traffic | Key | Unit | Description | |-----|------|-------------| | `user_activity` | events/user/hr | Endpoint user activity (logons, processes, connections) | -| `web` | requests/web_server/hr | Background HTTP requests to web_server hosts | +| `web` | top-level actions/web_server/hr | User-driven page/API/tool requests to web_server hosts; page assets are emitted as dependent requests and do not consume this budget | | `dns_interval` | seconds between queries | Lower = more DNS traffic | | `ntp` | syncs/host/hr | NTP time sync frequency | | `smb_interval` | seconds between SMB ops | Lower = more SMB/file share traffic | diff --git a/commands/eforge/references/config-host-activity.md b/commands/eforge/references/config-host-activity.md index 9dd21378..4ee80dbf 100644 --- a/commands/eforge/references/config-host-activity.md +++ b/commands/eforge/references/config-host-activity.md @@ -285,6 +285,36 @@ Failed-logon profiles control source-native Windows 4625 fields and DC-side vali --- +## Auth Noise (`auth_noise.yaml`) + +Controls baseline authentication noise that is not scenario-authored, especially stale scheduled credentials. + +```yaml +scheduled_stale_credentials: + account_base_names: [svc_backup, svc_monitor, svc_report, svc_deploy, svc_scan] + host_count_min: 1 + host_count_max: 2 + interval_ranges: + - min_minutes: 55 + max_minutes: 95 + weight: 30 + - min_minutes: 105 + max_minutes: 155 + weight: 45 + first_occurrence_seconds_min: 0 + first_occurrence_seconds_max: 2700 + jitter_seconds_min: -420 + jitter_seconds_max: 780 + skip_probability: 0.16 + backoff_probability: 0.10 + backoff_seconds_min: 900 + backoff_seconds_max: 3600 +``` + +`account_base_names` should be plausible disabled service or automation principals; the engine still avoids collisions with scenario users and service accounts. Interval ranges, jitter, skip probability, and backoff probability produce deterministic but non-modulo recurrence so stale scheduled-task failures do not land on exact hourly or two-hour cadences. Run `eforge validate-config` after overlay changes; ranges must be ordered, weights must be positive, and probabilities must be between 0 and 0.95. + +--- + ## timing_profiles.yaml Data-driven timing windows for causal relationships, source-native latency, teardown margins, and Windows/Sysmon same-timestamp collision spacing. Use this when tuning realism of correlated event gaps without changing scenario YAML. @@ -313,6 +343,32 @@ relationships: position: after min_ms: 800 max_ms: 2500 + web.session_navigation: + class: human_workflow + position: after + min_ms: 3000 + max_ms: 30000 + web.asset_stylesheet_script_after_page: + class: burst_fanout + position: after + min_ms: 50 + max_ms: 200 + web.tool_request_gap: + class: burst_fanout + position: after + min_ms: 120 + max_ms: 1500 + +network_sensor_observation: + default_profile: well_synced + profiles: + well_synced: + clock_skew_us: + min: -1500 + max: 1500 + path_delay_us: + min: 50 + max: 2000 windows_event_time: collision_spacing: @@ -334,6 +390,9 @@ windows_event_time: | `windows_event_time.collision_spacing.near_zero_until` | int | yes | Same-host tied-event collisions that can remain near-zero before larger spacing begins | | `windows_event_time.collision_spacing.near_gap_min_us` / `near_gap_max_us` | int | yes | Microsecond spacing for small tied clusters | | `windows_event_time.collision_spacing.large_gap_min_ms` / `large_gap_max_ms` | int | yes | Millisecond spacing for large tied clusters that would otherwise compress into synthetic-looking bursts | +| `network_sensor_observation.default_profile` | string | yes | Sensor timing profile used for multi-sensor Zeek observation offsets | +| `network_sensor_observation.profiles..clock_skew_us` | mapping | yes | `{min, max}` per-sensor clock skew in microseconds | +| `network_sensor_observation.profiles..path_delay_us` | mapping | yes | `{min, max}` per-flow tap/capture delay in microseconds | ### Conventions @@ -342,6 +401,8 @@ windows_event_time: `ssl.log` and `x509.log` timestamps should occur after conn start but before conn end for the same UID. - Use seconds or minutes for human or bulk workflow relationships; do not force everything into microseconds. +- Web session timing uses `web.session_navigation` for user-driven page-to-page actions and `web.asset_*_after_page` / `web.tool_request_gap` for render fanout and tool/API bursts. +- Keep the default `network_sensor_observation` profile in low milliseconds for well-synced Zeek fleets; use overlays only when modeling known sensor clock drift or queued/remote capture paths. - Run `eforge validate-config` after overlay changes; it rejects invalid relationship classes, positions, negative windows, and inverted min/max ranges. --- diff --git a/commands/eforge/references/config-validation.md b/commands/eforge/references/config-validation.md index 765ce880..3e5400ad 100644 --- a/commands/eforge/references/config-validation.md +++ b/commands/eforge/references/config-validation.md @@ -81,6 +81,8 @@ Run `eforge info ` to get specific values (e.g., `eforge info paths.activ | 34 | create_remote_thread_patterns.yaml structure | ERROR | Baseline pair missing source/target PID keys, image paths, or positive weight | | 35 | smb_file_transfers.yaml structure | ERROR | Missing SMB file-analysis thresholds/probabilities, invalid probability ranges, empty MIME/analyzer lists, invalid filename templates, or non-positive weights | | 36 | kerberos_realism.yaml structure | ERROR | Invalid Kerberos 4768 pre-auth/ticket/encryption distribution, unsupported hex values, PKINIT without certificate profile, non-PKINIT with certificate fields, excessive no-preauth/PKINIT/RC4 weights, or malformed certificate profile fields | +| 37 | web_session_profiles.yaml structure | ERROR | Invalid inbound web visitor class, missing User-Agent pool, malformed configured request, or invalid request-count range | +| 38 | auth_noise.yaml structure | ERROR | Invalid stale scheduled-credential account pool, host-count range, recurrence interval range, jitter range, skip probability, or backoff bounds | ## Scenario Validation: traffic_rates diff --git a/commands/eforge/references/evidence-formats.md b/commands/eforge/references/evidence-formats.md index 85dd2f44..7db99be7 100644 --- a/commands/eforge/references/evidence-formats.md +++ b/commands/eforge/references/evidence-formats.md @@ -175,7 +175,7 @@ EDR/XDR telemetry rendered in MITRE CAR-based eCAR format. Represents what an ED **File:** `syslog.log` **Format:** RFC 5424 syslog -Authentication and system logs from Linux hosts. Generated syslog uses RFC 5424 with year-bearing ISO/RFC3339 timestamps. `eforge eval` still accepts older BSD/RFC3164-style syslog as a legacy ingest fallback. All generated syslog entries are rendered from `SyslogContext` on `SecurityEvent` — the emitter doesn't derive messages from other contexts. This enables correlated dispatch: a logon event carries both `AuthContext` (for Windows 4624) and `SyslogContext` (for sshd accepted) on the same SecurityEvent. +Authentication and system logs from Linux hosts. Generated syslog uses RFC 5424 with year-bearing ISO/RFC3339 timestamps. `eforge eval` still accepts older BSD/RFC3164-style syslog as a legacy ingest fallback. All generated syslog entries are rendered from `SyslogContext` on `SecurityEvent` — the emitter doesn't derive messages from other contexts. This enables correlated dispatch: a logon event carries both `AuthContext` (for Windows 4624) and `SyslogContext` (for sshd accepted) on the same SecurityEvent. Remote Linux `sshd` failed-password rows reuse the same source port as the companion Zeek SSH connection tuple. | Program | Description | Notes | |---------|-------------|-------| @@ -313,7 +313,7 @@ Fields are whitespace-delimited; values with spaces, such as User-Agent strings, **Status and byte semantics:** For explicit proxy mode, client-side Zeek HTTP records describe the client-to-proxy exchange. Plain HTTP denials therefore show the proxy's status code and proxy response size, not the origin's status/body. For intercepted HTTPS, the CONNECT setup status is tracked separately from the inspected request status, so a successful tunnel setup can coexist with a denied inspected GET. -**Session depth:** Persona HTTP traffic generates multi-request browsing sessions with subresource cascades. Each page load triggers follow-on requests for JS, CSS, images, and fonts, producing realistic request clusters in the proxy log. The number of pages and subresources per session is controlled by the persona's `browsing_intensity` setting (light/normal/heavy). +**Session depth:** Persona HTTP traffic and inbound `web_server` human visitors generate multi-request browsing sessions with subresource cascades. Each page load triggers follow-on requests for JS, CSS, images, fonts, and same-origin API calls, producing realistic request clusters in proxy and web access logs. Persona browsing depth is controlled by `browsing_intensity`; inbound web visitor classes, tool/API requests, and User-Agent pools are controlled by `web_session_profiles.yaml`. **Known Limitations:** - Only generated for systems with the `forward_proxy` role declared diff --git a/commands/eforge/references/scenario-reference.md b/commands/eforge/references/scenario-reference.md index 7492c106..67eae45c 100644 --- a/commands/eforge/references/scenario-reference.md +++ b/commands/eforge/references/scenario-reference.md @@ -115,7 +115,7 @@ If `proxy_access` is requested and `environment.proxy` is omitted, validation wa The `roles` field declares a system's function in the network. The engine uses roles to generate both **outbound** traffic (connections the host initiates) and **inbound** traffic (connections the host receives): -- `web_server` — outbound: database queries, LDAP auth, API calls; inbound: HTTPS/HTTP from external clients and internal users +- `web_server` — outbound: database queries, LDAP auth, API calls; inbound: HTTPS/HTTP from external clients and internal users. Human inbound traffic is generated as browsing sessions: top-level page views consume the `web` traffic-rate budget, and required assets/API calls fan out from each page load. - `database` — outbound: replication, updates; inbound: SQL queries from web/app servers - `mail_server` — outbound: SMTP relay, LDAP lookups; inbound: SMTP from internet, webmail from users - `file_server` — outbound: Kerberos/LDAP auth; inbound: SMB file access from workstations. File-server roles also increase baseline SMB target selection beyond normal DC SYSVOL/GPO traffic. @@ -306,7 +306,7 @@ Work hours are automatically parsed into a `work_hours_parsed` dict containing: ### Browsing Intensity -The `browsing_intensity` field controls how much HTTP traffic a persona generates per browsing session. It affects proxy log depth (number of page loads and subresource cascades) for baseline web activity. +The `browsing_intensity` field controls how much HTTP traffic a persona generates per browsing session. It affects proxy log depth (number of page loads and subresource cascades) for baseline web activity. Inbound `web_server` background traffic uses the separate `web_session_profiles.yaml` visitor mix: `traffic_rates.web` counts top-level visitor actions, then page assets and same-origin API calls fan out automatically. ```yaml personas: @@ -524,7 +524,7 @@ The generation engine automatically provides several layers of realism in baseli **NTP time synchronization:** In AD environments, all domain-joined workstations sync NTP from the domain controller (W32Time service), not from external NIST servers. NTP stratum is stable per server — a DC serving as NTP always reports the same stratum value. External NTP servers are only used for non-domain environments. -**Multi-sensor timing realism:** When multiple Zeek sensors observe the same connection, each sensor's records have a deterministic propagation delay (100-500 microseconds) based on the sensor's position. Sensors farther from the packet source see events slightly later. Byte and packet counts are identical across sensors (both see the same packets on the wire), but timestamps and durations differ. +**Multi-sensor timing realism:** When multiple Zeek sensors observe the same connection, each sensor's records use the well-synced network sensor timing profile in `config/activity/timing_profiles.yaml`. The default profile keeps stable per-sensor clock skew within +/-1.5 ms and per-flow path/capture delay within 50-2000 microseconds. Byte and packet counts remain canonical unless sensor observation variance is explicitly allowed for that source-native row. **Linux syslog depth:** Linux hosts generate 18 categories of syslog messages: SSH login/key exchange (70% key / 30% password), package management, systemd timer execution, logrotate detail, journald statistics, plus systemd lifecycle, cron, UFW, logind, and more. Distro-aware (Ubuntu vs RHEL) with appropriate daemon names and paths. diff --git a/commands/eforge/scenario.md b/commands/eforge/scenario.md index d31cc6eb..5e1f1044 100644 --- a/commands/eforge/scenario.md +++ b/commands/eforge/scenario.md @@ -64,7 +64,7 @@ Inbound traffic respects network topology: DMZ-placed `web_server` hosts attract **Browsing patterns** — How much web browsing does each user role generate? Personas have a default `browsing_intensity` (light/normal/heavy) that controls proxy session depth — how many pages and subresources each browsing session produces. Ask whether any user roles are heavier or lighter web users than their persona default suggests, and set per-user `browsing_intensity` overrides where appropriate. -**Traffic volume** — For scenarios that output server-side logs (especially `web_access`), the `intensity` setting controls how much background traffic web servers receive (low: ~20/hr, medium: ~1000/hr, high: ~5000/hr). If the scenario focuses on server-side analysis (web scanners, access log anomalies), you likely need `intensity: high` or explicit `traffic_rates: {web: [5000, 12000]}` overrides to ensure attackers are buried in realistic background noise. Ask about expected noise-to-signal ratios for server-focused scenarios. +**Traffic volume** — For scenarios that output server-side logs (especially `web_access`), the `intensity` setting controls how many top-level visitor actions web servers receive (low: ~20/hr, medium: ~1000/hr, high: ~5000/hr). Human page views automatically fan out into required page assets (JS, CSS, images, fonts, same-origin API calls) without consuming additional `web` budget. If the scenario focuses on server-side analysis (web scanners, access log anomalies), you likely need `intensity: high` or explicit `traffic_rates: {web: [5000, 12000]}` overrides to ensure attackers are buried in realistic background noise. Ask about expected noise-to-signal ratios for server-focused scenarios. **Stale accounts** — Does the organization have any disabled or inactive accounts that haven't been fully cleaned up? Former employees, decommissioned service accounts, or un-revoked contractor access are common in real environments. Add 2-4 stale accounts to `environment.stale_accounts` with `username`, `last_active` (ISO date), and `reason`. The engine automatically generates background noise from these: failed logons, Kerberos pre-auth failures on DCs, scheduled task failures, and service startup failures — creating realistic "why is this disabled account still here?" ambiguity for analysts. diff --git a/docs/reference/CUSTOMIZING_CONFIG.md b/docs/reference/CUSTOMIZING_CONFIG.md index 24375dde..8e6f5c52 100644 --- a/docs/reference/CUSTOMIZING_CONFIG.md +++ b/docs/reference/CUSTOMIZING_CONFIG.md @@ -156,10 +156,12 @@ Configuration files are interconnected. When you add an entry to one file, other | A new domain | `proxy_uri_templates.yaml` (URI paths), `site_maps.yaml` (browsing depth) | | Certificate/update/telemetry proxy behavior | `proxy_uri_templates.yaml` (`domain_class`, infra-specific paths/content types, and `referrer_policy: none`; non-browser classes are excluded from site-map browsing sessions) | | New proxy User-Agent behavior | `proxy_user_agents.yaml` (workstation/server UA pools, package-manager host bindings, domain-specific update/cert/telemetry overrides) | +| Inbound web visitor mix | `web_session_profiles.yaml` (visitor classes, configured tool/API requests, and User-Agent pools). Human visitor sessions use `site_maps.yaml`; timing lives in `timing_profiles.yaml`; `traffic_rates.yaml` `web` counts top-level actions only. | | New TLS issuer behavior | `tls_issuers.yaml` (issuer validity, key-type weights, and domain CA overrides). RSA-branded issuer names should only advertise RSA key types unless the chain/signature model is also updated to distinguish issuer signature algorithm from leaf public-key algorithm. | | New TLS OCSP responder behavior | `tls_realism.yaml` (`ocsp.responders`) plus `dns_registry.yaml` for each responder hostname | | Kerberos TGT pre-auth realism | `kerberos_realism.yaml` (`tgt_success.pre_auth_types`, ticket options, encryption types, and PKINIT certificate profiles). Run `eforge validate-config`; PKINIT (`PreAuthType: 15`) requires populated certificate profile support. | | Windows auth realism | `windows_auth_realism.yaml` (`workstation_lock.min_unlock_gap_seconds`, failed-logon local/network profiles, and optional companion network connection rates) | +| Baseline auth noise | `auth_noise.yaml` (stale scheduled-credential account pools, host counts, recurrence intervals, jitter, skips, and backoff) | | Causal/source-native timing | `timing_profiles.yaml` (`relationships` for causal prerequisites, source latency, teardown margins, Zeek analyzer offsets and TLS duration floors, plus Windows/Sysmon collision spacing) | | Public NTP fallback servers and DNS tunnel timing | `network_params.yaml` (`public_ntp_servers`, `dns_tunnel_rtt`; scenario-defined internal/domain NTP servers still take precedence) | | A new application | `spawn_rules.yaml` (process tree), `process_network_map.yaml` (if it generates traffic) | diff --git a/docs/reference/EVIDENCE_FORMATS.md b/docs/reference/EVIDENCE_FORMATS.md index 85dd2f44..7db99be7 100644 --- a/docs/reference/EVIDENCE_FORMATS.md +++ b/docs/reference/EVIDENCE_FORMATS.md @@ -175,7 +175,7 @@ EDR/XDR telemetry rendered in MITRE CAR-based eCAR format. Represents what an ED **File:** `syslog.log` **Format:** RFC 5424 syslog -Authentication and system logs from Linux hosts. Generated syslog uses RFC 5424 with year-bearing ISO/RFC3339 timestamps. `eforge eval` still accepts older BSD/RFC3164-style syslog as a legacy ingest fallback. All generated syslog entries are rendered from `SyslogContext` on `SecurityEvent` — the emitter doesn't derive messages from other contexts. This enables correlated dispatch: a logon event carries both `AuthContext` (for Windows 4624) and `SyslogContext` (for sshd accepted) on the same SecurityEvent. +Authentication and system logs from Linux hosts. Generated syslog uses RFC 5424 with year-bearing ISO/RFC3339 timestamps. `eforge eval` still accepts older BSD/RFC3164-style syslog as a legacy ingest fallback. All generated syslog entries are rendered from `SyslogContext` on `SecurityEvent` — the emitter doesn't derive messages from other contexts. This enables correlated dispatch: a logon event carries both `AuthContext` (for Windows 4624) and `SyslogContext` (for sshd accepted) on the same SecurityEvent. Remote Linux `sshd` failed-password rows reuse the same source port as the companion Zeek SSH connection tuple. | Program | Description | Notes | |---------|-------------|-------| @@ -313,7 +313,7 @@ Fields are whitespace-delimited; values with spaces, such as User-Agent strings, **Status and byte semantics:** For explicit proxy mode, client-side Zeek HTTP records describe the client-to-proxy exchange. Plain HTTP denials therefore show the proxy's status code and proxy response size, not the origin's status/body. For intercepted HTTPS, the CONNECT setup status is tracked separately from the inspected request status, so a successful tunnel setup can coexist with a denied inspected GET. -**Session depth:** Persona HTTP traffic generates multi-request browsing sessions with subresource cascades. Each page load triggers follow-on requests for JS, CSS, images, and fonts, producing realistic request clusters in the proxy log. The number of pages and subresources per session is controlled by the persona's `browsing_intensity` setting (light/normal/heavy). +**Session depth:** Persona HTTP traffic and inbound `web_server` human visitors generate multi-request browsing sessions with subresource cascades. Each page load triggers follow-on requests for JS, CSS, images, fonts, and same-origin API calls, producing realistic request clusters in proxy and web access logs. Persona browsing depth is controlled by `browsing_intensity`; inbound web visitor classes, tool/API requests, and User-Agent pools are controlled by `web_session_profiles.yaml`. **Known Limitations:** - Only generated for systems with the `forward_proxy` role declared diff --git a/docs/reference/scenario-reference.md b/docs/reference/scenario-reference.md index 8c739830..f74e98f6 100644 --- a/docs/reference/scenario-reference.md +++ b/docs/reference/scenario-reference.md @@ -115,7 +115,7 @@ If `proxy_access` is requested and `environment.proxy` is omitted, validation wa The `roles` field declares a system's function in the network. The engine uses roles to generate both **outbound** traffic (connections the host initiates) and **inbound** traffic (connections the host receives): -- `web_server` — outbound: database queries, LDAP auth, API calls; inbound: HTTPS/HTTP from external clients and internal users +- `web_server` — outbound: database queries, LDAP auth, API calls; inbound: HTTPS/HTTP from external clients and internal users. Human inbound traffic is generated as browsing sessions: top-level page views consume the `web` traffic-rate budget, and required assets/API calls fan out from each page load. - `database` — outbound: replication, updates; inbound: SQL queries from web/app servers - `mail_server` — outbound: SMTP relay, LDAP lookups; inbound: SMTP from internet, webmail from users - `file_server` — outbound: Kerberos/LDAP auth; inbound: SMB file access from workstations. File-server roles also increase baseline SMB target selection beyond normal DC SYSVOL/GPO traffic. @@ -306,7 +306,7 @@ Work hours are automatically parsed into a `work_hours_parsed` dict containing: ### Browsing Intensity -The `browsing_intensity` field controls how much HTTP traffic a persona generates per browsing session. It affects proxy log depth (number of page loads and subresource cascades) for baseline web activity. +The `browsing_intensity` field controls how much HTTP traffic a persona generates per browsing session. It affects proxy log depth (number of page loads and subresource cascades) for baseline web activity. Inbound `web_server` background traffic uses the separate `web_session_profiles.yaml` visitor mix: `traffic_rates.web` counts top-level visitor actions, then page assets and same-origin API calls fan out automatically. ```yaml personas: @@ -524,7 +524,7 @@ The generation engine automatically provides several layers of realism in baseli **NTP time synchronization:** In AD environments, all domain-joined workstations sync NTP from the domain controller (W32Time service), not from external NIST servers. NTP stratum is stable per server — a DC serving as NTP always reports the same stratum value. External NTP servers are only used for non-domain environments. -**Multi-sensor timing realism:** When multiple Zeek sensors observe the same connection, each sensor's records have a deterministic propagation delay (100-500 microseconds) based on the sensor's position. Sensors farther from the packet source see events slightly later. Byte and packet counts are identical across sensors (both see the same packets on the wire), but timestamps and durations differ. +**Multi-sensor timing realism:** When multiple Zeek sensors observe the same connection, each sensor's records use the well-synced network sensor timing profile in `config/activity/timing_profiles.yaml`. The default profile keeps stable per-sensor clock skew within +/-1.5 ms and per-flow path/capture delay within 50-2000 microseconds. Byte and packet counts remain canonical unless sensor observation variance is explicitly allowed for that source-native row. **Linux syslog depth:** Linux hosts generate 18 categories of syslog messages: SSH login/key exchange (70% key / 30% password), package management, systemd timer execution, logrotate detail, journald statistics, plus systemd lifecycle, cron, UFW, logind, and more. Distro-aware (Ubuntu vs RHEL) with appropriate daemon names and paths. diff --git a/src/evidenceforge/cli/validate_config.py b/src/evidenceforge/cli/validate_config.py index 3d878205..f3cbf5c3 100644 --- a/src/evidenceforge/cli/validate_config.py +++ b/src/evidenceforge/cli/validate_config.py @@ -168,6 +168,9 @@ def validate_config() -> ValidationResult: "activity/process_access_patterns.yaml": { "list_fields": {"baseline_pairs": None}, }, + "activity/auth_noise.yaml": { + "dict_fields": {"scheduled_stale_credentials"}, + }, "activity/create_remote_thread_patterns.yaml": { "list_fields": {"baseline_pairs": None}, "dict_fields": {"start_locations", "target_overrides"}, @@ -233,11 +236,14 @@ def validate_config() -> ValidationResult: "activity/web_scan_presets.yaml": { "dict_fields": {"presets"}, }, + "activity/web_session_profiles.yaml": { + "dict_fields": {"visitor_classes", "user_agent_pools"}, + }, "activity/traffic_rates.yaml": { "dict_fields": {"low", "medium", "high"}, }, "activity/timing_profiles.yaml": { - "dict_fields": {"relationships", "windows_event_time"}, + "dict_fields": {"relationships", "windows_event_time", "network_sensor_observation"}, }, } @@ -433,6 +439,7 @@ def validate_config() -> ValidationResult: # Every config file should be loaded via its loader (not raw yaml.safe_load) # so that overlay customizations are visible to validation. from evidenceforge.generation.activity.application_catalog import load_catalog + from evidenceforge.generation.activity.auth_noise import load_auth_noise_config from evidenceforge.generation.activity.create_remote_thread_patterns import ( load_create_remote_thread_config, load_create_remote_thread_patterns, @@ -451,6 +458,7 @@ def validate_config() -> ValidationResult: from evidenceforge.generation.activity.timing_profiles import load_timing_profiles from evidenceforge.generation.activity.tls_realism import load_tls_realism from evidenceforge.generation.activity.traffic_profiles import load_traffic_profiles + from evidenceforge.generation.activity.web_session_profiles import load_web_session_profiles from evidenceforge.generation.activity.windows_auth_realism import load_windows_auth_realism dns_data = load_dns_registry() @@ -460,6 +468,7 @@ def validate_config() -> ValidationResult: spawn_data = load_spawn_rules() process_net_data = load_process_network_map() process_access_data = load_process_access_patterns() + auth_noise_data = load_auth_noise_config() create_remote_thread_data = load_create_remote_thread_patterns() create_remote_thread_config = load_create_remote_thread_config() proxy_data = load_proxy_uri_templates() @@ -469,6 +478,7 @@ def validate_config() -> ValidationResult: tls_realism_data = load_tls_realism() windows_auth_data = load_windows_auth_realism() timing_profiles_data = load_timing_profiles() + web_session_profiles_data = load_web_session_profiles() # Collect file count (package + overlay) yaml_files: list[Path] = [] @@ -948,6 +958,94 @@ def _record_ids_rule_identity( ) ) + sensor_timing = timing_profiles_data.get("network_sensor_observation", {}) + if not isinstance(sensor_timing, dict): + result.issues.append( + Issue("ERROR", "timing_profiles.yaml", "network_sensor_observation must be a mapping") + ) + else: + default_profile = sensor_timing.get("default_profile") + profiles = sensor_timing.get("profiles") + if not isinstance(default_profile, str) or not default_profile: + result.issues.append( + Issue( + "ERROR", + "timing_profiles.yaml", + "network_sensor_observation.default_profile must be a non-empty string", + ) + ) + if not isinstance(profiles, dict) or not profiles: + result.issues.append( + Issue( + "ERROR", + "timing_profiles.yaml", + "network_sensor_observation.profiles must be a non-empty mapping", + ) + ) + elif isinstance(default_profile, str) and default_profile not in profiles: + result.issues.append( + Issue( + "ERROR", + "timing_profiles.yaml", + f'network_sensor_observation.default_profile "{default_profile}" is not defined', + ) + ) + if isinstance(profiles, dict): + for profile_name, profile_data in profiles.items(): + if not isinstance(profile_data, dict): + result.issues.append( + Issue( + "ERROR", + "timing_profiles.yaml", + f'Network sensor profile "{profile_name}" must be a mapping', + ) + ) + continue + for field_name, minimum in { + "clock_skew_us": -1_000_000, + "path_delay_us": 0, + }.items(): + bounds = profile_data.get(field_name) + if not isinstance(bounds, dict): + result.issues.append( + Issue( + "ERROR", + "timing_profiles.yaml", + f"network_sensor_observation.profiles.{profile_name}.{field_name} must be a mapping", + ) + ) + continue + min_value = bounds.get("min") + max_value = bounds.get("max") + if not isinstance(min_value, int) or min_value < minimum: + result.issues.append( + Issue( + "ERROR", + "timing_profiles.yaml", + f"network_sensor_observation.profiles.{profile_name}.{field_name}.min must be an integer >= {minimum}", + ) + ) + if not isinstance(max_value, int) or max_value > 1_000_000: + result.issues.append( + Issue( + "ERROR", + "timing_profiles.yaml", + f"network_sensor_observation.profiles.{profile_name}.{field_name}.max must be an integer <= 1000000", + ) + ) + if ( + isinstance(min_value, int) + and isinstance(max_value, int) + and max_value < min_value + ): + result.issues.append( + Issue( + "ERROR", + "timing_profiles.yaml", + f"network_sensor_observation.profiles.{profile_name}.{field_name}.max must be >= min", + ) + ) + # Check 8: Orphaned site maps for domain in site_domains - dns_domain_set: result.issues.append( @@ -982,6 +1080,127 @@ def _record_ids_rule_identity( ) ) + # --- Inbound web visitor profile integrity --- + web_visitor_classes = web_session_profiles_data.get("visitor_classes", {}) + web_ua_pools = web_session_profiles_data.get("user_agent_pools", {}) + if not isinstance(web_visitor_classes, dict) or not web_visitor_classes: + result.issues.append( + Issue("ERROR", "web_session_profiles.yaml", "visitor_classes must be a mapping") + ) + if not isinstance(web_ua_pools, dict) or not web_ua_pools: + result.issues.append( + Issue("ERROR", "web_session_profiles.yaml", "user_agent_pools must be a mapping") + ) + if isinstance(web_visitor_classes, dict) and isinstance(web_ua_pools, dict): + for class_name, class_data in web_visitor_classes.items(): + if not isinstance(class_data, dict): + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" must be a mapping', + ) + ) + continue + if class_data.get("kind") not in {"session", "requests"}: + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" kind must be "session" or "requests"', + ) + ) + weight = class_data.get("weight") + if not isinstance(weight, int | float) or isinstance(weight, bool) or weight <= 0: + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" weight must be positive', + ) + ) + pool_name = class_data.get("user_agent_pool") + if not isinstance(pool_name, str) or pool_name not in web_ua_pools: + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" references missing user_agent_pool "{pool_name}"', + ) + ) + by_os = class_data.get("user_agent_pool_by_os") + if by_os is not None and not isinstance(by_os, dict): + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" user_agent_pool_by_os must be a mapping', + ) + ) + if isinstance(by_os, dict): + for os_name, os_pool in by_os.items(): + if not isinstance(os_name, str) or not isinstance(os_pool, str): + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" user_agent_pool_by_os must map strings to strings', + ) + ) + continue + if os_pool not in web_ua_pools: + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" references missing OS user_agent_pool "{os_pool}"', + ) + ) + if class_data.get("kind") == "requests": + request_count = class_data.get("request_count") + if ( + not isinstance(request_count, list) + or len(request_count) != 2 + or not all(isinstance(value, int) and value > 0 for value in request_count) + or request_count[1] < request_count[0] + ): + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" request_count must be [min, max] positive integers', + ) + ) + requests = class_data.get("requests") + if not isinstance(requests, list) or not requests: + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" requests must be a non-empty list', + ) + ) + continue + for index, request in enumerate(requests): + if not isinstance(request, dict): + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" request {index} must be a mapping', + ) + ) + continue + for required in ("path", "method", "status", "type"): + if required not in request: + result.issues.append( + Issue( + "ERROR", + "web_session_profiles.yaml", + f'Visitor class "{class_name}" request {index} missing "{required}"', + ) + ) + # --- Checks 11-13: Traffic Profile Integrity --- role_traffic = traffic_data.get("role_traffic", {}) persona_traffic = traffic_data.get("persona_traffic", {}) @@ -1462,6 +1681,7 @@ def _record_ids_rule_identity( # --- Schema validation: validate merged entries against Pydantic models --- from evidenceforge.config.schemas import ( ApplicationEntry, + AuthNoiseConfig, ConnectionEntry, CreateRemoteThreadNoiseConfig, CreateRemoteThreadPatternEntry, @@ -1831,6 +2051,10 @@ def _record_ids_rule_identity( if err: result.issues.append(Issue("ERROR", "windows_auth_realism.yaml", err)) + err = validate_entry(auth_noise_data, AuthNoiseConfig, "auth_noise.yaml") + if err: + result.issues.append(Issue("ERROR", "auth_noise.yaml", err)) + if isinstance(proxy_ua_data.get("domain_overrides"), dict): _SCHEMA_CHECKS.append( ( diff --git a/src/evidenceforge/config/activity/README.md b/src/evidenceforge/config/activity/README.md index 8ef4ca24..fb8221f8 100644 --- a/src/evidenceforge/config/activity/README.md +++ b/src/evidenceforge/config/activity/README.md @@ -21,12 +21,14 @@ caches data after first load. Two files (`network_params.yaml`, | `tls_realism.yaml` | `tls_realism.py` | TLS SAN, OCSP, certificate-chain, and destination-profile settings with overlay support. | | `kerberos_realism.yaml` | `kerberos_realism.py` | Kerberos 4768 TGT PreAuthType, TicketOptions, encryption, and PKINIT certificate field distributions with overlay support. | | `windows_auth_realism.yaml` | `windows_auth_realism.py` | Windows Security authentication realism knobs such as minimum 4800→4801 lock/unlock gap, failed-logon validation paths, companion network evidence, and 4672 privilege profiles. | +| `auth_noise.yaml` | `auth_noise.py` | Baseline authentication-noise profiles such as stale scheduled-credential account pools and irregular recurrence timing. | | `proxy_uri_templates.yaml` | `proxy_uri.py` | Per-domain URI path templates for proxy logs (Windows Update, CRL, OCSP, Azure AD, etc.). | | `network_params.yaml` | `network_params.py`, `engine/emitter_setup.py` | MAC address OUI prefixes, public NTP fallback servers, and DNS tunnel RTT bounds. | | `systemd_schedules.yaml` | `engine/baseline.py` | Systemd timer and cron job schedules (logrotate, fstrim, apt-daily, etc.). | | `extra_syslog_messages.yaml` | `extra_syslog.py` | Role/distro-tagged syslog program messages for baseline diversity. | | `application_catalog.yaml` | `application_catalog.py` | Unified app definitions: image paths, PE metadata, command templates, persona filtering, child processes. | | `traffic_profiles.yaml` | `traffic_profiles.py` | Role-based and persona-based network traffic profiles. See `docs/design/traffic-profiles-design.md`. | +| `web_session_profiles.yaml` | `web_session_profiles.py` | Inbound web server visitor classes, request profiles, and User-Agent pools. Human visitors use `site_maps.yaml`; top-level `web` traffic rates fan out into page assets. | | `process_network_map.yaml` | `process_network.py` | Process-to-network service mappings for PID attribution and process-network correlation. | | `process_access_patterns.yaml` | `process_access_patterns.py` | Sysmon Event 10 baseline source/target pairs and weighted GrantedAccess masks. | | `create_remote_thread_patterns.yaml` | `create_remote_thread_patterns.py` | Sysmon Event 8/eCAR THREAD benign source/target pairs plus weighted start module/function locations. | diff --git a/src/evidenceforge/config/activity/auth_noise.yaml b/src/evidenceforge/config/activity/auth_noise.yaml new file mode 100644 index 00000000..09f04fe0 --- /dev/null +++ b/src/evidenceforge/config/activity/auth_noise.yaml @@ -0,0 +1,46 @@ +# auth_noise.yaml — Baseline authentication noise profiles. +# +# User customizations go in: +# .eforge/config/activity/auth_noise.yaml +# +# Overlay behavior: nested dictionaries merge with package defaults. + +scheduled_stale_credentials: + # Candidate disabled service accounts for stale scheduled-task failures. + account_base_names: + - svc_backup + - svc_monitor + - svc_report + - svc_deploy + - svc_scan + - svc_patch + - svc_build + - svc_sync + - svc_jobs + - svc_batch + + host_count_min: 1 + host_count_max: 2 + + # Real stale scheduled tasks tend to cluster around automation intervals, + # but retries, queueing, and maintenance windows keep them from landing on + # exact modulo-hour boundaries. + interval_ranges: + - min_minutes: 55 + max_minutes: 95 + weight: 30 + - min_minutes: 105 + max_minutes: 155 + weight: 45 + - min_minutes: 170 + max_minutes: 260 + weight: 25 + + first_occurrence_seconds_min: 0 + first_occurrence_seconds_max: 2700 + jitter_seconds_min: -420 + jitter_seconds_max: 780 + skip_probability: 0.16 + backoff_probability: 0.10 + backoff_seconds_min: 900 + backoff_seconds_max: 3600 diff --git a/src/evidenceforge/config/activity/timing_profiles.yaml b/src/evidenceforge/config/activity/timing_profiles.yaml index bb650cbc..29f80824 100644 --- a/src/evidenceforge/config/activity/timing_profiles.yaml +++ b/src/evidenceforge/config/activity/timing_profiles.yaml @@ -111,6 +111,55 @@ relationships: position: after min_ms: 800 max_ms: 2500 + web.session_navigation: + class: human_workflow + position: after + min_ms: 3000 + max_ms: 30000 + web.asset_stylesheet_script_after_page: + class: burst_fanout + position: after + min_ms: 50 + max_ms: 200 + web.asset_image_after_page: + class: burst_fanout + position: after + min_ms: 200 + max_ms: 800 + web.asset_font_after_page: + class: burst_fanout + position: after + min_ms: 300 + max_ms: 600 + web.asset_api_after_page: + class: burst_fanout + position: after + min_ms: 500 + max_ms: 2000 + web.asset_other_after_page: + class: burst_fanout + position: after + min_ms: 100 + max_ms: 500 + web.tool_request_gap: + class: burst_fanout + position: after + min_ms: 120 + max_ms: 1500 + +network_sensor_observation: + # Default assumes security infrastructure with good time sync and local tap + # placement. These bounds apply when multiple Zeek sensors observe the same + # packet/flow and should stay in the low-millisecond range. + default_profile: well_synced + profiles: + well_synced: + clock_skew_us: + min: -1500 + max: 1500 + path_delay_us: + min: 50 + max: 2000 windows_event_time: collision_spacing: diff --git a/src/evidenceforge/config/activity/traffic_rates.yaml b/src/evidenceforge/config/activity/traffic_rates.yaml index d45f19e6..d6b706f6 100644 --- a/src/evidenceforge/config/activity/traffic_rates.yaml +++ b/src/evidenceforge/config/activity/traffic_rates.yaml @@ -14,7 +14,7 @@ low: user_activity: [5, 5] # events/user/hr - web: [10, 30] # requests/web_server/hr + web: [10, 30] # top-level web actions/web_server/hr dns_interval: [600, 1800] # seconds between DNS queries ntp: [1, 1] # syncs/host/hr smb_interval: [1200, 3000] # seconds between SMB ops diff --git a/src/evidenceforge/config/activity/web_session_profiles.yaml b/src/evidenceforge/config/activity/web_session_profiles.yaml new file mode 100644 index 00000000..afc1d4dc --- /dev/null +++ b/src/evidenceforge/config/activity/web_session_profiles.yaml @@ -0,0 +1,109 @@ +# Web server visitor profiles for inbound web_access baseline traffic. +# +# Human visitors use site_maps.yaml to generate page + subresource sessions. +# Synthetic tools use the request lists below so crawlers, health checks, API +# clients, and scanners keep source-native path/status/User-Agent behavior. +# +# Overlay behavior: nested dicts merge and lists extend. Project overlays can +# add visitor classes or append requests/User-Agents without copying this file. + +visitor_classes: + human_browser: + weight: 70 + kind: session + external: true + internal: true + browsing_intensity: normal + user_agent_pool: browser_any + user_agent_pool_by_os: + windows: browser_windows + linux: browser_linux + + crawler: + weight: 8 + kind: requests + external: true + internal: false + request_count: [1, 3] + user_agent_pool: crawler + referrer_mode: none + requests: + - {path: "/robots.txt", method: "GET", status: 200, type: "text/plain", weight: 45} + - {path: "/sitemap.xml", method: "GET", status: 200, type: "application/xml", weight: 35} + - {path: "/.well-known/security.txt", method: "GET", status: 200, type: "text/plain", weight: 20} + + health_check: + weight: 10 + kind: requests + external: false + internal: true + request_count: [1, 2] + user_agent_pool: health_check + referrer_mode: none + requests: + - {path: "/health", method: "GET", status: 200, type: "application/json", weight: 40} + - {path: "/api/v1/health", method: "GET", status: 200, type: "application/json", weight: 35} + - {path: "/status", method: "GET", status: 200, type: "text/plain", weight: 25} + + api_client: + weight: 7 + kind: requests + external: true + internal: true + request_count: [1, 4] + user_agent_pool: api_client + referrer_mode: same_origin + requests: + - {path: "/api/v1/status", method: "GET", status: 200, type: "application/json", weight: 35} + - {path: "/api/v1/data", method: "POST", status: 200, type: "application/json", weight: 35} + - {path: "/api/v2/events", method: "POST", status: 200, type: "application/json", weight: 20} + - {path: "/api/v1/auth/token", method: "POST", status: 200, type: "application/json", weight: 10} + + opportunistic_probe: + weight: 5 + kind: requests + external: true + internal: false + request_count: [1, 5] + user_agent_pool: scanner + referrer_mode: none + requests: + - {path: "/wp-login.php", method: "GET", status: 404, type: "text/html", weight: 22} + - {path: "/wp-admin/", method: "GET", status: 404, type: "text/html", weight: 18} + - {path: "/xmlrpc.php", method: "POST", status: 404, type: "text/html", weight: 14} + - {path: "/phpmyadmin/", method: "GET", status: 404, type: "text/html", weight: 14} + - {path: "/.env", method: "GET", status: 403, type: "text/html", weight: 14} + - {path: "/admin", method: "GET", status: 403, type: "text/html", weight: 10} + - {path: "/backup.sql", method: "GET", status: 404, type: "text/html", weight: 8} + +user_agent_pools: + browser_any: + - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" + - "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0" + - "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" + - "Mozilla/5.0 (iPhone; CPU iPhone OS 17_2 like Mac OS X) AppleWebKit/605.1.15 Mobile/15E148" + browser_windows: + - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + - "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0" + - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0" + browser_linux: + - "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + - "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" + crawler: + - "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" + - "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" + - "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" + health_check: + - "ELB-HealthChecker/2.0" + - "kube-probe/1.28" + - "Prometheus/2.47.0" + api_client: + - "python-requests/2.31.0" + - "Go-http-client/1.1" + - "Apache-HttpClient/4.5.14 (Java/17.0.9)" + - "curl/7.88.1" + scanner: + - "curl/7.88.1" + - "python-requests/2.31.0" + - "Mozilla/5.0 zgrab/0.x" diff --git a/src/evidenceforge/config/schemas.py b/src/evidenceforge/config/schemas.py index b832f800..707d1d38 100644 --- a/src/evidenceforge/config/schemas.py +++ b/src/evidenceforge/config/schemas.py @@ -680,6 +680,73 @@ def non_empty_weighted_lists(cls, v: list[Any]) -> list[Any]: return v +# --- Auth Noise --- + + +class AuthNoiseIntervalRange(BaseModel, extra="forbid"): + """A weighted interval range for auth-noise recurrence.""" + + min_minutes: int = Field(ge=1, le=1440) + max_minutes: int = Field(ge=1, le=1440) + weight: int = Field(gt=0) + + @model_validator(mode="after") + def valid_range(self) -> Self: + if self.max_minutes < self.min_minutes: + raise ValueError("max_minutes must be greater than or equal to min_minutes") + return self + + +class ScheduledStaleCredentialsConfig(BaseModel, extra="forbid"): + """Stale scheduled-task failed-logon noise profile.""" + + account_base_names: list[str] = Field(min_length=1) + host_count_min: int = Field(ge=1) + host_count_max: int = Field(ge=1) + interval_ranges: list[AuthNoiseIntervalRange] = Field(min_length=1) + first_occurrence_seconds_min: int = Field(ge=0, le=86_400) + first_occurrence_seconds_max: int = Field(ge=0, le=86_400) + jitter_seconds_min: int = Field(ge=-86_400, le=86_400) + jitter_seconds_max: int = Field(ge=-86_400, le=86_400) + skip_probability: float = Field(ge=0.0, le=0.95) + backoff_probability: float = Field(ge=0.0, le=0.95) + backoff_seconds_min: int = Field(ge=0, le=86_400) + backoff_seconds_max: int = Field(ge=0, le=86_400) + + @field_validator("account_base_names") + @classmethod + def account_base_names_non_empty(cls, v: list[str]) -> list[str]: + for name in v: + if not name or not name.strip(): + raise ValueError("account_base_names entries must be non-empty") + return v + + @model_validator(mode="after") + def valid_ranges(self) -> Self: + if self.host_count_max < self.host_count_min: + raise ValueError("host_count_max must be greater than or equal to host_count_min") + if self.first_occurrence_seconds_max < self.first_occurrence_seconds_min: + raise ValueError( + "first_occurrence_seconds_max must be greater than or equal to " + "first_occurrence_seconds_min" + ) + if self.jitter_seconds_max < self.jitter_seconds_min: + raise ValueError( + "jitter_seconds_max must be greater than or equal to jitter_seconds_min" + ) + if self.backoff_seconds_max < self.backoff_seconds_min: + raise ValueError( + "backoff_seconds_max must be greater than or equal to backoff_seconds_min" + ) + return self + + +class AuthNoiseConfig(BaseModel, extra="forbid"): + """Root schema for auth_noise.yaml.""" + + scheduled_stale_credentials: ScheduledStaleCredentialsConfig + + # --- Network Params --- diff --git a/src/evidenceforge/generation/activity/auth_noise.py b/src/evidenceforge/generation/activity/auth_noise.py new file mode 100644 index 00000000..781ffcae --- /dev/null +++ b/src/evidenceforge/generation/activity/auth_noise.py @@ -0,0 +1,38 @@ +# Copyright (c) 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: MIT + +"""Baseline authentication noise configuration loader.""" + +from __future__ import annotations + +from typing import Any + +from evidenceforge.config import get_activity_directory +from evidenceforge.config.overlay import deep_merge_dict, load_with_overlay + +_CONFIG_PATH = get_activity_directory() / "auth_noise.yaml" +_CACHED_DATA: dict[str, Any] | None = None + + +def load_auth_noise_config() -> dict[str, Any]: + """Load auth-noise config, merged with project-local overlay.""" + global _CACHED_DATA + if _CACHED_DATA is None: + _CACHED_DATA = load_with_overlay( + _CONFIG_PATH, + "activity/auth_noise.yaml", + deep_merge_dict, + ) + return _CACHED_DATA + + +def reset_auth_noise_cache() -> None: + """Clear cached auth-noise config. Intended for tests.""" + global _CACHED_DATA + _CACHED_DATA = None + + +def scheduled_stale_credentials_config() -> dict[str, Any]: + """Return stale scheduled-credential failure settings.""" + config = load_auth_noise_config().get("scheduled_stale_credentials", {}) + return config if isinstance(config, dict) else {} diff --git a/src/evidenceforge/generation/activity/browsing_session.py b/src/evidenceforge/generation/activity/browsing_session.py index a63bb667..0afa65a8 100644 --- a/src/evidenceforge/generation/activity/browsing_session.py +++ b/src/evidenceforge/generation/activity/browsing_session.py @@ -16,8 +16,10 @@ from dataclasses import dataclass from evidenceforge.generation.activity.http_content import ( + is_stable_resource_path, normalize_mime_type_for_path, response_size_for_mime, + response_size_for_status, ) from evidenceforge.generation.activity.proxy_uri import is_browser_like_proxy_domain from evidenceforge.generation.activity.site_maps import ( @@ -26,6 +28,7 @@ SubresourceDef, get_site_map, ) +from evidenceforge.generation.activity.timing_profiles import get_timing_window @dataclass @@ -63,8 +66,10 @@ class BrowsingRequest: } -def _response_size(rng: random.Random, content_type: str) -> int: +def _response_size(rng: random.Random, hostname: str, path: str, content_type: str) -> int: """Generate a realistic response size for a given content type.""" + if is_stable_resource_path(path): + return response_size_for_status(200, hostname, path) return response_size_for_mime(rng, content_type) @@ -75,6 +80,70 @@ def _request_size(rng: random.Random, method: str) -> int: return 0 +def _sample_profile_timing_ms( + rng: random.Random, + key: str, + *, + default_min_ms: int, + default_max_ms: int, + default_class: str, +) -> int: + """Sample a configured web-session timing window using the caller's RNG.""" + window = get_timing_window( + key, + default_min_ms=default_min_ms, + default_max_ms=default_max_ms, + default_position="after", + default_class=default_class, + ) + if window.max_ms <= window.min_ms: + return window.min_ms + return rng.randint(window.min_ms, window.max_ms) + + +def _subresource_delay_ms(rng: random.Random, content_type: str) -> int: + """Return render-pipeline timing for a page subresource.""" + if content_type in ("text/css", "application/javascript"): + return _sample_profile_timing_ms( + rng, + "web.asset_stylesheet_script_after_page", + default_min_ms=50, + default_max_ms=200, + default_class="burst_fanout", + ) + if content_type.startswith("font/"): + return _sample_profile_timing_ms( + rng, + "web.asset_font_after_page", + default_min_ms=300, + default_max_ms=600, + default_class="burst_fanout", + ) + if content_type.startswith("image/"): + return _sample_profile_timing_ms( + rng, + "web.asset_image_after_page", + default_min_ms=200, + default_max_ms=800, + default_class="burst_fanout", + ) + if content_type == "application/json": + return _sample_profile_timing_ms( + rng, + "web.asset_api_after_page", + default_min_ms=500, + default_max_ms=2_000, + default_class="burst_fanout", + ) + return _sample_profile_timing_ms( + rng, + "web.asset_other_after_page", + default_min_ms=100, + default_max_ms=500, + default_class="burst_fanout", + ) + + def _make_referrer(hostname: str, path: str, port: int = 443) -> str: """Build a full referrer URL from hostname and path.""" scheme = "https" if port == 443 else "http" @@ -117,6 +186,7 @@ def generate_browsing_session( source_os: str = "windows", browsing_intensity: str = "normal", port: int = 443, + require_browser_like_domain: bool = True, ) -> list[BrowsingRequest]: """Generate a complete browsing session as a list of HTTP requests. @@ -131,11 +201,14 @@ def generate_browsing_session( source_os: Source host OS ("windows" or "linux"). browsing_intensity: "light", "normal", or "heavy". port: Destination port (443 for HTTPS, 80 for HTTP). + require_browser_like_domain: When true, suppress sessions for + certificate/update/telemetry domains. Set false for inbound + web-server logs where the public host may not exist in dns_registry. Returns: List of BrowsingRequest objects sorted by time_offset_ms. """ - if not is_browser_like_proxy_domain(hostname): + if require_browser_like_domain and not is_browser_like_proxy_domain(hostname): return [] site_map = get_site_map(hostname, domain_tags, rng) @@ -187,8 +260,13 @@ def generate_browsing_session( next_idx = _pick_next_page(rng, site_map, current_page, visited_indices) current_page_idx = next_idx - # Inter-page navigation delay: 3-30 seconds - current_ms += rng.randint(3_000, 30_000) + current_ms += _sample_profile_timing_ms( + rng, + "web.session_navigation", + default_min_ms=3_000, + default_max_ms=30_000, + default_class="human_workflow", + ) page = site_map.pages[current_page_idx] page_content_type = normalize_mime_type_for_path(page.path, page.content_type) @@ -206,7 +284,7 @@ def generate_browsing_session( referrer=previous_page_url, trans_depth=1, is_page_load=True, - response_body_len=_response_size(rng, page_content_type), + response_body_len=_response_size(rng, hostname, page.path, page_content_type), request_body_len=_request_size(rng, "GET"), ) ) @@ -220,17 +298,7 @@ def generate_browsing_session( sub_hostname = sub.host or hostname sub_content_type = normalize_mime_type_for_path(sub.path, sub.content_type) - # Timing: CSS/JS load early, images later, API calls latest - if sub_content_type in ("text/css", "application/javascript"): - delay = rng.randint(50, 200) - elif sub_content_type.startswith("font/"): - delay = rng.randint(300, 600) - elif sub_content_type.startswith("image/"): - delay = rng.randint(200, 800) - elif sub_content_type == "application/json": - delay = rng.randint(500, 2_000) - else: - delay = rng.randint(100, 500) + delay = _subresource_delay_ms(rng, sub_content_type) requests.append( BrowsingRequest( @@ -242,7 +310,12 @@ def generate_browsing_session( referrer=page_url, trans_depth=sub_idx + 2, # Page is depth 1, subs start at 2 is_page_load=False, - response_body_len=_response_size(rng, sub_content_type), + response_body_len=_response_size( + rng, + sub_hostname, + sub.path, + sub_content_type, + ), request_body_len=_request_size(rng, sub.method), ) ) diff --git a/src/evidenceforge/generation/activity/generator.py b/src/evidenceforge/generation/activity/generator.py index 81e0b040..a766423c 100644 --- a/src/evidenceforge/generation/activity/generator.py +++ b/src/evidenceforge/generation/activity/generator.py @@ -1079,6 +1079,25 @@ def _dns_rtt(rng: random.Random, resolver_ip: str | None = None) -> float: return rng.uniform(0.080, 0.250) # Slow/distant: 80-250ms +def _jitter_default_connection_duration( + duration: float | None, + *, + caller_provided_duration: bool, + seed_parts: tuple[Any, ...], +) -> float | None: + """Diversify generator-owned placeholder durations without changing authored values.""" + if caller_provided_duration or duration is None: + return duration + anchors = (0.8, 2.0, 0.2, 0.1, 0.02, 0.01) + if not any(math.isclose(duration, anchor, rel_tol=0.0, abs_tol=1e-9) for anchor in anchors): + return duration + seed = _stable_seed("default_conn_duration:" + ":".join(str(part) for part in seed_parts)) + rng = random.Random(seed) + if duration <= 0.02: + return max(0.0005, duration * rng.uniform(0.55, 1.85) + rng.uniform(0.0002, 0.004)) + return max(0.001, duration * rng.uniform(0.82, 1.24) + rng.uniform(-0.015, 0.035)) + + def _dns_registrable_domain(hostname: str) -> str: """Return a practical DNS owner name for mail/TXT companion lookups.""" parts = [part for part in hostname.rstrip(".").split(".") if part] @@ -3433,6 +3452,22 @@ def generate_failed_logon( user_sid = self._get_sid(effective_username) failure_reason = "%%2307" + remote_linux_source = ( + _get_os_category(system.os) == "linux" + and source_ip not in (None, "-") + and source_ip != system.ip + ) + linux_ssh_source_port = None + if remote_linux_source and source_ip is not None: + linux_ssh_source_port = self._allocate_ephemeral_port( + source_ip, + system.ip, + 22, + "tcp", + time, + self._os_for_ip(source_ip), + ) + event = SecurityEvent( timestamp=time, event_type="failed_logon", @@ -3445,8 +3480,10 @@ def generate_failed_logon( failure_reason=failure_reason, failure_status="0xc000006d", failure_substatus=substatus, - source_ip=auth_source_ip, - source_port=failed_profile["source_port"], + source_ip=( + source_ip if remote_linux_source and source_ip is not None else auth_source_ip + ), + source_port=linux_ssh_source_port or failed_profile["source_port"], auth_package=failed_profile["auth_package"], logon_process=failed_profile["logon_process"], lm_package=failed_profile["lm_package"], @@ -3466,6 +3503,7 @@ def generate_failed_logon( from evidenceforge.events.contexts import SyslogContext if source_ip and source_ip != "-": + ssh_source_port = linux_ssh_source_port or _ephemeral_port(_get_rng(), "linux") event.syslog = SyslogContext( app_name="sshd", pid=_get_rng().randint(5000, 60000), @@ -3473,7 +3511,7 @@ def generate_failed_logon( severity=4, message=( f"Failed password for {effective_username} from {source_ip} " - f"port {_ephemeral_port(_get_rng(), 'linux')} ssh2" + f"port {ssh_source_port} ssh2" ), ) else: @@ -3488,6 +3526,15 @@ def generate_failed_logon( ), ) + if remote_linux_source and source_ip is not None and linux_ssh_source_port is not None: + self._emit_failed_linux_ssh_network_connection( + system=system, + time=time, + source_ip=source_ip, + source_port=linux_ssh_source_port, + rng=rng, + ) + self.dispatcher.dispatch(event) # Domain controller side: validation evidence only. The failed local logon @@ -3651,6 +3698,30 @@ def _workstation_name_for_source(source_ip: str) -> str: return rdns.split(".", 1)[0].upper() return source_ip + def _emit_failed_linux_ssh_network_connection( + self, + system: System, + time: datetime, + source_ip: str, + source_port: int, + rng: random.Random, + ) -> None: + """Emit source-matched Zeek SSH evidence for a failed Linux sshd logon.""" + conn_time = time - timedelta(milliseconds=rng.randint(35, 450)) + self.generate_connection( + src_ip=source_ip, + dst_ip=system.ip, + time=conn_time, + dst_port=22, + proto="tcp", + service="ssh", + duration=rng.uniform(0.12, 3.5), + orig_bytes=rng.randint(260, 1800), + resp_bytes=rng.randint(240, 2600), + src_port=source_port, + conn_state=rng.choices(["SF", "RSTR"], weights=[78, 22], k=1)[0], + ) + def _maybe_emit_failed_logon_network_connection( self, system: System, @@ -4876,6 +4947,7 @@ def generate_connection( """ from evidenceforge.events.contexts import NetworkContext + caller_provided_duration = duration is not None caller_provided_conn_state = conn_state is not None caller_provided_payload = ( service is not None @@ -5177,7 +5249,19 @@ def generate_connection( _stable_seed(f"proxy_egress_delay:{src_ip}:{dst_ip}:{time.timestamp()}") ).randint(proxy_delay_window.min_ms, proxy_delay_window.max_ms) ) - client_duration = min(duration or 0.2, 2.0) + proxy_client_cap = random.Random( + _stable_seed( + "proxy_client_duration_cap:" + f"{src_ip}:{proxy_sys.ip}:{dst_ip}:{dst_port}:{time.timestamp()}" + ) + ).uniform(1.72, 2.36) + client_duration = min(duration if duration is not None else 0.2, proxy_client_cap) + if duration is None: + client_duration = _jitter_default_connection_duration( + client_duration, + caller_provided_duration=False, + seed_parts=(src_ip, proxy_sys.ip, dst_ip, dst_port, time, "proxy_client"), + ) if dst_port == 443 and proxy_context.status_code < 400: client_duration = duration or _get_rng().uniform(0.5, 10.0) if proxy_context.method == "CONNECT": @@ -5195,7 +5279,11 @@ def generate_connection( client_orig_bytes += framing_rng.randint(160, 900) client_resp_bytes += framing_rng.randint(180, 2400) if will_emit_egress: - egress_duration = duration or 0.1 + egress_duration = duration or _jitter_default_connection_duration( + 0.1, + caller_provided_duration=False, + seed_parts=(proxy_sys.ip, dst_ip, dst_port, time, "proxy_egress"), + ) response_flush = random.Random( _stable_seed(f"proxy_response_flush:{src_ip}:{dst_ip}:{time.timestamp()}") ).uniform(0.02, 0.25) @@ -5495,7 +5583,15 @@ def generate_connection( resp_bytes=resp_bytes, ) elif service == "dns" and proto in ("udp", "tcp") and dst_port == 53: - duration = min(duration or 0.02, 0.08) + duration = min( + duration + or _jitter_default_connection_duration( + 0.02, + caller_provided_duration=False, + seed_parts=(src_ip, dst_ip, dst_port, time, "dns_default"), + ), + 0.08, + ) orig_bytes = min(max(orig_bytes or 40, 40), 260) if resp_bytes is None: resp_bytes = 120 @@ -5733,6 +5829,21 @@ def generate_connection( if duration is None or duration < http_min_duration: duration = http_min_duration + rng.uniform(0.0, 0.025) + duration_locked_to_dns_rtt = ( + service == "dns" + and proto in ("udp", "tcp") + and dst_port == 53 + and dns is not None + and dns.rtt is not None + and duration is not None + and math.isclose(duration, dns.rtt, rel_tol=0.0, abs_tol=1e-9) + ) + duration = _jitter_default_connection_duration( + duration, + caller_provided_duration=caller_provided_duration or duration_locked_to_dns_rtt, + seed_parts=(src_ip, src_port, dst_ip, dst_port, proto, service or "", time), + ) + # Calculate packet counts — enforce consistency with history if proto == "udp" and history: orig_pkts = max(history.count("D"), math.ceil((orig_bytes or 0) / 1232)) diff --git a/src/evidenceforge/generation/activity/timing_profiles.py b/src/evidenceforge/generation/activity/timing_profiles.py index c1711205..64f1058f 100644 --- a/src/evidenceforge/generation/activity/timing_profiles.py +++ b/src/evidenceforge/generation/activity/timing_profiles.py @@ -20,6 +20,7 @@ _MAX_COLLISION_NEAR_ZERO_UNTIL = 10_000 _MAX_COLLISION_GAP_US = 1_000_000 _MAX_COLLISION_GAP_MS = 60_000 +_MAX_SENSOR_TIMING_US = 1_000_000 @dataclass(frozen=True, slots=True) @@ -32,6 +33,16 @@ class TimingWindow: relationship_class: str = "" +@dataclass(frozen=True, slots=True) +class NetworkSensorObservationTiming: + """Per-sensor observation timing bounds for well-synced network sensors.""" + + clock_skew_min_us: int + clock_skew_max_us: int + path_delay_min_us: int + path_delay_max_us: int + + def load_timing_profiles() -> dict[str, Any]: """Load timing profiles, merged with project-local overlay.""" global _CACHED_DATA @@ -59,6 +70,24 @@ def _safe_int(value: Any, fallback: int, *, minimum: int, maximum: int) -> int: return max(minimum, min(parsed, maximum)) +def _safe_int_range( + value: Any, + *, + fallback_min: int, + fallback_max: int, + minimum: int, + maximum: int, +) -> tuple[int, int]: + """Read a ``{min, max}`` mapping and fall back when the range is invalid.""" + if not isinstance(value, dict): + return fallback_min, fallback_max + min_value = _safe_int(value.get("min"), fallback_min, minimum=minimum, maximum=maximum) + max_value = _safe_int(value.get("max"), fallback_max, minimum=minimum, maximum=maximum) + if max_value < min_value: + return fallback_min, fallback_max + return min_value, max_value + + def get_timing_window( key: str, *, @@ -119,6 +148,41 @@ def sample_packet_timing_delta(key: str, *, seed_parts: tuple[Any, ...] = ()) -> return base_delta + timedelta(microseconds=rng.randint(37, 997)) +def network_sensor_observation_timing() -> NetworkSensorObservationTiming: + """Return safe timing bounds for a well-synced Zeek/network sensor fleet.""" + data = load_timing_profiles().get("network_sensor_observation", {}) + if not isinstance(data, dict): + data = {} + profiles = data.get("profiles", {}) + if not isinstance(profiles, dict): + profiles = {} + default_profile = data.get("default_profile", "well_synced") + profile = profiles.get(default_profile, {}) + if not isinstance(profile, dict): + profile = {} + + skew_min, skew_max = _safe_int_range( + profile.get("clock_skew_us"), + fallback_min=-1_500, + fallback_max=1_500, + minimum=-_MAX_SENSOR_TIMING_US, + maximum=_MAX_SENSOR_TIMING_US, + ) + delay_min, delay_max = _safe_int_range( + profile.get("path_delay_us"), + fallback_min=50, + fallback_max=2_000, + minimum=0, + maximum=_MAX_SENSOR_TIMING_US, + ) + return NetworkSensorObservationTiming( + clock_skew_min_us=skew_min, + clock_skew_max_us=skew_max, + path_delay_min_us=delay_min, + path_delay_max_us=delay_max, + ) + + def windows_collision_spacing_config() -> dict[str, int]: """Return Windows/Sysmon same-timestamp collision spacing settings.""" spacing = load_timing_profiles().get("windows_event_time", {}).get("collision_spacing", {}) diff --git a/src/evidenceforge/generation/activity/web_session_profiles.py b/src/evidenceforge/generation/activity/web_session_profiles.py new file mode 100644 index 00000000..6d26c221 --- /dev/null +++ b/src/evidenceforge/generation/activity/web_session_profiles.py @@ -0,0 +1,132 @@ +# Copyright (c) 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: MIT + +"""Inbound web server visitor profile loader and selection helpers.""" + +from __future__ import annotations + +import random +from typing import Any + +from evidenceforge.config import get_activity_directory +from evidenceforge.config.overlay import deep_merge_dict, load_with_overlay + +_CONFIG_PATH = get_activity_directory() / "web_session_profiles.yaml" +_CACHED_DATA: dict[str, Any] | None = None + + +def load_web_session_profiles() -> dict[str, Any]: + """Load inbound web visitor profiles from YAML, merged with overlay. Cached.""" + global _CACHED_DATA + if _CACHED_DATA is None: + _CACHED_DATA = load_with_overlay( + _CONFIG_PATH, + "activity/web_session_profiles.yaml", + deep_merge_dict, + ) + return _CACHED_DATA + + +def reset_web_session_profiles_cache() -> None: + """Clear cached web visitor profile data. Intended for tests.""" + global _CACHED_DATA + _CACHED_DATA = None + + +def _positive_weight(value: Any, fallback: float = 1.0) -> float: + try: + parsed = float(value) + except (TypeError, ValueError): + return fallback + return parsed if parsed > 0 else fallback + + +def _visitor_candidates( + data: dict[str, Any], *, is_external: bool +) -> list[tuple[str, dict[str, Any]]]: + classes = data.get("visitor_classes", {}) + if not isinstance(classes, dict): + return [] + allowed_key = "external" if is_external else "internal" + candidates: list[tuple[str, dict[str, Any]]] = [] + for name, profile in classes.items(): + if not isinstance(profile, dict): + continue + if profile.get(allowed_key, True) is False: + continue + candidates.append((str(name), profile)) + return candidates + + +def pick_web_visitor_profile( + rng: random.Random, *, is_external: bool +) -> tuple[str, dict[str, Any]]: + """Pick a visitor profile appropriate for an internal or external client.""" + data = load_web_session_profiles() + candidates = _visitor_candidates(data, is_external=is_external) + if not candidates: + return ( + "human_browser", + { + "kind": "session", + "browsing_intensity": "normal", + "user_agent_pool": "browser_any", + }, + ) + weights = [_positive_weight(profile.get("weight")) for _, profile in candidates] + return rng.choices(candidates, weights=weights, k=1)[0] + + +def pick_web_user_agent( + rng: random.Random, + profile: dict[str, Any], + *, + source_os: str | None = None, +) -> str: + """Pick a User-Agent from the profile's configured pool.""" + data = load_web_session_profiles() + pools = data.get("user_agent_pools", {}) + if not isinstance(pools, dict): + pools = {} + + pool_name = None + by_os = profile.get("user_agent_pool_by_os") + if isinstance(by_os, dict) and source_os: + pool_name = by_os.get(source_os) + if not isinstance(pool_name, str): + pool_name = profile.get("user_agent_pool") + pool = pools.get(pool_name) if isinstance(pool_name, str) else None + if not isinstance(pool, list) or not pool: + pool = pools.get("browser_any", []) + if not isinstance(pool, list) or not pool: + return "Mozilla/5.0" + return str(rng.choice(pool)) + + +def pick_profile_request(rng: random.Random, profile: dict[str, Any]) -> dict[str, Any]: + """Pick a configured request entry from a non-session visitor profile.""" + requests = profile.get("requests", []) + if not isinstance(requests, list) or not requests: + return {"path": "/", "method": "GET", "status": 200, "type": "text/html"} + choices = [entry for entry in requests if isinstance(entry, dict)] + if not choices: + return {"path": "/", "method": "GET", "status": 200, "type": "text/html"} + weights = [_positive_weight(entry.get("weight")) for entry in choices] + return dict(rng.choices(choices, weights=weights, k=1)[0]) + + +def request_count_bounds(profile: dict[str, Any]) -> tuple[int, int]: + """Return safe per-visitor request count bounds for non-session profiles.""" + raw_bounds = profile.get("request_count", [1, 1]) + if not isinstance(raw_bounds, (list, tuple)) or len(raw_bounds) != 2: + return 1, 1 + try: + lo = int(raw_bounds[0]) + hi = int(raw_bounds[1]) + except (TypeError, ValueError): + return 1, 1 + lo = max(1, min(lo, 50)) + hi = max(1, min(hi, 50)) + if hi < lo: + return 1, 1 + return lo, hi diff --git a/src/evidenceforge/generation/emitters/zeek_base.py b/src/evidenceforge/generation/emitters/zeek_base.py index 54adf613..85f899fc 100644 --- a/src/evidenceforge/generation/emitters/zeek_base.py +++ b/src/evidenceforge/generation/emitters/zeek_base.py @@ -46,6 +46,7 @@ from typing import Any from evidenceforge.formats.format_def import FormatDefinition +from evidenceforge.generation.activity.timing_profiles import network_sensor_observation_timing from evidenceforge.generation.emitters.base import LogEmitter from evidenceforge.utils.paths import sanitize_path_component from evidenceforge.utils.rng import _stable_seed @@ -76,18 +77,21 @@ def _sensor_variation_fraction(hostname: str, uid: Any, field: str, magnitude: f def _sensor_clock_skew_us(hostname: str) -> int: """Return stable per-sensor clock skew in microseconds.""" + timing = network_sensor_observation_timing() seed = _stable_seed(f"zeek_sensor_clock_skew:{hostname}") - return (seed % 800_001) - 400_000 + width = timing.clock_skew_max_us - timing.clock_skew_min_us + 1 + return timing.clock_skew_min_us + (seed % max(1, width)) def _sensor_path_delay_us(hostname: str, original_uid: Any) -> int: """Return per-flow capture timestamp variance for a sensor observation.""" + timing = network_sensor_observation_timing() seed = _stable_seed(f"zeek_sensor_path_delay:{hostname}:{original_uid}") # Tap placement, NIC timestamping, Zeek scheduling, and capture buffering - # all add small positive path delay. The stable per-sensor clock skew owns - # the sign of cross-sensor offsets, so identical paths do not flip earlier - # and later flow-by-flow like independent synthetic jitter. - return 5_000 + (seed % 75_001) + # add a small positive delay. The profile keeps this consistent with a + # well-synced sensor fleet instead of synthetic hundreds-of-ms offsets. + width = timing.path_delay_max_us - timing.path_delay_min_us + 1 + return timing.path_delay_min_us + (seed % max(1, width)) def _jitter_numeric_observation( diff --git a/src/evidenceforge/generation/engine/baseline.py b/src/evidenceforge/generation/engine/baseline.py index 34949ae9..7dc9e7d9 100644 --- a/src/evidenceforge/generation/engine/baseline.py +++ b/src/evidenceforge/generation/engine/baseline.py @@ -39,6 +39,7 @@ from evidenceforge.config import get_activity_directory from evidenceforge.config.overlay import load_with_overlay, merge_keyed_list +from evidenceforge.generation.activity.auth_noise import scheduled_stale_credentials_config from evidenceforge.generation.activity.create_remote_thread_patterns import ( load_create_remote_thread_noise_config, load_create_remote_thread_patterns, @@ -233,6 +234,95 @@ def _pick_non_colliding_account_name( raise ValueError(msg) +def _as_int(value: Any, default: int) -> int: + """Return an integer config value or a default.""" + try: + return int(value) + except (TypeError, ValueError): + return default + + +def _as_probability(value: Any, default: float) -> float: + """Return a clamped probability config value.""" + try: + probability = float(value) + except (TypeError, ValueError): + probability = default + return max(0.0, min(probability, 0.95)) + + +def _weighted_interval_minutes( + rng: random.Random, + interval_ranges: list[dict[str, Any]], +) -> int: + """Pick an interval in minutes from weighted config ranges.""" + ranges = [entry for entry in interval_ranges if isinstance(entry, dict)] + if not ranges: + ranges = [ + {"min_minutes": 55, "max_minutes": 95, "weight": 30}, + {"min_minutes": 105, "max_minutes": 155, "weight": 45}, + {"min_minutes": 170, "max_minutes": 260, "weight": 25}, + ] + weights = [max(1, _as_int(entry.get("weight"), 1)) for entry in ranges] + selected = rng.choices(ranges, weights=weights, k=1)[0] + min_minutes = max(1, _as_int(selected.get("min_minutes"), 60)) + max_minutes = max(min_minutes, _as_int(selected.get("max_minutes"), min_minutes)) + return rng.randint(min_minutes, max_minutes) + + +def _scheduled_stale_failure_offsets( + *, + scenario_name: str, + account_name: str, + hostname: str, + hour_idx: int, + config: dict[str, Any], +) -> list[int]: + """Return stale scheduled-credential failure offsets for this generation hour.""" + if hour_idx < 0: + return [] + + profile_key = f"sched_fail_profile:{scenario_name}:{account_name}:{hostname}" + profile_rng = random.Random(_stable_seed(profile_key)) + first_min = max(0, _as_int(config.get("first_occurrence_seconds_min"), 0)) + first_max = max(first_min, _as_int(config.get("first_occurrence_seconds_max"), 2700)) + first_offset = profile_rng.randint(first_min, first_max) + + jitter_min = _as_int(config.get("jitter_seconds_min"), -420) + jitter_max = max(jitter_min, _as_int(config.get("jitter_seconds_max"), 780)) + skip_probability = _as_probability(config.get("skip_probability"), 0.16) + backoff_probability = _as_probability(config.get("backoff_probability"), 0.10) + backoff_min = max(0, _as_int(config.get("backoff_seconds_min"), 900)) + backoff_max = max(backoff_min, _as_int(config.get("backoff_seconds_max"), 3600)) + interval_ranges = config.get("interval_ranges") + if not isinstance(interval_ranges, list): + interval_ranges = [] + + window_start = hour_idx * 3600 + window_end = window_start + 3600 + offsets: list[int] = [] + nominal_second = first_offset + occurrence_idx = 0 + while nominal_second < window_end + max(0, -jitter_min) + backoff_max: + occurrence_rng = random.Random(_stable_seed(f"{profile_key}:occurrence:{occurrence_idx}")) + observed_second = nominal_second + occurrence_rng.randint(jitter_min, jitter_max) + if occurrence_rng.random() < backoff_probability: + observed_second += occurrence_rng.randint(backoff_min, backoff_max) + if ( + occurrence_rng.random() >= skip_probability + and window_start <= observed_second < window_end + ): + offsets.append(int(observed_second - window_start)) + + interval_minutes = _weighted_interval_minutes(occurrence_rng, interval_ranges) + nominal_second += interval_minutes * 60 + occurrence_idx += 1 + if occurrence_idx > 10_000: + break + + return sorted(set(offsets)) + + def _hawkes_params_from_persona(persona: Persona | None) -> dict: """Derive Hawkes kernel parameters from persona risk_profile. @@ -430,110 +520,6 @@ def _load_systemd_schedules() -> list[dict[str, Any]]: return _CACHED_SCHEDULES -# Weighted web request categories for realistic path diversity at high volume. -# (category_weight, paths_within_category) -_WEB_REQUEST_CATEGORIES: list[tuple[float, list[tuple[str, str, int, str]]]] = [ - # (weight, [(path, method, status, mime), ...]) - ( - 40, - [ # Page views - ("/", "GET", 200, "text/html"), - ("/index.html", "GET", 200, "text/html"), - ("/about", "GET", 200, "text/html"), - ("/contact", "GET", 200, "text/html"), - ("/products", "GET", 200, "text/html"), - ("/services", "GET", 200, "text/html"), - ("/blog", "GET", 200, "text/html"), - ("/login", "GET", 200, "text/html"), - ("/dashboard", "GET", 200, "text/html"), - ("/search?q=help", "GET", 200, "text/html"), - ], - ), - ( - 30, - [ # Static assets - ("/assets/main.css", "GET", 200, "text/css"), - ("/assets/app.js", "GET", 200, "application/javascript"), - ("/assets/vendor.js", "GET", 200, "application/javascript"), - ("/images/logo.png", "GET", 200, "image/png"), - ("/images/banner.jpg", "GET", 200, "image/jpeg"), - ("/favicon.ico", "GET", 200, "image/x-icon"), - ("/fonts/roboto.woff2", "GET", 200, "font/woff2"), - ("/assets/style.min.css", "GET", 200, "text/css"), - ], - ), - ( - 15, - [ # API calls - ("/api/v1/health", "GET", 200, "application/json"), - ("/api/v1/data", "POST", 200, "application/json"), - ("/api/v1/users", "GET", 200, "application/json"), - ("/api/v1/status", "GET", 200, "application/json"), - ("/api/v2/events", "POST", 200, "application/json"), - ("/api/v1/auth/token", "POST", 200, "application/json"), - ], - ), - ( - 8, - [ # Bot/crawler probes - ("/robots.txt", "GET", 200, "text/plain"), - ("/sitemap.xml", "GET", 200, "application/xml"), - ("/.well-known/security.txt", "GET", 200, "text/plain"), - ], - ), - ( - 7, - [ # 404/403 noise (opportunistic scanners, mistyped URLs) - ("/wp-login.php", "GET", 404, "text/html"), - ("/admin", "GET", 403, "text/html"), - ("/.env", "GET", 403, "text/html"), - ("/phpmyadmin/", "GET", 404, "text/html"), - ("/xmlrpc.php", "POST", 404, "text/html"), - ("/wp-admin/", "GET", 404, "text/html"), - ("/cgi-bin/", "GET", 403, "text/html"), - ("/backup.sql", "GET", 404, "text/html"), - ], - ), -] - -# Pre-compute flattened weights for fast sampling -_WEB_REQ_FLAT: list[tuple[str, str, int, str]] = [] -_WEB_REQ_WEIGHTS: list[float] = [] -for _cat_weight, _cat_paths in _WEB_REQUEST_CATEGORIES: - per_path_weight = _cat_weight / len(_cat_paths) - for _entry in _cat_paths: - _WEB_REQ_FLAT.append(_entry) - _WEB_REQ_WEIGHTS.append(per_path_weight) - -# Parameterized path templates for additional diversity at high volume -_PARAMETERIZED_PATHS: list[tuple[str, str, int, str]] = [ - ("/products/{id}", "GET", 200, "text/html"), - ("/users/{id}/profile", "GET", 200, "application/json"), - ("/api/v1/items/{id}", "GET", 200, "application/json"), - ("/blog/post-{id}", "GET", 200, "text/html"), - ("/images/gallery/{id}.jpg", "GET", 200, "image/jpeg"), - ("/docs/page/{id}", "GET", 200, "text/html"), -] - - -def _generate_web_request(rng: random.Random) -> tuple[str, str, int, str]: - """Generate a realistic web request (path, method, status, mime). - - Uses weighted categories for realistic URI distribution. Occasionally - generates parameterized paths for additional variety. - """ - from evidenceforge.generation.activity.http_content import normalize_mime_type_for_path - - # 20% chance of parameterized path for extra diversity - if rng.random() < 0.20: - template, method, status, mime = rng.choice(_PARAMETERIZED_PATHS) - path = template.replace("{id}", str(rng.randint(1, 9999))) - return (path, method, status, normalize_mime_type_for_path(path, mime)) - - path, method, status, mime = rng.choices(_WEB_REQ_FLAT, weights=_WEB_REQ_WEIGHTS, k=1)[0] - return (path, method, status, normalize_mime_type_for_path(path, mime)) - - def _machine_account_tgs_gap_ms(rng: random.Random, *, first: bool) -> int: """Return a realistic gap before machine-account service-ticket requests.""" if first: @@ -1124,10 +1110,25 @@ def _generate_baseline_failed_logons(self, current_hour: datetime) -> None: ) # Pattern 2: Scheduled task with stale creds (deterministic per scenario). - # Pick 1-2 hosts and a plausible service account name. + # Pick configured hosts and a plausible service account name. _sched_seed = _stable_seed(self.scenario.name + "_sched_fail") _sched_rng = random.Random(_sched_seed) - _svc_names = ["svc_backup", "svc_monitor", "svc_report", "svc_deploy", "svc_scan"] + _sched_config = scheduled_stale_credentials_config() + configured_names = _sched_config.get("account_base_names", []) + _svc_names = [ + str(name).strip() for name in configured_names if isinstance(name, str) and name.strip() + ] or [ + "svc_backup", + "svc_monitor", + "svc_report", + "svc_deploy", + "svc_scan", + "svc_patch", + "svc_build", + "svc_sync", + "svc_jobs", + "svc_batch", + ] # Ensure no collision with actual scenario accounts _existing = {u.username for u in self.scenario.environment.users} | set( self.scenario.environment.service_accounts @@ -1143,35 +1144,28 @@ def _generate_baseline_failed_logons(self, current_hour: datetime) -> None: email=f"{_sched_acct}@system.local", enabled=False, ) - n_sched_hosts = min(2, len(servers)) + host_min = max(1, _as_int(_sched_config.get("host_count_min"), 1)) + host_max = max(host_min, _as_int(_sched_config.get("host_count_max"), 2)) + n_sched_hosts = min(_sched_rng.randint(host_min, host_max), len(servers)) _sched_hosts = _sched_rng.sample(servers, n_sched_hosts) hour_idx = int((current_hour - self.start_time).total_seconds() / 3600) for host in _sched_hosts: - profile_rng = random.Random( - _stable_seed( - f"sched_fail_profile:{self.scenario.name}:{_sched_acct}:{host.hostname}" - ) - ) - sched_interval = profile_rng.choices([1, 2, 3], weights=[35, 45, 20], k=1)[0] - sched_phase = profile_rng.randint(0, sched_interval - 1) - if (hour_idx + sched_phase) % sched_interval != 0: - continue - hour_rng = random.Random( - _stable_seed( - f"sched_fail_time:{self.scenario.name}:{_sched_acct}:{host.hostname}:{hour_idx}" + for sched_second in _scheduled_stale_failure_offsets( + scenario_name=self.scenario.name, + account_name=_sched_acct, + hostname=host.hostname, + hour_idx=hour_idx, + config=_sched_config, + ): + sched_time = current_hour + timedelta(seconds=sched_second) + self.state_manager.set_current_time(sched_time) + self.activity_generator.generate_failed_logon( + user=_sched_user, + system=host, + time=sched_time, + logon_type=4, # batch (scheduled task) + source_ip=host.ip, ) - ) - nominal_second = profile_rng.randint(0, 900) - sched_second = max(0, min(3599, nominal_second + hour_rng.randint(-180, 540))) - sched_time = current_hour + timedelta(seconds=sched_second) - self.state_manager.set_current_time(sched_time) - self.activity_generator.generate_failed_logon( - user=_sched_user, - system=host, - time=sched_time, - logon_type=4, # batch (scheduled task) - source_ip=host.ip, - ) # Pattern 3: Management software sweep (1-2 per business day). # Use scenario-local time for business-hour gating. @@ -5314,159 +5308,228 @@ def _svc_pid(*keys: str, _pids: dict = sys_pids) -> int: # noqa: B006 # Web access logs if "web_access" in self.emitters: - _WEB_UAS_BROWSER = [ - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " - "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " - "(KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36", - "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0", - "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " - "(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36", - "Mozilla/5.0 (iPhone; CPU iPhone OS 17_2 like Mac OS X) AppleWebKit/605.1.15 Mobile/15E148", - "curl/7.88.1", - "python-requests/2.31.0", - ] - _WEB_UAS_BOT = [ - "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", - "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", - "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)", - ] for sys_obj in systems: - if "web_server" not in (sys_obj.roles or []): - continue - _web_lo, _web_hi = self._resolve_traffic_rate("web") - num_reqs = rng.randint(_web_lo, _web_hi) - - internal_ips = [s.ip for s in systems if s.ip != sys_obj.ip] - _segment = self._get_segment_for_system(sys_obj) - exposure = _segment.exposure if _segment else self._get_system_exposure(sys_obj) - ext_ratio = ( - _segment.external_ratio - if _segment is not None and _segment.external_ratio is not None - else 0.6 - ) + self._emit_web_server_access(sys_obj, systems, rng, current_hour) - # Build Zipf-weighted visitor IP pool for realistic frequency distribution - ext_pool_size = min(200, max(10, num_reqs // 10)) - ext_ip_pool = [self._generate_external_client_ip(rng) for _ in range(ext_pool_size)] - ext_ip_weights = [1.0 / (i + 1) for i in range(ext_pool_size)] - - # Zipf-weighted internal pool for non-uniform health-check / monitoring traffic - if internal_ips: - int_ip_weights = [1.0 / (i + 1) for i in range(len(internal_ips))] - else: - int_ip_weights = [] + def _emit_web_server_access( + self, + sys_obj: Any, + systems: list[Any], + rng: random.Random, + current_hour: datetime, + ) -> None: + """Emit inbound web server traffic as sessions and source-native tool requests.""" + if "web_server" not in (sys_obj.roles or []): + return - _pub_hosts = getattr(sys_obj, "public_hostnames", None) or [] + from evidenceforge.events.contexts import HttpContext + from evidenceforge.generation.activity.browsing_session import generate_browsing_session + from evidenceforge.generation.activity.http_content import ( + is_stable_resource_path, + normalize_mime_type_for_path, + response_size_for_mime, + response_size_for_status, + ) + from evidenceforge.generation.activity.timing_profiles import get_timing_window + from evidenceforge.generation.activity.web_session_profiles import ( + pick_profile_request, + pick_web_user_agent, + pick_web_visitor_profile, + request_count_bounds, + ) - from evidenceforge.events.contexts import HttpContext + web_lo, web_hi = self._resolve_traffic_rate("web") + top_level_budget = rng.randint(web_lo, web_hi) + if top_level_budget <= 0: + return - for _ in range(num_reqs): - offset = rng.uniform(0, 3599) - ts = current_hour + timedelta(seconds=offset) - path, method, status, mime = _generate_web_request(rng) - if exposure == "external": - client_ip = rng.choices(ext_ip_pool, weights=ext_ip_weights, k=1)[0] - elif exposure == "both": - if rng.random() < ext_ratio: - client_ip = rng.choices(ext_ip_pool, weights=ext_ip_weights, k=1)[0] - else: - client_ip = ( - rng.choices(internal_ips, weights=int_ip_weights, k=1)[0] - if internal_ips - else "10.0.0.1" - ) - else: - client_ip = ( - rng.choices(internal_ips, weights=int_ip_weights, k=1)[0] - if internal_ips - else "10.0.0.1" - ) + internal_ips = [s.ip for s in systems if s.ip != sys_obj.ip] + segment = self._get_segment_for_system(sys_obj) + exposure = segment.exposure if segment else self._get_system_exposure(sys_obj) + ext_ratio = ( + segment.external_ratio + if segment is not None and segment.external_ratio is not None + else 0.6 + ) - is_external_client = not _is_private_ip(client_ip) - dst_port = 80 - dst_service = "http" - if is_external_client and rng.random() < 0.85: - dst_port = 443 - dst_service = "ssl" - if is_external_client and _pub_hosts: - http_host = rng.choice(_pub_hosts) - else: - http_host = sys_obj.hostname - ip_map = getattr(self.activity_generator, "_ip_to_system", {}) - client_sys = ip_map.get(client_ip) - if client_sys and _get_os_category(client_sys.os) == "linux": - ua_pool = [ - "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " - "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", - "curl/7.88.1", - "python-requests/2.31.0", - ] - else: - ua_pool = _WEB_UAS_BROWSER + (_WEB_UAS_BOT if is_external_client else []) - from evidenceforge.generation.activity.http_content import ( - is_stable_resource_path, - response_size_for_mime, - response_size_for_status, - ) + ext_pool_size = min(200, max(10, top_level_budget // 10)) + ext_ip_pool = [self._generate_external_client_ip(rng) for _ in range(ext_pool_size)] + ext_ip_weights = [1.0 / (i + 1) for i in range(ext_pool_size)] + int_ip_weights = [1.0 / (i + 1) for i in range(len(internal_ips))] + public_hosts = getattr(sys_obj, "public_hostnames", None) or [] + ip_map = getattr(self.activity_generator, "_ip_to_system", {}) + + def _choose_client_ip() -> str: + if exposure == "external": + return rng.choices(ext_ip_pool, weights=ext_ip_weights, k=1)[0] + if exposure == "both" and rng.random() < ext_ratio: + return rng.choices(ext_ip_pool, weights=ext_ip_weights, k=1)[0] + if internal_ips: + return rng.choices(internal_ips, weights=int_ip_weights, k=1)[0] + return "10.0.0.1" + + def _effective_dst_ip(is_external_client: bool) -> str: + dispatcher = getattr(self, "dispatcher", None) + if is_external_client and dispatcher is not None: + visibility = getattr(dispatcher, "visibility_engine", None) + real_to_vip = getattr(visibility, "_real_ip_to_vip", None) if visibility else None + vip = real_to_vip.get(sys_obj.ip) if isinstance(real_to_vip, dict) else None + if vip: + return vip + return sys_obj.ip + + def _status_message(status: int) -> str: + return { + 200: "OK", + 403: "Forbidden", + 404: "Not Found", + 405: "Method Not Allowed", + 500: "Internal Server Error", + 503: "Service Unavailable", + }.get(status, "OK") + + tool_gap = get_timing_window( + "web.tool_request_gap", + default_min_ms=120, + default_max_ms=1500, + default_position="after", + default_class="burst_fanout", + ) - resp_bytes = ( - response_size_for_status(status, http_host, path) - if status != 200 or is_stable_resource_path(path) - else response_size_for_mime(rng, mime) - ) - ua_rng = random.Random( - _stable_seed(f"web_client_ua:{client_ip}:{sys_obj.hostname}") - ) - chosen_ua = ua_rng.choice(ua_pool) - _ua_is_bot = any( - bot in chosen_ua for bot in ("Googlebot", "bingbot", "AhrefsBot") - ) - from evidenceforge.generation.activity.referrer import pick_referrer + def _tool_gap_ms() -> int: + if tool_gap.max_ms <= tool_gap.min_ms: + return tool_gap.min_ms + return rng.randint(tool_gap.min_ms, tool_gap.max_ms) + + top_level_emitted = 0 + attempts = 0 + while top_level_emitted < top_level_budget and attempts < top_level_budget * 4: + attempts += 1 + client_ip = _choose_client_ip() + is_external_client = not _is_private_ip(client_ip) + dst_port = 443 if is_external_client and rng.random() < 0.85 else 80 + dst_service = "ssl" if dst_port == 443 else "http" + http_host = ( + rng.choice(public_hosts) + if is_external_client and public_hosts + else sys_obj.hostname + ) + client_sys = ip_map.get(client_ip) + source_os = _get_os_category(client_sys.os) if client_sys is not None else None + profile_name, profile = pick_web_visitor_profile( + rng, + is_external=is_external_client, + ) + ua_rng = random.Random( + _stable_seed( + f"web_client_ua:{client_ip}:{http_host}:{profile_name}:{source_os or 'external'}" + ) + ) + chosen_ua = pick_web_user_agent(ua_rng, profile, source_os=source_os) + base_ts = current_hour + timedelta(seconds=rng.uniform(0, 3599)) + effective_dst_ip = _effective_dst_ip(is_external_client) - _site_map = getattr(sys_obj, "site_map", None) - _referer = pick_referrer( - rng, - http_host, - site_map=_site_map, - is_bot=_ua_is_bot, - context="general", - port=dst_port, - ) - effective_dst_ip = sys_obj.ip - if is_external_client and hasattr(self, "dispatcher"): - visibility = self.dispatcher.visibility_engine - vip = visibility._real_ip_to_vip.get(sys_obj.ip) if visibility else None - if vip: - effective_dst_ip = vip + if profile.get("kind") == "session": + session_requests = generate_browsing_session( + rng=rng, + hostname=http_host, + domain_tags=["web"], + source_os=source_os or "windows", + browsing_intensity=str(profile.get("browsing_intensity", "normal")), + port=dst_port, + require_browser_like_domain=False, + ) + current_page_allowed = False + for req in session_requests: + if req.is_page_load: + if top_level_emitted >= top_level_budget: + break + top_level_emitted += 1 + current_page_allowed = True + elif not current_page_allowed: + continue + if req.hostname != http_host: + continue + req_ts = base_ts + timedelta(milliseconds=req.time_offset_ms) self.activity_generator.generate_connection( src_ip=client_ip, dst_ip=effective_dst_ip, - time=ts, + time=req_ts, dst_port=dst_port, proto="tcp", service=dst_service, - duration=rng.uniform(0.01, 2.0), - orig_bytes=rng.randint(200, 2000), - resp_bytes=resp_bytes, + duration=rng.uniform(0.03, 2.0), + orig_bytes=max(200, req.request_body_len), + resp_bytes=req.response_body_len, + source_system=client_sys, http=HttpContext( - method=method, + method=req.method, host=http_host, - uri=path, + uri=req.path, version="1.1", user_agent=chosen_ua, - request_body_len=rng.randint(0, 500) if method == "POST" else 0, - response_body_len=resp_bytes, - status_code=status, - status_msg={200: "OK", 403: "Forbidden", 404: "Not Found"}.get( - status, "OK" - ), - referrer=_referer, - resp_mime_types=[mime] if status == 200 else [], + request_body_len=req.request_body_len, + response_body_len=req.response_body_len, + status_code=200, + status_msg="OK", + referrer=req.referrer, + trans_depth=req.trans_depth, + resp_mime_types=[req.content_type] if req.content_type else [], tags=[], ), hostname=http_host, ) + continue + + lo, hi = request_count_bounds(profile) + count = min(top_level_budget - top_level_emitted, rng.randint(lo, hi)) + elapsed_ms = 0 + for request_index in range(count): + request = pick_profile_request(rng, profile) + path = str(request.get("path", "/")) + method = str(request.get("method", "GET")) + status = int(request.get("status", 200)) + mime = normalize_mime_type_for_path(path, str(request.get("type", "text/html"))) + resp_bytes = ( + response_size_for_status(status, http_host, path) + if status != 200 or is_stable_resource_path(path) + else response_size_for_mime(rng, mime) + ) + request_body_len = rng.randint(100, 5_000) if method == "POST" else 0 + referrer = "" + if profile.get("referrer_mode") == "same_origin" and rng.random() < 0.35: + referrer = f"{'https' if dst_port == 443 else 'http'}://{http_host}/" + req_ts = base_ts + timedelta(milliseconds=elapsed_ms) + if request_index < count - 1: + elapsed_ms += _tool_gap_ms() + self.activity_generator.generate_connection( + src_ip=client_ip, + dst_ip=effective_dst_ip, + time=req_ts, + dst_port=dst_port, + proto="tcp", + service=dst_service, + duration=rng.uniform(0.01, 1.5), + orig_bytes=max(200, request_body_len), + resp_bytes=resp_bytes, + source_system=client_sys, + http=HttpContext( + method=method, + host=http_host, + uri=path, + version="1.1", + user_agent=chosen_ua, + request_body_len=request_body_len, + response_body_len=resp_bytes, + status_code=status, + status_msg=_status_message(status), + referrer=referrer, + resp_mime_types=[mime] if status == 200 else [], + tags=[], + ), + hostname=http_host, + ) + top_level_emitted += 1 def _generate_rsat_sessions(self, current_hour: datetime, rng, local_dt) -> None: """Generate correlated RSAT sessions from admin workstations to DCs. diff --git a/src/evidenceforge/models/scenario.py b/src/evidenceforge/models/scenario.py index 1e7e8c29..f4a69aa5 100644 --- a/src/evidenceforge/models/scenario.py +++ b/src/evidenceforge/models/scenario.py @@ -314,7 +314,7 @@ class BaselineActivity(BaseModel): Defines the baseline ("normal") activity level and variation for the environment. The intensity field scales ALL background traffic types (user activity, web server - requests, DNS, SMB, Kerberos, LDAP, persona connections) via traffic_rates.yaml. + top-level actions, DNS, SMB, Kerberos, LDAP, persona connections) via traffic_rates.yaml. Attributes: description: Natural language description of baseline activity diff --git a/tests/unit/test_activity.py b/tests/unit/test_activity.py index 60f32af8..4f2009cf 100644 --- a/tests/unit/test_activity.py +++ b/tests/unit/test_activity.py @@ -39,7 +39,10 @@ _is_invalid_network_connection, ) from evidenceforge.generation.activity import generator as generator_module -from evidenceforge.generation.activity.generator import _extract_image_from_command +from evidenceforge.generation.activity.generator import ( + _extract_image_from_command, + _jitter_default_connection_duration, +) from evidenceforge.generation.activity.tls_realism import ( certificate_analyzer_delay_ms, certificate_file_size, @@ -3273,6 +3276,32 @@ def test_http_connection_duration_covers_zeek_http_offset( assert net.conn_state == "SF" assert net.duration is not None and net.duration >= 0.04 + def test_default_connection_duration_jitter_diversifies_reviewer_anchors(self): + """Generator-owned placeholder durations should not render as exact constants.""" + for anchor in (0.8, 2.0, 0.01): + samples = { + round( + _jitter_default_connection_duration( + anchor, + caller_provided_duration=False, + seed_parts=("duration-anchor", anchor, idx), + ), + 6, + ) + for idx in range(8) + } + assert len(samples) > 1 + assert anchor not in samples + + assert ( + _jitter_default_connection_duration( + anchor, + caller_provided_duration=True, + seed_parts=("authored", anchor), + ) + == anchor + ) + def test_generate_connection_with_duration(self, activity_gen, state_manager, mock_emitters): """generate_connection with duration sets a valid conn_state.""" timestamp = datetime(2024, 1, 15, 10, 0, 0, tzinfo=UTC) diff --git a/tests/unit/test_baseline_canonical.py b/tests/unit/test_baseline_canonical.py index 4f4d4273..72725d69 100644 --- a/tests/unit/test_baseline_canonical.py +++ b/tests/unit/test_baseline_canonical.py @@ -27,6 +27,7 @@ """ import random +import re from datetime import UTC, datetime, timedelta from unittest.mock import Mock @@ -510,6 +511,38 @@ def test_failed_logon_attaches_syslog_context( assert "Failed password" in event.syslog.message assert event.syslog.severity == 4 # Warning level + def test_remote_linux_failed_logon_reuses_ssh_source_port_for_zeek_tuple( + self, activity_gen, state_manager, mock_emitters, timestamp + ): + """Remote failed sshd auth should have a matching Zeek SSH tuple.""" + linux = System(hostname="LNX-01", ip="10.0.10.2", os="Linux Ubuntu 22.04", type="server") + source_ip = "10.0.10.99" + state_manager.set_current_time(timestamp) + + activity_gen.generate_failed_logon( + user=User(username="attacker", full_name="Attacker", email="a@t.com", enabled=True), + system=linux, + time=timestamp, + logon_type=3, + source_ip=source_ip, + ) + + syslog_event = mock_emitters["syslog"].emit.call_args[0][0] + match = re.search(r"from (?P\S+) port (?P\d+) ssh2", syslog_event.syslog.message) + assert match is not None + ssh_source_port = int(match.group("port")) + zeek_events = [call.args[0] for call in mock_emitters["zeek_conn"].emit.call_args_list] + + assert any( + event.network.src_ip == source_ip + and event.network.src_port == ssh_source_port + and event.network.dst_ip == linux.ip + and event.network.dst_port == 22 + and event.network.service == "ssh" + and event.timestamp < syslog_event.timestamp + for event in zeek_events + ) + def test_local_linux_failed_logon_does_not_render_ssh_from_dash( self, activity_gen, state_manager, mock_emitters, timestamp ): @@ -529,6 +562,13 @@ def test_local_linux_failed_logon_does_not_render_ssh_from_dash( assert event.syslog is not None assert event.syslog.app_name == "login" assert "from -" not in event.syslog.message + zeek_events = [call.args[0] for call in mock_emitters["zeek_conn"].emit.call_args_list] + assert not any( + event.event_type == "connection" + and event.network is not None + and event.network.dst_port == 22 + for event in zeek_events + ) def test_self_sourced_linux_failed_logon_renders_local_auth( self, activity_gen, state_manager, mock_emitters, timestamp @@ -1103,3 +1143,119 @@ def test_external_ratio_custom_low(self): """exposure=both, external_ratio=0.05 → ≤10% external clients.""" frac = self._simulate_both_branch(ext_ratio=0.05) assert frac <= 0.10, f"Expected ≤10% external with ratio=0.05, got {frac:.1%}" + + def test_web_server_access_uses_browsing_session_shape(self, monkeypatch): + """Human visitors should emit clustered page/subresource requests, not isolated paths.""" + from random import Random + from types import SimpleNamespace + from unittest.mock import MagicMock + + from evidenceforge.generation.activity import web_session_profiles + from evidenceforge.generation.engine.baseline import BaselineMixin + + monkeypatch.setattr( + web_session_profiles, + "pick_web_visitor_profile", + lambda rng, *, is_external: ( + "human_browser", + { + "kind": "session", + "browsing_intensity": "normal", + "user_agent_pool": "browser_any", + }, + ), + ) + + collected = [] + activity_gen = MagicMock() + activity_gen._ip_to_system = {} + activity_gen.generate_connection.side_effect = lambda **kw: collected.append(kw) + engine = MagicMock() + engine.activity_generator = activity_gen + engine._resolve_traffic_rate.return_value = (8, 8) + engine._get_segment_for_system.return_value = SimpleNamespace( + exposure="external", + external_ratio=None, + ) + engine._generate_external_client_ip.side_effect = [f"8.8.4.{idx}" for idx in range(1, 20)] + sys_obj = self._make_web_system("external", public_hostnames=["portal.example.com"]) + + BaselineMixin._emit_web_server_access( + engine, + sys_obj, + [sys_obj], + Random(42), + datetime(2024, 3, 15, 10, 0, 0, tzinfo=UTC), + ) + + page_loads = [kw for kw in collected if kw["http"].trans_depth == 1] + assert len(page_loads) == 8 + assert len(collected) > len(page_loads) + assert {kw["http"].host for kw in collected} == {"portal.example.com"} + by_client = {} + for kwargs in collected: + by_client.setdefault(kwargs["src_ip"], set()).add(kwargs["http"].user_agent) + assert all(len(user_agents) == 1 for user_agents in by_client.values()) + assert any(kw["http"].referrer == "https://portal.example.com/" for kw in collected) + assert any( + kw["http"].uri.endswith(".css") or kw["http"].uri.endswith(".js") for kw in collected + ) + + def test_web_server_access_keeps_scanner_requests_source_native(self, monkeypatch): + """Scanner visitors should keep configured error paths and blank referrers.""" + from random import Random + from types import SimpleNamespace + from unittest.mock import MagicMock + + from evidenceforge.generation.activity import web_session_profiles + from evidenceforge.generation.engine.baseline import BaselineMixin + + monkeypatch.setattr( + web_session_profiles, + "pick_web_visitor_profile", + lambda rng, *, is_external: ( + "opportunistic_probe", + { + "kind": "requests", + "request_count": [3, 3], + "user_agent_pool": "scanner", + "referrer_mode": "none", + "requests": [ + { + "path": "/wp-login.php", + "method": "GET", + "status": 404, + "type": "text/html", + "weight": 1, + } + ], + }, + ), + ) + + collected = [] + activity_gen = MagicMock() + activity_gen._ip_to_system = {} + activity_gen.generate_connection.side_effect = lambda **kw: collected.append(kw) + engine = MagicMock() + engine.activity_generator = activity_gen + engine._resolve_traffic_rate.return_value = (3, 3) + engine._get_segment_for_system.return_value = SimpleNamespace( + exposure="external", + external_ratio=None, + ) + engine._generate_external_client_ip.side_effect = [f"8.8.8.{idx}" for idx in range(1, 20)] + sys_obj = self._make_web_system("external", public_hostnames=["portal.example.com"]) + + BaselineMixin._emit_web_server_access( + engine, + sys_obj, + [sys_obj], + Random(7), + datetime(2024, 3, 15, 10, 0, 0, tzinfo=UTC), + ) + + assert len(collected) == 3 + assert {kw["http"].status_code for kw in collected} == {404} + assert {kw["http"].uri for kw in collected} == {"/wp-login.php"} + assert all(kw["http"].referrer == "" for kw in collected) diff --git a/tests/unit/test_browsing_session.py b/tests/unit/test_browsing_session.py index 7cb99aa5..864870f3 100644 --- a/tests/unit/test_browsing_session.py +++ b/tests/unit/test_browsing_session.py @@ -9,6 +9,7 @@ BrowsingRequest, generate_browsing_session, ) +from evidenceforge.generation.activity.timing_profiles import reset_timing_profiles_cache class TestBrowsingSessionBasics: @@ -222,6 +223,18 @@ def test_non_browser_proxy_domain_produces_no_browsing_session(self): requests = generate_browsing_session(rng, "ocsp.pki.goog", ["background"]) assert requests == [] + def test_inbound_web_server_can_use_generic_public_hostname(self): + rng = random.Random(42) + requests = generate_browsing_session( + rng, + "portal.customer.example", + [], + port=443, + require_browser_like_domain=False, + ) + assert len(requests) > 0 + assert requests[0].hostname == "portal.customer.example" + class TestResponseSizes: """Response body lengths should be realistic for content types.""" @@ -258,6 +271,54 @@ def test_extension_drives_content_type(self): assert request.content_type == "image/x-icon" assert 500 <= request.response_body_len <= 5_000 + def test_stable_static_asset_size_for_same_host_and_path(self): + first = generate_browsing_session( + random.Random(42), + "portal.customer.example", + [], + require_browser_like_domain=False, + ) + second = generate_browsing_session( + random.Random(43), + "portal.customer.example", + [], + require_browser_like_domain=False, + ) + first_favicon = next(r for r in first if r.path == "/favicon.ico") + second_favicon = next(r for r in second if r.path == "/favicon.ico") + + assert first_favicon.response_body_len == second_favicon.response_body_len + + def test_subresource_timing_uses_timing_profile_overlay(self, tmp_path, monkeypatch): + overlay = tmp_path / ".eforge" / "config" / "activity" + overlay.mkdir(parents=True) + (overlay / "timing_profiles.yaml").write_text( + """ +relationships: + web.asset_stylesheet_script_after_page: + class: burst_fanout + position: after + min_ms: 1000 + max_ms: 1000 +""".lstrip() + ) + monkeypatch.chdir(tmp_path) + reset_timing_profiles_cache() + + requests = generate_browsing_session(random.Random(42), "github.com", []) + first_page = requests[0] + first_page_referrer = f"https://{first_page.hostname}{first_page.path}" + css_js = [ + request + for request in requests + if request.referrer == first_page_referrer + and request.content_type in {"text/css", "application/javascript"} + ] + + assert css_js + assert {request.time_offset_ms for request in css_js} == {1000} + reset_timing_profiles_cache() + class TestDeterminism: """Same seed produces identical sessions.""" diff --git a/tests/unit/test_dns_realism.py b/tests/unit/test_dns_realism.py index 5ae9dd60..76eb839a 100644 --- a/tests/unit/test_dns_realism.py +++ b/tests/unit/test_dns_realism.py @@ -588,6 +588,34 @@ def test_dns_conn_duration_uses_rtt( event = mock_emitters["zeek_conn"].emit.call_args[0][0] assert event.network.duration == 0.35 + def test_dns_conn_duration_exact_anchor_still_uses_rtt( + self, activity_gen, timestamp, state_manager, mock_emitters + ): + """DNS RTT locks must not be jittered just because they equal old default anchors.""" + state_manager.set_current_time(timestamp) + + activity_gen.generate_connection( + src_ip="10.0.1.50", + dst_ip="10.0.0.1", + time=timestamp, + dst_port=53, + proto="udp", + service="dns", + dns=DnsContext( + query="anchor.example.com", + query_type="A", + qtype=1, + rcode="NOERROR", + rcode_num=0, + answers=["93.184.216.34"], + rtt=0.02, + ), + resp_bytes=120, + ) + + event = mock_emitters["zeek_conn"].emit.call_args[0][0] + assert event.network.duration == 0.02 + def test_explicit_dns_response_state_keeps_responder_accounting( self, activity_gen, timestamp, state_manager, mock_emitters ): diff --git a/tests/unit/test_remaining_expert_review.py b/tests/unit/test_remaining_expert_review.py index ab7c1431..f85c7f41 100644 --- a/tests/unit/test_remaining_expert_review.py +++ b/tests/unit/test_remaining_expert_review.py @@ -9,6 +9,7 @@ from evidenceforge.generation.engine.baseline import ( BaselineMixin, _pick_non_colliding_account_name, + _scheduled_stale_failure_offsets, ) from evidenceforge.models.scenario import AccountCreatedEventSpec, AccountDeletedEventSpec from evidenceforge.utils.rng import _stable_seed @@ -108,6 +109,35 @@ def test_management_sweep_targets_multiple_hosts(self): targets = rng.sample(servers, n_targets) assert 5 <= len(targets) <= 15 + def test_scheduled_stale_credentials_do_not_use_exact_two_hour_cadence(self): + """Stale scheduled-credential noise should not expose modulo-hour timing.""" + config = { + "interval_ranges": [{"min_minutes": 105, "max_minutes": 155, "weight": 1}], + "first_occurrence_seconds_min": 0, + "first_occurrence_seconds_max": 1200, + "jitter_seconds_min": -420, + "jitter_seconds_max": 780, + "skip_probability": 0.0, + "backoff_probability": 0.0, + "backoff_seconds_min": 0, + "backoff_seconds_max": 0, + } + event_seconds = [] + for hour_idx in range(12): + for offset in _scheduled_stale_failure_offsets( + scenario_name="cadence-test", + account_name="svc_deploy", + hostname="LNX-01", + hour_idx=hour_idx, + config=config, + ): + event_seconds.append(hour_idx * 3600 + offset) + + gaps = [right - left for left, right in zip(event_seconds, event_seconds[1:], strict=False)] + assert len(gaps) >= 3 + assert any(abs(gap - 7200) > 600 for gap in gaps) + assert len(set(gaps)) > 1 + def test_password_typo_pattern(self): """Password typo: 1-2 failures should precede success by seconds.""" import random diff --git a/tests/unit/test_timing_profiles.py b/tests/unit/test_timing_profiles.py index d328a24e..f36ffbce 100644 --- a/tests/unit/test_timing_profiles.py +++ b/tests/unit/test_timing_profiles.py @@ -9,6 +9,7 @@ from evidenceforge.generation.activity.timing_profiles import ( get_timing_window, + network_sensor_observation_timing, reset_timing_profiles_cache, sample_timing_delta, windows_collision_spacing_config, @@ -55,6 +56,29 @@ def test_timing_profiles_load_default_relationship(): assert tls_window.relationship_class == "same_observation" assert tls_window.min_ms >= 650 + navigation_window = get_timing_window( + "web.session_navigation", + default_min_ms=0, + default_max_ms=0, + default_position="after", + ) + asset_window = get_timing_window( + "web.asset_stylesheet_script_after_page", + default_min_ms=0, + default_max_ms=0, + default_position="after", + ) + assert navigation_window.relationship_class == "human_workflow" + assert navigation_window.min_ms >= 3000 + assert asset_window.relationship_class == "burst_fanout" + assert asset_window.max_ms <= 200 + + sensor_timing = network_sensor_observation_timing() + assert sensor_timing.clock_skew_min_us == -1500 + assert sensor_timing.clock_skew_max_us == 1500 + assert sensor_timing.path_delay_min_us == 50 + assert sensor_timing.path_delay_max_us == 2000 + def test_timing_profiles_overlay_overrides_relationship(tmp_path, monkeypatch): overlay = tmp_path / ".eforge" / "config" / "activity" @@ -74,6 +98,16 @@ def test_timing_profiles_overlay_overrides_relationship(tmp_path, monkeypatch): near_gap_max_us: 20 large_gap_min_ms: 2000 large_gap_max_ms: 3000 +network_sensor_observation: + default_profile: lab + profiles: + lab: + clock_skew_us: + min: -250 + max: 250 + path_delay_us: + min: 25 + max: 500 """.lstrip() ) monkeypatch.chdir(tmp_path) @@ -86,11 +120,14 @@ def test_timing_profiles_overlay_overrides_relationship(tmp_path, monkeypatch): default_position="after", ) spacing = windows_collision_spacing_config() + sensor_timing = network_sensor_observation_timing() assert window.min_ms == 250 assert window.max_ms == 750 assert spacing["near_zero_until"] == 3 assert spacing["large_gap_min_ms"] == 2000 + assert sensor_timing.clock_skew_min_us == -250 + assert sensor_timing.path_delay_max_us == 500 def test_sample_timing_delta_is_deterministic_and_bounded(): @@ -119,6 +156,16 @@ def test_timing_profiles_overlay_invalid_values_fall_back_safely(tmp_path, monke near_gap_max_us: 2000000 large_gap_min_ms: bad large_gap_max_ms: 999999999 +network_sensor_observation: + default_profile: bad + profiles: + bad: + clock_skew_us: + min: later + max: -later + path_delay_us: + min: 5000 + max: 100 """.lstrip() ) monkeypatch.chdir(tmp_path) @@ -131,6 +178,7 @@ def test_timing_profiles_overlay_invalid_values_fall_back_safely(tmp_path, monke default_position="before", ) spacing = windows_collision_spacing_config() + sensor_timing = network_sensor_observation_timing() assert window.min_ms == 20 assert window.max_ms == 86_400_000 @@ -139,3 +187,7 @@ def test_timing_profiles_overlay_invalid_values_fall_back_safely(tmp_path, monke assert spacing["near_gap_max_us"] == 1_000_000 assert spacing["large_gap_min_ms"] == 1000 assert spacing["large_gap_max_ms"] == 60_000 + assert sensor_timing.clock_skew_min_us == -1500 + assert sensor_timing.clock_skew_max_us == 1500 + assert sensor_timing.path_delay_min_us == 50 + assert sensor_timing.path_delay_max_us == 2000 diff --git a/tests/unit/test_validate_config.py b/tests/unit/test_validate_config.py index d3a60ce9..2b7833a6 100644 --- a/tests/unit/test_validate_config.py +++ b/tests/unit/test_validate_config.py @@ -380,6 +380,41 @@ def load_invalid_network_params(): for issue in result.issues ) + def test_validate_config_rejects_invalid_auth_noise_ranges(self, monkeypatch): + from evidenceforge.generation.activity import auth_noise + + def load_invalid_auth_noise_config(): + return { + "scheduled_stale_credentials": { + "account_base_names": ["svc_backup"], + "host_count_min": 3, + "host_count_max": 1, + "interval_ranges": [{"min_minutes": 120, "max_minutes": 60, "weight": 1}], + "first_occurrence_seconds_min": 0, + "first_occurrence_seconds_max": 2700, + "jitter_seconds_min": -420, + "jitter_seconds_max": 780, + "skip_probability": 0.10, + "backoff_probability": 0.10, + "backoff_seconds_min": 900, + "backoff_seconds_max": 3600, + } + } + + monkeypatch.setattr(auth_noise, "load_auth_noise_config", load_invalid_auth_noise_config) + + result = validate_config() + + assert any( + issue.severity == "ERROR" + and issue.file == "auth_noise.yaml" + and ( + "max_minutes must be greater than or equal to min_minutes" in issue.message + or "host_count_max must be greater than or equal to host_count_min" in issue.message + ) + for issue in result.issues + ) + def test_validate_config_rejects_too_short_workstation_unlock_gap(self, monkeypatch): from evidenceforge.generation.activity import windows_auth_realism diff --git a/tests/unit/test_web_session_profiles.py b/tests/unit/test_web_session_profiles.py new file mode 100644 index 00000000..49c255e0 --- /dev/null +++ b/tests/unit/test_web_session_profiles.py @@ -0,0 +1,58 @@ +# Copyright (c) 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: MIT + +"""Tests for inbound web visitor profile config.""" + +import random + +import pytest + +from evidenceforge.generation.activity.web_session_profiles import ( + load_web_session_profiles, + pick_profile_request, + pick_web_user_agent, + pick_web_visitor_profile, + request_count_bounds, + reset_web_session_profiles_cache, +) + + +@pytest.fixture(autouse=True) +def _reset_cache(): + reset_web_session_profiles_cache() + yield + reset_web_session_profiles_cache() + + +def test_web_session_profiles_load_default_classes(): + data = load_web_session_profiles() + + assert "visitor_classes" in data + assert "human_browser" in data["visitor_classes"] + assert data["visitor_classes"]["human_browser"]["kind"] == "session" + assert "user_agent_pools" in data + assert data["user_agent_pools"]["browser_any"] + + +def test_external_profile_selection_excludes_internal_health_checks(): + rng = random.Random(4) + + for _ in range(100): + name, _profile = pick_web_visitor_profile(rng, is_external=True) + assert name != "health_check" + + +def test_user_agent_honors_source_os_pool(): + profile = load_web_session_profiles()["visitor_classes"]["human_browser"] + ua = pick_web_user_agent(random.Random(1), profile, source_os="linux") + + assert "Linux" in ua + + +def test_profile_request_and_bounds_are_safe(): + profile = load_web_session_profiles()["visitor_classes"]["opportunistic_probe"] + request = pick_profile_request(random.Random(3), profile) + lo, hi = request_count_bounds(profile) + + assert request["status"] in {403, 404} + assert 1 <= lo <= hi diff --git a/tests/unit/test_zeek_multiplex.py b/tests/unit/test_zeek_multiplex.py index 0db22486..6329406c 100644 --- a/tests/unit/test_zeek_multiplex.py +++ b/tests/unit/test_zeek_multiplex.py @@ -107,7 +107,7 @@ def test_second_sensor_observation_preserves_lossless_packetization(self): assert core[field] == dmz[field] assert core["uid"] != dmz["uid"] assert core["ts"] != dmz["ts"] - assert abs(core["ts"] - dmz["ts"]) <= 1.5 + assert abs(core["ts"] - dmz["ts"]) <= 0.005 assert core["orig_bytes"] == dmz["orig_bytes"] == 23124 assert core["resp_bytes"] == dmz["resp_bytes"] == 80921 assert core["orig_pkts"] == dmz["orig_pkts"] == 52 @@ -161,9 +161,9 @@ def test_sensor_timestamp_offsets_vary_by_flow(self): for port in sorted(core_by_port) ] - assert max(offsets) - min(offsets) > 0.05 + assert max(offsets) - min(offsets) > 0.0005 assert len(set(offsets)) > 30 - assert all(offset > 0 for offset in offsets) or all(offset < 0 for offset in offsets) + assert max(abs(offset) for offset in offsets) <= 0.005 def test_second_sensor_observation_preserves_http_body_lengths(self): """HTTP body sizes are transaction facts, not per-sensor packet-counter jitter."""