Add observation profiles for source coverage

TODO.md

@@ -240,8 +240,9 @@ Replaced manual per-emitter field coordination with SecurityEvent intermediate r
 - [x] **P2** Scenario skill anti-curation guidance follow-up — Revised the dev scenario skill so attacker-controlled domains, service accounts, scheduled tasks, files, and process names blend into ordinary naming conventions without becoming semantic breadcrumbs that reveal the attack narrative. Verification: `uv run pytest tests/unit/test_install_skills.py -q --no-cov` passed (`30 passed`); the same focused test file also passed under the default coverage run, but that command failed the whole-repo coverage threshold because it intentionally ran only one test module.
 - [x] **P1** Web application response/session realism follow-up — Added data-driven inbound `web_server` visitor profiles so human visitors consume `traffic_rates.web` as top-level actions, then fan out into required page assets/API calls through `site_maps.yaml`; crawler, health-check, API-client, and opportunistic-probe traffic now uses source-native configured request/status/User-Agent profiles. Static resource sizes are stable per host/path, human navigation and render fanout timing use `timing_profiles.yaml`, and docs/skill references now explain the budget and config ownership. Verification passed: focused web/timing/baseline tests (`107 passed, 1 skipped`), config-related tests (`64 passed`), `uv run eforge validate-config`, repo-wide Ruff checks/format checks, full normal `uv run pytest -q` (`3012 passed, 15 skipped`), and `git diff --check`.
 - [x] **P1** Well-synced network sensor timing follow-up — Replaced hardcoded multi-sensor Zeek +/-400ms skew plus broad path delay with a validated `network_sensor_observation` timing profile. The default `well_synced` profile keeps stable per-sensor clock skew within +/-1.5ms and per-flow capture/path delay within 50-2000us while preserving canonical packet/byte truth unless source-native observation variance is explicitly enabled. Verification passed with focused Zeek/timing tests, `uv run eforge validate-config`, repo-wide Ruff checks/format checks, full normal `uv run pytest -q` (`3012 passed, 15 skipped`), and `git diff --check`.
-- [ ] **DEFERRED with observation/source coverage architecture** **P2** Endpoint/eCAR baseline variance follow-up — Loop 96 found workstation eCAR category volumes and Linux process lifecycle evidence too uniform and complete. Defer with the broader observation/profile sprint so host/persona-specific variance, long-lived process state, benign unmatched artifacts, and realistic endpoint observation gaps are modeled coherently rather than as eCAR-only omissions.
-- [ ] **Later architectural sprint: imperfect observation and source coverage** — defer the broad "too-complete telemetry" problem until after the sharper defects are gone. Model source-specific drop rates, ingestion delay, audit-policy gaps, endpoint coverage variance, and asymmetric Security/Sysmon/eCAR/Zeek visibility as a coherent observation/profile layer rather than one-off omissions. Bundle the related deferred items into this sprint: endpoint/eCAR baseline variance, source-specific process lifecycle completeness modeling, configurable cross-source evidence disagreement, per-host/source log coverage, and the host/activity profile items for per-entity artifact and volume variance.
+- [x] **P1** Source identity and endpoint baseline realism sprint — completed TLS/X.509 issuer-compatible chain signatures, Sysmon Event 7 native third-party module identity, config-driven Windows scheduled-process timing, and DHCP registry emission policy tied to lease activity. Verified with `uv run eforge validate-config`, focused regressions, Ruff, normal pytest, and slow-inclusive pytest.
+- [ ] **P2** Endpoint/eCAR baseline variance follow-up — Loop 96 found workstation eCAR category volumes and Linux process lifecycle evidence too uniform and complete. The realistic endpoint observation-gap portion is now handled by named observation profiles; remaining work should focus on host/persona-specific volume variance, long-lived process state, and benign unmatched endpoint artifacts.
+- [x] **Later architectural sprint: imperfect observation and source coverage** — implemented a training-friendly `complete` default plus overlay-compatible named observation profiles that apply deterministic source-level drop/delay/coverage semantics without modeling contradictions. The policy covers endpoint, network, proxy/web, firewall, IDS, Windows, Sysmon, Zeek, syslog, bash history, and eCAR source families, while ground truth preserves canonical truth and records source evidence status. Verification passed: focused observation/config/ground-truth tests, `uv run eforge validate-config`, Ruff checks/format checks, full normal `uv run pytest -v` (`3036 passed, 15 skipped`), and slow-inclusive `uv run pytest -v --include-slow` (`3050 passed, 1 skipped`).
 - [x] Full slow-suite regression cleanup after loop-65 merge — explicit-proxy storyline beacons now preserve authored hostname+destination IP pairs only when the storyline marks that pair as intentional, normal proxy-origin DNS resolution remains intact, and the parallel-generation LogonID assertion treats Type 7 unlock reuse as valid slice-of-time Windows behavior. Verified with targeted proxy/parallel tests, `uv run ruff check .`, `uv run ruff format --check .`, and `uv run pytest -v --include-slow` (`2875 passed, 23 skipped`).
   Detection Engineer blind review completed for the regenerated Loop 61 dataset at `scenarios/iteration-test/data`; reviewer verdict: Synthetic, 63/100 confidence. Main findings: one PROXY-01 sshd accepted-login lifecycle gap/self-source artifact and Windows 4648 explicit-credential caller PID/image provenance ambiguity around `WS-MCHEN-01`.
 
@@ -580,7 +581,7 @@ Data works but experienced analysts spot tells. Grouped by format for efficient
 
 **Cisco ASA:**
 - [x] Security: bound threat-detection deny timestamp tracking window to prevent unbounded memory/CPU growth
-- [ ] ASA imperfect-observation realism — deferred to a general solution for configurable evidence gaps. Built/Teardown counts are currently perfectly balanced, while real logs can have orphans from rotation boundaries, packet loss, sensor downtime, or collection windows. Keep exact pairing as the training-friendly default unless a realism profile enables dropped/partial firewall evidence.
+- [x] ASA imperfect-observation realism — addressed by the general observation profile layer. `complete` preserves paired training-friendly firewall evidence, while non-default profiles can apply deterministic ASA source-family gaps that create realistic missing/partial firewall evidence without rewriting canonical truth.
 - [ ] ASA message type diversity limited to 106023/302013-16/305011-12 — missing 111008, 113004, 733100, 106001, 725001, 304001
 - [ ] ASA deny baseline burstiness/profile variance — defer to a general per-source activity profile rather than a one-off ASA fix. Current deny events are uniformly spaced (3-7s); real scans should have configurable burst/quiet periods, campaign-level cadence, and source-specific variance.
 - [ ] ASA deny metadata diversity — defer to a general field-distribution realism layer. Current deny events use `[0x0, 0x0]` hash values uniformly; a later profile should model when hashes remain zero vs vary by platform/message/context.
@@ -595,12 +596,12 @@ Data works but experienced analysts spot tells. Grouped by format for efficient
 - [x] Template variable leak — literal `{psql_db}` appearing in eCAR output; stale audit finding: Linux query placeholders are handled by `_parameterize_command()`, with `tests/unit/test_activity_helpers.py` covering `{psql_db}` replacement.
 
 **Cross-Source / General:**
-- [ ] Configurable cross-source evidence disagreement — deferred by design. Perfect cross-source correlation is useful for training/huntability and should remain the default feature unless a scenario/evaluation profile asks for realism gaps. Later design a deterministic setting for dropped/partial/ambiguous corroborating evidence across Zeek, web, proxy, firewall, IDS, Windows, Sysmon, and eCAR without breaking ground-truth traceability. Include broader sensor-observation timing realism beyond the current per-event jitter: sensor clock skew/drift, NTP corrections, capture-path latency, log buffering, occasional source-specific missing/late records, and policy differences between proxy access and Zeek HTTP.
+- [x] Configurable cross-source evidence disagreement — implemented as named observation profiles with `complete` as the default. Non-default profiles can introduce deterministic dropped/delayed/filtered/out-of-window evidence across Zeek, web, proxy, firewall, IDS, Windows, Sysmon, syslog, bash history, and eCAR without contradictions or ambiguous rewrites; ground truth retains source evidence status for traceability.
 - [x] Cross-sensor timestamp precision identical to 15+ decimal places — microsecond jitter added in snort.py, windows.py, and storyline.py
 - [ ] **P2** Per-host-type event rate multiplier — Domain controllers generate ~50 events/hr but real DCs running AD/DNS/DFS/GPO produce thousands/hr. `system.type` is used for routing but never for volume scaling. Need `event_rate_multiplier` on System model (or implicit per-type defaults) applied in `_calculate_events_for_hour()` and `_generate_system_traffic()`. DCs should be 3-5x workstation baseline; file servers and web servers similarly elevated.
 - [ ] Configurable per-entity artifact variation — deferred to the general host/activity profile layer. Encoded PowerShell baseline noise is currently identical across hosts (same Get-Service blob); later profiles should derive stable per-host command variants, encoded payloads, tool versions, and operator habits.
 - [ ] Configurable per-host volume variance — deferred to the general host/activity profile layer. Workstation connection counts are suspiciously uniform (808-1068 range); later profiles should widen variance by role, persona, weekday, installed apps, and stable host-specific multipliers.
-- [ ] Configurable per-host/source log coverage — deferred to the general imperfect-observation/profile layer. Uniform log file sets across all hosts can be useful for training, but a later setting should allow host-specific telemetry coverage differences, disabled sensors, partial deployment, and collection gaps.
+- [ ] Configurable per-host/source log deployment coverage — observation profiles now support source-family gaps and host-scoped missingness multipliers, but explicit per-host source enablement/disablement remains future work. A later setting should model named host groups, disabled sensors, partial deployments, and collection windows when users need topology-level telemetry coverage differences rather than event-level missingness.
 - [x] DNS IP pool reuse causes cross-provider resolution (CloudFront→Microsoft IPs, etc.) — domain-first selection ensures consistent domain→IP mapping via FORWARD_DNS
 - [x] AWS region mismatch between DNS PTR and SSL SNI for same IP — AWS hostname/PTR generation now derives a stable per-IP region/edge identity and PTR generation respects known forward hostname context.
 - [x] TLS volume clustering design — added data-driven TLS destination profiles with overlay support and `eforge validate-config` schema/tag checks. Auto-generated external TLS now uses weighted enterprise, certificate-infra, package-update, developer-tool, and long-tail browsing profiles with stable per-host preferences. Smoke output had 28,544 TLS SNI rows, 116 distinct names, top SNI share 5.5%, and top-5 share 18.0%.

commands/eforge/config.md

@@ -66,8 +66,11 @@ When writing to the overlay, files are partial — they contain ONLY the user's
 | Modify CallTrace patterns | `calltrace_patterns.yaml` | (standalone — Event 10 ProcessAccess call chain templates) |
 | Modify ProcessAccess masks | `process_access_patterns.yaml` | (standalone — Event 10 baseline source/target pairs and GrantedAccess masks) |
 | Modify CreateRemoteThread pairs | `create_remote_thread_patterns.yaml` | (standalone — Event 8 baseline source/target pairs) |
+| Modify TLS chain/OCSP/SNI realism | `tls_realism.yaml` | `dns_registry.yaml` for OCSP responder hosts and domains selected by `dns_tags` |
 | Modify Windows auth realism | `windows_auth_realism.yaml` | (standalone — Security log auth timing and failed-logon profile knobs) |
 | Modify baseline auth noise | `auth_noise.yaml` | (standalone — stale scheduled-credential accounts and irregular recurrence timing) |
+| Modify endpoint background noise | `endpoint_noise.yaml` | (standalone — scheduled-process timing and DHCP registry emission policy) |
+| Modify source observation coverage | `observation_profiles.yaml` | Scenario `observation_profile` selects the named profile; keep `complete` as the default training profile |
 | Modify causal/source timing | `timing_profiles.yaml` | (standalone — causal prerequisite, source latency, teardown, and Windows/Sysmon collision-spacing knobs) |
 | ~~Format definitions~~ | Not user-customizable | Engine internals — requires code changes |
 | ~~Evaluation rules~~ | Not user-customizable | Must match format definitions — requires code changes |
@@ -88,7 +91,7 @@ Also read the relevant reference doc for field schemas and conventions:
 | Applications, spawn rules, processes | `references/config-apps-processes.md` |
 | Sysmon filters, EDR pools, CallTrace, ProcessAccess masks, CreateRemoteThread pairs | `references/config-apps-processes.md` (Sysmon sections) |
 | Persona file structure | `references/config-personas.md` |
-| Host activity (bash, systemd, syslog) | `references/config-host-activity.md` |
+| Host activity (bash, systemd, syslog, endpoint noise) | `references/config-host-activity.md` |
 | Timing profiles | `references/config-host-activity.md` |
 | Format definitions | `references/config-formats.md` (read-only reference — not user-customizable) |
 | Evaluation rules | `references/config-evaluation.md` (read-only reference — not user-customizable) |

commands/eforge/references/config-apps-processes.md

@@ -80,16 +80,17 @@ applications:
 
 ### Loaded Module Fields (Windows only)
 
-DLLs characteristically loaded by this process, used for Sysmon Event 7 (ImageLoaded) generation. All fields except `path` have defaults — only specify what differs.
+DLLs characteristically loaded by this process, used for Sysmon Event 7 (ImageLoaded) generation. Microsoft OS loader DLLs can rely on defaults. Third-party modules should set source-native signer metadata, and known vendor modules should carry PE metadata so rendered `Company`, `Product`, `Description`, and `FileVersion` do not fall back to Microsoft or blank values.
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
 | `path` | string | (required) | Full Windows path to the DLL (must contain `\`) |
 | `signed` | bool | `true` | Whether the DLL is digitally signed |
 | `signature` | string | `"Microsoft Windows"` | Signer name (e.g., `"Google LLC"`, `"Mozilla Corporation"`) |
 | `signature_status` | string | `"Valid"` | One of: `Valid`, `Expired`, `Revoked`, `Unavailable` |
+| `pe_metadata` | object | inherited or blank | Optional DLL-specific PE fields: `file_version`, `description`, `product`, `company`, `original_filename` |
 
-Every Windows process also receives the common OS loader DLLs (ntdll.dll, kernel32.dll, etc.) defined in `system_processes.yaml` under `common_loaded_modules.windows` — you don't need to repeat those in per-app profiles.
+Application DLLs inherit the owning app's PE version/product/company if `pe_metadata` is omitted. Every Windows process also receives the common OS loader DLLs (ntdll.dll, kernel32.dll, etc.) defined in `system_processes.yaml` under `common_loaded_modules.windows` — you don't need to repeat those in per-app profiles.
 
 ### Valid Categories
 
@@ -369,7 +370,7 @@ Provides file path, registry key, and DLL pools for probabilistic background eve
 - `registry_keys_hklm:` — `[key, value_name, details]` triples for HKLM writes (Run, Defender, WDigest, Firewall)
 - `dll_pool:` — System32 and application DLL paths for module load events
 
-Overlay replaces entire sections (section-replace merge). Details values use Sysmon format: `"DWORD (0x00000001)"` for REG_DWORD, string for REG_SZ. Registry and DLL entries may use `{user}`, `{rand}`, `{hex}`, `{guid}`, `{mru}`, `{doc}`, `{package}`, and `{version}` placeholders; these are materialized per emitted event to avoid repetitive TargetObject paths.
+Overlay replaces entire sections (section-replace merge). Details values use Sysmon format: `"DWORD (0x00000001)"` for REG_DWORD, string for REG_SZ. Registry and DLL entries may use `{user}`, `{rand}`, `{hex}`, `{guid}`, `{mru}`, `{doc}`, `{package}`, and `{version}` placeholders; these are materialized per emitted event to avoid repetitive TargetObject paths. DHCP interface registry values are additionally controlled by `endpoint_noise.yaml`, which reserves them for actual DHCP lease/reconfigure activity unless explicitly relaxed.
 
 ---
 

commands/eforge/references/config-dependency-graph.md

@@ -151,6 +151,20 @@ Each row is a file; columns show what it depends on and what depends on it.
 | depends on | nothing | Standalone authentication-noise profile data |
 | **depended on by** | Engine (runtime) | Drives stale scheduled-credential account pools, recurrence timing, jitter, skips, and backoff |
 
+### endpoint_noise.yaml
+| Direction | File | Relationship |
+|-----------|------|-------------|
+| depends on | nothing | Standalone endpoint background timing and registry-emission policy data |
+| **depended on by** | Engine (runtime) | Drives Windows scheduled-process trigger windows, host drift, skips, and DHCP interface registry write policy |
+| validated by | `eforge validate-config` | Enforces coherent timing bounds, probability ranges, and non-empty DHCP registry value lists |
+
+### observation_profiles.yaml
+| Direction | File | Relationship |
+|-----------|------|-------------|
+| depends on | scenario `observation_profile` | The scenario selects a named profile; the profile file owns source-level missingness/delay values |
+| **depended on by** | Event dispatcher, GROUND_TRUTH.md | Applies deterministic source-observation drops/delays after canonical state updates and reports source evidence status |
+| validated by | `eforge validate-config` and `eforge validate` | Config validation checks source-family names/ranges; scenario validation checks that the named profile exists |
+
 ### network_params.yaml
 | Direction | File | Relationship |
 |-----------|------|-------------|
@@ -166,8 +180,9 @@ Each row is a file; columns show what it depends on and what depends on it.
 ### tls_realism.yaml
 | Direction | File | Relationship |
 |-----------|------|-------------|
-| depends on | tls_issuers.yaml, dns_registry.yaml | Chain templates match issuer names/patterns selected from issuer config; OCSP responder hosts must exist in dns_registry; destination profiles can pull domains by DNS tag |
-| **depended on by** | Engine (runtime) | Drives Zeek TLS SAN, x509 chain depth, OCSP cache/status behavior, and profiled TLS SNI/destination selection |
+| depends on | tls_issuers.yaml, dns_registry.yaml | Chain templates and subject-key profiles match issuer names/patterns selected from issuer config; OCSP responder hosts must exist in dns_registry; destination profiles can pull domains by DNS tag |
+| **depended on by** | Engine (runtime) | Drives Zeek TLS SAN, x509 chain depth, issuer-compatible certificate signature algorithms, OCSP cache/status behavior, and profiled TLS SNI/destination selection |
+| validated by | `eforge validate-config` | Enforces coherent chain profile structure, non-empty subject-key patterns, and RSA/ECDSA child signature compatibility |
 
 ### smb_file_transfers.yaml
 | Direction | File | Relationship |