diff --git a/docs/paper-validation-runbook.md b/docs/paper-validation-runbook.md index 6de5b80..d6bff18 100644 --- a/docs/paper-validation-runbook.md +++ b/docs/paper-validation-runbook.md @@ -30,7 +30,10 @@ The Azure Functions isolated worker runs **locally on the dev box**, not in Azur ## 2. Daily preflight (before the pre-market timer) Run this check each trading morning **before 5:00 AM PT in winter (PST) / 6:00 AM PT in summer -(PDT)** — the winter deadline is the earlier one (see section 3): +(PDT)** — the winter deadline is the earlier one (see section 3) — or run +`tools/ops/preflight.ps1` (S6-004), which scripts the same checks with bounded timeouts: +TWS port probe (step 1), gateway health (step 2 — gateway-down is a WARN, not a FAIL), +worker admin-endpoint 4-function list (steps 3–5), and the snapshots data directory: 1. **TWS paper** is running and logged in to the paper account; API enabled with socket port **7497** and trusted IP **127.0.0.1** (Configure → API → Settings). @@ -174,7 +177,8 @@ KD-005/KD-006 — so its absence is by design, not a gap to fix.) | Symptom | Likely cause | Action | |---|---|---| -| **Nothing at all** — no report, no alert, on a trading day | Worker (or the whole box) was not running — the dead-man case | **Check this first each morning.** Confirm the worker process is up and the startup banner shows all 4 functions (section 2 step 5). Distinguish crash from never-started: startup banner present in the console = the worker ran and then exited (check the exit reason in the console tail / `%LOCALAPPDATA%\TradingSystem\logs`); no banner at all = it was never started. If the box slept through a timer, that run is gone (no catch-up); log the gap day in the Run Log | +| **Nothing at all** — no report, no alert, on a trading day | Worker (or the whole box) was not running — the dead-man case | **Check this first each morning.** Confirm the worker process is up and the startup banner shows all 4 functions (section 2 step 5). Distinguish crash from never-started: startup banner present in the console = the worker ran and then exited (check the exit reason in the console tail / `%LOCALAPPDATA%\TradingSystem\logs`); no banner at all = it was never started. If the box slept through a timer, that run is gone (no catch-up); log the gap day in the Run Log. The `TradingSystem-GapDayMonitor` task (2:30 PM weekdays — see the gap-day monitor subsection below) automates this check and alerts when the day's snapshot is missing | +| `Dead-Man Alert — No EOD Snapshot for ` (orange; sent by the **gap-day monitor task**, not the worker) | The 2:30 PM `TradingSystem-GapDayMonitor` task found no entry for today in `data/snapshots.json` — the EOD path never ran (worker down, box asleep, or EOD failure: the dead-man case above). **On a market holiday this is an expected false positive** — the monitor is weekday-only with no holiday calendar (~9/year, accepted by design); ignore it | Holiday → ignore. Otherwise triage the dead-man row above (worker up? banner? local logs), then record the gap day in the Run Log | | **Worker IS running, timers fired (visible in log), but no Discord message arrived** | Discord unreachable (or webhook broken) at the moment an ops/risk alert was sent — the alert was dropped (dead-man gap on the alert path) | The ONLY signal is the local worker log (console, or the files under `%LOCALAPPDATA%\TradingSystem\logs`): the signature is a `LogError` line from `DiscordRiskAlertService` matching `"… alert NOT delivered and will not be retried — … AlertDropped=True …"` (rendered console form — capital `T`); search the log files for `AlertDropped=True`. Operator action: treat the logged title as the alert you never received — triage its underlying failure first, then fix Discord delivery (webhook URL valid? Discord up?), and log the incident in the Run Log | | No daily report by ~15 min after the EOD timer, but no orange alert either | Report send failed silently is *not* possible without a log — check worker log for report-path warnings; also re-check the dead-man cases above | Worker log around 20:30 UTC (12:30 PM PT winter / 1:30 PM PT summer); `data/snapshots.json` tells you whether the EOD run itself happened | | `Broker Connect Failure — …` alert | TWS not running, not logged in, API disabled, or port/client-id mismatch | Match TWS API settings against `IBKR:Host/Port/ClientId` in `local.settings.json`; restart TWS; pre-market failure skips the options sleeve, EOD failure skips the snapshot | @@ -183,6 +187,45 @@ KD-005/KD-006 — so its absence is by design, not a gap to fix.) | Gateway down (health check fails, or log line `Direct API fallback disabled; gateway miss → deterministic rules`) | claude-gateway process not running or CLI session expired | **Expected degrade, not an incident:** regime classification falls back to deterministic rules with zero metered spend. Restart the gateway; check `GET /health/cli` for credential status. Do NOT enable the metered fallback as a remediation | | Snapshot missing for a day (`data/snapshots.json`) | Any of the above on the EOD path | Identify which from logs/alerts; record the gap day in the Run Log | +### Gap-day monitor (dead-man) — one-time manual registration + +`tools/ops/Check-DailySnapshot.ps1` (S6-004) closes the dead-man blind spot in the first +triage row: a down worker can never tell you it is down, so an independent scheduled task +checks each weekday at **2:30 PM local** (≥45 min after the EOD timer in both DST regimes) +that today's entry exists in `data/snapshots.json`, and posts an orange +`Dead-Man Alert — No EOD Snapshot for ` embed when it does not. The webhook URL is +read from the gitignored `local.settings.json` (`Discord:WebhookUrl`) and is never logged. + +**Registration is a one-time manual operator step** — it is never performed by CI, tests, +hooks, or agent pipelines: + +```powershell +pwsh -NoProfile -File D:\Source\TradingSystem\tools\ops\Register-GapDayMonitorTask.ps1 +``` + +Run this as the normal dev-box user — no elevation required; the task is registered to +run as the current user. + +The script is idempotent (re-running replaces the task). Verify with +`schtasks /query /tn TradingSystem-GapDayMonitor /v /fo LIST`. + +At the end of the 12-week run, decommission the task: +`Unregister-ScheduledTask -TaskName TradingSystem-GapDayMonitor -Confirm:$false`. + +**Exit-code legend** (visible in Task Scheduler history — the code tells you what happened +without opening logs): + +| Exit code | Meaning | +|---|---| +| 0 | Snapshot present (or weekend — nothing expected) | +| 1 | Snapshot MISSING; alert skipped (Discord disabled or webhook not configured) — still investigate | +| 2 | Snapshot MISSING; alert delivery FAILED (Discord unreachable) — investigate both | +| 3 | Snapshot MISSING; alert delivered — triage per the table above | + +Optional controlled drill (renders a real alert without waiting for a genuine gap day): +`Check-DailySnapshot.ps1 -Date ` — note it in the Run Log as +a drill. Use `-WhatIf` to evaluate without posting. + ## 6. Locked posture (ops policy) Restating the repo trading rules as standing ops policy for the entire validation window: diff --git a/tools/ops/Check-DailySnapshot.ps1 b/tools/ops/Check-DailySnapshot.ps1 new file mode 100644 index 0000000..c4ba3c4 --- /dev/null +++ b/tools/ops/Check-DailySnapshot.ps1 @@ -0,0 +1,163 @@ +<# +.SYNOPSIS + Dead-man gap-day monitor: verifies today's DailySnapshot entry exists (S6-004). + +.DESCRIPTION + Observability only — reads a JSON file and (on a gap) posts one Discord embed. + It never touches TWS, the worker, or any trading state, and is safe to run while + the live host is up. + + Snapshot location of record: JsonSnapshotRepository writes + Path.Combine(LocalStorage:DataDirectory, "snapshots.json") with DataDirectory="data" + (relative), resolved against the worker's working directory. The + TradingSystem-FuncHost scheduled task action is + `cmd /c cd /d D:\Source\TradingSystem\src\TradingSystem.Functions && func start ...`, + so the worker cwd — verified against the registered task on 2026-06-10 — is + D:\Source\TradingSystem\src\TradingSystem.Functions and the file of record is + D:\Source\TradingSystem\src\TradingSystem.Functions\data\snapshots.json (the + -SnapshotPath default). JSON is camelCase with ISO-8601 dates ("date" property). + + Trading-day logic is weekday-only (Mon-Fri). Market-holiday false positives + (~9/year) are ACCEPTED and benign: "no snapshot today" on a holiday is expected — + the operator ignores the alert (see docs/paper-validation-runbook.md section 5). + No holiday calendar, no Polygon call from PowerShell — by design (spec D11). + + Secrets: the Discord webhook URL is read from the gitignored local.settings.json + and flows file -> variable -> Invoke-RestMethod -Uri ONLY. It is never written to + console, logs, errors, or transcript, and never interpolated into any string. + +.PARAMETER Date + The trading date to check (default: today). Override for fixture-driven testing + and for controlled drills against past dates. + +.PARAMETER SnapshotPath + Path to snapshots.json (default: the live worker's file of record, see above). + +.PARAMETER SettingsPath + Path to the gitignored local.settings.json holding Discord:Enabled / + Discord:WebhookUrl (default: the live worker's settings file). + +.PARAMETER WhatIf + Evaluate and print the verdict, but never POST. Exits with the code the real run + would produce assuming delivery success (so fixtures are deterministic). + +.NOTES + Exit codes (diagnosable from scheduled-task history): + 0 = snapshot present (or weekend — nothing expected) + 1 = snapshot missing, alert skipped (Discord disabled, settings/webhook absent) + 2 = snapshot missing, alert delivery FAILED + 3 = snapshot missing, alert delivered (or would be, under -WhatIf) + + Never auto-registered or run by CI, tests, hooks, or agent pipelines. + Registration is an explicit operator action via Register-GapDayMonitorTask.ps1. + Requires PowerShell 7 (pwsh). +#> +[CmdletBinding()] +param( + [DateTime]$Date = (Get-Date), + [string]$SnapshotPath = 'D:\Source\TradingSystem\src\TradingSystem.Functions\data\snapshots.json', + [string]$SettingsPath = 'D:\Source\TradingSystem\src\TradingSystem.Functions\local.settings.json', + [switch]$WhatIf +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$dateLabel = $Date.ToString('yyyy-MM-dd') + +# --- 1. Trading-day check (weekday-only; holiday false positives accepted, see header) --- +if ($Date.DayOfWeek -in @([DayOfWeek]::Saturday, [DayOfWeek]::Sunday)) { + exit 0 +} + +# --- 2. Look for a snapshot entry whose date part equals -Date --- +$found = $false +if (Test-Path -LiteralPath $SnapshotPath) { + try { + $entries = Get-Content -LiteralPath $SnapshotPath -Raw | ConvertFrom-Json + foreach ($entry in @($entries)) { + if ($null -eq $entry -or -not ($entry.PSObject.Properties.Name -contains 'date')) { continue } + $entryDate = if ($entry.date -is [DateTime]) { + $entry.date + } else { + [DateTime]::Parse([string]$entry.date, + [System.Globalization.CultureInfo]::InvariantCulture, + [System.Globalization.DateTimeStyles]::RoundtripKind) + } + if ($entryDate.Date -eq $Date.Date) { $found = $true; break } + } + } + catch { + # Unreadable/malformed file counts as missing — still a useful dead-man signal. + Write-Warning "Could not parse snapshot file ($($_.Exception.GetType().Name)); treating as missing." + } +} + +if ($found) { + Write-Host "OK: DailySnapshot entry present for $dateLabel." + exit 0 +} + +Write-Warning "GAP: no DailySnapshot entry for $dateLabel (weekday). If today is a market holiday this is a benign false positive — see runbook section 5." + +# --- 3. Read Discord settings from the gitignored local.settings.json --- +$discordEnabled = $false +$webhookUrl = $null # flows file -> this variable -> -Uri only; NEVER logged or interpolated +if (Test-Path -LiteralPath $SettingsPath) { + try { + $cfg = Get-Content -LiteralPath $SettingsPath -Raw | ConvertFrom-Json + $discordEnabled = ([string]$cfg.Values.'Discord:Enabled') -eq 'true' + $webhookUrl = [string]$cfg.Values.'Discord:WebhookUrl' + } + catch { + Write-Warning "Could not parse settings file ($($_.Exception.GetType().Name))." + } +} +else { + Write-Warning 'Settings file not found.' +} + +if (-not $discordEnabled -or [string]::IsNullOrWhiteSpace($webhookUrl) -or $webhookUrl -eq 'YOUR_DISCORD_WEBHOOK_URL') { + # Name the exact skip condition so the operator knows what to fix (message-level only; URL value never echoed). + $skipReason = if (-not $discordEnabled) { + "Discord is disabled in config — set Discord:Enabled to 'true' in local.settings.json" + } + elseif ([string]::IsNullOrWhiteSpace($webhookUrl)) { + 'webhook URL is blank — set Discord:WebhookUrl in local.settings.json' + } + else { + 'webhook URL is still the placeholder — replace YOUR_DISCORD_WEBHOOK_URL with the real webhook in local.settings.json' + } + Write-Warning "Alert skipped: $skipReason. Snapshot is still MISSING — investigate per runbook section 5." + exit 1 +} + +# --- 4. Alert (orange embed; date + runbook pointer only — no paths, no config values) --- +$title = "Dead-Man Alert — No EOD Snapshot for $dateLabel" +if ($WhatIf) { + Write-Host "WhatIf: would POST '$title' to the configured Discord webhook (not sent)." + exit 3 +} + +$payload = @{ + username = 'TradingSystem Ops' + embeds = @( + @{ + title = $title + description = "No DailySnapshot entry was written for $dateLabel. The worker may be down (dead-man case) — triage per docs/paper-validation-runbook.md section 5, first row. If $dateLabel was a market holiday, this is an expected false positive: ignore." + color = 15105570 # orange — operational failure class + } + ) +} | ConvertTo-Json -Depth 5 + +try { + Invoke-RestMethod -Uri $webhookUrl -Method Post -ContentType 'application/json' ` + -Body $payload -TimeoutSec 10 | Out-Null + Write-Host 'Alert delivered.' + exit 3 +} +catch { + # Exception text could echo the request URI — report the type name only. + Write-Error -ErrorAction Continue "Alert delivery FAILED ($($_.Exception.GetType().Name)). Snapshot is still MISSING for $dateLabel." + exit 2 +} diff --git a/tools/ops/Register-GapDayMonitorTask.ps1 b/tools/ops/Register-GapDayMonitorTask.ps1 new file mode 100644 index 0000000..848f9eb --- /dev/null +++ b/tools/ops/Register-GapDayMonitorTask.ps1 @@ -0,0 +1,69 @@ +<# +.SYNOPSIS + Registers the TradingSystem-GapDayMonitor scheduled task (one-time MANUAL operator step). + +.DESCRIPTION + Registers a Windows scheduled task that runs Check-DailySnapshot.ps1 on weekdays at + 2:30 PM local time — at least 45 minutes after the EOD timer in BOTH DST regimes + (EOD = 1:30 PM PDT / 12:30 PM PST), so the snapshot has had time to land. + + Idempotent: re-running unregisters the existing task and registers it again + (useful after moving the repo or changing the schedule). + + Quoting note: the -File path is wrapped in literal double quotes inside the + argument string (the S5 TradingSystem-Azurite cmd-quoting bug is the cautionary + precedent). Verify after registration with: + schtasks /query /tn TradingSystem-GapDayMonitor /v /fo LIST + +.PARAMETER TaskName + Scheduled task name (default: TradingSystem-GapDayMonitor — repo task convention). + +.PARAMETER At + Local wall-clock run time (default 2:30 PM). Do not move earlier than 2:15 PM: + the EOD timer fires at 1:30 PM local in PDT (12:30 PM in PST), and the monitor + must stay >=45 minutes behind it in both DST regimes. + +.PARAMETER ScriptPath + Absolute path to Check-DailySnapshot.ps1 (default: sibling of this script). + +.NOTES + NEVER invoked by CI, tests, hooks, or any agent pipeline — registration is a + human/runbook action (docs/paper-validation-runbook.md, gap-day monitor subsection). + Requires PowerShell 7 (pwsh) and rights to register scheduled tasks. +#> +[CmdletBinding()] +param( + [string]$TaskName = 'TradingSystem-GapDayMonitor', + [string]$At = '14:30', + [string]$ScriptPath = (Join-Path $PSScriptRoot 'Check-DailySnapshot.ps1') +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$ScriptPath = (Resolve-Path -LiteralPath $ScriptPath).Path +$repoRoot = Split-Path (Split-Path (Split-Path $ScriptPath)) # tools/ops/.. /.. = repo root +$pwshPath = (Get-Command pwsh).Source + +# Literal quotes around the -File path — the Azurite task's quoting bug is the precedent. +$arguments = "-NoProfile -ExecutionPolicy Bypass -File `"$ScriptPath`"" + +$action = New-ScheduledTaskAction -Execute $pwshPath -Argument $arguments -WorkingDirectory $repoRoot +$trigger = New-ScheduledTaskTrigger -Weekly -At $At -DaysOfWeek Monday, Tuesday, Wednesday, Thursday, Friday +$settings = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Minutes 5) + +$existing = Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue +if ($existing) { + Write-Host "Task '$TaskName' already exists — unregistering for idempotent re-register." + Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false +} + +Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger -Settings $settings ` + -Description 'TradingSystem dead-man gap-day monitor: checks for today''s DailySnapshot entry, weekdays 2:30 PM local (S6-004). Exit codes: 0 present/weekend, 1 missing+skipped, 2 missing+alert-failed, 3 missing+alert-delivered.' | Out-Null + +Write-Host "Registered scheduled task '$TaskName':" +Get-ScheduledTask -TaskName $TaskName | Select-Object -ExpandProperty Actions | + Format-List Execute, Arguments, WorkingDirectory +Get-ScheduledTask -TaskName $TaskName | Select-Object -ExpandProperty Triggers | + Format-List StartBoundary, DaysOfWeek +Write-Host "Verify quoting with: schtasks /query /tn $TaskName /v /fo LIST" diff --git a/tools/ops/preflight.ps1 b/tools/ops/preflight.ps1 new file mode 100644 index 0000000..c0722cc --- /dev/null +++ b/tools/ops/preflight.ps1 @@ -0,0 +1,110 @@ +<# +.SYNOPSIS + Daily preflight checks for the paper-validation run (automates runbook section 2). + +.DESCRIPTION + Read-only probes with bounded timeouts — safe to run while the live host is up; + touches no trading state and reads no secrets at all. + + Checks (docs/paper-validation-runbook.md section 2): + (a) TWS paper API port reachable (TcpClient, 3s timeout — not Test-NetConnection, + which can stall) -> PASS/FAIL + (b) claude-gateway /health (5s timeout). Gateway-down is the DESIGNED + rules-fallback degrade (ADR-029/ADR-030) -> PASS/WARN + (c) Functions worker admin endpoint lists all 4 functions: + DailyOrchestrator_PreMarket, DailyOrchestrator_EndOfDay, + IncomeSleeve_MonthlyReinvest, IncomeSleeve_QuarterlyAudit -> PASS/FAIL + (d) snapshots data directory exists -> PASS/FAIL + + Exit code: 0 = no FAIL (WARNs allowed), 1 = at least one FAIL. + +.NOTES + Never auto-run by CI, tests, hooks, or agent pipelines — operator tooling only. + Requires PowerShell 7 (pwsh). +#> +[CmdletBinding()] +param( + [string]$TwsHost = '127.0.0.1', + [int]$TwsPort = 7497, + [string]$GatewayHealthUrl = 'http://localhost:3131/health', + [string]$WorkerAdminUrl = 'http://127.0.0.1:7071/admin/functions', + [string]$DataDirectory = 'D:\Source\TradingSystem\src\TradingSystem.Functions\data' +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$expectedFunctions = @( + 'DailyOrchestrator_PreMarket', + 'DailyOrchestrator_EndOfDay', + 'IncomeSleeve_MonthlyReinvest', + 'IncomeSleeve_QuarterlyAudit' +) + +$results = [System.Collections.Generic.List[object]]::new() +function Add-Result([string]$Check, [string]$Status, [string]$Detail) { + $results.Add([pscustomobject]@{ Check = $Check; Status = $Status; Detail = $Detail }) +} + +# --- (a) TWS API port: TcpClient connect with a 3s timeout --- +$client = [System.Net.Sockets.TcpClient]::new() +try { + $connectTask = $client.ConnectAsync($TwsHost, $TwsPort) + if ($connectTask.Wait(3000) -and $client.Connected) { + Add-Result 'TWS API port' 'PASS' "TCP connect to ${TwsHost}:${TwsPort} succeeded" + } + else { + Add-Result 'TWS API port' 'FAIL' "No TCP connect to ${TwsHost}:${TwsPort} within 3s — TWS down, not logged in, or API disabled (runbook section 2 step 1)" + } +} +catch { + # ConnectAsync faults (e.g. connection refused) surface here via Task.Wait — same verdict as a timeout. + Add-Result 'TWS API port' 'FAIL' "No TCP connect to ${TwsHost}:${TwsPort} (refused or unreachable) — TWS down, not logged in, or API disabled (runbook section 2 step 1)" +} +finally { + $client.Dispose() +} + +# --- (b) claude-gateway health: WARN, not FAIL (designed degrade per ADR-029/ADR-030) --- +try { + Invoke-RestMethod -Uri $GatewayHealthUrl -TimeoutSec 5 | Out-Null + Add-Result 'claude-gateway' 'PASS' 'Health endpoint responded' +} +catch { + Add-Result 'claude-gateway' 'WARN' 'Gateway down — system degrades to deterministic rule-based regime classification (ADR-029/ADR-030): expected behavior, not an incident. Restart the gateway when convenient' +} + +# --- (c) Functions worker: admin endpoint must list all 4 functions --- +try { + $functions = Invoke-RestMethod -Uri $WorkerAdminUrl -TimeoutSec 5 + $names = @($functions | ForEach-Object { $_.name }) + $missing = @($expectedFunctions | Where-Object { $_ -notin $names }) + if ($missing.Count -eq 0) { + Add-Result 'Functions worker' 'PASS' "All 4 functions registered ($($names.Count) total)" + } + else { + Add-Result 'Functions worker' 'FAIL' "Worker is up but missing function(s): $($missing -join ', ') — broken build or wrong project started (runbook section 2 step 5)" + } +} +catch { + Add-Result 'Functions worker' 'FAIL' 'Admin endpoint unreachable — worker not running or still booting (runbook section 2 steps 3-4)' +} + +# --- (d) snapshots data directory --- +if (Test-Path -LiteralPath $DataDirectory -PathType Container) { + Add-Result 'Data directory' 'PASS' 'Snapshots data directory exists' +} +else { + Add-Result 'Data directory' 'FAIL' "Data directory not found at the expected worker cwd location: $DataDirectory" +} + +# --- Summary --- +$results | Format-Table -AutoSize -Wrap | Out-String | Write-Host + +$failCount = @($results | Where-Object Status -eq 'FAIL').Count +if ($failCount -gt 0) { + Write-Host "Preflight: $failCount FAIL check(s) — do not assume the day will run." + exit 1 +} +Write-Host 'Preflight: all checks passed (WARNs, if any, are designed degrades).' +exit 0