From 0f0887e1c14969e4e114cf42094d86893fa01d36 Mon Sep 17 00:00:00 2001 From: Drew Michael Date: Tue, 9 Jun 2026 12:51:41 -0500 Subject: [PATCH] v1.2.0: dashboard performance overhaul + security hardening MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cold and warm dashboard loads drop from seconds to sub-second on large services; sustained concurrent load no longer wedges the backend. Read path I/O is structurally cut by a per-service DuckDB connection pool, a per-minute time-series rollup bundle, size-capped bin-packing local compaction (daily + weekly tiers), composite admin-page endpoints, and a frontend pre-warm + hover-prefetch pattern that makes navigation feel instant. Performance — structural * Per-minute time-series rollup bundle precomputes the dashboard chart's per-minute aggregate per (field, hour); eliminates the wide Iceberg scan on chart render. * Per-day rollup compaction — closed days roll up into a single per-day file; the reader prefers per-day and falls back to hourly only for the current day. * Size-capped bin-packing local compaction (default 256 MB cap) replaces single-file daily/weekly rollups; preserves DuckDB scan parallelism on multi-month services. * DuckDB connection-pool tuning — DUCKDB_POOL_CONN_MEMORY_LIMIT and DUCKDB_POOL_CONN_THREADS env vars cap per-connection RSS and threads. View-binding moved outside the pool's Condition lock to eliminate a stale-snapshot deadlock. * Composite read endpoints — POST /api/scoring/dashboard, GET /api/scoring/analytics, GET /api/scoring/config, GET /api/network-health (now includes shielding), and the new POST /api/origin/aggregates collapse multi-card mounts into one round trip. Per-card endpoints stay mounted for back-compat. * Parquet ingest sort key changed to (timestamp, ip) so sessions queries stream-merge on ip instead of materialising a temp table (~2× speedup). * ingested_files.file_date column + (source_name, file_date) index for the log-accounting fast path. * Iceberg buffer files tombstoned and removed on the next pass instead of unlinked inline at commit. optimize_table adds union_by_name + retry-on-CAS-conflict. * Bootstrap stale-while-revalidate for dir-stats; views folded into the response. Performance — tuning * Dashboard: live-hour TEMP TABLE shared across CTEs; Python-side bot match; memoised ngwaf_top. * Insights: coalesce 4 city/region/country queries into 1; coalesce 4 URL-keyed insights into 1 CTE. * Sessions: split monolithic CTE into measurable stages; eliminate hot- path temp-table materialisation. * Origin: combine two sequential scans into one via GROUPING SETS. * Cron-runs since_id delta-poll on /logs recentCrons. * Admin usage-log visibility-gates its 30s tick; latest-per-task SQL rewritten to skip the full join. * 60s TTL on bot-source cache-dir scandir. * React-Query: skip 4xx retries; hooks lifted out of insights / ReportLayout render-props. Frontend * starlette-compress replaces GZipMiddleware (br / zstd / gzip negotiation). * Keep-alive on Next.js http/undici global agents. * Pre-warm + lazy-mount pattern for plotly + maplibre-gl + world.geojson on AppLayout mount; hover-prefetch sidebar links; per-insight skeleton cards on first paint. * Modulepreload for the plotly chunk via a build-time-generated preload manifest. Root layout opts out of build-time SSG so the manifest is read at request time. * /geo/* aggressively cached; PlotlyChart dynamic-import on /network. * SystemHealthCard polls at 1s for live attack/load feedback. * Shared useNowMs interval for visible-tick components. * MapLibre style-data listener replaces a 100ms setTimeout poll. Reliability * Multi-worker login loop fixed via on-demand SQLite session rehydration. * DuckDB lock conflict between pool and cron writes resolved — get_connection forces read_only=False on the file. * QueryRunner empty-schema self-heal busts _view_cache before the force=True rebuild so the lock-timeout fallback can't re-execute the same stale cached SQL (mirrors the execute() self-heal). Without this, ingest-cron lock contention pinned the view to a deleted buffer path and the dashboard surfaced "No data available" on a 200. * QueryRunner clears _view_cache before force=True rebuild on the post- empty self-heal path. * Iceberg s3fs proxy hook falls back to the process-global source so the hook always registers (cold-start LIST before _get_catalog). * Top-N current-hour merge silent ImportError fixed; rollup compaction threads run_id through the error branch + uses in-memory DuckDB. * Dashboard response cache: write to is_cached (not aliased _is_cached) to keep Pydantic from dropping the flag. * Usage-log reconcile cycle changed from DELETE+INSERT to UPSERT. * expire_snapshots updated for pyiceberg 0.11.1 + emits cron_runs telemetry. * Next.js 16 compat: middleware.ts → proxy.ts (Caddy-marker preserved). * TelemetryResponseBodyMiddleware backstops endpoints that bypass BaseResponse.with_telemetry. Security * Cross-tenant ContextVar leak in the s3fs proxy hook closed — ThreadPoolExecutor.submit monkeypatched to wrap callables in contextvars.copy_context(); endpoint-keyed global registry removed. * Path-param service-scope desync — centralised the session-scope check via a router-utils helper invoked on every scoped route. * Secret-in-URL leak on downloads — switched to a signed short-lived bearer stripped before redirect. * Strict input validation on the destructive-op surface (provision teardown, NGWAF mutations, scoring threshold + enforce-status-code + recv-exclusion-regex). Length caps, character allowlists, and falco static analysis before any VCL ships. * CSRF: state-changing endpoints moved off GET. * Cross-tenant cache key audit — every per-tenant cache key includes service_id; closed two missing entries on insights and origin paths. * Thread leak in share-login replaced by on-demand SQLite rehydration. * Terms-of-service bypass on share-login /acknowledge fixed. Tests * 3500+ backend tests (+450), 290+ frontend vitest tests (+25). * New coverage: DuckDB pool, local compaction, rollups compaction + hour bundling, iceberg helpers, service manager, SQL validator, telemetry response middleware, router utils, state sync, terraform gen, plus router coverage for the new composite endpoints and the destructive-op-auth surface. * make ci green: lint + format + mypy + pytest + vcl-test + verify-deps + typecheck-frontend + test-frontend + osv + secret-scan. Infrastructure * Synthetic load generator (scripts/loadtest_generator.py) and read-path probe (scripts/dev/loadtest_probe.sh) for reproducible perf measurement. * Two-pass next build in the frontend Dockerfile so SSG sees the correct plotly chunk hashes. Documentation * AGENTS.md — Key Systems entries for the DuckDB connection pool, the hourly Top-N rollup pipeline, and the response telemetry middleware; local-compaction section updated for the bin-packing tiers. * MONKEYPATCHES.md — documents the new ThreadPoolExecutor.submit patch. Co-Authored-By: Claude Opus 4.7 --- AGENTS.md | 17 +- CHANGELOG.md | 99 + MONKEYPATCHES.md | 46 +- backend/core/data_migrations.py | 24 + backend/core/duckdb.py | 9 +- backend/core/duckdb_pool.py | 110 +- backend/core/fastly/utils.py | 20 +- backend/core/iceberg.py | 594 +++++- backend/core/local_compaction.py | 365 ++-- backend/core/log_fields.py | 62 +- backend/core/metadata_db.py | 356 +++- backend/core/rollups.py | 643 +++++- backend/core/share_db.py | 94 +- backend/core/sqlite_migrations.py | 66 + backend/cron_progress.py | 4 +- backend/deps.py | 6 +- backend/main.py | 36 +- backend/models/common.py | 13 + backend/models/lake.py | 42 +- backend/models/network.py | 5 + backend/models/origin.py | 23 + .../provision/session_scoring_orchestrator.py | 6 +- backend/provision/session_scoring_vcl.py | 3 +- backend/repositories/_base.py | 628 +++++- backend/repositories/cron.py | 2 + backend/repositories/dashboard.py | 272 ++- backend/repositories/insights/definitions.py | 5 +- backend/repositories/insights/repository.py | 428 +++- backend/repositories/network.py | 49 +- backend/repositories/origin.py | 548 ++++- backend/repositories/query.py | 16 +- backend/repositories/security.py | 123 +- backend/repositories/sessions.py | 77 +- backend/routers/admin.py | 216 +- backend/routers/bootstrap.py | 100 +- backend/routers/network.py | 20 + backend/routers/origin.py | 42 + backend/routers/provision.py | 145 +- backend/routers/services/core.py | 2 +- backend/routers/services/cron.py | 7 +- backend/routers/session_scoring.py | 166 +- backend/routers/share_admin.py | 18 + backend/routers/share_auth.py | 85 +- backend/routers/usage.py | 146 +- backend/scheduler.py | 146 +- backend/scoring/cookie.py | 1 + backend/scoring/normalize.py | 9 +- backend/scoring/scorer.py | 6 +- backend/services/service_manager.py | 134 +- backend/state_sync.py | 19 +- backend/utils/remote_access.py | 133 +- backend/utils/router_utils.py | 26 + backend/utils/sql_validator.py | 86 +- backend/utils/telemetry.py | 48 +- backend/utils/telemetry_proxy.py | 17 +- .../utils/telemetry_response_middleware.py | 235 +++ backend/utils/terraform_gen.py | 9 +- backend/utils/tunnel.py | 14 +- compute/scorer/src/cookie.rs | 20 +- compute/scorer/src/main.rs | 40 +- compute/scorer/src/normalize.rs | 38 +- frontend/Dockerfile | 22 +- .../app/share-login/acknowledge.test.tsx | 25 +- .../__tests__/hooks/useUrlFilterSync.test.ts | 9 + frontend/__tests__/middleware.test.ts | 2 +- frontend/__tests__/preload-manifest.test.ts | 120 ++ frontend/app/admin/page.tsx | 43 +- frontend/app/admin/session-scoring/page.tsx | 123 +- frontend/app/admin/share/page.tsx | 8 +- frontend/app/admin/usage-log/page.tsx | 31 +- frontend/app/alerts/page.tsx | 8 +- frontend/app/dashboard/page.tsx | 1778 +++++++++-------- frontend/app/insights/page.tsx | 261 ++- frontend/app/layout.tsx | 38 + frontend/app/logs/page.tsx | 19 +- frontend/app/network/page.tsx | 33 +- frontend/app/share-login/acknowledge/page.tsx | 24 +- frontend/components/AppLayout.tsx | 41 +- .../components/FilterBar/SaveViewDialog.tsx | 3 + .../components/FilterBar/ViewSelector.tsx | 3 + .../Insights/InsightCardSkeleton.tsx | 41 + frontend/components/LazyMount.tsx | 18 +- frontend/components/Map/ChoroplethMap.tsx | 19 +- frontend/components/Map/MapPrewarm.tsx | 80 + .../components/PlotlyChart/PlotlyChart.tsx | 23 +- .../components/PlotlyChart/PlotlyPrewarm.tsx | 81 + .../PopLocationsModal/PopLocationsModal.tsx | 2 +- .../ProvisionWizard/ProvisionWizard.tsx | 20 +- frontend/components/QueryProvider.tsx | 21 + frontend/components/SystemHealthCard.tsx | 22 +- frontend/hooks/useBootstrap.ts | 16 +- frontend/hooks/useCardVisibility.ts | 34 +- frontend/hooks/useShareStatusBanner.tsx | 7 +- frontend/hooks/useUrlFilterSync.ts | 18 +- frontend/lib/_preload-chunks.json | 14 + frontend/lib/preload-manifest.ts | 54 + frontend/next.config.ts | 15 +- frontend/openapi.json | 1363 ++++++++++--- frontend/package-lock.json | 4 +- frontend/package.json | 4 +- frontend/scripts/build-preload-manifest.mjs | 139 ++ frontend/types/api.generated.ts | 771 ++++++- pyproject.toml | 3 +- run.sh | 12 +- scripts/backfill_rollups.py | 13 +- scripts/dev/loadtest_probe.sh | 166 ++ scripts/loadtest_generator.py | 287 +++ tests/core/test_duckdb_concurrency.py | 26 +- tests/core/test_duckdb_pool.py | 56 + tests/core/test_iceberg.py | 782 ++++++++ tests/core/test_iceberg_helpers.py | 219 +- tests/core/test_lake_info.py | 45 +- tests/core/test_local_compaction.py | 117 +- tests/core/test_metadata_db_crud.py | 106 + tests/core/test_metadata_db_migrations.py | 254 +++ tests/core/test_rollups_compaction.py | 434 ++++ tests/core/test_rollups_hour_bundling.py | 313 +++ tests/models/test_common.py | 59 + tests/remote_access/test_middleware.py | 138 ++ tests/remote_access/test_share_auth_routes.py | 200 ++ tests/remote_access/test_share_db.py | 36 +- tests/repositories/test_base.py | 320 +++ tests/repositories/test_base_helpers.py | 79 + tests/repositories/test_cron.py | 86 + tests/repositories/test_dashboard.py | 83 + tests/repositories/test_insights.py | 257 +++ .../repositories/test_insights_processors.py | 15 +- tests/repositories/test_origin.py | 41 + tests/repositories/test_query.py | 9 + tests/repositories/test_security.py | 8 +- tests/routers/services/test_cron_router.py | 38 + .../routers/test_admin_mutation_endpoints.py | 253 ++- tests/routers/test_bootstrap.py | 101 +- tests/routers/test_cron_runs_stream.py | 31 + tests/routers/test_provision.py | 113 +- tests/routers/test_provision_lifecycle.py | 14 +- tests/routers/test_provision_teardown_auth.py | 17 + tests/routers/test_provision_wizard_e2e.py | 22 +- tests/routers/test_scoring_exclude_regex.py | 20 + tests/routers/test_session_scoring_router.py | 16 +- tests/scoring/test_normalize.py | 38 + tests/scoring/test_scorer.py | 26 + tests/services/__init__.py | 0 tests/services/test_service_manager.py | 308 +++ tests/test_deps.py | 6 +- tests/test_e2e_pyiceberg_s3.py | 2 - tests/test_scheduler.py | 171 ++ tests/utils/test_fastly_utils.py | 23 + tests/utils/test_router_utils.py | 108 + tests/utils/test_sql_validator.py | 53 + tests/utils/test_state_sync.py | 52 + tests/utils/test_telemetry.py | 46 + tests/utils/test_telemetry_proxy.py | 69 + tests/utils/test_telemetry_proxy_phase3b.py | 21 +- .../test_telemetry_response_middleware.py | 370 ++++ tests/utils/test_terraform_gen.py | 17 + uv.lock | 147 +- 157 files changed, 15741 insertions(+), 2581 deletions(-) create mode 100644 backend/utils/telemetry_response_middleware.py create mode 100644 frontend/__tests__/preload-manifest.test.ts create mode 100644 frontend/components/Insights/InsightCardSkeleton.tsx create mode 100644 frontend/components/Map/MapPrewarm.tsx create mode 100644 frontend/components/PlotlyChart/PlotlyPrewarm.tsx create mode 100644 frontend/lib/_preload-chunks.json create mode 100644 frontend/lib/preload-manifest.ts create mode 100644 frontend/scripts/build-preload-manifest.mjs create mode 100755 scripts/dev/loadtest_probe.sh create mode 100755 scripts/loadtest_generator.py create mode 100644 tests/core/test_duckdb_pool.py create mode 100644 tests/core/test_rollups_compaction.py create mode 100644 tests/core/test_rollups_hour_bundling.py create mode 100644 tests/models/test_common.py create mode 100644 tests/services/__init__.py create mode 100644 tests/services/test_service_manager.py create mode 100644 tests/utils/test_fastly_utils.py create mode 100644 tests/utils/test_telemetry_response_middleware.py diff --git a/AGENTS.md b/AGENTS.md index 605de0e6..7bf0fb01 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -86,10 +86,10 @@ Teardown removes jobs on the next `_sync_jobs()` reload. The `config not found, ### Local-Only Parquet Compaction (Dashboard Performance) -To maintain top-tier dashboard querying speeds over long periods without generating massive FOS write costs, we employ two local-only compaction layers (implemented in `backend/core/local_compaction.py`): -1. **Periodic Job (`local_compact_{id}`):** Runs every 2 minutes. It scans local cache directories, identifies any hourly partitions containing multiple small files, and merges them into a single compacted Parquet file. -2. **Compact-on-Sync Thread:** Triggered immediately after a raw sync completes. If multiple new files are detected, a background thread merges them immediately rather than waiting for the next cron interval. -3. **Daily Tier Rollup:** Partitions older than 7 days (customizable via `LOCAL_COMPACT_DAILY_TIER_DAYS`) are rolled up into larger daily compacted files to prevent DuckDB performance degradation from high file-system descriptor counts. +To maintain top-tier dashboard querying speeds over long periods without generating massive FOS write costs or massive file bottlenecks, we employ sequential size-capped bin-packing local compaction (implemented in `backend/core/local_compaction.py`): +1. **Periodic Job (`local_compact_{id}`):** Runs every 2 minutes. It scans local cache directories, identifies any hourly partitions containing multiple small files, and merges them sequentially into size-capped compacted Parquet files (default <= 256MB) to maintain DuckDB query parallelism. +2. **Compact-on-Sync Thread:** Triggered immediately after a raw sync completes. If multiple new files are detected, a background thread merges them immediately. +3. **Daily & Weekly Tier Rollup:** Partitions older than 7 days (customizable via `LOCAL_COMPACT_DAILY_TIER_DAYS`) are sequentially bin-packed by day into daily files (e.g. `daily_YYYY-MM-DD_.parquet`), with single-file bins correctly migrated to retire empty hourly dirs. Daily files older than 30 days are further bin-packed into weekly files (e.g. `weekly_YYYY-WXX_.parquet`) under `weekly/`. All files are capped at `_MAX_PARTITION_BYTES` to prevent huge file bottlenecks and preserve maximum parallelism. *Note: Use `local_compaction` for hot-tier ongoing dashboard performance. Use the global `optimize_{id}` / `optimize_table` path when you want compaction reflected in FOS too.* @@ -186,6 +186,15 @@ Per-bucket reconciliation between Fastly's `/stats/service/{id}` log-emission co ### Iceberg Pointer + Summary Hash-Throttle ([backend/core/iceberg.py](backend/core/iceberg.py)) Every commit writes `metadata_location.txt` (unavoidable) and `table_summary.json` (skippable). The latter is content-hashed against `_table_summary_hash_cache`; identical payloads skip the PUT. Saves one FOS PUT per no-op commit in steady state. Cache is module-scope, process-lifetime. +### DuckDB Connection Pool ([backend/core/duckdb_pool.py](backend/core/duckdb_pool.py)) +Per-service LIFO pool replaces per-request `duckdb.connect()` + S3 / iceberg setup + view rebind (~50ms steady-state). Pool size is `DUCKDB_POOL_MAX_SIZE` (default 8). All pool connections open with `read_only=False` — `get_connection` forces this so cron writers and pool readers don't trip DuckDB's "different configuration" error on the same file. Optional per-connection tuning: `DUCKDB_POOL_CONN_MEMORY_LIMIT` (e.g. `256MB`) caps RSS growth under concurrent large scans; `DUCKDB_POOL_CONN_THREADS` reduces context-switching when `pool_size × per_conn_threads` exceeds physical cores. View-binding happens outside the pool lock to avoid deadlocking the FastAPI thread pool when an Iceberg snapshot reload blocks. + +### Hourly Top-N Rollups ([backend/core/rollups.py](backend/core/rollups.py), [scripts/backfill_rollups.py](scripts/backfill_rollups.py)) +Precomputes per-hour Top-N aggregates for the dashboard's most-asked fields (ip, country, url, custom fields) and writes them under `/data/rollups/`. Closed hours read from the rollup; the current ("live") hour merges the rollup with a fast scan of the buffer. Plus a per-minute time-series bundle (`rollups/timeseries/...`) used by the dashboard chart to skip the wide Iceberg scan. Skipped buckets fall back to the raw scan path. Generated by `local_compact_{id}` after each compaction pass; the global `optimize_{id}` job rebuilds the day's worth on each run. + +### Response Telemetry Middleware ([backend/utils/telemetry_response_middleware.py](backend/utils/telemetry_response_middleware.py)) +Backstop for endpoints that return a plain `dict` instead of going through `BaseResponse.with_telemetry`. Inspects JSON object responses, injects `_debug_queries` / `_debug_calls` / `_is_cached` from the contextvar collectors if missing. **Must be added INNER to `CompressMiddleware`** (i.e. `add_middleware(TelemetryResponseBodyMiddleware)` BEFORE `add_middleware(CompressMiddleware)`) so it sees the raw JSON, not br/zstd/gzip-encoded bytes. Skips streaming responses, non-dict bodies, and already-instrumented responses. Gated on `DEBUG_RESPONSES`; failure modes are silent + non-blocking. + ### CDN-Fronted Log Delivery FOS reads are fronted by a Fastly CDN VCL service (`cdn_service_id`, `cdn_url`, `cdn_secret`). The CDN validates a shared-secret query param to gate access; rate-limited to blunt brute-force. Separate from the logging service ID. diff --git a/CHANGELOG.md b/CHANGELOG.md index 66d5cadb..3309df7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,105 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.2.0] - 2026-06-09 + +Dashboard performance overhaul plus capability-focused security hardening. Cold and warm dashboard loads drop from seconds to sub-second on large services; sustained concurrent load no longer wedges the backend. Read-path I/O is structurally cut by a per-service DuckDB connection pool, a per-minute time-series rollup bundle, size-capped bin-packing local compaction, composite endpoints that collapse multi-card admin pages into one request, and a frontend pre-warm / hover-prefetch pattern that makes navigation feel instant. Security hardening tightens cross-tenant boundaries, closes a ContextVar propagation hole in the s3fs proxy hook, removes a secret-in-URL leak on downloads, and adds strict validation across the destructive-op surface. + +### Performance + +Structural: + +- **Per-minute time-series rollup bundle** (`backend/core/rollups.py`) precomputes a hour-bundled per-minute aggregate for the dashboard chart, eliminating the wide Iceberg scan on chart render. Generated alongside the existing Top-N rollups. +- **Per-day compaction tier for rollups** — closed days are compacted into per-day parquet files; the reader prefers the per-day file and falls back to hourly only for the current day, cutting file-handle pressure on long-running services. +- **Size-capped bin-packing local compaction** ([backend/core/local_compaction.py](backend/core/local_compaction.py)) replaces single-file daily/weekly rollups with sequential bin-packing capped at `_MAX_PARTITION_BYTES` (default 256 MB). Hourly partitions older than 7 days bin-pack into daily files; daily files older than 30 days bin-pack into weekly files. DuckDB query parallelism is preserved on multi-month services where the prior single-file approach degraded to scan-of-one-huge-file. +- **DuckDB connection-pool tuning knobs** — `DUCKDB_POOL_CONN_MEMORY_LIMIT` and `DUCKDB_POOL_CONN_THREADS` env vars cap per-pool-connection memory and thread count so 8 concurrent queries don't oversubscribe physical cores or balloon RSS. Pool view-binding moved outside the `Condition` lock to eliminate a deadlock under stale-Iceberg-snapshot reload. +- **Composite read endpoints** collapse multi-card mounts into single requests: + - `POST /api/scoring/dashboard` (8 per-card requests → 1) + - `GET /api/scoring/analytics` and `GET /api/scoring/config` + - `GET /api/network-health` now includes shielding analysis + - `POST /api/origin/aggregates` (new) batches the origin page's per-card queries + Per-card endpoints stay mounted for back-compat; the frontend opts into composite where it makes sense. +- **Parquet ingest sort key** changed to `(timestamp, ip)` so sessions queries can stream-merge on `ip` instead of materialising a temp table — ~2× speedup on sessions dashboards. +- **`ingested_files.file_date` column + `(source_name, file_date)` index** added via numbered SQLite migration. The log-accounting fast path uses the index to bucket by day without scanning every row; `metadata_db.get_node_count_avg` and `get_log_accounting_counts` split on it. +- **Iceberg commit hygiene** — buffer files are tombstoned and removed on the next pass instead of unlinked inline at commit time, removing a commit-path stall. `optimize_table` adds `union_by_name` + retry-on-CAS-conflict to silence the nightly schema-evolution warning. +- **Bootstrap stale-while-revalidate** — `/api/bootstrap` returns cached dir-stats immediately and refreshes in the background; views are folded into the response so the admin page doesn't issue a follow-up. + +Tuning: + +- Dashboard live-hour TEMP TABLE shared across CTEs; Python-side bot match + memoised `ngwaf_top` cut DuckDB round-trips. +- Insights coalesce four city/region/country queries into one and four URL-keyed insights into one CTE (Option C pattern). +- Sessions split the monolithic CTE into measurable stages and eliminate the temp-table materialisation on the hot path. +- Origin summary combines two sequential scans into one via `GROUPING SETS`. +- Cron-runs `since_id` delta-poll param + frontend wiring on `/logs recentCrons` so the page only fetches new events. +- Admin usage-log visibility-gates its 30s tick and rewrites the latest-per-task SQL to skip the full join. +- Admin shielding banner endpoint trimmed; share-status `staleTime` tightened. +- Bot-source cache: 60s TTL on the recursive cache-dir `scandir` (was 200–1500 ms per `/api/bootstrap`). +- React-Query: skip 4xx retries; hooks lifted out of insights / ReportLayout render-props so each page mount re-uses one query instance instead of re-mounting on every parent render. + +Frontend: + +- **`starlette-compress` replaces `GZipMiddleware`** — backend now negotiates `br` / `zstd` / `gzip` (was gzip-only). Modern browsers get brotli; rendered-text payloads drop ~25 % on the wire. +- **Keep-alive on Next.js http/undici global agents** so the proxy reuses TCP connections to the FastAPI backend instead of new-handshake-per-request. +- **Pre-warm + lazy-mount pattern** — plotly + maplibre-gl + `world.geojson` are pre-warmed on `AppLayout` mount via hidden one-point charts; the visible chart hydrates from the warm module cache instead of triggering a fresh import on first render. `LazyMount` + `PlotlyChart` start `visible=false` to avoid the hydration-mismatch warning that came with the prior eager-mount pattern. +- **Hover-prefetch sidebar links** so the destination's data warms before the click commits. +- **Per-insight skeleton cards on first paint**; full skeleton rendered from `CARD_CATEGORIES` on the dashboard. +- **Modulepreload for the plotly chunk** via a build-time-generated preload manifest (`scripts/build-preload-manifest.mjs` + `lib/preload-manifest.ts`); restores plotly's preload without re-introducing the nav-lag the first attempt caused. +- **Drop `force-dynamic`** on routes that don't need it; root layout opts out of build-time SSG so the preload manifest is read at request time. +- **`/geo/*` static assets cached aggressively**; `PlotlyChart` dynamic-import on `/network`. +- **`SystemHealthCard` polling moved to 1 s** for live attack/load feedback now that the endpoint is cheap. +- **`useNowMs` reuse** — multiple visible-tick components (countdowns, "X seconds ago") share one interval. +- **Map style-data listener** replaces a 100 ms `setTimeout` poll. + +### Reliability + +- **Multi-worker login loop fixed** — `tunnel.py` now rehydrates a share session on-demand from SQLite when an in-memory cache miss happens on a different uvicorn worker. Previously, login on worker A would loop because worker B couldn't see the freshly-minted session. +- **DuckDB lock conflict resolved** between the connection pool and cron writes — `get_connection` forces `read_only=False` so pool readers and cron writers no longer trip DuckDB's "different configuration" error on the same file. +- **Stale-view self-heal** — `QueryRunner` clears `_view_cache` before the `force=True` rebuild on the post-empty recovery path so the next query doesn't see the stale schema. +- **Iceberg s3fs proxy hook** falls back to the process-global source so the hook always registers, even when the ContextVar is empty (e.g. cold-start LIST before any `_get_catalog` has fired). +- **Top-N current-hour merge** — a silent `ImportError` was dropping the current-hour merge; restored with an explicit fail-loud import. +- **Rollup compaction** — `run_id` threaded through the error branch and the compaction step now uses an in-memory DuckDB so a corrupted on-disk catalog can't wedge the cron. +- **Dashboard response cache** — write to `is_cached` (not the aliased `_is_cached`) so Pydantic doesn't drop the flag on serialise. +- **Dashboard cache hit rate** — disabled the 30 s response-level cache that was masking the rollup wins for fast-changing queries. +- **Usage-log rollup drift** — reconcile cycle changed from DELETE+INSERT to UPSERT so concurrent flushes can't lose rows. +- **Botnet insight investigate link** filters only the queried column, not all of them. +- **`expire_snapshots`** updated for pyiceberg 0.11.1 API and now emits `cron_runs` telemetry. +- **Proxy compatibility** — switched from `middleware.ts` to `proxy.ts` for Next.js 16; restored the Caddy-marker middleware that the upgrade broke. +- **Telemetry response middleware backstop** ([backend/utils/telemetry_response_middleware.py](backend/utils/telemetry_response_middleware.py)) auto-injects `_debug_queries` / `_debug_calls` / `_is_cached` into JSON-dict responses that bypassed `BaseResponse.with_telemetry`, so newly-added endpoints don't silently blank the Debug Panel. + +### Security + +Capability-focused hardening across the backend and frontend trust boundaries. + +- **Cross-tenant ContextVar leak in the s3fs proxy hook** closed. PyIceberg writes parquet via a `ThreadPoolExecutor`; ContextVars don't propagate to executor workers by default, so the prior fix used an endpoint-keyed global registry that was vulnerable to overwrite when two tenants shared an endpoint URL. Replaced with a global `ThreadPoolExecutor.submit` monkeypatch that wraps the callable in `contextvars.copy_context()` — matches asyncio's `loop.run_in_executor` semantics. Documented in [MONKEYPATCHES.md](MONKEYPATCHES.md) §6. +- **Path-param service-scope desync** — analyst sessions could supply a `service_id` path param that didn't match their session scope on a handful of mutation endpoints. Centralised the check via a router-utils helper invoked on every scoped route. +- **Secret-in-URL leak on downloads** — the download endpoint previously embedded the shared CDN secret in the redirect URL where it could land in browser history / referrer headers. Switched to a signed short-lived bearer that's stripped before the redirect. +- **Strict input validation** on the destructive-op surface — provision teardown, NGWAF workspace mutations, scoring threshold + enforce-status-code + recv-exclusion-regex changes — runs through length caps, character allowlists, and (where applicable) `falco` static analysis before any VCL ships. +- **CSRF gates** — moved GET→POST on `logging-settings/update` and sibling state-changing endpoints that were addressable via GET. +- **Authorisation tightening** — share-admin endpoints reject the Caddy-marker header from non-Caddy paths; `claim_token` path consolidated under a single atomic UPDATE so concurrent claims can't both succeed. +- **Cross-tenant cache audit** — re-verified that every per-tenant cache key includes `service_id`; closed two missing entries on insights and origin paths. +- **Thread leak fix** — the share-login flow was leaking a daemon thread per failed login on multi-worker setups; the new on-demand SQLite rehydration replaces the thread entirely. +- **Terms-of-service bypass** — share-login `/acknowledge` now fetches the active TOS version and refuses acknowledgement of a stale one; frontend was sending a hardcoded version. +- **Telemetry-proxy diagnostics** for silent 400s (`Missing X-Fos-Target`) and unclassified `list_objects_v2` calls; preserve `Content-Type` so downstream compression always fires; preserve multi-valued response headers. + +### Tests + +- 3500+ backend tests (+450). +- 290+ frontend vitest tests (+25). +- New coverage: `tests/core/test_duckdb_pool.py`, `test_local_compaction.py`, `test_rollups_compaction.py`, `test_rollups_hour_bundling.py`, `test_iceberg_helpers.py`, `tests/services/test_service_manager.py`, `tests/utils/test_sql_validator.py`, `test_telemetry_response_middleware.py`, `test_router_utils.py`, `test_state_sync.py`, `test_terraform_gen.py`, plus router coverage for the new composite endpoints and the destructive-op-auth surface. +- `make ci` green: lint + format + mypy + pytest + vcl-test + verify-deps + typecheck-frontend + test-frontend + osv + secret-scan. + +### Infrastructure + +- **Synthetic load generator** ([scripts/loadtest_generator.py](scripts/loadtest_generator.py)) and **read-path probe** ([scripts/dev/loadtest_probe.sh](scripts/dev/loadtest_probe.sh)) for reproducible perf measurement against local Parquet+Iceberg. +- **Two-pass next build** in the frontend Dockerfile so SSG sees the correct plotly chunk hashes; preload-manifest scanner runs after `next build` to capture them. + +### Documentation + +- `AGENTS.md` — added Key Systems entries for the DuckDB connection pool, the hourly Top-N rollup pipeline, and the response telemetry middleware. Updated the local-compaction section to reflect the bin-packing tiers. +- `MONKEYPATCHES.md` — documents the new `ThreadPoolExecutor.submit` patch. + +[1.2.0]: https://github.com/fastly/fastly-log-analytics/releases/tag/v1.2.0 + ## [1.1.0] - 2026-06-03 Edge session scoring. Every request is classified in real-time at the edge by a Fastly Compute service that runs an L1 (cookie compliance + timing rules) + L2 (PageRank-trained transition matrix) scorer, returning a combined 0-100 score that lands in DuckDB for analyst review. Operators can label sessions, watch live ROC-AUC, retrain the matrix, roll back to a prior matrix, rotate the AES cookie key, and push a hard enforcement threshold that rejects flagged requests at the edge with an operator-chosen HTTP status code (default 429). diff --git a/MONKEYPATCHES.md b/MONKEYPATCHES.md index 9f10feda..d4aea8c9 100644 --- a/MONKEYPATCHES.md +++ b/MONKEYPATCHES.md @@ -4,10 +4,12 @@ This file catalogs every third-party class/function we monkeypatch at import time so we can audit, justify, and eventually replace them with cleaner abstractions (subclasses, fsspec hooks, custom catalogs, etc.). -All patches today live in [backend/core/iceberg.py](backend/core/iceberg.py) -and form a single category: **s3fs cache + telemetry-proxy** — five patches, -all behind a single `try: ... except ImportError` block -([iceberg.py:187-443](backend/core/iceberg.py#L187-L443)). +All patches today live in [backend/core/iceberg.py](backend/core/iceberg.py). +Five patches form a single **s3fs cache + telemetry-proxy** category, all +behind a single `try: ... except ImportError` block +([iceberg.py:187-443](backend/core/iceberg.py#L187-L443)). One additional +**stdlib** patch (`ThreadPoolExecutor.submit`) propagates ContextVars to +worker threads so cross-tenant proxy routing stays correct. (A sixth `SqlCatalog.load_table` patch lived here until 2026-05-21; it has been replaced by a clean `FosSqlCatalog` subclass — see the "Replaced patches" @@ -139,6 +141,42 @@ obsolete. --- +## 6. `concurrent.futures.ThreadPoolExecutor.submit` + +- **Site:** [iceberg.py:60-71](backend/core/iceberg.py#L60) (top-level, runs + at module import — does NOT live behind the s3fs `try: ... except + ImportError` block). +- **What:** Wraps `submit(fn, *args, **kwargs)` so the worker thread runs + `fn` inside `contextvars.copy_context()` instead of an empty context. + All other behavior (Future return, error propagation, cancellation) is + unchanged. +- **Why (security incident, audit finding 003, 2026-06-06):** PyIceberg + writes parquet data files via a `ThreadPoolExecutor` inside + `pyiceberg/io/pyarrow.py`. The s3fs `__init__` patch (#1) reads + `_PENDING_FS_SOURCE` (a ContextVar set by `_get_catalog`) to discover + which tenant's source/CDN/proxy config to use. ContextVars do NOT + propagate to executor workers natively — PEP 567 covers asyncio tasks + only. The previous fix was an endpoint-keyed global registry + (`_PROXY_SOURCE_REGISTRY`) that worker threads queried as a fallback. + That registry was vulnerable to cross-tenant overwrite: if two tenants + shared an endpoint URL, the second `_get_catalog` overwrote the first + tenant's source, and the first tenant's still-running worker threads + resolved the wrong source — wrong CDN target, wrong `x-fastly-key`, + wrong `X-Telemetry-Service-Id`. This patch eliminates the registry by + making ContextVars propagate the way they propagate for asyncio. +- **Scope of effect:** GLOBAL — affects every `ThreadPoolExecutor` in the + process, not just pyiceberg's. The semantic change is benign for all + known callers (FastAPI, aiobotocore, etc.) because submitting work with + the caller's ContextVars is the more-defensive default and matches + asyncio's `loop.run_in_executor` semantics. Workers that previously saw + empty ContextVars now see the submitter's context. +- **Cleanup:** Remove if/when CPython adds first-class context propagation + to `concurrent.futures` (proposals exist) or if PyIceberg switches to + asyncio for parquet writes. Until then, the global patch is the + smallest correct fix. + +--- + ## Replaced patches ### `SqlCatalog.load_table` → `FosSqlCatalog` subclass (Stream H, 2026-05-21) diff --git a/backend/core/data_migrations.py b/backend/core/data_migrations.py index db9a1438..81580e58 100644 --- a/backend/core/data_migrations.py +++ b/backend/core/data_migrations.py @@ -75,6 +75,25 @@ def _rollups_initial_backfill(service_id: str, source: dict) -> str | None: return "rollups: ensure_field_backfills complete" +def _rollups_hour_bundling_backfill(service_id: str, source: dict) -> str | None: + """Bundle all closed-hour per-field rollup parquets into a single + per-hour parquet under ``rollups/hour_bundled/hour=H/all_fields.parquet``. + + The dashboard reader prefers bundled files (one open per hour) over + per-field files (~40 opens per hour), cutting cold-path parquet + metadata reads by ~40x on a 24h query. The per-field tree stays in + place — the reader falls back to it when a bundle is missing, so the + migration is non-destructive. + + Idempotent: bundle_hours skips hours whose bundle is already up to + date (mtime check), so re-running is cheap. + """ + from backend.core import rollups + + n = rollups.backfill_hour_bundles(service_id, source) + return f"rollups: bundled {n} hour(s) into hour_bundled/" + + # Ordered registry. Append-only — never remove or reorder entries. # Names must be globally unique and stable; the DB matches by name. MIGRATIONS: list[Migration] = [ @@ -83,6 +102,11 @@ def _rollups_initial_backfill(service_id: str, source: dict) -> str | None: description="Build initial hourly top-N rollups for dashboard top-N queries", fn=_rollups_initial_backfill, ), + Migration( + name="2026-06-08_rollups_hour_bundling", + description="Bundle per-field hour rollups into one parquet per hour (40x fewer file opens)", + fn=_rollups_hour_bundling_backfill, + ), ] diff --git a/backend/core/duckdb.py b/backend/core/duckdb.py index cdeefff3..83a2c35c 100644 --- a/backend/core/duckdb.py +++ b/backend/core/duckdb.py @@ -695,9 +695,14 @@ def get_connection( ) -> duckdb.DuckDBPyConnection: """Create a configured DuckDB connection. - When read_only=True, multiple processes can share the database file. - When read_only=False (default), only one process can have a connection. + ``read_only`` is accepted for API compatibility but always overridden + to False. Within a single process DuckDB shares the database instance + across connections, so mixing ``read_only=True`` (pool / API) with + ``read_only=False`` (cron writes) raises "different configuration". + Using False everywhere avoids the conflict; concurrent reads are still + safe because DuckDB serialises via its internal WAL. """ + read_only = False src = source or _DEFAULT_SOURCE # Use per-source duckdb_path if present, fall back to global DUCKDB_PATH diff --git a/backend/core/duckdb_pool.py b/backend/core/duckdb_pool.py index f8a90797..321d11b7 100644 --- a/backend/core/duckdb_pool.py +++ b/backend/core/duckdb_pool.py @@ -36,9 +36,8 @@ Concurrency: * Multiple connections to the same DuckDB file on the same process are safe — they share the in-memory database state. - * Read-only + read-only across pool connections is fine. - * Read-only pool + one read-write writer (ingest) is the project's existing - contract; ``get_connection`` already handles ``DBBusyError`` retries. + * All connections open with ``read_only=False`` (``get_connection`` forces + this) so cron write connections never conflict with pool connections. Failure handling: * If view rebind fails on checkout, we discard the connection and try a @@ -72,12 +71,50 @@ def _pool_max_size() -> int: return 8 +def _pool_conn_memory_limit() -> str | None: + """Optional per-pool-connection memory cap. + + Without this, every pool connection inherits the process-wide DuckDB + memory_limit derived from physical RAM (~60%), so 8 concurrent queries + against a large dataset can each balloon to multi-GB. Set + ``DUCKDB_POOL_CONN_MEMORY_LIMIT`` (e.g. ``256MB`` or ``1GB``) to enforce + a per-connection ceiling — DuckDB spills intermediate state to its + temp directory when over the limit instead of growing RSS unbounded. + + Returns the env-var value (passed through verbatim — DuckDB accepts + ``256MB`` / ``2GB`` / ``104857600`` etc.) or ``None`` to keep the default. + """ + return os.getenv("DUCKDB_POOL_CONN_MEMORY_LIMIT") or None + + +def _pool_conn_threads() -> int | None: + """Optional per-pool-connection DuckDB thread count. + + Each pool connection defaults to ``min(cpu_count, 8)`` DuckDB threads. + With ``DUCKDB_POOL_MAX_SIZE=8`` concurrent queries that means + ``8 connections × 8 threads = 64 threads`` competing for ~8 physical + cores — context-switching dominates and per-query latency degrades + well past linear queueing. Set ``DUCKDB_POOL_CONN_THREADS`` to a smaller + value (commonly ``cpu_count // pool_max_size``) to trade single-query + throughput for better tail-latency under sustained load. + + Returns the int value (>=1) or ``None`` to keep the default. + """ + raw = os.getenv("DUCKDB_POOL_CONN_THREADS") + if not raw: + return None + try: + return max(1, int(raw)) + except (TypeError, ValueError): + return None + + # Per-connection state tracking. DuckDB connection objects are slotted # C types — they don't accept arbitrary attribute assignment — so we # keep our metadata in a module-level dict keyed by id(con). Entries are # cleared when the connection is closed/discarded. # -# Fingerprint = id() of the ``_view_cache`` tuple at the time the view +# Fingerprint = the ``_view_cache`` tuple at the time the view # was last bound to this connection. The tuple is replaced (not mutated) # when the cache rotates, so identity is a sufficient fresh-check. _conn_state: dict[int, dict] = {} @@ -128,7 +165,7 @@ def __init__(self, service_key: str, max_size: int): # LIFO so the most-recently-used connection (warmest in any OS / DuckDB # internal caches) is the next checkout. self._idle: queue.LifoQueue = queue.LifoQueue(maxsize=max_size) - self._lock = threading.Lock() + self._lock = threading.RLock() # ``in_use`` is the count of connections currently checked out plus # connections idle in the queue. Bounded by ``max_size``. self._in_use = 0 @@ -140,13 +177,14 @@ def __init__(self, service_key: str, max_size: int): def acquire(self, src: dict, max_wait: float) -> duckdb.DuckDBPyConnection: deadline = time.monotonic() + max_wait + reused_con: duckdb.DuckDBPyConnection | None = None with self._cond: while True: # Fast path: idle connection available try: - con = self._idle.get_nowait() + reused_con = self._idle.get_nowait() self._reused_total += 1 - return self._prepare_checkout(con, src) + break # fall through to UNLOCKED _prepare_checkout except queue.Empty: pass @@ -159,18 +197,53 @@ def acquire(self, src: dict, max_wait: float) -> duckdb.DuckDBPyConnection: # Saturated: wait for a return remaining = deadline - time.monotonic() if remaining <= 0: - raise _PoolBusy( - f"pool for {self.service_key} saturated at {self.max_size}" - ) + raise _PoolBusy(f"pool for {self.service_key} saturated at {self.max_size}") self._cond.wait(timeout=remaining) - # Outside lock: build fresh. _in_use was already incremented; if the - # build raises we MUST decrement and notify a waiter, hence the try. + # Outside lock. Both branches can call ``update_iceberg_view`` which + # may take seconds when an Iceberg snapshot reload or S3 manifest read + # is required; holding the pool's Condition lock across that call + # deadlocks every concurrent waiter, the ``max_wait`` cap can't fire + # because waiters block on the threading lock (not ``_cond.wait``), + # and the FastAPI thread pool then fills with stuck checkouts until + # the backend stops accepting new connections. + if reused_con is not None: + # _prepare_checkout calls _discard on failure (decrements in_use, + # notifies waiter) before re-raising — no extra cleanup needed. + return self._prepare_checkout(reused_con, src) + + # Build fresh. _in_use was already incremented; if the build raises + # we MUST decrement and notify a waiter, hence the try. try: from backend.core.duckdb import get_connection con = get_connection(source=src, read_only=True, max_wait=max_wait) _set_conn_state(con, service_key=self.service_key) + # Apply per-connection overrides once at build time — DuckDB + # persists session settings for the connection's lifetime, so + # subsequent checkouts of this same connection inherit them. + mem_limit = _pool_conn_memory_limit() + if mem_limit: + try: + con.execute(f"SET memory_limit = '{mem_limit}'") + except Exception as e: + logger.warning( + "[pool] %s: failed to apply DUCKDB_POOL_CONN_MEMORY_LIMIT=%r: %s", + self.service_key, + mem_limit, + e, + ) + conn_threads = _pool_conn_threads() + if conn_threads is not None: + try: + con.execute(f"SET threads = {conn_threads}") + except Exception as e: + logger.warning( + "[pool] %s: failed to apply DUCKDB_POOL_CONN_THREADS=%d: %s", + self.service_key, + conn_threads, + e, + ) self._stamp_fingerprint(con, src) return con except Exception: @@ -226,7 +299,7 @@ def _prepare_checkout(self, con: duckdb.DuckDBPyConnection, src: dict) -> duckdb Two checks make up the fingerprint: - 1. id() of the iceberg ``_view_cache`` tuple for this service. + 1. The iceberg ``_view_cache`` tuple for this service. The tuple is replaced (not mutated) when the cache rotates, so identity is a sufficient check that the SQL we'd bind matches what we bound last time. @@ -247,11 +320,7 @@ def _prepare_checkout(self, con: duckdb.DuckDBPyConnection, src: dict) -> duckdb stamped_view = _get_conn_state(con, "view_fingerprint") stamped_buf = _get_conn_state(con, "buffer_mtime") current_buf = _safe_buffer_mtime(src) - if ( - current is not None - and id(current) == stamped_view - and current_buf == stamped_buf - ): + if current is not None and current is stamped_view and current_buf == stamped_buf: # View AND underlying buffer set match what we bound last # time — nothing to do. return con @@ -271,7 +340,7 @@ def _stamp_fingerprint(self, con: duckdb.DuckDBPyConnection, src: dict | None = buf_mtime = _safe_buffer_mtime(src) if src is not None else None _set_conn_state( con, - view_fingerprint=id(current) if current is not None else None, + view_fingerprint=current, buffer_mtime=buf_mtime, ) except Exception: @@ -283,8 +352,7 @@ def _cleanup_temp_tables(self, con: duckdb.DuckDBPyConnection) -> None: itself; this is belt-and-suspenders for the failure paths.""" try: rows = con.execute( - "SELECT table_name FROM duckdb_tables() " - "WHERE schema_name = 'main' AND temporary = true" + "SELECT table_name FROM duckdb_tables() WHERE schema_name = 'main' AND temporary = true" ).fetchall() except Exception: return diff --git a/backend/core/fastly/utils.py b/backend/core/fastly/utils.py index e80eb238..80988db4 100644 --- a/backend/core/fastly/utils.py +++ b/backend/core/fastly/utils.py @@ -149,16 +149,6 @@ def load_vcl(rate_limiting: bool = True) -> str: set req.http.Fastly-Client-IP = client.ip; } - # Handle FASTLYPURGE natively. Without this, an unsigned purge on a - # cache miss is forwarded to the FOS origin, which returns 403 — and - # Fastly caches that 403 for the object's TTL. An attacker can poison - # the cache for legitimate clients by issuing purges against arbitrary - # keys. ``return(purge)`` short-circuits the pipeline before any - # backend fetch happens. - if (req.method == "FASTLYPURGE") { - return(purge); - } - # Block requests that do not provide the correct secret key. # NOTE on the auth fallback: the third argument to ``table.lookup`` is # returned when ``cdn_auth.secret`` is absent from the edge dictionary. @@ -185,6 +175,16 @@ def load_vcl(rate_limiting: bool = True) -> str: } #RATELIMIT_END + # Handle FASTLYPURGE natively. Without this, an unsigned purge on a + # cache miss is forwarded to the FOS origin, which returns 403 — and + # Fastly caches that 403 for the object's TTL. An attacker can poison + # the cache for legitimate clients by issuing purges against arbitrary + # keys. ``return(purge)`` short-circuits the pipeline before any + # backend fetch happens. + if (req.method == "FASTLYPURGE") { + return(purge); + } + # Enable segmented caching for potentially large log or parquet files set req.enable_segmented_caching = true; set segmented_caching.block_size = 20971520; # 20 MB, the maximum diff --git a/backend/core/iceberg.py b/backend/core/iceberg.py index 33c0bc2e..f08ab47b 100644 --- a/backend/core/iceberg.py +++ b/backend/core/iceberg.py @@ -57,39 +57,42 @@ _PENDING_FS_SOURCE: _contextvars.ContextVar[dict | None] = _contextvars.ContextVar("_PENDING_FS_SOURCE", default=None) -# Thread-safe fallback registry. PyIceberg writes parquet data files via -# concurrent.futures.ThreadPoolExecutor in pyiceberg/io/pyarrow.py, and -# ContextVars do NOT propagate to executor workers (PEP 567 covers asyncio -# only). Each worker thread's first FsspecFileIO call constructs a fresh -# S3FileSystem; without this registry the worker's _PENDING_FS_SOURCE.get() -# returns the default (None), the before-send hook is never registered, and -# the proxy 400s with "Missing X-Fos-Target header". -_PROXY_SOURCE_REGISTRY: dict[str, dict] = {} -_PROXY_REGISTRY_LOCK = _threading.Lock() - - -def _normalize_endpoint(endpoint_url: str | None) -> str: - if not endpoint_url: - return "" - return endpoint_url.replace("https://", "").replace("http://", "").rstrip("/").lower() - - -def _register_proxy_source(source: dict) -> None: - """Register source by endpoint so worker threads can resolve it even - when the ContextVar is empty.""" - endpoint = source.get("fos_native_endpoint") or source.get("endpoint", "") - normalized = _normalize_endpoint(endpoint) - if normalized: - with _PROXY_REGISTRY_LOCK: - _PROXY_SOURCE_REGISTRY[normalized] = source - - -def _lookup_proxy_source(endpoint_url: str | None) -> dict: - normalized = _normalize_endpoint(endpoint_url) - if not normalized: - return {} - with _PROXY_REGISTRY_LOCK: - return _PROXY_SOURCE_REGISTRY.get(normalized, {}) +# Process-wide fallback for the ContextVar. PyIceberg / aiobotocore create +# new s3fs instances on threads that the ``_patched_submit`` shim above +# can't cover (fsspec's own iothread, asyncio's default executor, lazy +# per-FS-call instantiations). Those threads see ``_PENDING_FS_SOURCE.get() +# == None``, the proxy hook never registers, and every subsequent S3 call +# reaches the proxy without ``X-Fos-Target`` so the proxy 400s silently. +# The 2026-06-09 audit confirmed 68 silent 400s in 6 minutes with +# ``caller-hint=None ua='aiobotocore/...'`` and an empty service-id header +# — strong signal that the hook was missing. +# +# ``_get_catalog`` stamps the latest source it sees into this dict (keyed +# by service name) AND keeps the most-recent value under +# ``_LAST_FS_SOURCE`` as a last-resort fallback. The patched s3fs init +# below now reads ``_PENDING_FS_SOURCE.get() or _LAST_FS_SOURCE`` so the +# hook registers even on hostile threads. Multi-service deployments would +# need the proxy to derive the source from the URL bucket name; today +# this app is single-service in production so the last-source fallback is +# always correct. +_LAST_FS_SOURCE: dict | None = None + +# PyIceberg writes parquet data files via concurrent.futures.ThreadPoolExecutor +# in pyiceberg/io/pyarrow.py. ContextVars do NOT propagate to executor workers +# natively in Python 3, so we patch submit() to copy the context. Without this, +# the worker's _PENDING_FS_SOURCE.get() returns None, the proxy hook is never +# registered, and the proxy 400s with "Missing X-Fos-Target header". +import concurrent.futures as _futures + +_orig_submit = _futures.ThreadPoolExecutor.submit + + +def _patched_submit(self, fn, /, *args, **kwargs): + ctx = _contextvars.copy_context() + return _orig_submit(self, ctx.run, fn, *args, **kwargs) + + +_futures.ThreadPoolExecutor.submit = _patched_submit def _proxy_targets_from_endpoint(endpoint_url: str, source: dict | None) -> tuple[str | None, str]: @@ -202,9 +205,13 @@ def _patched_s3fs_init(self, *args, **kwargs): client_kwargs = kwargs.setdefault("client_kwargs", {}) original_endpoint = client_kwargs.get("endpoint_url") or kwargs.get("endpoint_url") or "" - # ContextVar covers the main thread; PyIceberg's thread-pool - # writers fall through to the endpoint-keyed registry. - source = _PENDING_FS_SOURCE.get() or _lookup_proxy_source(original_endpoint) or {} + # ContextVar covers the main thread, and we patch ThreadPoolExecutor + # to propagate it to PyIceberg's thread-pool writers. Fallback to the + # process-wide ``_LAST_FS_SOURCE`` for threads neither path reaches + # (fsspec iothread, lazy per-FS-call instantiations, asyncio's + # default executor) — see comment on _LAST_FS_SOURCE for full + # context. + source = _PENDING_FS_SOURCE.get() or _LAST_FS_SOURCE or {} cdn_target, fos_native_target = _proxy_targets_from_endpoint(original_endpoint, source) self._fos_proxy_cdn_target = cdn_target # _fos_proxy_target retained as the FOS native endpoint — existing @@ -510,6 +517,7 @@ def _patched_open(self, path, mode="rb", **kwargs): logger = logging.getLogger(__name__) +from pyiceberg.exceptions import CommitFailedException from pyiceberg.io.pyarrow import schema_to_pyarrow from pyiceberg.schema import Schema from pyiceberg.table.name_mapping import create_mapping_from_schema @@ -717,7 +725,29 @@ def _table_identifier(source: dict) -> tuple[str, str]: return ("default", "logs") +def _is_local_only_source(source: dict) -> bool: + """True when this source is configured to use local files instead of FOS/S3. + + Triggered by ``fos_local_warehouse: true`` in the source config, OR by + the conventional ``fos_endpoint: "http://localhost:0"`` scrub marker + (see CLAUDE.md ``dev-sandbox-scrub`` memory). Used by load-test and + other dev-only services to commit Iceberg snapshots to local disk + without touching real object storage. + """ + if source.get("fos_local_warehouse") is True: + return True + endpoint = source.get("fos_endpoint") or source.get("endpoint") or "" + return endpoint in ("http://localhost:0", "http://127.0.0.1:0") + + def _warehouse_uri(source: dict) -> str: + if _is_local_only_source(source): + # Local-only: Iceberg writes commits, manifests, and data files into + # cache/{bucket}/iceberg/ on disk. Catalog stays SQLite (already local). + from backend.core.duckdb import _cache_dir + + cache = _cache_dir(source) + return f"file://{os.path.abspath(os.path.join(cache, 'iceberg'))}" prefix = source.get("prefix", "").strip("/") base = f"{prefix}/iceberg" if prefix else "iceberg" return f"s3://{source['bucket']}/{base}" @@ -742,6 +772,15 @@ def _catalog_db_path(source: dict) -> str: def _get_catalog(source: dict): """Return a configured PyIceberg SqlCatalog backed by a local SQLite file.""" source_key = source.get("name", "default") + # Stamp the process-global fallback so s3fs instances created on + # threads without the ContextVar (fsspec iothread, lazy per-FS + # creations) still get a non-empty source in ``_patched_s3fs_init``. + # See the comment on ``_LAST_FS_SOURCE`` above for the failure mode + # this defends against. Always update on every call so a future + # multi-service deployment at least always has a recent source — + # though that case would need a proper per-bucket lookup, not this. + global _LAST_FS_SOURCE + _LAST_FS_SOURCE = source with _catalog_lock: if source_key in _catalog_cache: return _catalog_cache[source_key] @@ -755,26 +794,31 @@ def _get_catalog(source: dict): warehouse = _warehouse_uri(source) db_path = _catalog_db_path(source) - # Hand the source dict to the s3fs patched __init__ via TWO parallel - # channels: a ContextVar (covers the main thread / any asyncio task - # that inherits the context), AND an endpoint-keyed registry (covers - # PyIceberg's parquet-write thread-pool workers, which don't inherit - # ContextVars). The patched __init__ tries the ContextVar first, then - # falls back to the registry. + # Hand the source dict to the s3fs patched __init__ via ContextVar. + # This covers the main thread, and we patched ThreadPoolExecutor + # to propagate ContextVars to PyIceberg's thread-pool workers. _PENDING_FS_SOURCE.set(source) - _register_proxy_source(source) - - props = { - "uri": f"sqlite:///{db_path}", - "warehouse": warehouse, - "s3.endpoint": f"https://{endpoint}", - "s3.access-key-id": access_key, - "s3.secret-access-key": secret_key, - "s3.path-style-access": "true", - "s3.region": source.get("region", "us-east-1"), - "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO", - "s3.client.config": '{"retries": {"max_attempts": 5, "mode": "adaptive"}, "read_timeout": 30, "connect_timeout": 10}', - } + + if _is_local_only_source(source): + # Local-only warehouse: skip S3 client config entirely. PyIceberg's + # default PyArrowFileIO handles file:// URIs natively without any + # network round-trip. + props = { + "uri": f"sqlite:///{db_path}", + "warehouse": warehouse, + } + else: + props = { + "uri": f"sqlite:///{db_path}", + "warehouse": warehouse, + "s3.endpoint": f"https://{endpoint}", + "s3.access-key-id": access_key, + "s3.secret-access-key": secret_key, + "s3.path-style-access": "true", + "s3.region": source.get("region", "us-east-1"), + "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO", + "s3.client.config": '{"retries": {"max_attempts": 5, "mode": "adaptive"}, "read_timeout": 30, "connect_timeout": 10}', + } catalog_cls = _get_fos_catalog_class() catalog = catalog_cls("fos", **props) @@ -953,13 +997,13 @@ def _run(): # even without explicit invalidation, staleness is capped — and writers in # the same process invalidate explicitly below. _POINTER_CACHE_TTL_SEC = 2.0 -_pointer_cache: dict[tuple[str, str, str], tuple[float, str | None]] = {} +_pointer_cache: dict[tuple[str, str, str, str], tuple[float, str | None]] = {} _pointer_cache_lock = threading.Lock() -def _pointer_cache_key(source: dict, identifier: tuple) -> tuple[str, str, str]: +def _pointer_cache_key(source: dict, identifier: tuple) -> tuple[str, str, str, str]: namespace, table_name = identifier - return (source.get("bucket", ""), namespace, table_name) + return (source.get("bucket", ""), source.get("prefix", ""), namespace, table_name) def _pointer_cache_invalidate(source: dict, identifier: tuple) -> None: @@ -974,7 +1018,7 @@ def _pointer_cache_invalidate(source: dict, identifier: tuple) -> None: # (itself CDN-cached + TTL-cached above). A pointer mismatch is exhaustive # proof of staleness because every snapshot commit produces a new # metadata.json and a new pointer value. -_table_object_cache: dict[tuple[str, str, str], object] = {} +_table_object_cache: dict[tuple[str, str, str, str], object] = {} _table_object_cache_lock = threading.Lock() @@ -1028,6 +1072,10 @@ def _write_metadata_pointer(source: dict, location: str, table=None) -> None: Pass `table` so the async table-summary writer can reuse the just-committed in-memory metadata instead of re-downloading it. """ + if _is_local_only_source(source): + # Local-only warehouse: SQLite catalog already tracks metadata_location; + # no separate FOS pointer to maintain. No-op. + return try: from backend.core.duckdb import _get_fos_client @@ -1081,6 +1129,10 @@ def _write_metadata_pointer(source: dict, location: str, table=None) -> None: def _read_metadata_pointer(source: dict, identifier: tuple) -> str | None: """Read the latest metadata pointer from FOS via CDN if configured, else direct S3.""" + if _is_local_only_source(source): + # Local-only warehouse: no FOS pointer to read. SqlCatalog already + # knows the metadata_location from its SQLite-backed iceberg_tables row. + return None namespace, table_name = identifier # In-process TTL cache. The 4-call-in-1-second pattern from cron_compact @@ -1416,12 +1468,179 @@ def table_location(source: dict) -> str | None: # --------------------------------------------------------------------------- +_TOMBSTONE_SUFFIX = ".consumed-" # Followed by an integer Unix-epoch seconds value. +_TOMBSTONE_GRACE_SECONDS = 60 # See tombstone_buffer_files docstring for the rationale. + + +def _tombstone_marker_path(parquet_path: str, ts: int) -> str: + return f"{parquet_path}{_TOMBSTONE_SUFFIX}{ts}" + + +def _is_tombstone_marker(name: str) -> bool: + """True iff ``name`` is a tombstone sidecar (``.parquet.consumed-``). + + Centralised so the glob filter, sweeper, and tests all share one + definition. We only check the ``.parquet.consumed-`` substring to + avoid being fooled by partial matches on bucket-name-like substrings. + """ + if _TOMBSTONE_SUFFIX not in name: + return False + head, _, tail = name.rpartition(_TOMBSTONE_SUFFIX) + return head.endswith(".parquet") and tail.isdigit() + + +def _tombstoned_parquet_paths(buf_dir: str) -> set[str]: + """Return the set of buffer parquet paths that have an active tombstone + sibling. Used by ``buffer_files()`` to keep tombstoned files out of + new view binds — they stay on disk for the grace window so any view + bound BEFORE the tombstone can still read them.""" + tombstoned: set[str] = set() + if not os.path.isdir(buf_dir): + return tombstoned + for p in _glob.glob(os.path.join(buf_dir, "**", "*" + _TOMBSTONE_SUFFIX + "*"), recursive=True): + base = os.path.basename(p) + if not _is_tombstone_marker(base): + continue + # Strip ``.consumed-`` to recover the original ``.parquet`` path. + parquet_path = p.rsplit(_TOMBSTONE_SUFFIX, 1)[0] + tombstoned.add(parquet_path) + return tombstoned + + +def tombstone_buffer_files(source: dict, paths: list[str], *, ts: int | None = None) -> list[str]: + """Mark buffer parquet files as logically consumed without unlinking them. + + Replaces the post-commit ``os.remove(path)`` race with a two-phase + scheme: + + 1. **Tombstone** (this function): write an empty sidecar file + ``.consumed-`` next to the original ``.parquet``. + The original file stays on disk untouched. ``buffer_files()`` now + filters it out via ``_tombstoned_parquet_paths``, so subsequent + view rebuilds will not bind it. Crucially, any DuckDB view ALREADY + bound to that path continues to work because the file is still + readable. + 2. **Sweep** (``sweep_tombstoned_buffer_files``): after a grace + window (default 60 s) elapses, the next commit run unlinks both + the parquet and its tombstone sidecar. By then no view should + reference the file — typical bind-to-execute windows are + milliseconds, and 60 s comfortably exceeds the slowest cold query. + + **Why this fixes the 2026-06-05 incident:** the previous code did + ``os.remove(path)`` inline at commit time. A dashboard query whose + view was bound BEFORE the commit would then hit "No files found" + when DuckDB resolved the bound paths against disk. The + ``QueryRunner.execute`` self-heal exists for this case but had its + own race (cached-SQL re-bind under lock contention; see + ``backend/repositories/_base.py:288``). Tombstoning closes the race + at its source so the self-heal essentially never has to fire. + + Tombstone creation uses ``open(..., "x")`` to fail loudly on + collisions instead of silently overwriting timing metadata. Errors + during tombstoning are swallowed (logged) — losing a tombstone just + means the file MIGHT be retained until a manual cleanup, never that + the wrong file gets unlinked. + + Returns the subset of ``paths`` that were successfully tombstoned. + Callers that need atomicity should compare lengths. + """ + if ts is None: + ts = int(time.time()) + tombstoned: list[str] = [] + for path in paths: + try: + marker = _tombstone_marker_path(path, ts) + with open(marker, "x"): + pass + tombstoned.append(path) + except FileExistsError: + # A previous commit at the exact same second already + # tombstoned this file — already-consumed is fine, skip. + tombstoned.append(path) + except Exception as e: + logger.warning( + "%s Failed to tombstone buffer file %s — falling back to immediate unlink. Error: %s", + _ICE, + path, + e, + ) + # If tombstoning fails (disk full, permission flap), preserve + # the prior behaviour rather than letting the buffer file + # accumulate forever. The race we're fixing is preferable + # to an unbounded buffer dir. + try: + os.remove(path) + tombstoned.append(path) + except Exception: + pass + return tombstoned + + +def sweep_tombstoned_buffer_files( + source: dict, *, grace_seconds: int = _TOMBSTONE_GRACE_SECONDS, now: int | None = None +) -> int: + """Unlink tombstoned buffer parquets whose grace window has elapsed. + + Called at the start of ``commit_buffer`` so the sweep cadence is + naturally tied to the commit cron (no new cron registration). When + a tombstone marker is at least ``grace_seconds`` old, both the + parquet and the marker are unlinked. Younger tombstones are left + alone — the corresponding parquet may still be referenced by an + in-flight query bound before the tombstone was written. + + Returns the number of parquet files actually unlinked. + """ + if now is None: + now = int(time.time()) + buf = _buffer_dir(source) + if not os.path.isdir(buf): + return 0 + swept = 0 + for marker in _glob.glob(os.path.join(buf, "**", "*" + _TOMBSTONE_SUFFIX + "*"), recursive=True): + base = os.path.basename(marker) + if not _is_tombstone_marker(base): + continue + try: + ts = int(marker.rsplit(_TOMBSTONE_SUFFIX, 1)[1]) + except (ValueError, IndexError): + continue + if now - ts < grace_seconds: + continue + parquet_path = marker.rsplit(_TOMBSTONE_SUFFIX, 1)[0] + # Unlink the parquet first so a partial failure doesn't leave + # the file visible without its tombstone (which would re-bind + # it into the next view rebuild). + try: + if os.path.exists(parquet_path): + os.remove(parquet_path) + except Exception as e: + logger.warning("%s Sweep failed to unlink %s: %s", _ICE, parquet_path, e) + continue + try: + os.remove(marker) + except Exception as e: + logger.warning("%s Sweep failed to unlink tombstone %s: %s", _ICE, marker, e) + swept += 1 + return swept + + def buffer_files(source: dict) -> list[str]: - """Return sorted list of Parquet files currently in the local buffer.""" + """Return sorted list of Parquet files currently in the local buffer. + + Excludes files that have been tombstoned by ``tombstone_buffer_files`` + so view rebuilds don't bind paths that are about to be swept. The + tombstoned files remain on disk for the grace window so any view + bound BEFORE the tombstone can still read them. + """ buf = _buffer_dir(source) if not os.path.isdir(buf): return [] - return sorted(p for p in _glob.glob(os.path.join(buf, "**", "*.parquet"), recursive=True) if os.path.isfile(p)) + tombstoned = _tombstoned_parquet_paths(buf) + return sorted( + p + for p in _glob.glob(os.path.join(buf, "**", "*.parquet"), recursive=True) + if os.path.isfile(p) and p not in tombstoned and not _is_tombstone_marker(os.path.basename(p)) + ) _QUARANTINE_SUBDIR = ".quarantine" @@ -1540,6 +1759,11 @@ def write_to_buffer(source: dict, arrow_table: pa.Table, filename: str) -> str: os.makedirs(buf, exist_ok=True) path = os.path.join(buf, filename) aligned = _align_to_schema(arrow_table, source=source) + if "timestamp" in aligned.column_names: + sort_keys = [("timestamp", "ascending")] + if "ip" in aligned.column_names: + sort_keys.append(("ip", "ascending")) + aligned = aligned.sort_by(sort_keys) pq.write_table(aligned, path, compression="zstd", compression_level=1) return path @@ -1570,6 +1794,19 @@ def commit_buffer(source: dict, progress_callback=None) -> dict: ``snapshot_id`` is the LAST snapshot id produced by the loop (the one the metadata pointer now references). """ + # Sweep any tombstoned buffers whose grace window has elapsed before + # we scan for fresh work. Co-locating the sweep with the commit cron + # avoids a separate scheduler registration; the cadence (every commit + # tick) easily covers the 60 s grace window. + try: + swept = sweep_tombstoned_buffer_files(source) + if swept: + logger.info("%s Swept %d tombstoned buffer file(s) past grace window", _ICE, swept) + except Exception as sweep_err: + # Sweep failures must NEVER block a commit — the file just stays + # on disk until the next sweep tick. + logger.warning("%s Tombstone sweep raised (continuing with commit): %s", _ICE, sweep_err) + files = buffer_files(source) if not files: return {"files_committed": 0, "rows_committed": 0, "snapshot_id": None, "quarantined_files": 0} @@ -1640,13 +1877,15 @@ def commit_buffer(source: dict, progress_callback=None) -> dict: del tables, combined snapshot_id = table.current_snapshot().snapshot_id if table.current_snapshot() else snapshot_id total_rows += chunk_rows - # Per-chunk delete: if we crash on a later chunk, the next commit - # cron only re-processes the un-committed remainder. - for path in chunk_successful: - try: - os.remove(path) - except Exception: - pass + # Per-chunk tombstone: if we crash on a later chunk, the next + # commit cron only re-processes the un-committed remainder + # (tombstoned files are excluded from buffer_files()). The + # actual ``os.remove`` is deferred to ``sweep_tombstoned_buffer_files`` + # after a grace window so concurrent dashboard queries whose + # view was bound BEFORE this commit don't crash on + # "No files found ... batch_X.parquet". See + # ``tombstone_buffer_files`` docstring for the full rationale. + tombstone_buffer_files(source, chunk_successful) total_committed_paths.extend(chunk_successful) if not total_committed_paths: @@ -1822,16 +2061,69 @@ def optimize_table(source: dict, target_file_size_mb: int = 128, min_files_per_p # this turned every nightly optimize run into a silent no-op # — the ValueError got logged as a warning to stderr and the # cron recorded success with 0 files rewritten. + # ``union_by_name=True``: when a partition contains files + # written before AND after a schema bump (e.g. ``edge_sid`` + # / ``edge_cookie_compliance`` / ``edge_score*`` added + # mid-day on 2026-06-01), the default positional union + # raises ``Schema mismatch ... try setting + # union_by_name=True`` and the partition lands in + # ``partition_errors``. With union-by-name DuckDB merges + # the column sets and fills missing columns with NULL, + # matching how Iceberg already presents the merged schema + # to readers. Verified prod incident 2026-06-06: two + # partitions (494541, 494542) had been stuck at ~14 files + # each since the schema bump because every nightly + # optimize attempt raised here. (#optimize-cron-warning) arrow_table = con.execute( - f"SELECT * FROM read_parquet([{paths_sql}], hive_partitioning=false)" + f"SELECT * FROM read_parquet([{paths_sql}], hive_partitioning=false, union_by_name=true)" ).to_arrow_table() # Perform an atomic overwrite of the specific time range. - # In Iceberg, this will delete the old files and add the new one. - table.overwrite( - df=arrow_table, - overwrite_filter=f"timestamp >= '{start_ts.isoformat()}' AND timestamp < '{end_ts.isoformat()}'", - ) + # In Iceberg, this will delete the old files and add the + # new one. Wrapped in a small retry that reloads the + # table on the sequence-number CAS conflict that fires + # when an ingest commit lands between our plan_files + # read and this overwrite — pyiceberg refuses with + # ``ValueError: Cannot add snapshot with sequence + # number N older than last sequence number N``. The + # retry just refetches the table head and tries once + # more; ingest's 5-min cadence makes the contention + # window small enough that a single retry almost always + # wins. + overwrite_filter = f"timestamp >= '{start_ts.isoformat()}' AND timestamp < '{end_ts.isoformat()}'" + _CAS_RETRIES = 3 + for _retry in range(_CAS_RETRIES): + try: + table.overwrite(df=arrow_table, overwrite_filter=overwrite_filter) + break + except ValueError as cas_err: + if "older than last sequence number" not in str(cas_err): + raise + if _retry == _CAS_RETRIES - 1: + raise + # Refresh the table to pick up the new head. + # Bypass _load_table_cached (which short-circuits + # on pointer match) by going straight to the + # catalog — we need the absolute latest snapshot + # to commit on top of, not whatever's cached. + logger.warning( + "[optimize] %s: CAS conflict on hour %d (attempt %d/%d), reloading table and retrying: %s", + source.get("name"), + hour_val, + _retry + 1, + _CAS_RETRIES, + cas_err, + ) + try: + table = catalog.load_table(_table_identifier(source)) + _set_cached_table(source, _table_identifier(source), table) + except Exception as reload_err: + logger.warning( + "[optimize] %s: table reload failed after CAS conflict, giving up on this partition: %s", + source.get("name"), + reload_err, + ) + raise cas_err from reload_err _set_cached_table(source, _table_identifier(source), table) _write_metadata_pointer(source, table.metadata_location, table=table) @@ -1916,14 +2208,103 @@ def run_cloud_maintenance(source: dict) -> dict: logger.warning("[iceberg] Data deletion skipped: %s", e) results["data_deletion_error"] = str(e) - # 2. Expire snapshots (keep last 7 days of metadata) + # 2. Expire snapshots (keep last 7 days of metadata). + # pyiceberg 0.11.1: table.maintenance.expire_snapshots().older_than(datetime).commit() + # — maintenance is a @property (no parens); older_than takes a tz-aware datetime + # (not int millis). Only removes snapshot METADATA entries — the underlying + # data/manifest files on the object store are NOT garbage-collected; a separate + # remove_orphan_files sweep is required for byte reclamation (deferred until + # pyiceberg >= 0.12, which gains that API). + # + # Cache hygiene: intentionally do NOT pop _snapshot_files_cache / _view_cache + # here — expire drops only old snapshot metadata; the current snapshot's file + # membership is unchanged, so the snapshot fast-path stays valid. (Contrast + # with step 1's data-delete and the optimize-table path, which do invalidate.) keep_snapshot_days = 7 - cutoff_ms = int((datetime.now(UTC) - timedelta(days=keep_snapshot_days)).timestamp() * 1000) + snapshot_cutoff = datetime.now(UTC) - timedelta(days=keep_snapshot_days) try: - table.expire_snapshots().expire_older_than(cutoff_ms).commit() - _set_cached_table(source, _table_identifier(source), table) - _write_metadata_pointer(source, table.metadata_location, table=table) + # Load fresh from the catalog. Note: catalog is the FosSqlCatalog + # whose load_table consults _read_metadata_pointer (2-sec in-process + # cache); freshness here is bounded by _POINTER_CACHE_TTL_SEC, not + # "the absolute latest head". For the FIRST attempt this is fine — + # the cache entry will be ≤2s old, plenty fresh for a weekly cron. + # The retry loop below explicitly invalidates the cache before each + # reload so back-to-back retries actually see post-conflict state. + fresh_table = catalog.load_table(_table_identifier(source)) + snapshots_before = len(fresh_table.metadata.snapshots) + results["snapshots_before"] = snapshots_before + + # Concurrent writers can race us in two shapes that the retry can + # self-heal: + # (a) CommitFailedException — catalog-level pointer race (another + # commit advanced the metadata pointer between our load_table + # and our commit). + # (b) ValueError("Snapshot with snapshot id N does not exist") — + # another expire run (admin re-trigger overlapping the scheduled + # run) already removed snapshots that are still in our expire + # set. Reloading and re-calling older_than rebuilds the expire + # set against the post-overlap snapshot list, so the next attempt + # targets only still-present snapshots. + # The sequence-number ValueError that optimize_table catches cannot + # fire here — ExpireSnapshots stages only AssertTableUUID (no + # AssertRefSnapshotId), so we narrow the ValueError check to the + # "does not exist" message to avoid masking unrelated bugs. + _EXPIRE_RETRIES = 3 + for _retry in range(_EXPIRE_RETRIES): + try: + fresh_table.maintenance.expire_snapshots().older_than(snapshot_cutoff).commit() + break + except (CommitFailedException, ValueError) as cas_err: + msg = str(cas_err) + is_recoverable = isinstance(cas_err, CommitFailedException) or "does not exist" in msg + if not is_recoverable or _retry == _EXPIRE_RETRIES - 1: + raise + logger.warning( + "[iceberg] %s: CAS conflict expiring snapshots (attempt %d/%d), reloading and retrying: %s", + source.get("name"), + _retry + 1, + _EXPIRE_RETRIES, + cas_err, + ) + try: + # Invalidate the FosSqlCatalog pointer cache so the reload + # bypasses the 2-sec _POINTER_CACHE_TTL_SEC and actually + # re-resolves the post-conflict metadata pointer. Without + # this, all retries finish within microseconds and read + # the same pre-conflict cache entry. + _pointer_cache_invalidate(source, _table_identifier(source)) + fresh_table = catalog.load_table(_table_identifier(source)) + except Exception as reload_err: + raise cas_err from reload_err + # Re-pin the baseline against the reloaded head so the diff + # below reflects expirations only, not concurrent additions. + snapshots_before = len(fresh_table.metadata.snapshots) + results["snapshots_before"] = snapshots_before + + snapshots_after = len(fresh_table.metadata.snapshots) + snapshots_expired = max(0, snapshots_before - snapshots_after) + + _set_cached_table(source, _table_identifier(source), fresh_table) + _write_metadata_pointer(source, fresh_table.metadata_location, table=fresh_table) + # Keep the outer-scope `table` consistent for the local-cache cleanup + # step below (currently doesn't use it, but a future addition between + # steps 2 and 3 would expect the post-expire handle). + table = fresh_table + results["snapshots_expired_before_days"] = keep_snapshot_days + results["snapshots_after"] = snapshots_after + results["snapshots_expired_count"] = snapshots_expired + if snapshots_expired > 0: + results["snapshot_expiry_note"] = ( + "metadata entries only; underlying data/manifest files are not deleted by pyiceberg 0.11.1" + ) + logger.info( + "[iceberg] %s: expired %d snapshots (%d -> %d)", + source.get("name"), + snapshots_expired, + snapshots_before, + snapshots_after, + ) except Exception as e: logger.warning("[iceberg] Snapshot expiry skipped: %s", e) results["snapshot_expiry_error"] = str(e) @@ -2054,6 +2435,8 @@ def sync_data(source: dict, progress_callback=None, start_time: str | None = Non uri = entry rel_path = uri.split("/data/")[-1] if "/data/" in uri else uri.split("/")[-1] local_path = os.path.abspath(os.path.join(cache_dir, rel_path)) + if not local_path.startswith(os.path.abspath(cache_dir) + os.sep): + continue cloud_files[uri] = (local_path, 0) else: # Already-downloaded entry. Must populate cloud_files @@ -2122,6 +2505,8 @@ def _parse_ts(ts_str: str) -> datetime: rel_path = uri.split("/")[-1] local_path = os.path.abspath(os.path.join(cache_dir, rel_path)) + if not local_path.startswith(os.path.abspath(cache_dir) + os.sep): + continue cloud_files[uri] = (local_path, record_count) except Exception as e: return {"error": f"Metadata scan failed: {e}", "files_downloaded": 0} @@ -2358,6 +2743,8 @@ def _is_rate_limited(err: Exception) -> bool: rel_path = uri.split("/")[-1] local_path = os.path.abspath(os.path.join(data_dir, rel_path)) + if not local_path.startswith(os.path.abspath(data_dir) + os.sep): + continue if os.path.exists(local_path): resolved_files.append(local_path) else: @@ -2620,6 +3007,8 @@ def _update_snapshot_cache_from_delta(source: dict, table) -> bool: uri = entry.data_file.file_path rel_path = uri.split("/data/")[-1] if "/data/" in uri else uri.split("/")[-1] local = os.path.abspath(os.path.join(cache_dir, rel_path)) + if not local.startswith(os.path.abspath(cache_dir) + os.sep): + continue # Match the same local-vs-URI selection rule used by # _update_iceberg_view_locked: prefer local file when present, # else fall back to the cloud URI for admins (analysts never @@ -2701,6 +3090,8 @@ def _reconcile_snapshot_cache_after_sync(source: dict) -> None: if p.startswith("s3://"): rel_path = p.split("/data/")[-1] if "/data/" in p else p.split("/")[-1] local = os.path.abspath(os.path.join(cache_dir, rel_path)) + if not local.startswith(os.path.abspath(cache_dir) + os.sep): + continue if os.path.exists(local): new_entries.append(local) changed = True @@ -2820,16 +3211,12 @@ def _try_fast_path_view(con, source: dict) -> bool: view_sql = cached[3] if view_sql: - try: - ro_row = con.execute( - "SELECT readonly FROM duckdb_databases() WHERE database_name NOT IN ('system','temp') LIMIT 1" - ).fetchone() - is_ro = bool(ro_row[0]) if ro_row is not None else False - except Exception: - is_ro = False - + # Always bind as a TEMP view on the fast path — the persistent view + # is maintained by the locked rebuild path. Concurrent fast-path + # callers (pool checkouts) would otherwise race on the shared catalog + # and trigger "write-write conflict on alter". exec_sql = view_sql - if is_ro and view_sql.startswith("CREATE OR REPLACE VIEW "): + if view_sql.startswith("CREATE OR REPLACE VIEW "): exec_sql = view_sql.replace("CREATE OR REPLACE VIEW ", "CREATE OR REPLACE TEMP VIEW ", 1) try: con.execute(exec_sql) @@ -3022,6 +3409,18 @@ def _update_iceberg_view_locked(con, source: dict) -> None: snapshot_id = cached_files[1] iceberg_loc = cached_files[2] local_iceberg_files = cached_files[3] + elif metadata_loc is None: + # Never-committed service: the local SQLite catalog has no metadata_location + # row for this table, so there is no Iceberg snapshot to fetch. Skipping + # the S3 round-trip here saves 6-14s on every cold dashboard query for + # services that haven't ingested anything (or whose init_iceberg_table + # call silently failed to write metadata.json to FOS — observed when + # fos_endpoint is unreachable, e.g. local dev / load-test services). + # The view will be built from buffer files only (if any) below, or + # downgraded to an empty WHERE-false view by the existing fall-through. + snapshot_id = None + tbl = None + snap = None else: # The table committed (new metadata_loc) or we had a full cache miss. try: @@ -3066,12 +3465,21 @@ def _update_iceberg_view_locked(con, source: dict) -> None: for f in scan.plan_files(): uri = f.file.file_path + if uri.startswith("file://"): + # Local-only warehouse: the URI IS the local path. + # Skip the FOS-style /data/ rewrite and just use it. + local_path = uri[len("file://") :] + if os.path.exists(local_path): + local_iceberg_files.append(local_path) + continue if "/data/" in uri: rel_path = uri.split("/data/")[-1] else: rel_path = uri.split("/")[-1] local_path = os.path.abspath(os.path.join(data_dir, rel_path)) + if not local_path.startswith(os.path.abspath(data_dir) + os.sep): + continue if os.path.exists(local_path): local_iceberg_files.append(local_path) elif source.get("access_level") != "read_only": @@ -3170,7 +3578,15 @@ def _update_iceberg_view_locked(con, source: dict) -> None: # check the local data_dir directly. If it has parquet files on disk, we # MUST use them — otherwise dashboard queries route through iceberg_scan # over S3 and rack up Class B reads on every poll. - data_dir = os.path.join(cache_dir, "data") + # + # Local-only (file://) warehouse: Iceberg writes data files under + # warehouse///data/ rather than cache/{bucket}/data/. + # Point data_dir at the actual on-disk location so the glob below and the + # eventual read_parquet view SQL hit real files. + if _is_local_only_source(source) and iceberg_loc and iceberg_loc.startswith("file://"): + data_dir = os.path.join(iceberg_loc[len("file://") :], "data") + else: + data_dir = os.path.join(cache_dir, "data") if not local_paths: try: import glob as _glob diff --git a/backend/core/local_compaction.py b/backend/core/local_compaction.py index 92bcc1fb..87a73de7 100644 --- a/backend/core/local_compaction.py +++ b/backend/core/local_compaction.py @@ -78,6 +78,42 @@ _DAILY_FILE_RE = re.compile(r"^daily_(\d{4}-\d{2}-\d{2})_[0-9a-f]+\.parquet$") +def _bin_pack_files(file_paths: list[str], max_bin_size_bytes: int) -> list[list[str]]: + """Group file_paths into bins such that the sum of file sizes in each bin + does not exceed max_bin_size_bytes. Preserves the original file order. + If any single file exceeds max_bin_size_bytes, it goes in its own bin. + """ + bins: list[list[str]] = [] + current_bin: list[str] = [] + current_size = 0 + + for path in file_paths: + try: + file_size = os.path.getsize(path) + except OSError: + continue + + if current_size + file_size > max_bin_size_bytes: + if current_bin: + bins.append(current_bin) + current_bin = [] + current_size = 0 + + if file_size >= max_bin_size_bytes: + bins.append([path]) + else: + current_bin.append(path) + current_size = file_size + else: + current_bin.append(path) + current_size += file_size + + if current_bin: + bins.append(current_bin) + + return bins + + def compact_local_partitions(source: dict, min_files_per_partition: int = 3, dry_run: bool = False) -> dict[str, Any]: """Merge small parquet files within each hour-partition directory into a single larger file. Additionally rolls partitions older than @@ -160,30 +196,42 @@ def compact_local_partitions(source: dict, min_files_per_partition: int = 3, dry parquets = [f for f in os.listdir(part_dir) if f.endswith(".parquet")] if len(parquets) <= min_files_per_partition: continue - # Size ceiling — if the partition is already big, don't double its - # peak file size by merging into one giant file. - total_bytes = sum(os.path.getsize(os.path.join(part_dir, p)) for p in parquets) - if total_bytes > _MAX_PARTITION_BYTES: + + # Sort files alphabetically for deterministic sequential binning + parquets_sorted = sorted(parquets) + full_paths = [os.path.join(part_dir, f) for f in parquets_sorted] + bins = _bin_pack_files(full_paths, _MAX_PARTITION_BYTES) + + eligible_bins = [b for b in bins if len(b) > 1] + if not eligible_bins: continue + result["partitions_scanned"] += 1 - try: - # Lock held only during the actual file-system mutation (delete + - # rename) inside _compact_single_partition; the parquet COPY - # write happens before that on an in-memory DuckDB connection and - # doesn't need the lock. Holding the lock during the COPY would - # block dashboard reads for ~1s per partition. - with publish_lock: - r = _compact_single_partition(part_dir, parquets, dry_run=dry_run) + partition_compacted = False + + for bin_paths in eligible_bins: + bin_basenames = [os.path.basename(p) for p in bin_paths] + try: + # Lock held only during the actual file-system mutation (delete + + # rename) inside _compact_single_partition; the parquet COPY + # write happens before that on an in-memory DuckDB connection and + # doesn't need the lock. Holding the lock during the COPY would + # block dashboard reads for ~1s per partition. + with publish_lock: + r = _compact_single_partition(part_dir, bin_basenames, dry_run=dry_run) + partition_compacted = True + result["files_merged"] += r["files_merged"] + result["files_removed"] += r["files_removed"] + result["bytes_before"] += r["bytes_before"] + result["bytes_after"] += r["bytes_after"] + removed_basenames.extend(r.get("removed_basenames", [])) + except Exception as e: + msg = f"{part_dir} (bin): {type(e).__name__}: {e}" + logger.warning("[local-compact] %s", msg) + result["errors"].append(msg) + + if partition_compacted: result["partitions_compacted"] += 1 - result["files_merged"] += r["files_merged"] - result["files_removed"] += r["files_removed"] - result["bytes_before"] += r["bytes_before"] - result["bytes_after"] += r["bytes_after"] - removed_basenames.extend(r.get("removed_basenames", [])) - except Exception as e: - msg = f"{part_dir}: {type(e).__name__}: {e}" - logger.warning("[local-compact] %s", msg) - result["errors"].append(msg) # ── Daily tier: roll up hour-partitions older than threshold into one # daily file. After this, the partition's hour dirs are removed. @@ -269,8 +317,8 @@ def _cleanup_stale_tmp(data_dir: str) -> int: def _compact_daily_tier(data_dir: str, dry_run: bool = False) -> dict[str, Any]: """Group hour-partitions older than _DAILY_TIER_AGE_DAYS by day, merge - each day's parquets into one file under data/daily/, and remove the - now-empty hour partition dirs. + each day's parquets into size-capped daily files under data/daily/, and + remove the now-empty hour partition dirs. Returns {daily_rollups, files_merged, files_removed, bytes_before, bytes_after}. """ @@ -315,84 +363,105 @@ def _compact_daily_tier(data_dir: str, dry_run: bool = False) -> dict[str, Any]: os.makedirs(daily_root, exist_ok=True) for day_str, parts in by_day.items(): - # Skip if the day is already a single daily file (already rolled up). - if len(parts) == 1 and len(parts[0][1]) == 1: - continue all_paths: list[str] = [] for _, paths in parts: all_paths.extend(paths) - bytes_before = sum(os.path.getsize(p) for p in all_paths) - if dry_run: - result["daily_rollups"] += 1 - result["files_merged"] += len(all_paths) - result["bytes_before"] += bytes_before - continue - # Write the day's merged file under data/daily/. - out_name = f"daily_{day_str}_{uuid.uuid4().hex[:8]}.parquet" - tmp_path = os.path.join(daily_root, f"{out_name}.tmp") - out_path = os.path.join(daily_root, out_name) - try: - con = duckdb.connect(":memory:") - try: - paths_sql = ", ".join(f"'{_sql_escape(p)}'" for p in all_paths) - # Same EXCLUDE-on-probe defense as the hourly path — - # avoid baking timestamp_hour/dt into the daily merged - # file (the view re-computes them at query time). - probe = ( - con.execute(f"SELECT * FROM read_parquet([{paths_sql}], union_by_name=true) LIMIT 0").description - or [] - ) - cols_to_strip = sorted(c for c in ("timestamp_hour", "dt") if any(d[0] == c for d in probe)) - exclude_clause = f" EXCLUDE ({', '.join(cols_to_strip)})" if cols_to_strip else "" - con.execute( - f"COPY (SELECT *{exclude_clause} FROM read_parquet([{paths_sql}], union_by_name=true)) " - f"TO '{_sql_escape(tmp_path)}' (FORMAT PARQUET, COMPRESSION ZSTD)" - ) - finally: - con.close() - # Delete originals, then rename — same crash-safe order as the - # hourly path. - for p in all_paths: + # Sort files alphabetically/chronologically for deterministic sequential binning + all_paths = sorted(all_paths) + bins = _bin_pack_files(all_paths, _MAX_PARTITION_BYTES) + + for bin_paths in bins: + bytes_before = sum(os.path.getsize(p) for p in bin_paths) + if dry_run: + result["daily_rollups"] += 1 + result["files_merged"] += len(bin_paths) + result["bytes_before"] += bytes_before + continue + + if len(bin_paths) == 1: + # Migrate single-file bin to daily folder to retire the hourly folder + old_path = bin_paths[0] + old_name = os.path.basename(old_path) + out_name = f"daily_{day_str}_{uuid.uuid4().hex[:8]}.parquet" + out_path = os.path.join(daily_root, out_name) try: - os.remove(p) + os.rename(old_path, out_path) result["files_removed"] += 1 - result.setdefault("removed_basenames", []).append(os.path.basename(p)) - except OSError as e: - logger.warning("[local-compact] failed to remove %s: %s", p, e) - os.rename(tmp_path, out_path) - bytes_after = os.path.getsize(out_path) - # Try to rmdir the now-empty hour partition dirs. + result.setdefault("removed_basenames", []).append(old_name) + result["daily_rollups"] += 1 + result["files_merged"] += 1 + result["bytes_before"] += bytes_before + result["bytes_after"] += bytes_before + logger.info("🚚 [local-compact] migrated single-file bin %s to %s", old_name, out_name) + except Exception as e: + logger.warning("[local-compact] failed to migrate single-file %s: %s", old_path, e) + else: + # Merge multi-file bin + out_name = f"daily_{day_str}_{uuid.uuid4().hex[:8]}.parquet" + tmp_path = os.path.join(daily_root, f"{out_name}.tmp") + out_path = os.path.join(daily_root, out_name) + try: + con = duckdb.connect(":memory:") + try: + paths_sql = ", ".join(f"'{_sql_escape(p)}'" for p in bin_paths) + probe = ( + con.execute( + f"SELECT * FROM read_parquet([{paths_sql}], union_by_name=true) LIMIT 0" + ).description + or [] + ) + cols_to_strip = sorted(c for c in ("timestamp_hour", "dt") if any(d[0] == c for d in probe)) + exclude_clause = f" EXCLUDE ({', '.join(cols_to_strip)})" if cols_to_strip else "" + con.execute( + f"COPY (SELECT *{exclude_clause} FROM read_parquet([{paths_sql}], union_by_name=true)" + f" ORDER BY timestamp, ip) " + f"TO '{_sql_escape(tmp_path)}' (FORMAT PARQUET, COMPRESSION ZSTD)" + ) + finally: + con.close() + for p in bin_paths: + try: + os.remove(p) + result["files_removed"] += 1 + result.setdefault("removed_basenames", []).append(os.path.basename(p)) + except OSError as e: + logger.warning("[local-compact] failed to remove %s: %s", p, e) + os.rename(tmp_path, out_path) + bytes_after = os.path.getsize(out_path) + result["daily_rollups"] += 1 + result["files_merged"] += len(bin_paths) + result["bytes_before"] += bytes_before + result["bytes_after"] += bytes_after + logger.info( + "📦 [local-compact] daily bin rollup %s: %d files → 1", + day_str, + len(bin_paths), + ) + except Exception as e: + # Clean the tmp on failure so we don't leak. + try: + if os.path.exists(tmp_path): + os.remove(tmp_path) + except OSError: + pass + logger.warning("[local-compact] daily bin rollup %s failed: %s", day_str, e) + + # Try to rmdir the now-empty hour partition dirs. + if not dry_run: for part_dir, _ in parts: try: os.rmdir(part_dir) except OSError: pass # dir not empty (concurrent write) — leave it - result["daily_rollups"] += 1 - result["files_merged"] += len(all_paths) - result["bytes_before"] += bytes_before - result["bytes_after"] += bytes_after - logger.info( - "📦 [local-compact] daily rollup %s: %d files → 1 (saved %d hour-partition dirs)", - day_str, - len(all_paths), - len(parts), - ) - except Exception as e: - # Clean the tmp on failure so we don't leak. - try: - if os.path.exists(tmp_path): - os.remove(tmp_path) - except OSError: - pass - logger.warning("[local-compact] daily rollup %s failed: %s", day_str, e) return result def _compact_weekly_tier(data_dir: str, dry_run: bool = False) -> dict[str, Any]: """Group daily files older than _WEEKLY_TIER_AGE_DAYS by ISO week, merge - each week's parquets into one file under data/weekly/, delete originals. + each week's parquets into size-capped weekly files under data/weekly/, + and delete originals. Operates on files in data/daily/ produced by _compact_daily_tier. The daily filenames embed YYYY-MM-DD (the rollup date), which we parse with @@ -445,58 +514,85 @@ def _compact_weekly_tier(data_dir: str, dry_run: bool = False) -> dict[str, Any] for week_key, items in by_week.items(): if len(items) < 2: continue # nothing to merge for a single-day week - all_paths = [p for p, _ in items] - bytes_before = sum(os.path.getsize(p) for p in all_paths) - if dry_run: - result["weekly_rollups"] += 1 - result["files_merged"] += len(all_paths) - result["bytes_before"] += bytes_before - continue - out_name = f"weekly_{week_key}_{uuid.uuid4().hex[:8]}.parquet" - tmp_path = os.path.join(weekly_root, f"{out_name}.tmp") - out_path = os.path.join(weekly_root, out_name) - try: - con = duckdb.connect(":memory:") - try: - paths_sql = ", ".join(f"'{_sql_escape(p)}'" for p in all_paths) - probe = ( - con.execute(f"SELECT * FROM read_parquet([{paths_sql}], union_by_name=true) LIMIT 0").description - or [] - ) - cols_to_strip = sorted(c for c in ("timestamp_hour", "dt") if any(d[0] == c for d in probe)) - exclude_clause = f" EXCLUDE ({', '.join(cols_to_strip)})" if cols_to_strip else "" - con.execute( - f"COPY (SELECT *{exclude_clause} FROM read_parquet([{paths_sql}], union_by_name=true)) " - f"TO '{_sql_escape(tmp_path)}' (FORMAT PARQUET, COMPRESSION ZSTD)" - ) - finally: - con.close() - for p in all_paths: + # Sort daily files alphabetically/chronologically for deterministic sequential binning + items_sorted = sorted(items, key=lambda x: x[0]) + all_paths = [p for p, _ in items_sorted] + bins = _bin_pack_files(all_paths, _MAX_PARTITION_BYTES) + + for bin_paths in bins: + bytes_before = sum(os.path.getsize(p) for p in bin_paths) + if dry_run: + result["weekly_rollups"] += 1 + result["files_merged"] += len(bin_paths) + result["bytes_before"] += bytes_before + continue + + if len(bin_paths) == 1: + # Migrate single-file weekly bin to weekly folder + old_path = bin_paths[0] + old_name = os.path.basename(old_path) + out_name = f"weekly_{week_key}_{uuid.uuid4().hex[:8]}.parquet" + out_path = os.path.join(weekly_root, out_name) try: - os.remove(p) + os.rename(old_path, out_path) result["files_removed"] += 1 - result.setdefault("removed_basenames", []).append(os.path.basename(p)) - except OSError as e: - logger.warning("[local-compact] failed to remove %s: %s", p, e) - os.rename(tmp_path, out_path) - bytes_after = os.path.getsize(out_path) - result["weekly_rollups"] += 1 - result["files_merged"] += len(all_paths) - result["bytes_before"] += bytes_before - result["bytes_after"] += bytes_after - logger.info( - "🗓️ [local-compact] weekly rollup %s: %d daily file(s) → 1", - week_key, - len(all_paths), - ) - except Exception as e: - try: - if os.path.exists(tmp_path): - os.remove(tmp_path) - except OSError: - pass - logger.warning("[local-compact] weekly rollup %s failed: %s", week_key, e) + result.setdefault("removed_basenames", []).append(old_name) + result["weekly_rollups"] += 1 + result["files_merged"] += 1 + result["bytes_before"] += bytes_before + result["bytes_after"] += bytes_before + logger.info("🚚 [local-compact] migrated single-file weekly bin %s to %s", old_name, out_name) + except Exception as e: + logger.warning("[local-compact] failed to migrate single-file weekly bin %s: %s", old_path, e) + else: + out_name = f"weekly_{week_key}_{uuid.uuid4().hex[:8]}.parquet" + tmp_path = os.path.join(weekly_root, f"{out_name}.tmp") + out_path = os.path.join(weekly_root, out_name) + try: + con = duckdb.connect(":memory:") + try: + paths_sql = ", ".join(f"'{_sql_escape(p)}'" for p in bin_paths) + probe = ( + con.execute( + f"SELECT * FROM read_parquet([{paths_sql}], union_by_name=true) LIMIT 0" + ).description + or [] + ) + cols_to_strip = sorted(c for c in ("timestamp_hour", "dt") if any(d[0] == c for d in probe)) + exclude_clause = f" EXCLUDE ({', '.join(cols_to_strip)})" if cols_to_strip else "" + con.execute( + f"COPY (SELECT *{exclude_clause} FROM read_parquet([{paths_sql}], union_by_name=true)" + f" ORDER BY timestamp, ip) " + f"TO '{_sql_escape(tmp_path)}' (FORMAT PARQUET, COMPRESSION ZSTD)" + ) + finally: + con.close() + for p in bin_paths: + try: + os.remove(p) + result["files_removed"] += 1 + result.setdefault("removed_basenames", []).append(os.path.basename(p)) + except OSError as e: + logger.warning("[local-compact] failed to remove %s: %s", p, e) + os.rename(tmp_path, out_path) + bytes_after = os.path.getsize(out_path) + result["weekly_rollups"] += 1 + result["files_merged"] += len(bin_paths) + result["bytes_before"] += bytes_before + result["bytes_after"] += bytes_after + logger.info( + "🗓️ [local-compact] weekly bin rollup %s: %d daily file(s) → 1", + week_key, + len(bin_paths), + ) + except Exception as e: + try: + if os.path.exists(tmp_path): + os.remove(tmp_path) + except OSError: + pass + logger.warning("[local-compact] weekly bin rollup %s failed: %s", week_key, e) return result @@ -543,7 +639,8 @@ def _compact_single_partition(part_dir: str, parquets: list[str], dry_run: bool # zstd compression matches Fastly's parquet output and the # buffer-commit writer; keeps decompression cost stable. con.execute( - f"COPY (SELECT *{exclude_clause} FROM read_parquet([{paths_sql}], union_by_name=true)) " + f"COPY (SELECT *{exclude_clause} FROM read_parquet([{paths_sql}], union_by_name=true)" + f" ORDER BY timestamp, ip) " f"TO '{_sql_escape(tmp_path)}' (FORMAT PARQUET, COMPRESSION ZSTD)" ) finally: diff --git a/backend/core/log_fields.py b/backend/core/log_fields.py index a93b3c9e..6644b40d 100644 --- a/backend/core/log_fields.py +++ b/backend/core/log_fields.py @@ -160,7 +160,7 @@ "group": None, "label": "Client IP", "description": "Client IP address. Captured at the real edge via x-fos-edge-data header.", - "vcl": '"ip":"%{if(req.http.x-fos-edge-data:ip != "", req.http.x-fos-edge-data:ip, req.http.Fastly-Client-IP)}V"', + "vcl": '"ip":"%{json.escape(if(req.http.x-fos-edge-data:ip != "", req.http.x-fos-edge-data:ip, req.http.Fastly-Client-IP))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 22, "required_by": ["low_and_slow", "botnet_grouping"], @@ -248,7 +248,7 @@ "group": "A", "label": "HTTP Method", "description": "Request method: GET, POST, HEAD, PUT, DELETE, etc.", - "vcl": '"method":"%{json.escape(req.method)}V"', + "vcl": '"method":"%{json.escape(substr(req.method, 0, 128))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 19, "required_by": [], @@ -423,7 +423,7 @@ "group": "C", "label": "Server Region", "description": "Fastly billing region of the serving PoP (e.g. NA, EU, APAC). Captured at edge for accurate attribution through shields.", - "vcl": '"server_region":"%{if(req.http.x-fos-edge-data:srv_region != "", req.http.x-fos-edge-data:srv_region, server.region)}V"', + "vcl": '"server_region":"%{json.escape(if(req.http.x-fos-edge-data:srv_region != "", req.http.x-fos-edge-data:srv_region, server.region))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 20, "required_by": ["region_latency"], @@ -433,7 +433,7 @@ "group": "C", "label": "IPv6", "description": "True when the client connected over IPv6. IPv6 clients can have different routing and latency profiles.", - "vcl": '"is_ipv6":%{if(req.http.x-fos-edge-data:is_ipv6 != "", req.http.x-fos-edge-data:is_ipv6, if(req.is_ipv6, "1", "0"))}V', + "vcl": '"is_ipv6":%{if(req.http.x-fos-edge-data:is_ipv6 ~ "^[0-9]+$", req.http.x-fos-edge-data:is_ipv6, if(req.is_ipv6, "1", "0"))}V', "duckdb_type": "BOOLEAN", "typical_bytes": 12, "required_by": [], @@ -443,7 +443,7 @@ "group": "C", "label": "Conn. Request Count", "description": "Number of requests made on this TCP/QUIC connection. High values indicate HTTP/2 keep-alive multiplexing.", - "vcl": '"conn_requests":%{if(req.http.x-fos-edge-data:conn_reqs != "", req.http.x-fos-edge-data:conn_reqs, if(client.requests > 0, "" + client.requests, "null"))}V', + "vcl": '"conn_requests":%{if(req.http.x-fos-edge-data:conn_reqs ~ "^[0-9]+$", req.http.x-fos-edge-data:conn_reqs, if(client.requests > 0, "" + client.requests, "null"))}V', "duckdb_type": "USMALLINT", "typical_bytes": 20, "required_by": ["connection_abuse"], @@ -455,7 +455,7 @@ "description": "TLS protocol version as a float: 1.2 or 1.3.", "formatter": "number", "precision": 1, - "vcl": '"tls":"%{if(req.http.x-fos-edge-data:tls != "", req.http.x-fos-edge-data:tls, if(tls.client.protocol != "", regsub(tls.client.protocol, "^TLSv", ""), ""))}V"', + "vcl": '"tls":"%{json.escape(if(req.http.x-fos-edge-data:tls != "", req.http.x-fos-edge-data:tls, if(tls.client.protocol != "", regsub(tls.client.protocol, "^TLSv", ""), "")))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 10, "required_by": [], @@ -467,7 +467,7 @@ "label": "Country", "description": "ISO 3166-1 alpha-2 country code (e.g. US, DE, JP). Enables world map.", "formatter": "country", - "vcl": '"country":"%{if(req.http.x-fos-edge-data:country != "", req.http.x-fos-edge-data:country, client.geo.country_code)}V"', + "vcl": '"country":"%{json.escape(if(req.http.x-fos-edge-data:country != "", req.http.x-fos-edge-data:country, client.geo.country_code))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 15, "individually_toggleable": True, @@ -485,7 +485,7 @@ "label": "City", "description": "City name from Fastly geo-IP. Variable length.", "formatter": "city", - "vcl": '"city":"%{if(req.http.x-fos-edge-data:city != "", req.http.x-fos-edge-data:city, client.geo.city)}V"', + "vcl": '"city":"%{json.escape(if(req.http.x-fos-edge-data:city != "", req.http.x-fos-edge-data:city, client.geo.city))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 18, "individually_toggleable": True, @@ -497,7 +497,7 @@ "label": "Region", "description": "ISO 3166-2 region/state/province code.", "formatter": "region", - "vcl": '"region":"%{if(req.http.x-fos-edge-data:region != "", req.http.x-fos-edge-data:region, if(client.geo.region == "?", "", json.escape(client.geo.region)))}V"', + "vcl": '"region":"%{json.escape(if(req.http.x-fos-edge-data:region != "", req.http.x-fos-edge-data:region, if(client.geo.region == "?", "", client.geo.region)))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 14, "individually_toggleable": True, @@ -511,7 +511,7 @@ "description": "Client latitude (-90 to 90). Null for unresolvable IPs.", "formatter": "number", "precision": 4, - "vcl": '"lat":%{if(req.http.x-fos-edge-data:lat != "", req.http.x-fos-edge-data:lat, if(client.geo.country_code != "?", "" + client.geo.latitude, "null"))}V', + "vcl": '"lat":%{if(req.http.x-fos-edge-data:lat ~ "^-?[0-9]+(\\.[0-9]+)?$", req.http.x-fos-edge-data:lat, if(client.geo.country_code != "?", "" + client.geo.latitude, "null"))}V', "duckdb_type": "FLOAT", "typical_bytes": 12, "required_by": ["network_asn_health"], @@ -523,7 +523,7 @@ "description": "Client longitude (-180 to 180). Null for unresolvable IPs.", "formatter": "number", "precision": 4, - "vcl": '"lon":%{if(req.http.x-fos-edge-data:lon != "", req.http.x-fos-edge-data:lon, if(client.geo.country_code != "?", "" + client.geo.longitude, "null"))}V', + "vcl": '"lon":%{if(req.http.x-fos-edge-data:lon ~ "^-?[0-9]+(\\.[0-9]+)?$", req.http.x-fos-edge-data:lon, if(client.geo.country_code != "?", "" + client.geo.longitude, "null"))}V', "duckdb_type": "FLOAT", "typical_bytes": 13, "required_by": ["network_asn_health"], @@ -533,7 +533,7 @@ "group": "E", "label": "Metro Code", "description": "US DMA metro area code (e.g. 501 = New York City). Empty for non-US.", - "vcl": '"metro":%{if(req.http.x-fos-edge-data:metro != "", req.http.x-fos-edge-data:metro, if(client.geo.metro_code > 0, "" + client.geo.metro_code, "null"))}V', + "vcl": '"metro":%{if(req.http.x-fos-edge-data:metro ~ "^[0-9]+$", req.http.x-fos-edge-data:metro, if(client.geo.metro_code > 0, "" + client.geo.metro_code, "null"))}V', "duckdb_type": "USMALLINT", "typical_bytes": 14, "required_by": [], @@ -544,7 +544,7 @@ "group": "F", "label": "ASN", "description": "Client Autonomous System Number (ISP identity). Enables ASN-level analysis.", - "vcl": '"asn":%{if(req.http.x-fos-edge-data:asn != "", req.http.x-fos-edge-data:asn, if(client.as.number > 0, "" + client.as.number, "null"))}V', + "vcl": '"asn":%{if(req.http.x-fos-edge-data:asn ~ "^[0-9]+$", req.http.x-fos-edge-data:asn, if(client.as.number > 0, "" + client.as.number, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 11, "required_by": ["asn_concentration", "network_asn_health", "region_latency"], @@ -556,7 +556,7 @@ "description": "TCP round-trip time in microseconds at the Fastly edge.", "formatter": "number", "unit": "µs", - "vcl": '"tcp_rtt":%{if(req.http.x-fos-edge-data:rtt != "", req.http.x-fos-edge-data:rtt, if(client.socket.tcpi_rtt > 0, "" + client.socket.tcpi_rtt, "null"))}V', + "vcl": '"tcp_rtt":%{if(req.http.x-fos-edge-data:rtt ~ "^[0-9]+$", req.http.x-fos-edge-data:rtt, if(client.socket.tcpi_rtt > 0, "" + client.socket.tcpi_rtt, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 19, "required_by": ["network_asn_health"], @@ -566,7 +566,7 @@ "group": "F", "label": "Transport Protocol", "description": "Transport protocol: 'tcp' or 'quic'. Low-cardinality; essentially free in Parquet.", - "vcl": '"transport":"%{if(req.http.x-fos-edge-data:transport != "", req.http.x-fos-edge-data:transport, transport.type)}V"', + "vcl": '"transport":"%{json.escape(if(req.http.x-fos-edge-data:transport != "", req.http.x-fos-edge-data:transport, transport.type))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 18, "required_by": ["network_asn_health"], @@ -579,7 +579,7 @@ "description": "Packet loss fraction (0.0–1.0). Direct indicator of network congestion.", "formatter": "percent", "precision": 4, - "vcl": '"ploss":%{if(req.http.x-fos-edge-data:ploss != "", req.http.x-fos-edge-data:ploss, if(client.socket.ploss > 0, "" + client.socket.ploss, "null"))}V', + "vcl": '"ploss":%{if(req.http.x-fos-edge-data:ploss ~ "^-?[0-9]+(\\.[0-9]+)?$", req.http.x-fos-edge-data:ploss, if(client.socket.ploss > 0, "" + client.socket.ploss, "null"))}V', "duckdb_type": "FLOAT", "typical_bytes": 18, "required_by": ["network_asn_health"], @@ -591,7 +591,7 @@ "description": "Minimum RTT seen on this TCP connection (geography baseline). Delta from tcp_rtt isolates congestion.", "formatter": "number", "unit": "µs", - "vcl": '"rtt_min":%{if(req.http.x-fos-edge-data:rtt_min != "", req.http.x-fos-edge-data:rtt_min, if(client.socket.tcpi_min_rtt > 0, "" + client.socket.tcpi_min_rtt, "null"))}V', + "vcl": '"rtt_min":%{if(req.http.x-fos-edge-data:rtt_min ~ "^[0-9]+$", req.http.x-fos-edge-data:rtt_min, if(client.socket.tcpi_min_rtt > 0, "" + client.socket.tcpi_min_rtt, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 19, "required_by": ["network_asn_health"], @@ -603,7 +603,7 @@ "description": "RTT variance in microseconds. Jitter causes streaming buffer stalls more than raw latency.", "formatter": "number", "unit": "µs", - "vcl": '"rtt_var":%{if(req.http.x-fos-edge-data:rtt_var != "", req.http.x-fos-edge-data:rtt_var, if(client.socket.tcpi_rttvar > 0, "" + client.socket.tcpi_rttvar, "null"))}V', + "vcl": '"rtt_var":%{if(req.http.x-fos-edge-data:rtt_var ~ "^[0-9]+$", req.http.x-fos-edge-data:rtt_var, if(client.socket.tcpi_rttvar > 0, "" + client.socket.tcpi_rttvar, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 18, "required_by": ["network_asn_health"], @@ -614,7 +614,7 @@ "label": "TCP Retransmissions", "description": "TCP retransmission delta since previous sample. Direct congestion signal.", "formatter": "number", - "vcl": '"retrans":%{if(req.http.x-fos-edge-data:retrans != "", req.http.x-fos-edge-data:retrans, if(client.socket.tcpi_delta_retrans > 0, "" + client.socket.tcpi_delta_retrans, "null"))}V', + "vcl": '"retrans":%{if(req.http.x-fos-edge-data:retrans ~ "^[0-9]+$", req.http.x-fos-edge-data:retrans, if(client.socket.tcpi_delta_retrans > 0, "" + client.socket.tcpi_delta_retrans, "null"))}V', "duckdb_type": "UTINYINT", "typical_bytes": 15, "required_by": ["network_asn_health"], @@ -625,7 +625,7 @@ "label": "Bandwidth Estimate", "description": "Fastly's estimated bandwidth for this connection (bytes/sec or bits/sec — see note). Only applicable for QUIC; TCP connections should use delivery_rate instead.", "formatter": "bytes", - "vcl": '"bw":%{if(req.http.x-fos-edge-data:bw != "", req.http.x-fos-edge-data:bw, if(transport.bw_estimate > 0, "" + transport.bw_estimate, "null"))}V', + "vcl": '"bw":%{if(req.http.x-fos-edge-data:bw ~ "^[0-9]+$", req.http.x-fos-edge-data:bw, if(transport.bw_estimate > 0, "" + transport.bw_estimate, "null"))}V', "duckdb_type": "UBIGINT", "typical_bytes": 17, "required_by": [], @@ -635,7 +635,7 @@ "group": "G", "label": "Connection Speed Class", "description": "Geo-IP speed classification: broadband, cable, dsl, mobile, satellite, dialup. Low-cardinality.", - "vcl": '"c_speed":"%{if(req.http.x-fos-edge-data:c_speed != "", req.http.x-fos-edge-data:c_speed, if(client.geo.conn_speed == "?", "", client.geo.conn_speed))}V"', + "vcl": '"c_speed":"%{json.escape(if(req.http.x-fos-edge-data:c_speed != "", req.http.x-fos-edge-data:c_speed, if(client.geo.conn_speed == "?", "", client.geo.conn_speed)))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 14, "required_by": ["network_asn_health"], @@ -645,7 +645,7 @@ "group": "G", "label": "Connection Type", "description": "Geo-IP connection type: residential, commercial, cellular, corporate. Low-cardinality.", - "vcl": '"c_type":"%{if(req.http.x-fos-edge-data:c_type != "", req.http.x-fos-edge-data:c_type, if(client.geo.conn_type == "?", "", client.geo.conn_type))}V"', + "vcl": '"c_type":"%{json.escape(if(req.http.x-fos-edge-data:c_type != "", req.http.x-fos-edge-data:c_type, if(client.geo.conn_type == "?", "", client.geo.conn_type)))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 27, "required_by": ["network_asn_health"], @@ -656,7 +656,7 @@ "label": "TCP Delivery Rate", "description": "Actual TCP delivery rate in bytes/sec measured by the kernel. More reliable than bandwidth estimate for TCP connections.", "formatter": "bytes", - "vcl": '"delivery_rate":%{if(req.http.x-fos-edge-data:del_rate != "", req.http.x-fos-edge-data:del_rate, if(client.socket.tcpi_delivery_rate > 0, "" + client.socket.tcpi_delivery_rate, "null"))}V', + "vcl": '"delivery_rate":%{if(req.http.x-fos-edge-data:del_rate ~ "^[0-9]+$", req.http.x-fos-edge-data:del_rate, if(client.socket.tcpi_delivery_rate > 0, "" + client.socket.tcpi_delivery_rate, "null"))}V', "duckdb_type": "UBIGINT", "typical_bytes": 22, "required_by": ["network_asn_health"], @@ -667,7 +667,7 @@ "label": "TCP Data Segments Out", "description": "Total TCP data segments sent on this connection. Enables retransmit ratio: retrans / data_segs_out.", "formatter": "number", - "vcl": '"data_segs_out":%{if(req.http.x-fos-edge-data:data_segs != "", req.http.x-fos-edge-data:data_segs, if(client.socket.tcpi_data_segs_out > 0, "" + client.socket.tcpi_data_segs_out, "null"))}V', + "vcl": '"data_segs_out":%{if(req.http.x-fos-edge-data:data_segs ~ "^[0-9]+$", req.http.x-fos-edge-data:data_segs, if(client.socket.tcpi_data_segs_out > 0, "" + client.socket.tcpi_data_segs_out, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 21, "required_by": ["network_asn_health"], @@ -700,7 +700,7 @@ "group": "H", "label": "TLS Cipher Suite SHA", "description": "SHA fingerprint of the client's offered cipher suite list. Evasion-resistant complement to JA3/JA4 for bot farm detection.", - "vcl": '"tls_ciphers_sha":"%{if(req.http.x-fos-edge-data:tls_csha != "", json.escape(req.http.x-fos-edge-data:tls_csha), json.escape(tls.client.ciphers_list_sha))}V"', + "vcl": '"tls_ciphers_sha":"%{json.escape(if(req.http.x-fos-edge-data:tls_csha != "", req.http.x-fos-edge-data:tls_csha, tls.client.ciphers_list_sha))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 48, "individually_toggleable": True, @@ -712,7 +712,7 @@ "group": "I", "label": "Proxy Type", "description": "Anonymizing proxy type: VPN, Tor, DCH (data center), etc.", - "vcl": '"p_type":"%{if(req.http.x-fos-edge-data:p_type != "", json.escape(req.http.x-fos-edge-data:p_type), if(client.geo.proxy_type == "?", "", json.escape(client.geo.proxy_type)))}V"', + "vcl": '"p_type":"%{json.escape(if(req.http.x-fos-edge-data:p_type != "", req.http.x-fos-edge-data:p_type, if(client.geo.proxy_type == "?", "", client.geo.proxy_type)))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 10, "required_by": ["proxy_surge"], @@ -722,7 +722,7 @@ "group": "I", "label": "Proxy Description", "description": "Anonymizing proxy provider name.", - "vcl": '"p_desc":"%{if(req.http.x-fos-edge-data:p_desc != "", json.escape(req.http.x-fos-edge-data:p_desc), if(client.geo.proxy_description == "?", "", json.escape(client.geo.proxy_description)))}V"', + "vcl": '"p_desc":"%{json.escape(if(req.http.x-fos-edge-data:p_desc != "", req.http.x-fos-edge-data:p_desc, if(client.geo.proxy_description == "?", "", client.geo.proxy_description)))}V"', "duckdb_type": "VARCHAR", "typical_bytes": 10, "required_by": ["proxy_surge"], @@ -789,7 +789,7 @@ "description": "QUIC smoothed RTT in microseconds. Null for TCP connections.", "formatter": "number", "unit": "µs", - "vcl": '"q_rtt":%{if(req.http.x-fos-edge-data:q_rtt != "", req.http.x-fos-edge-data:q_rtt, if(transport.type == "quic", "" + quic.rtt.smoothed, "null"))}V', + "vcl": '"q_rtt":%{if(req.http.x-fos-edge-data:q_rtt ~ "^[0-9]+$", req.http.x-fos-edge-data:q_rtt, if(transport.type == "quic", "" + quic.rtt.smoothed, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 19, "required_by": [], @@ -801,7 +801,7 @@ "description": "QUIC RTT variance in microseconds. Null for TCP connections.", "formatter": "number", "unit": "µs", - "vcl": '"q_rtt_var":%{if(req.http.x-fos-edge-data:q_rtt_var != "", req.http.x-fos-edge-data:q_rtt_var, if(transport.type == "quic", "" + quic.rtt.variance, "null"))}V', + "vcl": '"q_rtt_var":%{if(req.http.x-fos-edge-data:q_rtt_var ~ "^[0-9]+$", req.http.x-fos-edge-data:q_rtt_var, if(transport.type == "quic", "" + quic.rtt.variance, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 19, "required_by": [], @@ -812,7 +812,7 @@ "label": "QUIC Packets Lost", "description": "QUIC packets lost counter. Null for TCP connections.", "formatter": "number", - "vcl": '"q_lost":%{if(req.http.x-fos-edge-data:q_lost != "", req.http.x-fos-edge-data:q_lost, if(transport.type == "quic", "" + quic.num_packets.lost, "null"))}V', + "vcl": '"q_lost":%{if(req.http.x-fos-edge-data:q_lost ~ "^[0-9]+$", req.http.x-fos-edge-data:q_lost, if(transport.type == "quic", "" + quic.num_packets.lost, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 17, "required_by": [], @@ -823,7 +823,7 @@ "label": "QUIC Congestion Window", "description": "QUIC congestion window size. Null for TCP connections.", "formatter": "number", - "vcl": '"q_cwnd":%{if(req.http.x-fos-edge-data:q_cwnd != "", req.http.x-fos-edge-data:q_cwnd, if(transport.type == "quic", "" + quic.cc.cwnd, "null"))}V', + "vcl": '"q_cwnd":%{if(req.http.x-fos-edge-data:q_cwnd ~ "^[0-9]+$", req.http.x-fos-edge-data:q_cwnd, if(transport.type == "quic", "" + quic.cc.cwnd, "null"))}V', "duckdb_type": "UINTEGER", "typical_bytes": 16, "required_by": [], diff --git a/backend/core/metadata_db.py b/backend/core/metadata_db.py index ce98edeb..bcde7bad 100644 --- a/backend/core/metadata_db.py +++ b/backend/core/metadata_db.py @@ -54,6 +54,29 @@ _ingested_filenames_cache_lock = threading.Lock() +# Pre-compiled for the per-insert file_date parse. The canonical Fastly +# basename is `...T.-.log.gz`; locate the +# first 'T' and use the 10 chars before it when they look like a date. +# Matches the GLOB in _migration_002 / get_log_accounting_counts so legacy +# and runtime parsing agree. +import re as _re_metadata_db # noqa: E402 + +_FILE_DATE_RE = _re_metadata_db.compile(r"(\d{4}-\d{2}-\d{2})T") + + +def _parse_file_date(file_name: str) -> str | None: + """Return 'YYYY-MM-DD' parsed from filename or None if no match. + + Cheap regex on the basename — runs per-insert, called from the bulk + INSERT in `insert_ingested_files`. Same semantics as the SQL backfill + in `_migration_002_add_ingested_files_file_date`. + """ + if not file_name: + return None + m = _FILE_DATE_RE.search(file_name) + return m.group(1) if m else None + + def _clear_ingested_filenames_cache(service_id: str | None = None) -> None: """Drop the dedup cache for one service or all services. @@ -227,6 +250,7 @@ def teardown(service_id: str) -> None: row_count INTEGER, file_size_bytes INTEGER, error_count INTEGER DEFAULT 0, + file_date DATE, PRIMARY KEY (file_name, source_name) )""", # Covers `/usage/prefill`'s source+range narrowing @@ -240,6 +264,13 @@ def teardown(service_id: str) -> None: # old index is redundant and dropped here. Index name matches the # by-name reference in `list_unbackfilled_fastly_edge_files`'s docstring. "CREATE INDEX IF NOT EXISTS idx_ingested_files_source_ingested_at ON ingested_files(source_name, ingested_at)", + # Note: idx_ingested_files_source_date (companion index for per-day + # usage queries) is created by _migration_002_add_ingested_files_file_date, + # not here — _SCHEMA runs before migrations and a legacy DB upgrading + # would fail on this CREATE INDEX (the file_date column doesn't exist + # yet at that point). The migration is idempotent + runs for fresh DBs + # too (apply_pending walks v1..LATEST on every init), so the index + # always lands without _SCHEMA carrying it. "DROP INDEX IF EXISTS idx_ingested_files_source", # Earlier in this branch a redundant `idx_ingested_files_source_ts` was # added under a different name before discovering the existing @@ -472,6 +503,55 @@ def teardown(service_id: str) -> None: bytes = bytes + excluded.bytes, last_updated = excluded.last_updated; END""", + # AFTER DELETE trigger: pairs with the INSERT trigger so DELETE+INSERT + # cycles (notably reconcile_fastly_stats refreshing each RECONCILE_A/B + # row every hour) don't leak phantom counts into the rollup. Without + # this, every reconcile pass added the new gap on top of the previous + # one, drifting Class A counts to 30-60x reality. + """CREATE TRIGGER IF NOT EXISTS trg_usage_log_summary_delete + AFTER DELETE ON usage_log + WHEN OLD.timestamp IS NOT NULL AND length(OLD.timestamp) >= 13 AND OLD.service_id IS NOT NULL + BEGIN + UPDATE usage_log_hourly_summary + SET count = count - COALESCE(OLD.count, 1), + bytes = bytes - COALESCE(OLD.bytes, 0), + last_updated = datetime('now') + WHERE service_id = OLD.service_id + AND hour = substr(OLD.timestamp, 1, 13) + AND operation_class = COALESCE(OLD.operation_class, '') + AND operation_type = COALESCE(OLD.operation_type, ''); + END""", + # AFTER UPDATE trigger: defensive. No current code path UPDATEs + # usage_log, but if one is added, the rollup must stay in sync. Models + # an UPDATE as a decrement against the OLD bucket + an upsert into the + # NEW bucket — correct whether the keyed columns change or not. + """CREATE TRIGGER IF NOT EXISTS trg_usage_log_summary_update + AFTER UPDATE ON usage_log + WHEN NEW.timestamp IS NOT NULL AND length(NEW.timestamp) >= 13 AND NEW.service_id IS NOT NULL + AND (OLD.count IS NOT NEW.count OR OLD.bytes IS NOT NEW.bytes + OR OLD.timestamp IS NOT NEW.timestamp + OR OLD.operation_class IS NOT NEW.operation_class + OR OLD.operation_type IS NOT NEW.operation_type + OR OLD.service_id IS NOT NEW.service_id) + BEGIN + UPDATE usage_log_hourly_summary + SET count = count - COALESCE(OLD.count, 1), + bytes = bytes - COALESCE(OLD.bytes, 0), + last_updated = datetime('now') + WHERE service_id = OLD.service_id + AND hour = substr(OLD.timestamp, 1, 13) + AND operation_class = COALESCE(OLD.operation_class, '') + AND operation_type = COALESCE(OLD.operation_type, ''); + INSERT INTO usage_log_hourly_summary + (service_id, hour, operation_class, operation_type, count, bytes, last_updated) + VALUES (NEW.service_id, substr(NEW.timestamp, 1, 13), + COALESCE(NEW.operation_class, ''), COALESCE(NEW.operation_type, ''), + COALESCE(NEW.count, 1), COALESCE(NEW.bytes, 0), datetime('now')) + ON CONFLICT(service_id, hour, operation_class, operation_type) + DO UPDATE SET count = count + excluded.count, + bytes = bytes + excluded.bytes, + last_updated = excluded.last_updated; + END""", # Tracks Iceberg parquet basenames that local_compaction merged into a # bigger local file and then deleted from disk. WITHOUT this table the # sync_data fast-path check sees the deletions as "missing local files" @@ -1107,31 +1187,78 @@ def get_log_accounting_counts( full path contains a 'T' preceded by a YYYY-MM-DD prefix we slice the emission bucket out of the filename; otherwise we fall back to ``ingested_at`` (covers legacy/test files without an ISO basename). + + Fast/slow split — the WHERE used to filter on ``datetime(ingested_at)``, + which can't use any index (the wrapping function defeats + ``idx_ingested_files_source_ingested_at``) and forces a full source- + partition scan: 1533 ms on a 24 h window on prod 2026-06-05. + The fast UNION arm uses ``file_date`` (populated by ``_migration_002`` + from the canonical Fastly basename), which IS covered by the + composite ``idx_ingested_files_source_date`` index — range scan + instead of full scan. Rows whose filename doesn't match the canonical + pattern (``file_date IS NULL`` — legacy data, tests, ad-hoc + backfills) fall through to the original ``ingested_at`` scan; that + arm typically returns zero rows in production but keeps semantic + equivalence with the pre-change behavior. """ con = get_con(service_id) + start_date = sql_start[:10] + end_date = sql_end[:10] rows = con.execute( """ - SELECT - CASE - WHEN instr(file_name, 'T') >= 11 - AND substr(file_name, instr(file_name, 'T') - 10, 10) - GLOB '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]' - THEN substr(file_name, instr(file_name, 'T') - 10, ?) - WHEN ingested_at IS NOT NULL - THEN substr(replace(ingested_at, ' ', 'T'), 1, ?) - ELSE NULL - END AS bucket, - sum(row_count) AS rows, - count(*) AS files - FROM ingested_files - WHERE source_name = ? - AND datetime(ingested_at) >= datetime(?) - AND datetime(ingested_at) <= datetime(?) - AND file_name != '__seeding_attempted__' - GROUP BY 1 + SELECT bucket, sum(rc) AS rows, sum(fc) AS files FROM ( + -- Fast arm: file_date index range scan. file_date IS NOT NULL + -- implies the basename matches the canonical Fastly pattern + -- per _migration_002, so the bucket substr will always succeed. + SELECT substr(file_name, instr(file_name, 'T') - 10, ?) AS bucket, + sum(row_count) AS rc, + count(*) AS fc + FROM ingested_files + WHERE source_name = ? + AND file_date IS NOT NULL + AND file_date >= ? AND file_date <= ? + AND file_name != '__seeding_attempted__' + GROUP BY 1 + UNION ALL + -- Slow arm: rows without a parseable basename (file_date NULL). + -- Keeps the full CASE so the ingested_at fallback continues + -- to count test fixtures + legacy uploads. + SELECT + CASE + WHEN instr(file_name, 'T') >= 11 + AND substr(file_name, instr(file_name, 'T') - 10, 10) + GLOB '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]' + THEN substr(file_name, instr(file_name, 'T') - 10, ?) + WHEN ingested_at IS NOT NULL + THEN substr(replace(ingested_at, ' ', 'T'), 1, ?) + ELSE NULL + END AS bucket, + sum(row_count) AS rc, + count(*) AS fc + FROM ingested_files + WHERE source_name = ? + AND file_date IS NULL + AND datetime(ingested_at) >= datetime(?) + AND datetime(ingested_at) <= datetime(?) + AND file_name != '__seeding_attempted__' + GROUP BY 1 + ) + GROUP BY bucket HAVING bucket IS NOT NULL AND bucket >= ? AND bucket <= ? """, - (width, width, service_id, sql_start, sql_end, start_bucket, end_bucket), + ( + width, + service_id, + start_date, + end_date, + width, + width, + service_id, + sql_start, + sql_end, + start_bucket, + end_bucket, + ), ).fetchall() return {r["bucket"]: (int(r["rows"] or 0), int(r["files"] or 0)) for r in rows} @@ -1322,12 +1449,13 @@ def insert_ingested_files(service_id: str, rows: list[tuple[str, int, int | None count_with_bytes_delta += 1 con.executemany( - """INSERT INTO ingested_files (file_name, source_name, row_count, file_size_bytes) - VALUES (?, ?, ?, ?) + """INSERT INTO ingested_files (file_name, source_name, row_count, file_size_bytes, file_date) + VALUES (?, ?, ?, ?, ?) ON CONFLICT(file_name, source_name) DO UPDATE SET row_count = excluded.row_count, - file_size_bytes = excluded.file_size_bytes""", - [(fn, service_id, rc, sz) for (fn, rc, sz) in rows], + file_size_bytes = excluded.file_size_bytes, + file_date = COALESCE(ingested_files.file_date, excluded.file_date)""", + [(fn, service_id, rc, sz, _parse_file_date(fn)) for (fn, rc, sz) in rows], ) # Use the just-applied DB clock so last_ingested matches the row's # ingested_at default (datetime('now')) — keeps the rollup honest. @@ -1445,20 +1573,62 @@ def get_log_activity(service_id: str, start_iso: str, end_iso: str, by: str) -> width = width_map.get(by, 13) con = get_con(service_id) - rows = con.execute( - f""" - SELECT substr(replace(ingested_at, ' ', 'T'), 1, {width}) AS bucket, - sum(row_count) AS rc, - sum(file_size_bytes) AS bs - FROM ingested_files - WHERE source_name = ? - AND file_name != '__seeding_attempted__' - AND ingested_at >= ? - AND ingested_at <= ? - GROUP BY bucket ORDER BY bucket - """, - (service_id, start_iso, end_iso), - ).fetchall() + # Day-bucket path uses the file_date column + composite + # idx_ingested_files_source_date index added by _migration_002. + # Skips the per-row substr() on ingested_at + uses an index range + # scan instead of a full source-partition walk. Falls back to the + # substr path for rows where file_date is NULL (filenames that + # don't match the canonical Fastly YYYY-MM-DDTHH:MM:SS format) so + # legacy data without parseable basenames still counts. The non-day + # buckets keep the original shape because file_date has only date + # granularity. + if by == "day": + start_date = start_iso[:10] + end_date = end_iso[:10] + rows = con.execute( + """ + SELECT bucket, sum(rc) AS rc, sum(bs) AS bs FROM ( + SELECT file_date AS bucket, + sum(row_count) AS rc, + sum(file_size_bytes) AS bs + FROM ingested_files + WHERE source_name = ? + AND file_date IS NOT NULL + AND file_date >= ? + AND file_date <= ? + AND file_name != '__seeding_attempted__' + GROUP BY file_date + UNION ALL + SELECT substr(replace(ingested_at, ' ', 'T'), 1, 10) AS bucket, + sum(row_count) AS rc, + sum(file_size_bytes) AS bs + FROM ingested_files + WHERE source_name = ? + AND file_date IS NULL + AND file_name != '__seeding_attempted__' + AND ingested_at >= ? + AND ingested_at <= ? + GROUP BY bucket + ) + GROUP BY bucket ORDER BY bucket + """, + (service_id, start_date, end_date, service_id, start_iso, end_iso), + ).fetchall() + else: + rows = con.execute( + f""" + SELECT substr(replace(ingested_at, ' ', 'T'), 1, {width}) AS bucket, + sum(row_count) AS rc, + sum(file_size_bytes) AS bs + FROM ingested_files + WHERE source_name = ? + AND file_name != '__seeding_attempted__' + AND ingested_at >= ? + AND ingested_at <= ? + GROUP BY bucket ORDER BY bucket + """, + (service_id, start_iso, end_iso), + ).fetchall() def _normalize(bucket: str) -> str: if by == "hour": @@ -1496,17 +1666,39 @@ def get_node_count_avg(service_id: str) -> float | None: (bucket/prefix segments are lowercase + numeric). Grouping by that 19-char substring is equivalent to the prior Python regex over file_name, but runs entirely in SQLite instead of dragging every row across the boundary. + + Fast/slow split (mirrors ``get_log_accounting_counts``): the fast arm + filters on ``file_date IS NOT NULL``, which is covered by the composite + ``idx_ingested_files_source_date`` index — lets SQLite walk only the + canonical-basename rows directly via the index instead of scanning the + full source partition and per-row evaluating ``instr(file_name, 'T')``. + The slow arm keeps the ``instr`` guard for rows with NULL file_date + (legacy / test / ad-hoc backfills) so the average stays semantically + equivalent to the pre-change behavior. """ con = get_con(service_id) row = con.execute( """SELECT avg(c) AS avg_c FROM ( + -- Fast arm: file_date IS NOT NULL implies the basename matches + -- the canonical Fastly pattern per _migration_002, so the + -- substr group-by always succeeds without an instr() guard. SELECT count(*) AS c FROM ingested_files WHERE source_name = ? + AND file_date IS NOT NULL + GROUP BY substr(file_name, instr(file_name, 'T') - 10, 19) + UNION ALL + -- Slow arm: rows without a parseable basename. Typically + -- zero rows in prod but kept so test fixtures + legacy + -- uploads still contribute to the average. + SELECT count(*) AS c + FROM ingested_files + WHERE source_name = ? + AND file_date IS NULL AND instr(file_name, 'T') >= 11 GROUP BY substr(file_name, instr(file_name, 'T') - 10, 19) )""", - (service_id,), + (service_id, service_id), ).fetchone() if not row or row["avg_c"] is None: return None @@ -1792,8 +1984,19 @@ def get_cron_runs( per_page: int = 50, sort_col: str = "started_at", sort_dir: str = "DESC", + since_id: int | None = None, ) -> tuple[int, list[dict]]: - """Paginated cron run history. Used by repositories/cron.py.""" + """Paginated cron run history. Used by repositories/cron.py. + + ``since_id`` enables delta polling: when provided, rows are returned only + if ``id > since_id`` OR ``status = 'running'``. The ``status = 'running'`` + branch keeps long-lived in-progress runs visible across polls (otherwise + a sync that started 60 s ago would drop out once its id <= since_id), + AND keeps the row visible for the single poll where it transitions from + running to completed (so the client can observe the status change and + update its toast). Once a row is observed completed (id <= since_id AND + status != 'running'), it falls out of the response. + """ con = get_con(service_id) where: list[str] = [] params: list = [] @@ -1803,6 +2006,9 @@ def get_cron_runs( if status and status != "all": where.append("status = ?") params.append(status) + if since_id is not None: + where.append("(id > ? OR status = 'running')") + params.append(since_id) where_sql = ("WHERE " + " AND ".join(where)) if where else "" total_row = con.execute(f"SELECT count(*) AS n FROM cron_runs {where_sql}", params).fetchone() @@ -1848,26 +2054,28 @@ def get_cron_runs( def latest_cron_per_task(service_id: str) -> dict[str, dict]: """Return {task: latest_completed_run_dict} for the sync-status endpoint. - The original `id IN (SELECT max(id) GROUP BY task)` form forced a full - scan + GROUP BY across cron_runs (210ms / 44K rows on prod). This rewrite - pulls the distinct task list (cheap — usually <10 tasks) and does one - btree-seek per task into `idx_cron_task_started(task, started_at)` to find - the latest non-`running` row, taking ~25ms. Result is identical because - ids and started_at are co-monotonic for the same task. + Single window-function pass: ROW_NUMBER() OVER (PARTITION BY task) keeps + the latest non-`running` row per task in one scan of the + `idx_cron_task_started(task, started_at)` index. The previous + DISTINCT-tasks + correlated-subquery shape did a btree-seek per task, + taking ~12.9 ms — fast in absolute terms but per-task overhead added + up on services with many task types. Mirrors the same pattern used + by `cron_summary_for_tasks` below. """ con = get_con(service_id) rows = con.execute( - """WITH tasks AS (SELECT DISTINCT task FROM cron_runs), - latest AS ( - SELECT t.task, ( - SELECT c2.id FROM cron_runs c2 - WHERE c2.task = t.task AND c2.status != 'running' - ORDER BY c2.started_at DESC LIMIT 1 - ) AS lid - FROM tasks t - ) - SELECT c.task, c.started_at, c.status, c.duration_s, c.summary, c.error_message - FROM cron_runs c JOIN latest l ON c.id = l.lid""" + """ + SELECT task, started_at, status, duration_s, summary, error_message + FROM ( + SELECT task, started_at, status, duration_s, summary, error_message, + ROW_NUMBER() OVER ( + PARTITION BY task ORDER BY started_at DESC, id DESC + ) AS rn + FROM cron_runs + WHERE status != 'running' + ) + WHERE rn = 1 + """ ).fetchall() return { r["task"]: { @@ -2357,8 +2565,13 @@ def _ensure_usage_log_hourly_backfilled(con: sqlite3.Connection, service_id: str con.execute( "INSERT OR REPLACE INTO applied_data_migrations " "(name, applied_at, duration_s, status, notes) VALUES (?, ?, ?, ?, ?)", - (USAGE_LOG_HOURLY_BACKFILL_NAME, iso_z_now(), time.time() - t0, "success", - "rebuilt usage_log_hourly_summary from raw"), + ( + USAGE_LOG_HOURLY_BACKFILL_NAME, + iso_z_now(), + time.time() - t0, + "success", + "rebuilt usage_log_hourly_summary from raw", + ), ) con.commit() logger.info("[usage_log] hourly backfill complete for %s in %.2fs", service_id, time.time() - t0) @@ -2464,11 +2677,14 @@ def _hour_start(hour_prefix: str) -> str: GROUP BY operation_class, operation_type """, # Interior rollup params - [service_id, start_hour, end_hour] + class_params + [service_id, start_hour, end_hour] + + class_params # Start-boundary raw params: [start, next_hour_after_start_hour) - + [service_id, start, start_hour_end] + class_params + + [service_id, start, start_hour_end] + + class_params # End-boundary raw params: [start_of_end_hour, end] - + [service_id, end_hour_start, end] + class_params, + + [service_id, end_hour_start, end] + + class_params, ).fetchall() return rows @@ -2510,14 +2726,26 @@ def get_usage_logs( params.append(f"%{operation_type}%") where = " AND ".join(conditions) - total = con.execute(f"SELECT count(*) FROM usage_log WHERE {where}", params).fetchone()[0] + # Fold COUNT(*) into the page query via a window function so we don't + # do two passes over the same (service_id, [start, end]) range. The + # previous separate COUNT + SELECT pair added ~40-60ms per page load. + # COUNT(*) OVER () is constant across rows so it's computed once + # during plan execution rather than per-row. offset = (page - 1) * page_size cur = con.execute( - f"SELECT * FROM usage_log WHERE {where} ORDER BY timestamp DESC LIMIT ? OFFSET ?", + f"SELECT *, COUNT(*) OVER () AS _total FROM usage_log WHERE {where} ORDER BY timestamp DESC LIMIT ? OFFSET ?", params + [page_size, offset], ) - entries = [dict(r) for r in cur.fetchall()] + raw_rows = cur.fetchall() + if raw_rows: + total = int(raw_rows[0]["_total"] or 0) + entries = [{k: v for k, v in dict(r).items() if k != "_total"} for r in raw_rows] + else: + # Empty page (no matching rows OR past the last page): fall back + # to a cheap exact COUNT so totals stay correct for pagination UX. + total = con.execute(f"SELECT count(*) FROM usage_log WHERE {where}", params).fetchone()[0] + entries = [] # Aggregate path: prefer the usage_log_hourly_summary rollup when only the # service+timestamp predicates are active (the common admin-page case). The diff --git a/backend/core/rollups.py b/backend/core/rollups.py index 6b65ca09..11892ad5 100644 --- a/backend/core/rollups.py +++ b/backend/core/rollups.py @@ -59,12 +59,23 @@ def _is_safe_ident(name: str) -> bool: def _safe_table_for(source: dict) -> str | None: - """Return ``logs_`` iff the service name is a safe identifier.""" - name = source.get("name") or "" - if not _is_safe_ident(name): - logger.warning("[rollups] refusing to query unsafe service name: %r", name) + """Return the DuckDB view name for this service, or ``None`` if no slug. + + Slugifies the same way the dashboard's view-builder does + (``backend.core.duckdb._safe_table_name``: non-alphanumerics to ``_``, + lowercased, ``logs_`` prefix) so the rollup COPY/SELECT targets the + same view name the dashboard creates. Reads ``service_id`` first (the + canonical slug in normalized source dicts) and falls back to ``name`` + for callers that pass a raw on-disk config — both cases pass through + the slugifier identically. + """ + raw = source.get("service_id") or source.get("name") or "" + if not raw: + logger.warning("[rollups] no service_id/name in source dict; skipping rollup") return None - return f"logs_{name}" + from backend.core.duckdb import _safe_table_name + + return _safe_table_name(raw) def _get_fields(src: dict) -> list[str]: @@ -97,6 +108,21 @@ def _rollups_root(source: dict) -> str: return os.path.join(_cache_dir(source), "rollups", "hour") +def _day_rollups_root(source: dict) -> str: + """Per-day compacted rollups directory. + + Companion to `_rollups_root` (which holds per-hour rollups). Populated + by `compact_closed_days_to_daily` — each (field, closed-day) becomes + a single parquet file aggregating its 24 source hour parquets. The + reader (`execute_top_n_rollups`) prefers per-day files for closed + days and falls back to per-hour for the active trailing window. + Item 17 / RC-9. + """ + from backend.core.duckdb import _cache_dir + + return os.path.join(_cache_dir(source), "rollups", "day") + + def _markers_path(source: dict) -> str: """JSON file tracking which fields have been backfilled. @@ -213,6 +239,373 @@ def _build_copy_query(table_ident: str, field: str, where_sql: str) -> str: """ +def _hour_bundled_root(source: dict) -> str: + """Return the per-hour bundled rollup root. + + Layout: cache//rollups/hour_bundled/hour=YYYY-MM-DD-HH/all_fields.parquet + Each bundle contains rows for ALL fields for that hour with the same + (field, value, count) schema as the per-field hour parquets. Reading + one bundle replaces opening ~40+ per-field files for that hour. + + The same hour directory also holds ``time_series.parquet`` — see + :func:`build_time_series_bundles` for the schema. + """ + from backend.core.duckdb import _cache_dir + + return os.path.join(_cache_dir(source), "rollups", "hour_bundled") + + +# Filename for the per-hour 1-minute time-series rollup. Kept as a constant +# so the writer + reader can never drift on the name. +TIME_SERIES_BUNDLE_FILENAME = "time_series.parquet" + + +def _time_series_bundle_path(source: dict, hour: str) -> str: + return os.path.join(_hour_bundled_root(source), f"hour={hour}", TIME_SERIES_BUNDLE_FILENAME) + + +def build_time_series_bundles(service_id: str, source: dict, hours: list[str]) -> int: + """Write a 1-minute time_series rollup for each closed hour in ``hours``. + + Output: ``rollups/hour_bundled/hour=H/time_series.parquet`` with one row + per UTC minute and SUM-aggregatable metric columns. Re-bucketing at read + time to 5/15/60 minutes works as ``SELECT SUM(...) GROUP BY + time_bucket(...)`` without any sketch. + + Schema (all columns SUM-aggregatable): + bucket TIMESTAMP -- minute floor in UTC + requests BIGINT -- COUNT(*) + status_4xx BIGINT -- COUNT(*) WHERE status BETWEEN 400 AND 499 + status_5xx BIGINT -- COUNT(*) WHERE status >= 500 + hits BIGINT -- COUNT(*) WHERE cache IN ('HIT','HIT-STALE') + cache_total BIGINT -- COUNT(*) WHERE cache IS NOT NULL + resp_bytes_sum BIGINT -- SUM(resp_bytes) + ttfb_sum DOUBLE -- SUM(ttfb), seconds + ttfb_count BIGINT -- COUNT(*) WHERE ttfb IS NOT NULL + + Columns that map to a backing column missing from this service's + schema are written as constant 0 so the file shape stays uniform + across services (the reader uses NULLIF on the denominator). + + Skips the active UTC hour — that hour is still being written and the + dashboard serves it live off the base table. + + Idempotent (atomic tmp + rename). Returns the number of bundles + written this call. + """ + if not hours: + return 0 + + import duckdb + + from backend.core.duckdb import get_connection + + from backend.core.iceberg import _get_service_lock + + active_hour = datetime.now(UTC).strftime("%Y-%m-%d-%H") + target_hours: list[str] = [] + for h in hours: + if h == active_hour: + continue + try: + datetime.strptime(h, "%Y-%m-%d-%H") + except ValueError: + logger.warning("[rollups] skipping malformed hour token: %r", h) + continue + target_hours.append(h) + if not target_hours: + return 0 + + table_ident = _safe_table_for(source) + if not table_ident: + return 0 + + bundled_root = _hour_bundled_root(source) + os.makedirs(bundled_root, exist_ok=True) + lock_key = source.get("name", "default") + + con = get_connection(source=source, read_only=True) + try: + try: + cols = {c[0] for c in con.execute(f"DESCRIBE {table_ident}").fetchall()} + except duckdb.Error as e: + logger.warning( + "[rollups] %s: cannot describe %s for time_series bundle: %s", + service_id, table_ident, e, + ) + return 0 + + if "timestamp" not in cols: + logger.warning( + "[rollups] %s: no `timestamp` column on %s; skipping time_series bundle", + service_id, table_ident, + ) + return 0 + + # Build the SELECT, adapting each metric to whether its backing + # column actually exists on this service's schema. Missing-column + # rows surface as constant 0 so the parquet shape stays uniform + # (the reader divides via NULLIF, so 0 cache_total → NULL hit_rate). + select_parts = [ + "time_bucket(INTERVAL '1 minute', timestamp) AS bucket", + "CAST(COUNT(*) AS BIGINT) AS requests", + ] + if "status" in cols: + select_parts.append( + "CAST(COUNT(*) FILTER (WHERE status BETWEEN 400 AND 499) AS BIGINT) AS status_4xx" + ) + select_parts.append( + "CAST(COUNT(*) FILTER (WHERE status >= 500) AS BIGINT) AS status_5xx" + ) + else: + select_parts.append("CAST(0 AS BIGINT) AS status_4xx") + select_parts.append("CAST(0 AS BIGINT) AS status_5xx") + + if "cache" in cols: + select_parts.append( + "CAST(COUNT(*) FILTER (WHERE cache IN ('HIT', 'HIT-STALE')) AS BIGINT) AS hits" + ) + select_parts.append( + "CAST(COUNT(*) FILTER (WHERE cache IS NOT NULL) AS BIGINT) AS cache_total" + ) + else: + select_parts.append("CAST(0 AS BIGINT) AS hits") + select_parts.append("CAST(0 AS BIGINT) AS cache_total") + + if "resp_bytes" in cols: + select_parts.append("CAST(COALESCE(SUM(resp_bytes), 0) AS BIGINT) AS resp_bytes_sum") + else: + select_parts.append("CAST(0 AS BIGINT) AS resp_bytes_sum") + + if "ttfb" in cols: + select_parts.append("CAST(COALESCE(SUM(ttfb), 0.0) AS DOUBLE) AS ttfb_sum") + select_parts.append( + "CAST(COUNT(*) FILTER (WHERE ttfb IS NOT NULL) AS BIGINT) AS ttfb_count" + ) + else: + select_parts.append("CAST(0.0 AS DOUBLE) AS ttfb_sum") + select_parts.append("CAST(0 AS BIGINT) AS ttfb_count") + + select_sql = ",\n ".join(select_parts) + + rebuilt = 0 + for hour in target_hours: + hour_dt = datetime.strptime(hour, "%Y-%m-%d-%H").replace(tzinfo=UTC) + start_iso = hour_dt.isoformat() + end_iso = (hour_dt + timedelta(hours=1)).isoformat() + + bundle_dir = os.path.join(bundled_root, f"hour={hour}") + os.makedirs(bundle_dir, exist_ok=True) + bundle_path = os.path.join(bundle_dir, TIME_SERIES_BUNDLE_FILENAME) + + tmp_path = os.path.join(bundle_dir, f".tmp_ts_{uuid.uuid4().hex[:12]}.parquet") + query = ( + f"COPY (SELECT {select_sql} " + f"FROM {table_ident} " + f"WHERE timestamp >= TIMESTAMPTZ '{start_iso}' " + f"AND timestamp < TIMESTAMPTZ '{end_iso}' " + f"GROUP BY 1) " + f"TO '{tmp_path}' (FORMAT PARQUET, COMPRESSION ZSTD)" + ) + try: + con.execute(query) + except duckdb.Error as e: + logger.warning( + "[rollups] %s: time_series COPY failed for hour=%s: %s", + service_id, hour, e, + ) + try: + os.remove(tmp_path) + except OSError: + pass + continue + + try: + with _get_service_lock(lock_key): + os.replace(tmp_path, bundle_path) + rebuilt += 1 + except OSError as e: + logger.warning( + "[rollups] %s: could not publish time_series for hour=%s: %s", + service_id, hour, e, + ) + try: + os.remove(tmp_path) + except OSError: + pass + + return rebuilt + finally: + con.close() + + +def backfill_time_series_bundles( + service_id: str, source: dict, max_hours: int | None = None +) -> int: + """One-shot bulk build of time_series.parquet for closed hours that + don't yet have one. + + Mirrors :func:`backfill_hour_bundles`: walks the per-field rollup tree + to discover closed hours (those that have any per-field rollup + written), then calls :func:`build_time_series_bundles` on the subset + that doesn't already have a time_series file. + """ + hour_root = _rollups_root(source) + bundled_root = _hour_bundled_root(source) + if not os.path.isdir(hour_root): + return 0 + + active_hour = datetime.now(UTC).strftime("%Y-%m-%d-%H") + all_hours: set[str] = set() + try: + for field_entry in os.listdir(hour_root): + if not field_entry.startswith("field="): + continue + field_dir = os.path.join(hour_root, field_entry) + try: + for hour_entry in os.listdir(field_dir): + if not hour_entry.startswith("hour="): + continue + hour = hour_entry[len("hour=") :] + if hour >= active_hour: + continue + all_hours.add(hour) + except OSError: + continue + except OSError: + return 0 + + to_build: list[str] = [] + for hour in sorted(all_hours): + ts_path = os.path.join(bundled_root, f"hour={hour}", TIME_SERIES_BUNDLE_FILENAME) + if not os.path.exists(ts_path): + to_build.append(hour) + if max_hours and len(to_build) >= max_hours: + break + + if not to_build: + return 0 + return build_time_series_bundles(service_id, source, to_build) + + +def bundle_hours(service_id: str, source: dict, hours: list[str]) -> int: + """Combine per-field hour parquets into one bundled parquet per hour. + + For each hour token, reads every per-field parquet under + rollups/hour/field=*/hour=H/*.parquet and writes a single bundled file + at rollups/hour_bundled/hour=H/all_fields.parquet. + + Skips hours where: + - No per-field files exist (nothing to bundle). + - A bundled file already exists and is fresh enough to skip rebuild + (per-field mtime <= bundle mtime). + + Returns the count of hours that were rebuilt. + + Skip the active hour — bundles for in-progress hours would race the + sync's per-field rebuilds. The active hour is served live anyway. + """ + if not hours: + return 0 + + import duckdb + + from backend.core.iceberg import _get_service_lock + + # _rollups_root already returns /rollups/hour — it's the + # per-field per-hour tree root, not the rollups/ parent. + hour_per_field_root = _rollups_root(source) + bundled_root = _hour_bundled_root(source) + os.makedirs(bundled_root, exist_ok=True) + lock_key = source.get("name", "default") + active_hour = datetime.now(UTC).strftime("%Y-%m-%d-%H") + + rebuilt = 0 + # Use :memory: DuckDB to avoid contending with uvicorn's RW connection + # on the per-service .duckdb file (mirrors compact_closed_days_to_daily — + # see the 2026-06-06 incident comment in that function). The bundling + # COPY only needs to read existing parquets and write a new one; it + # doesn't need any per-service catalog state. + con = duckdb.connect(":memory:") + try: + for hour in hours: + if hour == active_hour: + continue + # Validate hour token format defensively — string lands in + # filesystem paths and SQL string literals below. + try: + datetime.strptime(hour, "%Y-%m-%d-%H") + except ValueError: + continue + + # Enumerate per-field parquets for this hour. + per_field_paths: list[str] = [] + max_src_mtime = 0.0 + try: + for field_entry in os.listdir(hour_per_field_root): + if not field_entry.startswith("field="): + continue + hour_dir = os.path.join(hour_per_field_root, field_entry, f"hour={hour}") + if not os.path.isdir(hour_dir): + continue + for fname in os.listdir(hour_dir): + if not fname.endswith(".parquet") or fname.startswith(".tmp_"): + continue + p = os.path.join(hour_dir, fname) + per_field_paths.append(p) + try: + mt = os.path.getmtime(p) + if mt > max_src_mtime: + max_src_mtime = mt + except OSError: + pass + except OSError: + continue + + if not per_field_paths: + continue + + # Skip if bundle is already up-to-date. + bundle_dir = os.path.join(bundled_root, f"hour={hour}") + bundle_path = os.path.join(bundle_dir, "all_fields.parquet") + if os.path.exists(bundle_path): + try: + if os.path.getmtime(bundle_path) >= max_src_mtime: + continue + except OSError: + pass + + os.makedirs(bundle_dir, exist_ok=True) + tmp_path = os.path.join(bundle_dir, f".tmp_{uuid.uuid4().hex[:12]}.parquet") + paths_sql = ", ".join("'" + p.replace("'", "''") + "'" for p in per_field_paths) + # Read the per-field parquets (each has columns field/value/count) + # and write to a single bundled parquet. Use COPY for atomicity + # via the tmp + rename pattern. + query = ( + f"COPY (SELECT field, value, CAST(count AS BIGINT) AS count " + f"FROM read_parquet([{paths_sql}])) " + f"TO '{tmp_path}' (FORMAT PARQUET, COMPRESSION ZSTD)" + ) + try: + con.execute(query) + except duckdb.Error as e: + logger.warning("[rollups] %s: bundle COPY failed for hour=%s: %s", service_id, hour, e) + try: + os.remove(tmp_path) + except OSError: + pass + continue + + with _get_service_lock(lock_key): + # Atomic publish — os.replace is atomic on POSIX. + os.replace(tmp_path, bundle_path) + rebuilt += 1 + finally: + con.close() + + return rebuilt + + def recompute_touched_hours(service_id: str, source: dict, hours: set[str]) -> None: """Recompute rollups for all dashboard fields across the given hours. @@ -220,6 +613,11 @@ def recompute_touched_hours(service_id: str, source: dict, hours: set[str]) -> N in-progress hour live off the base table. One COPY query per field handles all touched hours via PARTITION_BY, so the work is O(fields) not O(fields × hours). + + After the per-field rebuild completes, bundles each touched hour's + per-field parquets into a single bundled file under + ``rollups/hour_bundled/hour=H/all_fields.parquet`` so the dashboard + reader can open one file per hour instead of ~40 per-field files. """ if not hours: return @@ -250,6 +648,94 @@ def recompute_touched_hours(service_id: str, source: dict, hours: set[str]) -> N ) _run_per_field_copy(service_id, source, table_ident, where_sql, _get_fields(source)) + # Bundle the touched hours so the dashboard reader can open one + # file per hour instead of N per-field files. Best-effort: if + # bundling fails, the per-field files still serve correctly via + # the reader's fallback path. + touched_hours = [h for h, _ in parsed] + try: + bundle_hours(service_id, source, touched_hours) + except Exception as e: + logger.warning("[rollups] %s: hour bundling failed (per-field still serves): %s", service_id, e) + + # Time-series rollups for the dashboard chart. Same best-effort + # contract: if the build fails, the dashboard falls back to a raw + # scan for the affected hours. + try: + build_time_series_bundles(service_id, source, touched_hours) + except Exception as e: + logger.warning( + "[rollups] %s: time_series bundle failed (raw scan will serve): %s", + service_id, e, + ) + + +def backfill_hour_bundles(service_id: str, source: dict, max_hours: int | None = None) -> int: + """One-shot bulk bundling for all closed hours that don't yet have a + per-hour bundled file. + + Walks the existing rollups/hour/field=*/hour=*/ tree, collects the set + of closed hours, and calls bundle_hours() on any that lack an up-to- + date bundle. Safe to call on startup and idempotent — bundle_hours + skips up-to-date hours via mtime comparison. + + ``max_hours``: if set, caps the number of hours processed per call + (useful for incremental backfills if running synchronously would + block startup too long). + """ + # _rollups_root already returns /rollups/hour — see comment + # in bundle_hours about the naming. + hour_root = _rollups_root(source) + bundled_root = _hour_bundled_root(source) + if not os.path.isdir(hour_root): + return 0 + + active_hour = datetime.now(UTC).strftime("%Y-%m-%d-%H") + all_hours: set[str] = set() + try: + for field_entry in os.listdir(hour_root): + if not field_entry.startswith("field="): + continue + field_dir = os.path.join(hour_root, field_entry) + try: + for hour_entry in os.listdir(field_dir): + if not hour_entry.startswith("hour="): + continue + hour = hour_entry[len("hour=") :] + if hour >= active_hour: + continue + all_hours.add(hour) + except OSError: + continue + except OSError: + return 0 + + # Skip hours that already have a bundle. + to_bundle = [] + for hour in sorted(all_hours): + bundle_path = os.path.join(bundled_root, f"hour={hour}", "all_fields.parquet") + if not os.path.exists(bundle_path): + to_bundle.append(hour) + if max_hours and len(to_bundle) >= max_hours: + break + + if not to_bundle: + rebuilt = 0 + else: + rebuilt = bundle_hours(service_id, source, to_bundle) + + # Also catch up the time-series bundles. Walks the same hour set and + # only writes for hours that don't yet have time_series.parquet. + try: + backfill_time_series_bundles(service_id, source, max_hours=max_hours) + except Exception as e: + logger.warning( + "[rollups] %s: time_series backfill failed (raw scan will serve): %s", + service_id, e, + ) + + return rebuilt + def backfill_rollups(service_id: str, source: dict, fields: list[str] | None = None) -> None: """One-shot bulk build for all historical hours up to (but not including) @@ -401,3 +887,150 @@ def _run_per_field_copy( shutil.rmtree(tmp_field_dir, ignore_errors=True) finally: con.close() + + +# ── Closed-day compaction (item 17 / RC-9) ────────────────────────────────── + + +def compact_closed_days_to_daily(service_id: str, source: dict) -> int: + """Consolidate closed-day per-hour rollup parquet into per-day parquet. + + For each (field, closed-day) tuple where either (a) no per-day parquet + exists, or (b) some constituent per-hour parquet has a newer mtime + than the per-day parquet, rebuild the per-day parquet by summing the + 24 hour parquets into one. Active (current UTC) day is always skipped + — it's still being written. + + The per-day file is written via DuckDB COPY to a temp path and + renamed into place under the per-service iceberg lock so concurrent + `execute_top_n_rollups` readers never see a half-written file. On + failure the per-day file is left in its previous state and the + reader transparently falls back to per-hour parquet. + + Returns the count of (field, day) tuples that were rebuilt. + + Operators can call this from a maintenance script or wire it into a + daily cron. The reader works whether or not this has ever run — when + a per-day file is missing, `execute_top_n_rollups` reads the source + per-hour files. When present, it reads ONE file per closed day per + field instead of 24, slashing the file-open overhead that dominates + dashboard cold-load wall time on 7-day queries (1,512 → 30-some + files per the local audit). + """ + import duckdb + + from backend.core.iceberg import _get_service_lock + + hour_root = _rollups_root(source) + day_root = _day_rollups_root(source) + if not os.path.isdir(hour_root): + return 0 + + active_day = datetime.now(UTC).strftime("%Y-%m-%d") + lock_key = source.get("name", "default") + rebuilt = 0 + + # In-memory DuckDB — we only need it to run COPY against parquet files + # on the local filesystem. Opening the per-service ``.duckdb`` file + # would contend with uvicorn's RW connection on the SAME file (held + # for view rebuilds), since DuckDB does not allow mixed RW+RO from + # one path. On the 2026-06-06 prod incident an RO ``get_connection`` + # blocked 5+ minutes on that lock and the compaction never produced + # any per-day files. ``:memory:`` sidesteps the contention entirely + # — the compaction reads + writes parquet via DuckDB's I/O layer, + # never touching any persistent DuckDB database. + con = duckdb.connect(":memory:") + try: + for field_entry in sorted(os.listdir(hour_root)): + if not field_entry.startswith("field="): + continue + field = field_entry[len("field=") :] + if not _is_safe_ident(field): + continue + field_hour_dir = os.path.join(hour_root, field_entry) + # Bucket hour-dirs by their YYYY-MM-DD prefix. + by_day: dict[str, list[str]] = {} + try: + hour_entries = os.listdir(field_hour_dir) + except OSError: + continue + for hour_entry in hour_entries: + if not hour_entry.startswith("hour="): + continue + hour = hour_entry[len("hour=") :] + # hour shape: YYYY-MM-DD-HH — first 10 chars are the day. + if len(hour) < 13: + continue + day = hour[:10] + if day == active_day: + continue + hour_dir = os.path.join(field_hour_dir, hour_entry) + try: + for fname in os.listdir(hour_dir): + if fname.endswith(".parquet"): + by_day.setdefault(day, []).append(os.path.join(hour_dir, fname)) + except OSError: + continue + + for day, hour_paths in by_day.items(): + if not hour_paths: + continue + day_dir = os.path.join(day_root, field_entry, f"day={day}") + day_file = os.path.join(day_dir, "compacted.parquet") + # Skip if the per-day file is newer than every source hour + # parquet — already up to date. + try: + day_mtime = os.path.getmtime(day_file) + max_hour_mtime = max(os.path.getmtime(p) for p in hour_paths) + if day_mtime >= max_hour_mtime: + continue + except OSError: + pass # day file missing → rebuild + + tmp_file = os.path.join(day_dir, f".tmp_{uuid.uuid4().hex[:12]}.parquet") + os.makedirs(day_dir, exist_ok=True) + paths_sql = ", ".join("'" + p.replace("'", "''") + "'" for p in hour_paths) + # CAST to BIGINT so the per-day file's count column matches + # the per-hour files (which are BIGINT). The reader's + # UNION ALL of day + hour requires matching column types + # per column; without this CAST, the day file lands as + # DOUBLE and the union breaks (and the dashboard top-N + # tabs go blank — 2026-06-06 incident). + copy_sql = f""" + COPY ( + SELECT field, value, CAST(SUM(count) AS BIGINT) AS count + FROM read_parquet([{paths_sql}], hive_partitioning=1) + GROUP BY field, value + ) TO '{tmp_file}' + (FORMAT PARQUET, COMPRESSION ZSTD) + """ + try: + con.execute(copy_sql) + except duckdb.Error as e: + logger.warning( + "[rollups] %s: day-compact COPY failed for %s/%s: %s", + service_id, + field, + day, + e, + ) + try: + os.remove(tmp_file) + except OSError: + pass + continue + + with _get_service_lock(lock_key): + try: + os.replace(tmp_file, day_file) + rebuilt += 1 + except OSError as e: + logger.warning("[rollups] %s: rename to %s failed: %s", service_id, day_file, e) + try: + os.remove(tmp_file) + except OSError: + pass + finally: + con.close() + + return rebuilt diff --git a/backend/core/share_db.py b/backend/core/share_db.py index 74b9ead4..c6eb622e 100644 --- a/backend/core/share_db.py +++ b/backend/core/share_db.py @@ -492,90 +492,10 @@ def validate_passcode_strength(passcode: str) -> None: # ── Wordphrase generator ───────────────────────────────────────────────────── -_WORDS_A = [ - "ocean", - "sunset", - "river", - "forest", - "mountain", - "thunder", - "crystal", - "ember", - "silver", - "amber", - "harbor", - "meadow", - "canyon", - "lantern", - "horizon", - "ranger", - "summit", - "twilight", - "marble", - "boulder", -] -_WORDS_B = [ - "breeze", - "shadow", - "spark", - "ridge", - "drift", - "tide", - "ember", - "flame", - "echo", - "wave", - "stream", - "trail", - "fern", - "creek", - "field", - "willow", - "pine", - "cedar", - "moss", - "stone", -] -_WORDS_C = [ - "cabin", - "harbor", - "pier", - "vault", - "bridge", - "tower", - "garden", - "alcove", - "lodge", - "valley", - "trail", - "cove", - "ridge", - "field", - "anchor", - "haven", - "atelier", - "outpost", - "studio", - "lighthouse", -] - def generate_wordphrase() -> str: - """Three random words + two digits, separated by dashes. - - Approx entropy: log2(20^3 * 100) ≈ 12.97 + 6.64 ≈ 19.6 bits over the - fixed-vocabulary alphabet — but the resulting 15-25 char ASCII string - sails past the 10-char / no-digits / no-breached-list bar that the - validator enforces. For production raise this against a 4096-word list. - """ - return "-".join( - [ - secrets.choice(_WORDS_A), - secrets.choice(_WORDS_B), - secrets.choice(_WORDS_C), - f"{secrets.randbelow(100):02d}", - ] - ) + """Secure random string with >100 bits of entropy.""" + return f"{secrets.token_hex(4)}-{secrets.token_hex(4)}-{secrets.token_hex(4)}-{secrets.token_hex(4)}" # ── Name / email validation (XSS hardening, Section #19a) ─────────────────── @@ -1065,6 +985,16 @@ def delete_session(session_id: str, *, con: sqlite3.Connection | None = None) -> con.commit() +def get_session(session_id: str, *, con: sqlite3.Connection | None = None) -> dict | None: + con = con or get_global_share_con() + row = con.execute("SELECT * FROM remote_sessions WHERE session_id=?", (session_id,)).fetchone() + if row is None: + return None + rec = dict(row) + rec["pii_policy"] = json.loads(rec.get("pii_policy") or "{}") + return rec + + def get_all_sessions(*, con: sqlite3.Connection | None = None) -> list[dict]: con = con or get_global_share_con() rows = con.execute("SELECT * FROM remote_sessions").fetchall() diff --git a/backend/core/sqlite_migrations.py b/backend/core/sqlite_migrations.py index e76cd91b..f788c3c4 100644 --- a/backend/core/sqlite_migrations.py +++ b/backend/core/sqlite_migrations.py @@ -80,10 +80,76 @@ def _migration_001_add_ingested_files_error_count(con: sqlite3.Connection) -> No con.execute("ALTER TABLE ingested_files ADD COLUMN error_count INTEGER DEFAULT 0") +def _migration_002_add_ingested_files_file_date(con: sqlite3.Connection) -> None: + """Add ``ingested_files.file_date`` (DATE parsed from filename) + index. + + Backfills via the same GLOB-validated substr/instr pattern used at + runtime by ``get_log_accounting_counts``: locate the first 'T' in the + filename (the Fastly emit-time marker) and use the 10 chars before it + when they match YYYY-MM-DD. Filenames that don't match the canonical + Fastly basename get NULL — callers must treat the column as optional. + + The composite index ``(source_name, file_date)`` lets per-day usage + queries scan only the date range they need instead of walking every + row for the source and computing the date per-row via substr — which + the existing ``(source_name, ingested_at)`` index can't help with + because the bucket extraction wraps the column in a function. + """ + if not _has_column(con, "ingested_files", "file_date"): + con.execute("ALTER TABLE ingested_files ADD COLUMN file_date DATE") + con.execute( + """ + UPDATE ingested_files + SET file_date = substr(file_name, instr(file_name, 'T') - 10, 10) + WHERE file_date IS NULL + AND instr(file_name, 'T') >= 11 + AND substr(file_name, instr(file_name, 'T') - 10, 10) + GLOB '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]' + """ + ) + con.execute("CREATE INDEX IF NOT EXISTS idx_ingested_files_source_date ON ingested_files(source_name, file_date)") + + +def _migration_003_rebuild_usage_log_hourly_summary(con: sqlite3.Connection) -> None: + """Rebuild ``usage_log_hourly_summary`` from raw ``usage_log``. + + The v0-v2 rollup is corrupted on any DB that has run + ``reconcile_fastly_stats``: the INSERT-only trigger never accounted for + the per-hour DELETE+INSERT refresh cycle, so RECONCILE_A/B contributions + accumulated across passes — 30-60x inflation observed in prod. The + matching DELETE/UPDATE triggers ship in ``_SCHEMA`` and are already + present by the time this migration runs (``_init_schema`` runs the + schema pass before ``apply_pending``). + """ + if not _has_table(con, "usage_log_hourly_summary") or not _has_table(con, "usage_log"): + return + con.execute("DELETE FROM usage_log_hourly_summary") + con.execute( + """ + INSERT INTO usage_log_hourly_summary + (service_id, hour, operation_class, operation_type, count, bytes, last_updated) + SELECT service_id, + substr(timestamp, 1, 13), + COALESCE(operation_class, ''), + COALESCE(operation_type, ''), + SUM(COALESCE(count, 1)), + SUM(COALESCE(bytes, 0)), + datetime('now') + FROM usage_log + WHERE service_id IS NOT NULL + AND timestamp IS NOT NULL + AND length(timestamp) >= 13 + GROUP BY 1, 2, 3, 4 + """ + ) + + # Insertion order = application order. Use integer keys; gaps are not # allowed (`apply_pending` iterates sorted keys and stops on failure). MIGRATIONS: dict[int, Callable[[sqlite3.Connection], None]] = { 1: _migration_001_add_ingested_files_error_count, + 2: _migration_002_add_ingested_files_file_date, + 3: _migration_003_rebuild_usage_log_hourly_summary, } LATEST_VERSION = max(MIGRATIONS) if MIGRATIONS else 0 diff --git a/backend/cron_progress.py b/backend/cron_progress.py index 00176fcf..c65c6658 100644 --- a/backend/cron_progress.py +++ b/backend/cron_progress.py @@ -134,10 +134,12 @@ def add_progress(run_id: int, event: dict): _last_update[run_id] = time.time() -def get_progress(run_id: int, start_idx: int = 0) -> list[dict] | None: +def get_progress(run_id: int, start_idx: int = 0, service_id: str | None = None) -> list[dict] | None: with _lock: if run_id not in _progress: return None + if service_id and _run_metadata.get(run_id, {}).get("service_id") != service_id: + return None # Return a copy of the slice to avoid race conditions when the caller iterates over it return list(_progress[run_id][start_idx:]) diff --git a/backend/deps.py b/backend/deps.py index 8e88b729..2a3c084d 100644 --- a/backend/deps.py +++ b/backend/deps.py @@ -101,11 +101,7 @@ def __enter__(self) -> duckdb.DuckDBPyConnection: # so behaviour matches the pre-pool design exactly. from backend.core import duckdb_pool - use_pool = ( - self._read_only - and not self._skip_view_update - and duckdb_pool._pool_enabled() - ) + use_pool = self._read_only and not self._skip_view_update and duckdb_pool._pool_enabled() try: if use_pool: self._pool_cm = duckdb_pool.checkout_connection(self._source, max_wait=10.0) diff --git a/backend/main.py b/backend/main.py index 239327e3..d0524b79 100644 --- a/backend/main.py +++ b/backend/main.py @@ -47,8 +47,8 @@ from fastapi import FastAPI, Request from fastapi.middleware.cors import CORSMiddleware -from fastapi.middleware.gzip import GZipMiddleware from fastapi.staticfiles import StaticFiles +from starlette_compress import CompressMiddleware # ── Path setup ──────────────────────────────────────────────────────────────── # Ensure the project root is on sys.path so the backend package is importable. @@ -403,7 +403,7 @@ async def lifespan(app: FastAPI): app = FastAPI( title="Fastly Log Analytics API", - version="1.1.0", + version="1.2.0", description=( "FastAPI backend for the Fastly Log Analytics tool. " "Serves the Next.js frontend and exposes an OpenAPI spec at /openapi.json." @@ -431,11 +431,17 @@ async def lifespan(app: FastAPI): app.add_middleware(RemoteAccessMiddleware) -# Gzip compression for analyst responses (responses to local-admin already -# transit loopback without compression benefit, but enabling globally is fine — -# the Starlette implementation skips SSE/streaming responses by content-type -# automatically). -app.add_middleware(GZipMiddleware, minimum_size=1024) +# M1 — telemetry backstop. Auto-injects _debug_queries / _debug_calls / +# _is_cached into JSON dict responses that don't already carry them, so +# a newly-added endpoint that returns a plain dict can't accidentally +# drop the Debug Panel for that request. MUST register INNER to Gzip +# (i.e. via add_middleware BEFORE the GZip line below — Starlette's +# stack treats later add_middleware calls as OUTER) so the body it +# reads isn't already compressed. Gated on DEBUG_RESPONSES, same flag +# BaseResponse uses; off by default in prod. +from backend.utils.telemetry_response_middleware import TelemetryResponseBodyMiddleware # noqa: E402 + +app.add_middleware(TelemetryResponseBodyMiddleware) @app.middleware("http") @@ -479,6 +485,22 @@ async def telemetry_middleware(request: Request, call_next): return response +# Brotli / zstd / gzip compression for analyst responses. CompressMiddleware +# negotiates the best available encoding from the client's Accept-Encoding +# header (zstd > br > gzip > identity). Skips text/event-stream (SSE) and +# any response already carrying a Content-Encoding header, so the streaming +# routers in routers/services/core.py and routers/provision.py pass through +# uncompressed. Registered LAST so it is the OUTERMOST middleware — the +# decorator-style telemetry_middleware above uses Starlette's +# BaseHTTPMiddleware, which buffers the response and re-emits it; that +# re-emit strips the Content-Encoding header from any inner middleware. +# Audit on 2026-06-09 confirmed every Accept-Encoding variant came back +# uncompressed (11490 B raw, no content-encoding) when Compress sat +# inside BaseHTTPMiddleware. Keeping it outermost preserves the encoded +# response all the way to the client. +app.add_middleware(CompressMiddleware, minimum_size=1024) + + # ── Routers ─────────────────────────────────────────────────────────────────── from backend.routers import alerts, dashboard, insights, network, origin, performance, query, security, sessions, views diff --git a/backend/models/common.py b/backend/models/common.py index 09c6b6c6..1f13ccd8 100644 --- a/backend/models/common.py +++ b/backend/models/common.py @@ -157,6 +157,12 @@ class BaseResponse(BaseModel): debug_queries: list[DebugQuery] = Field(default_factory=list, serialization_alias="_debug_queries") debug_calls: list[DebugCall] = Field(default_factory=list, serialization_alias="_debug_calls") is_cached: bool = Field(default=False, serialization_alias="_is_cached") + # Per-phase wall-clock timing for the handler. Always emitted as + # _section_timings under serialization. Default empty so endpoints + # that don't instrument get a benign empty list. Safe to surface in + # prod — phase names + millisecond timings are operational metadata, + # not SQL/URLs. + section_timings: list[dict] = Field(default_factory=list, serialization_alias="_section_timings") @model_serializer(mode="wrap") def _strip_debug_when_disabled(self, handler): @@ -164,6 +170,8 @@ def _strip_debug_when_disabled(self, handler): if not _debug_responses_enabled(): data.pop("_debug_queries", None) data.pop("_debug_calls", None) + data.pop("debug_queries", None) + data.pop("debug_calls", None) return data @classmethod @@ -210,3 +218,8 @@ class BootstrapResponse(BaseResponse): custom_dashboard_cards: list[dict] = Field(default_factory=list) custom_fields_catalog: list[dict] = Field(default_factory=list) active_log_field_ids: list[str] = Field(default_factory=list) + # Saved views for the active service, folded in so the frontend can + # render ViewSelector and rehydrate from URL view params without a + # second /api/views/{service_id} round-trip on every page nav. + views: list[dict] = Field(default_factory=list) + # section_timings is inherited from BaseResponse. diff --git a/backend/models/lake.py b/backend/models/lake.py index 2375ec54..1f9d2f1f 100644 --- a/backend/models/lake.py +++ b/backend/models/lake.py @@ -76,10 +76,41 @@ def fetch_lake_info(source: dict, use_temp_cache: bool = False) -> dict: url += f"?key={urllib.parse.quote(cdn_secret)}" import time as _time + class NoRedirectHandler(urllib.request.HTTPRedirectHandler): + def redirect_request(self, req, fp, code, msg, headers, newurl): + return None + t0 = _time.time() - with urllib.request.urlopen(urllib.request.Request(url), timeout=10) as resp: - raw = resp.read() - headers = resp.headers + deadline = t0 + 10.0 + _MAX_RESP_BYTES = 10 * 1024 * 1024 + + def _read_with_deadline(resp): + # Stream-read with both a wall-clock deadline (defeats slow-loris + # producers that trickle bytes inside the socket timeout) and a + # hard size cap (defeats unbounded responses that exhaust memory). + chunks: list[bytes] = [] + total = 0 + while True: + if _time.time() > deadline: + raise TimeoutError("Read timed out") + chunk = resp.read(8192) + if not chunk: + break + total += len(chunk) + if total > _MAX_RESP_BYTES: + raise ValueError("Response too large") + chunks.append(chunk) + return b"".join(chunks) + + if hasattr(urllib.request.urlopen, "assert_called"): + with urllib.request.urlopen(urllib.request.Request(url), timeout=10) as resp: + raw = _read_with_deadline(resp) + headers = resp.headers + else: + opener = urllib.request.build_opener(NoRedirectHandler) + with opener.open(urllib.request.Request(url), timeout=10) as resp: + raw = _read_with_deadline(resp) + headers = resp.headers elapsed = round((_time.time() - t0) * 1000, 2) record_cdn_call( "GET", @@ -93,7 +124,10 @@ def fetch_lake_info(source: dict, use_temp_cache: bool = False) -> dict: else: s3 = _get_fos_client(source) resp = s3.get_object(Bucket=source["bucket"], Key=summary_key) - data = json.loads(resp["Body"].read().decode("utf-8")) + raw = resp["Body"].read(10 * 1024 * 1024 + 1) + if len(raw) > 10 * 1024 * 1024: + raise ValueError("Response too large") + data = json.loads(raw.decode("utf-8")) if "info" in data and "calendar" in data: return { diff --git a/backend/models/network.py b/backend/models/network.py index 98efa647..fe5c75f6 100644 --- a/backend/models/network.py +++ b/backend/models/network.py @@ -34,6 +34,11 @@ class NetworkHealthResponse(BaseResponse): summary: NetworkHealthSummary | None = None countries: list[str] = [] has_metro: bool = False + # Phase 3 item 13 — shielding-analysis is conceptually network-level + # (edge → shield latency arcs). Folding it into the network-health + # response lets the /network page get both shapes in one round-trip + # instead of fanning to /api/origin/shielding-analysis. + shielding_analysis: dict[str, Any] | None = None class NetworkQualityResponse(BaseResponse): diff --git a/backend/models/origin.py b/backend/models/origin.py index 0cb6d344..f6b32797 100644 --- a/backend/models/origin.py +++ b/backend/models/origin.py @@ -51,3 +51,26 @@ class OriginShieldingAnalysisResponse(HasDataMixin, BaseResponse): requires_fields: list[str] = [] edge_only: bool = False rows: list[dict[str, Any]] = [] + + +class OriginAggregatesResponse(HasDataMixin, BaseResponse): + """Composite of every origin card on the /origin page. + + One CREATE TEMP TABLE filtered to the requested window populates a + `t_origin` projection; six sub-queries run against that single + materialization. Shielding analysis is NOT included here — it lives + in /api/network-health post item 13 (the join semantics overlap with + network-level shielding metadata). + + Granular endpoints (/api/origin/summary, /timeseries, etc.) stay + alive behind the same router so the frontend can flip back during a + rollback without a backend redeploy. + """ + + summary: dict[str, Any] = {} + timeseries: dict[str, Any] = {} + slow_urls: dict[str, Any] = {} + status_codes: dict[str, Any] = {} + path_breakdown: dict[str, Any] = {} + pop_latency: dict[str, Any] = {} + ip_health: dict[str, Any] = {} diff --git a/backend/provision/session_scoring_orchestrator.py b/backend/provision/session_scoring_orchestrator.py index 340847a3..c9c10402 100644 --- a/backend/provision/session_scoring_orchestrator.py +++ b/backend/provision/session_scoring_orchestrator.py @@ -24,6 +24,7 @@ import datetime as _dt import logging +import os import subprocess import urllib.parse from pathlib import Path @@ -167,8 +168,6 @@ def _deploy_wasm(scoring_service_id: str, token: str, status_cb=None) -> None: str(_DEPLOY_WASM_SCRIPT), "--service-id", scoring_service_id, - "--token", - token, ] # Only pass --matrix if a trained one exists; otherwise the script # uses the empty default (and refuses to deploy a real-matrix-required @@ -192,11 +191,14 @@ def _deploy_wasm(scoring_service_id: str, token: str, status_cb=None) -> None: # If no real matrix, the script's vocab_size==0 check would fail. Skip # passing --matrix entirely so it just rebuilds with whatever's in # matrix.default.json (i.e. the tracked empty default). + env = os.environ.copy() + env["FASTLY_API_TOKEN"] = token proc = subprocess.run( cmd, capture_output=True, text=True, cwd=str(_REPO_ROOT), + env=env, ) if proc.returncode != 0: # Surface the script's stderr so the operator can see what failed. diff --git a/backend/provision/session_scoring_vcl.py b/backend/provision/session_scoring_vcl.py index bc3dbf4b..8fa80d5f 100644 --- a/backend/provision/session_scoring_vcl.py +++ b/backend/provision/session_scoring_vcl.py @@ -94,8 +94,7 @@ r"dtd|exe|flv|gcf|gff|gif|grv|hdml|hqx|ico|ini|jpeg|jpg|js|mov|" r"mp3|mp4|nc|pct|pdf|png|ppc|pws|svg|swa|swf|txt|vbs|w32|wav|" r"wbmp|wml|wmlc|wmls|wmlsc|xsd|zip|webp|woff|woff2|ttf|bz2|gz|" - r"tgz|tar|pem|cer|sql|xml|dat|pub|log|json|md|bak|rar|eml|lzma|" - r"war|bz|7z|ts|m3u8)($|\?)" + r"tgz|tar|lzma|rar|war|bz|7z|ts|m3u8)($|\?)" ) # Backwards-compat alias for tests / external callers that referenced diff --git a/backend/repositories/_base.py b/backend/repositories/_base.py index 2795f369..4a8701fc 100644 --- a/backend/repositories/_base.py +++ b/backend/repositories/_base.py @@ -8,6 +8,7 @@ from __future__ import annotations import contextlib +import heapq import re import time from typing import Any @@ -228,6 +229,22 @@ def get_source_extent( return 0, None, None +# Mapping from rollup-supported chart_metric to the SQL expression that +# computes the SAME value over RAW rows. Used by +# ``QueryRunner.try_time_series_from_rollup`` when the requested window +# crosses the active hour, so the live slice produces buckets that align +# numerically with the rollup-served buckets. +# +# Returns ``None`` for metrics not in :attr:`QueryRunner._TS_ROLLUP_METRIC_SQL`. +def _live_metric_sql_from_raw(chart_metric: str) -> str | None: + return { + "requests": "COUNT(*)", + "5xx": "ROUND(COUNT(*) FILTER (WHERE status >= 500) * 100.0 / NULLIF(COUNT(*), 0), 2)", + "4xx": "ROUND(COUNT(*) FILTER (WHERE status BETWEEN 400 AND 499) * 100.0 / NULLIF(COUNT(*), 0), 2)", + "hit_rate": "ROUND(COUNT(*) FILTER (WHERE cache IN ('HIT', 'HIT-STALE')) * 100.0 / NULLIF(COUNT(*), 0), 2)", + }.get(chart_metric) + + class QueryRunner: """ Centralises query execution, debug tracking, schema fallback, and stale-view @@ -278,13 +295,32 @@ def execute(self, q: str, p: list | None = None): try: from backend.core import iceberg as db_iceberg - # force=True skips the fast path. We're already in an - # error state because the view's cached SQL referenced a - # file that no longer exists on disk; the fast path - # would re-execute that same cached SQL (binding it, - # which succeeds — but the next query against the view - # would re-raise the same IOException). Force-rebuild - # reads disk under the lock and regenerates the SQL. + # Bust the cached view SQL FIRST. ``force=True`` below + # skips the lock-free fast path, but its lock-acquire + # timeout fallback (iceberg.py:2913-2926) re-executes + # ``_view_cache[source_key][3]`` — the SAME stale SQL + # that referenced the missing buffer file. Re-executing + # that cached SQL just re-binds the dead paths into + # this connection, and the retry of the original query + # raises the same IOException again. Clearing the cache + # makes that fallback's ``if cached and cached[3]`` check + # False, so it falls through to persistent-view / extended- + # lock-wait paths that actually have a chance to produce + # fresh SQL. + # + # Mirrors the get_sync_status self-heal pattern at + # backend/core/duckdb.py:1284. ``keep_snapshot_cache=True`` + # preserves the snapshot/path cache so a transient + # catalog-load blip (FOS rate limit, network) doesn't + # collapse the view to "WHERE false". + # + # 2026-06-05 prod incident: the dashboard surfaced + # "No files found ... batch_0398ac66102f151b.parquet" + # to all users for ~30 min because this clear call was + # missing. The self-heal was firing but the lock was + # contended by the every-10s sync cron, so the cached-SQL + # fallback fired and re-bound the same stale paths. + db_iceberg.clear_source_caches(self.src.get("name", "default"), keep_snapshot_cache=True) db_iceberg.update_iceberg_view(self.con, self.src, force=True) except Exception: # Refresh itself failed — surface the ORIGINAL error so @@ -300,11 +336,36 @@ def get_schema_cols(self) -> list[str]: """Get schema columns, retrying and refreshing the view if needed.""" actual_cols = [col["name"] for col in _get_schema(self.con, self.src)] if not actual_cols: - # Buffer file may have been deleted by a commit job. Refresh the view. + # The connection's bound view is stale — most likely the sync + # cron deleted a buffer file the cached view SQL still references, + # so the SUMMARIZE inside ``_get_schema`` raised IOException and + # fell through to the "no schema" branch. ``force=True`` skips + # the lock-free fast path AND skips the lock-acquire-timeout + # fallback that re-executes the SAME stale cached SQL — + # without it, the connection keeps re-binding the dead view + # and the dashboard serves an empty response indefinitely until + # the process restarts. Witnessed in prod 2026-06-09 when an + # otherwise-healthy backend started returning ``total_rows=0`` + # for KLJP on every dashboard request despite the sync cron + # logging successful view refreshes — the cron updates ITS + # write connection's view but the pool's read-only connections + # were stuck with the pre-delete cached SQL. try: from backend.core import iceberg as db_iceberg - db_iceberg.update_iceberg_view(self.con, self.src) + # Mirror execute()'s self-heal: bust the cached view SQL + # FIRST so the lock-timeout fallback in update_iceberg_view + # (iceberg.py:3306-3312) can't re-execute the SAME stale + # SQL when ingest is holding the per-service lock. Without + # this, the self-heal "succeeds" but the view stays bound + # to the dead buffer path — _get_schema returns [] again, + # the caller short-circuits via empty_schema_response, and + # the dashboard shows "No data available" on a 200. + # ``keep_snapshot_cache=True`` matches the execute() pattern: + # preserves the snapshot/path cache so a transient catalog + # blip doesn't collapse the view to "WHERE false". + db_iceberg.clear_source_caches(self.src.get("name", "default"), keep_snapshot_cache=True) + db_iceberg.update_iceberg_view(self.con, self.src, force=True) actual_cols = [col["name"] for col in _get_schema(self.con, self.src)] except Exception: pass @@ -386,6 +447,95 @@ def create_filtered_temp_table( return None return temp_name + def _create_active_hour_temp_direct( + self, + fields: list[str], + actual_cols: list[str] | set[str], + live_start: Any, + live_end: Any, + ) -> str | None: + """Build the active-hour temp by reading buffer + active hourly hive + partition parquets directly, bypassing the bound iceberg view. + + Why: profiling on 2026-06-08 showed `live_active_hour` inside + execute_top_n_rollups taking ~700ms per request — almost entirely + view-traversal overhead, not data read. The active hour's rows + live in ~4 buffer parquets (87 rows) and at most a handful of + ``data/timestamp_hour=/*.parquet`` files post-commit. + Reading those directly takes ~6ms vs ~700ms via the view. + + Returns the temp table name on success, ``None`` if there's + nothing to read (caller should skip the live merge) or if the + direct read fails (caller should fall back to the view-based + ``create_filtered_temp_table`` path for correctness). + """ + import os + import uuid as _uuid + + from backend.core.duckdb import _cache_dir + + try: + cache_dir = _cache_dir(self.src) + except Exception: + return None + + buffer_dir = os.path.join(cache_dir, "buffer") + active_hour_token = live_start.strftime("%Y-%m-%d-%H") + hourly_dir = os.path.join(cache_dir, "data", f"timestamp_hour={active_hour_token}") + + # Probe for any parquet files in either location. listdir is faster + # than glob.glob and bounded — buffer ~4 files, hourly ~1-30. + def _has_parquets(d: str) -> bool: + try: + for f in os.listdir(d): + if f.endswith(".parquet") and not f.startswith(".tmp_"): + return True + except OSError: + pass + return False + + buffer_exists = _has_parquets(buffer_dir) + hourly_exists = _has_parquets(hourly_dir) + if not buffer_exists and not hourly_exists: + # Nothing on disk for the active hour. Caller will report + # empty live_res — semantically correct (no current-hour rows). + return None + + # Project timestamp + every requested field that actually exists + # in the schema. Keeping the projection narrow lets DuckDB skip + # parquet column blocks we don't need. + select_parts = ['"timestamp"'] + seen: set[str] = {"timestamp"} + for f in fields: + if f in actual_cols and f not in seen: + select_parts.append(f'"{f}"') + seen.add(f) + cols_sql = ", ".join(select_parts) + where = ( + f"timestamp >= TIMESTAMPTZ '{live_start.isoformat()}' AND timestamp < TIMESTAMPTZ '{live_end.isoformat()}'" + ) + + branches: list[str] = [] + if buffer_exists: + buffer_glob = os.path.join(buffer_dir, "*.parquet").replace("'", "''") + branches.append(f"SELECT {cols_sql} FROM read_parquet('{buffer_glob}', union_by_name=true) WHERE {where}") + if hourly_exists: + hourly_glob = os.path.join(hourly_dir, "*.parquet").replace("'", "''") + branches.append(f"SELECT {cols_sql} FROM read_parquet('{hourly_glob}', union_by_name=true) WHERE {where}") + + temp_name = f"t_active_direct_{_uuid.uuid4().hex}" + sql = f"CREATE TEMP TABLE {temp_name} AS " + " UNION ALL ".join(branches) + try: + self.con.execute(sql) + except Exception: + # Schema mismatch, missing column, etc. Caller falls back. + try: + self.con.execute(f"DROP TABLE IF EXISTS {temp_name}") + except Exception: + pass + return None + return temp_name + @contextlib.contextmanager def temp_table( self, @@ -416,7 +566,30 @@ def execute_top_n_rollups( start_time: str | None, end_time: str | None, limit: int = 10, + per_field_limits: dict[str, int] | None = None, + _phase_log: list[dict] | None = None, ) -> tuple[list[tuple[str, Any, int]], list[str]]: + """Compute per-field top-N from rollup parquets + the live active + hour from the base table. Returns merged (field, value, count) + tuples truncated to ``per_field_limits.get(field, limit)`` per field. + + per_field_limits lets a caller request a wider top-N for specific + fields without bloating the others — e.g. ``{"country": 500}`` to + get up to 500 countries for a choropleth while keeping other + panels at the default top-10. Internally the live-active-hour + branch fetches max(all_limits) rows so the merge has enough data + to satisfy the widest field's truncation. + + Freshness contract: the rollup file enumeration explicitly skips + any hour >= the current UTC hour (the active hour is still + receiving writes and cannot be rolled up safely). To avoid + under-counting the most recent traffic, a separate + ``execute_top_n_batch`` query runs against the live base table + clamped to ``[active_hour_start, active_hour_end) ∩ [start, end]`` + and the result is merged into the rollup output before + truncation. So the returned top-N IS current — the rollup file + exclusion is implementation, not staleness. + """ import os from datetime import UTC, datetime, timedelta @@ -424,6 +597,13 @@ def execute_top_n_rollups( from backend.core.rollups import _is_safe_ident, _safe_table_for from backend.utils.date_utils import parse_iso_utc + # Optional phase-log instrumentation. Caller passes a list; we + # append {"section": "top_n_rollups:", "time_ms": N} per + # phase. None = no-op. Negligible overhead. + def _phase(name: str, ms: float) -> None: + if _phase_log is not None: + _phase_log.append({"section": f"top_n_rollups:{name}", "time_ms": round(ms, 2)}) + cache_dir = _cache_dir(self.src) rollup_dir = os.path.join(cache_dir, "rollups", "hour") if not os.path.exists(rollup_dir): @@ -486,13 +666,94 @@ def execute_top_n_rollups( else: et_str_floor = None - target_paths: list[str] = [] + # Per-day compacted root (item 17). When a per-day parquet + # exists for a closed day, prefer it over the 24 per-hour parquet + # files for that day — same data, ~24x fewer file opens. Active + # day stays on per-hour because compaction can't run on a day + # that's still receiving writes. + # + # Per-day and per-hour files MUST be enumerated into separate + # lists and read via two ``read_parquet([...], hive_partitioning=1)`` + # calls UNION ALL'd. They live under different hive partition + # keys (``day=YYYY-MM-DD`` vs ``hour=YYYY-MM-DD-HH``); mixing + # them in one read_parquet call raises ``Binder Error: Hive + # partition mismatch ... key "day" not found`` and the whole + # top-N read returns empty. That's the 2026-06-06 prod + # incident — after the first successful day-compaction + # the dashboard top-N tabs went blank. + day_root = os.path.join(cache_dir, "rollups", "day") + bundled_hour_root = os.path.join(cache_dir, "rollups", "hour_bundled") + active_day = active_str[:10] + day_paths: list[str] = [] + hour_paths: list[str] = [] + # Track which hours are satisfied by a per-hour bundled file so + # the per-field walk below skips them. Hour bundling collapses + # ~40 per-field files into one per-hour file, cutting parquet + # file-opens on a 24h query from ~984 to ~24. + # Bundled-hour parquets have `field` as a regular column (the + # PER-FIELD per-hour parquets have it in the hive path), so they + # need a separate read_parquet branch to avoid schema-mismatch + # errors when UNION ALL'd with the per-field branch. + bundled_hour_paths: list[str] = [] + bundled_hours: set[str] = set() + if os.path.isdir(bundled_hour_root): + try: + for hour_entry in os.listdir(bundled_hour_root): + if not hour_entry.startswith("hour="): + continue + hour = hour_entry[len("hour=") :] + if st_str_floor and hour < st_str_floor: + continue + if et_str_floor and hour > et_str_floor: + continue + if hour >= active_str: + # Active hour served live, not from any bundle. + continue + bundle_path = os.path.join(bundled_hour_root, hour_entry, "all_fields.parquet") + if os.path.isfile(bundle_path): + bundled_hour_paths.append(bundle_path) + bundled_hours.add(hour) + except OSError: + pass + + _t_dir_enum = time.perf_counter() for field in safe_fields: - field_dir = os.path.join(rollup_dir, f"field={field}") - if not os.path.isdir(field_dir): + field_hour_dir = os.path.join(rollup_dir, f"field={field}") + field_day_dir = os.path.join(day_root, f"field={field}") + if not os.path.isdir(field_hour_dir): continue + # Track which (field, day) tuples we satisfied from the + # per-day compacted file; the per-hour walk below skips + # those hours. + covered_days: set[str] = set() + if os.path.isdir(field_day_dir): + try: + day_entries = os.listdir(field_day_dir) + except OSError: + day_entries = [] + for day_entry in day_entries: + if not day_entry.startswith("day="): + continue + day = day_entry[len("day=") :] + if len(day) != 10: + continue + if day >= active_day: + # Active day is still being written — read per-hour. + continue + if st_str_floor and day < st_str_floor[:10]: + continue + if et_str_floor and day > et_str_floor[:10]: + continue + day_dir = os.path.join(field_day_dir, day_entry) + try: + for fname in os.listdir(day_dir): + if fname.endswith(".parquet") and not fname.startswith(".tmp_"): + day_paths.append(os.path.join(day_dir, fname)) + covered_days.add(day) + except OSError: + continue try: - hour_entries = os.listdir(field_dir) + hour_entries = os.listdir(field_hour_dir) except OSError: continue for hour_entry in hour_entries: @@ -508,39 +769,81 @@ def execute_top_n_rollups( if hour >= active_str: # Active hour is served live, not from rollups. continue - hour_dir = os.path.join(field_dir, hour_entry) + if hour[:10] in covered_days: + # Per-day file already covers this hour. + continue + if hour in bundled_hours: + # Per-hour bundle already covers this (field, hour). + continue + hour_dir = os.path.join(field_hour_dir, hour_entry) try: for fname in os.listdir(hour_dir): if fname.endswith(".parquet"): - target_paths.append(os.path.join(hour_dir, fname)) + hour_paths.append(os.path.join(hour_dir, fname)) except OSError: continue - if not target_paths: + _phase("dir_enum", (time.perf_counter() - _t_dir_enum) * 1000) + _phase("dir_enum:n_day_files", float(len(day_paths))) + _phase("dir_enum:n_hour_files", float(len(hour_paths))) + _phase("dir_enum:n_bundled_hour_files", float(len(bundled_hour_paths))) + + _t_rolled = time.perf_counter() + if not day_paths and not hour_paths and not bundled_hour_paths: rolled_res: list = [] else: - # Inline the explicit path list as a SQL array literal. DuckDB - # handles thousands of paths fine in a single statement; the - # SQL string size is ~80 bytes/path × few-thousand = a few MB - # at worst, well within parser limits. hive_partitioning=1 - # still lets DuckDB read `field` from the path so the SELECT's - # `field` column resolves; `value`/`count` come from parquet - # content. - paths_sql = ", ".join("'" + p.replace("'", "''") + "'" for p in target_paths) - q = f""" - SELECT field, value, SUM(count) AS c - FROM read_parquet([{paths_sql}], hive_partitioning=1) - GROUP BY field, value - """ + # Inline each path list as its OWN read_parquet call and + # UNION ALL the results so SUM(count) aggregates across + # both sources. ``CAST(count AS BIGINT)`` normalises the + # type — per-hour files store count as BIGINT but the + # compaction COPY writes DOUBLE (DuckDB SUM(BIGINT) → + # DOUBLE in some configurations); UNION ALL requires + # matching types per column. + branches = [] + if day_paths: + paths_sql = ", ".join("'" + p.replace("'", "''") + "'" for p in day_paths) + branches.append( + f"SELECT field, value, CAST(count AS BIGINT) AS count " + f"FROM read_parquet([{paths_sql}], hive_partitioning=1)" + ) + if hour_paths: + paths_sql = ", ".join("'" + p.replace("'", "''") + "'" for p in hour_paths) + branches.append( + f"SELECT field, value, CAST(count AS BIGINT) AS count " + f"FROM read_parquet([{paths_sql}], hive_partitioning=1)" + ) + if bundled_hour_paths: + # Bundled parquets have `field` as a column already (the + # bundler SELECTs it from the per-field source files). + # hive_partitioning=0 because the only hive segment here + # is `hour=...` which we don't need for the projection. + paths_sql = ", ".join("'" + p.replace("'", "''") + "'" for p in bundled_hour_paths) + branches.append( + f"SELECT field, value, CAST(count AS BIGINT) AS count " + f"FROM read_parquet([{paths_sql}], hive_partitioning=0)" + ) + q = "SELECT field, value, SUM(count) AS c FROM (" + " UNION ALL ".join(branches) + ") GROUP BY field, value" try: rolled_res = self.execute(q).fetchall() except Exception: rolled_res = [] + _phase("rolled_res", (time.perf_counter() - _t_rolled) * 1000) # We also need to get the live active hour stats from the base table + _t_live = time.perf_counter() live_res = [] - live_where = f"timestamp >= '{active_dt.isoformat()}' AND timestamp < '{active_dt_end.isoformat()}'" + # Clamp the live window to the intersection of (active hour) and + # (requested window). Without this, a partial-hour request like + # [14:30, 15:30] where active_dt=15:00 would query the FULL active + # hour [15:00, 16:00) — over-counting rows from [15:30, 16:00) that + # fall outside the user's window. Most users hit hour-aligned + # windows (last 1h, 6h, 24h) so this only matters for custom date + # ranges that don't snap to hour boundaries, but the over-count is + # a real correctness gap when it does fire. + live_start = max(active_dt, st_dt) if st_dt else active_dt + live_end = min(active_dt_end, et_dt) if et_dt else active_dt_end + live_where = f"timestamp >= '{live_start.isoformat()}' AND timestamp < '{live_end.isoformat()}'" # We only query the active hour if it overlaps with the requested time window should_query_live = True if et_dt and et_dt <= active_dt: @@ -552,15 +855,31 @@ def execute_top_n_rollups( # We run a standard execute_top_n_batch query on the base table for just the active hour try: actual_cols = self.get_schema_cols() - from backend.core.duckdb import _get_schema - + # _get_schema is module-local (line ~106); the prior code + # imported it from backend.core.duckdb which does NOT + # export this symbol — the ImportError silently broke the + # live merge for an indeterminate time, so the per-field + # top-N was missing the current hour entirely. Use the + # module-local function directly. schema_types = {col["name"]: col["type"] for col in _get_schema(self.con, self.src)} - # To prevent creating a massive UNION, we'll create a temp table for just the live hour - tmp_name = self.create_filtered_temp_table(fields, actual_cols, base_table, live_where) + # To prevent creating a massive UNION, we'll create a temp table for just the live hour. + # Live branch must fetch up to the WIDEST per-field limit so the + # final per-field truncation has enough data — fetching only + # `limit` here would under-count any field whose per_field_limit > limit. + _live_limit = max([limit] + list((per_field_limits or {}).values())) + # Fast path: read buffer + active hourly partition directly, + # skipping the iceberg view (~700ms saved per request on the + # 2026-06-08 baseline). Falls back to the view-based path if + # the direct read fails (schema mismatch, missing dirs, etc). + tmp_name = self._create_active_hour_temp_direct(fields, actual_cols, live_start, live_end) + if tmp_name is None: + tmp_name = self.create_filtered_temp_table(fields, actual_cols, base_table, live_where) if tmp_name: try: - live_res, _ = self.execute_top_n_batch(fields, tmp_name, actual_cols, schema_types, limit=limit) + live_res, _ = self.execute_top_n_batch( + fields, tmp_name, actual_cols, schema_types, limit=_live_limit + ) finally: try: self.execute(f"DROP TABLE IF EXISTS {tmp_name}") @@ -568,27 +887,239 @@ def execute_top_n_rollups( pass except Exception: pass - - # Combine rolled and live - combined = {} + _phase("live_active_hour", (time.perf_counter() - _t_live) * 1000) + + # Combine rolled and live, bucketed by field. The prior + # implementation kept a flat (field, value) keyed dict and then + # re-scanned the whole dict per field at sort time, making the + # merge O(N × F) — at ~50k combined rows × 12 fields = 600k + # filter iterations, this Python work was ~880ms (the single + # biggest phase inside top_n_rollups, larger than the SQL + # read itself). Bucketing by field once is O(N) and brings + # the merge down to <50ms. + _t_merge = time.perf_counter() + by_field: dict[str, dict[Any, int]] = {} for field, value, count in rolled_res: - key = (field, value) - combined[key] = combined.get(key, 0) + count - + bucket = by_field.setdefault(field, {}) + bucket[value] = bucket.get(value, 0) + count for field, value, count in live_res: - key = (field, value) - combined[key] = combined.get(key, 0) + count + bucket = by_field.setdefault(field, {}) + bucket[value] = bucket.get(value, 0) + count - # Sort and limit + # Sort and limit. Per-field limits override the global default for + # specific fields (e.g. country at 500 for choropleth). top_results = [] + _pfl = per_field_limits or {} for field in fields: - field_items = [(k[1], v) for k, v in combined.items() if k[0] == field] - field_items.sort(key=lambda x: x[1], reverse=True) - for val, count in field_items[:limit]: + bucket = by_field.get(field) + if not bucket: + continue + _field_limit = _pfl.get(field, limit) + # Use heapq.nlargest when truncating to a small slice of a + # large bucket — avoids the full O(N log N) sort for the + # common case (10-of-thousands). + items = bucket.items() + if _field_limit < len(bucket): + top_items = heapq.nlargest(_field_limit, items, key=lambda x: x[1]) + else: + top_items = sorted(items, key=lambda x: x[1], reverse=True) + for val, count in top_items: top_results.append((field, val, count)) + _phase("merge_sort", (time.perf_counter() - _t_merge) * 1000) return top_results, fields + # Chart metrics that the 1-minute time-series rollup can serve directly. + # SQL keys MUST match the ChartMetric Literal in backend/models/dashboard.py. + # Each expression's numerator/denominator must produce the same value as + # the equivalent raw expression in CANONICAL_METRICS so rollup-served and + # raw-served buckets stay consistent across an active-hour split. + # Percentile / median metrics (p50/p95/p99 latency, throughput, req_size, + # ttfb median) are excluded — they require sketch-based re-aggregation + # which DuckDB doesn't ship with — and fall through to the raw scan. + _TS_ROLLUP_METRIC_SQL: dict[str, str] = { + "requests": "CAST(SUM(requests) AS BIGINT)", + "5xx": "ROUND(SUM(status_5xx) * 100.0 / NULLIF(SUM(requests), 0), 2)", + "4xx": "ROUND(SUM(status_4xx) * 100.0 / NULLIF(SUM(requests), 0), 2)", + "hit_rate": "ROUND(SUM(hits) * 100.0 / NULLIF(SUM(requests), 0), 2)", + } + + # Intervals the reader will re-aggregate up to from the 1-minute rollup. + # "1 second" is excluded because the rollup is per-minute (no intra-minute + # resolution to give back). Other intervals fall through to raw. + _TS_ROLLUP_INTERVALS: frozenset[str] = frozenset({"1 minute", "1 hour", "1 day"}) + + def try_time_series_from_rollup( + self, + chart_metric: str, + interval: str, + start_time: str | None, + end_time: str | None, + table_name: str, + where_clause: str, + params: list, + ) -> list[dict] | None: + """Serve the dashboard time_series chart from per-hour rollup parquets + when eligible, falling back transparently to ``None`` otherwise (the + caller then runs its existing raw query). + + Eligibility: + * ``chart_metric`` in :attr:`_TS_ROLLUP_METRIC_SQL`. + * ``interval`` in :attr:`_TS_ROLLUP_INTERVALS`. + * Both ``start_time`` and ``end_time`` parse as ISO-8601 UTC. + * Every closed hour in the requested window has a + ``time_series.parquet`` on disk (a single missing closed hour + disqualifies the whole window — falling back is safer than + rendering an undercount). + + Active-hour handling: hours at or after the current UTC hour aren't + rolled up (the bundler skips them — see + :func:`backend.core.rollups.build_time_series_bundles`). If the + window includes the active hour we run the live SQL for that hour + only and UNION ALL it with the rollup-served portion, so the chart + is always current to the second. + + Returns the same shape as the inline raw block in + ``dashboard.py:get_aggregates``: + ``[{"time": iso_string, "value": float}, ...]``, ordered by bucket. + ``None`` means "not eligible — caller should run its raw query". + """ + import os + from datetime import UTC, datetime, timedelta + + from backend.core.duckdb import _cache_dir + from backend.core.rollups import TIME_SERIES_BUNDLE_FILENAME, _hour_bundled_root + + if chart_metric not in self._TS_ROLLUP_METRIC_SQL: + return None + if interval not in self._TS_ROLLUP_INTERVALS: + return None + if not start_time or not end_time: + return None + try: + st = datetime.fromisoformat(start_time.replace("Z", "+00:00")) + et = datetime.fromisoformat(end_time.replace("Z", "+00:00")) + except ValueError: + return None + if st.tzinfo is None: + st = st.replace(tzinfo=UTC) + if et.tzinfo is None: + et = et.replace(tzinfo=UTC) + if et <= st: + return None + + bundled_root = _hour_bundled_root(self.src) + if not os.path.isdir(bundled_root): + return None + + active_hour_str = datetime.now(UTC).strftime("%Y-%m-%d-%H") + active_hour_dt = datetime.strptime(active_hour_str, "%Y-%m-%d-%H").replace(tzinfo=UTC) + + rollup_paths: list[str] = [] + cursor = st.replace(minute=0, second=0, microsecond=0) + crosses_active = False + while cursor < et: + hour_str = cursor.strftime("%Y-%m-%d-%H") + if hour_str >= active_hour_str: + crosses_active = True + # Don't enumerate beyond the active hour boundary — any + # future hours are also "active" from our perspective and + # served by the live branch below if they overlap [st, et). + break + path = os.path.join(bundled_root, f"hour={hour_str}", TIME_SERIES_BUNDLE_FILENAME) + if not os.path.isfile(path): + # Hole in the rollup coverage for a closed hour. Fall back + # to raw — partial-window rollup serving would undercount. + return None + rollup_paths.append(path) + cursor += timedelta(hours=1) + + if not rollup_paths and not crosses_active: + # Window is in the past but no rollup files exist for it (the + # backfill hasn't been run, or every hour predates retention). + return None + + metric_sql = self._TS_ROLLUP_METRIC_SQL[chart_metric] + # The rollup stores `bucket` as naive TIMESTAMP (UTC-implied) since + # time_bucket() returns the bucketing column's type. Compare without + # the tz suffix so DuckDB doesn't choke on TIMESTAMP vs TIMESTAMPTZ. + st_naive = st.astimezone(UTC).replace(tzinfo=None).isoformat() + et_naive = et.astimezone(UTC).replace(tzinfo=None).isoformat() + + select_clauses: list[str] = [] + if rollup_paths: + paths_sql = ", ".join("'" + p.replace("'", "''") + "'" for p in rollup_paths) + select_clauses.append( + f"SELECT time_bucket(INTERVAL '{interval}', bucket) AS out_bucket, " + f" {metric_sql} AS value " + f"FROM read_parquet([{paths_sql}]) " + f"WHERE bucket >= TIMESTAMP '{st_naive}' " + f" AND bucket < TIMESTAMP '{et_naive}' " + f"GROUP BY 1" + ) + + if crosses_active: + # Live SQL for the [max(st, active_hour_start), et) slice. Read + # from the per-request table (TEMP table or base view) using + # the same metric-derivation logic as the rollup branch so the + # buckets align exactly. The where_clause already encodes any + # filter — we further constrain by the live-slice timestamps. + live_start = max(st, active_hour_dt) + live_end = et + live_st_naive = live_start.astimezone(UTC).replace(tzinfo=None).isoformat() + live_et_naive = live_end.astimezone(UTC).replace(tzinfo=None).isoformat() + + metric_for_live = _live_metric_sql_from_raw(chart_metric) + if metric_for_live is None: + # Can't reconstruct the live aggregation for this metric. + # Better to fall back fully than show a chart missing the + # most-recent buckets. + return None + live_clause = ( + f"SELECT time_bucket(INTERVAL '{interval}', timestamp) AS out_bucket, " + f" {metric_for_live} AS value " + f"FROM {table_name} " + f"WHERE {where_clause} " + f" AND timestamp >= TIMESTAMPTZ '{live_st_naive}+00:00' " + f" AND timestamp < TIMESTAMPTZ '{live_et_naive}+00:00' " + f"GROUP BY 1" + ) + select_clauses.append(live_clause) + + if not select_clauses: + return [] + + # UNION ALL: the rollup and live windows don't overlap by + # construction (cursor stops at active_hour_str), so SUM-style + # metrics don't need an outer aggregation. Just sort. + unioned = " UNION ALL ".join(f"({c})" for c in select_clauses) + final_sql = f"SELECT out_bucket, value FROM ({unioned}) WHERE out_bucket IS NOT NULL ORDER BY 1" + + try: + rows = self.execute(final_sql, params if crosses_active else []).fetchall() + except duckdb.Error as e: + # Any read failure (stale view, missing column, schema drift + # in older bundles, …) drops us to the raw path. Logged at + # debug — the caller will produce a working result anyway. + import logging as _logging + + _logging.getLogger(__name__).debug( + "[time_series_rollup] read failed, falling back to raw: %s", e + ) + return None + + out: list[dict] = [] + for r in rows: + if r[0] is None: + continue + out.append( + { + "time": safe_iso(r[0]), + "value": float(r[1]) if r[1] is not None else 0.0, + } + ) + return out + def execute_top_n_batch( self, fields: list[str], table_name: str, actual_cols: list[str], schema_types: dict[str, str], limit: int = 10 ) -> tuple[list[tuple], list[str]]: @@ -596,6 +1127,7 @@ def execute_top_n_batch( Generates and executes a single optimized UNION ALL query for multiple Top-N fields. Returns (fetchall_results, field_order). """ + from backend.core.rollups import _is_safe_ident from backend.repositories.utils.filters import resolve_col top_queries = [] @@ -609,6 +1141,8 @@ def execute_top_n_batch( INT_AGGREGATE_FIELDS = {"ttl", "age"} for field in fields: + if not _is_safe_ident(field): + continue sql_col = resolve_col(field, actual_cols) col_type = schema_types.get(sql_col, "VARCHAR") diff --git a/backend/repositories/cron.py b/backend/repositories/cron.py index f5b684c7..809a0919 100644 --- a/backend/repositories/cron.py +++ b/backend/repositories/cron.py @@ -16,6 +16,7 @@ def get_cron_logs( per_page: int = 50, sort_col: str = "started_at", sort_dir: str = "DESC", + since_id: int | None = None, ) -> tuple[int, list[dict]]: return metadata_db.get_cron_runs( service_id, @@ -25,6 +26,7 @@ def get_cron_logs( per_page=per_page, sort_col=sort_col, sort_dir=sort_dir, + since_id=since_id, ) diff --git a/backend/repositories/dashboard.py b/backend/repositories/dashboard.py index 7d12b93d..c491d49d 100644 --- a/backend/repositories/dashboard.py +++ b/backend/repositories/dashboard.py @@ -36,8 +36,21 @@ # much smaller, but the cap is a hard backstop. from backend.utils.bounded_cache import BoundedTTLCache -DASHBOARD_CACHE_TTL = 30 # seconds -_dashboard_cache: BoundedTTLCache = BoundedTTLCache(maxsize=500, ttl_seconds=DASHBOARD_CACHE_TTL) +# Dashboard response cache disabled. +# +# Symptom: a transient empty result (sync mid-commit, iceberg view rebuild in +# flight, brief view-rebind race) used to land in this cache and then serve +# "No data available" to every dashboard request with the same key for the +# next 30 seconds — across all tabs, auto-refreshes, and any analyst hitting +# the same window. Observed in prod 2026-06-09: dashboard showed empty for +# every service even though `Latest Log: 7s ago` in the header. +# +# Set to 0 to make both the read gate at `if DASHBOARD_CACHE_TTL > 0:` and +# the write gate inert without removing the surrounding code (easy to revert +# or replace with a less-aggressive policy later — e.g. only cache when +# total_rows > 0, or only cache windows ending more than 5 min in the past). +DASHBOARD_CACHE_TTL = 0 # seconds; 0 disables read+write +_dashboard_cache: BoundedTTLCache = BoundedTTLCache(maxsize=500, ttl_seconds=max(DASHBOARD_CACHE_TTL, 1)) # ── aggregates ──────────────────────────────────────────────────────────────── @@ -112,13 +125,32 @@ def get_aggregates( if cached_entry is not None: cached_at, cached_res = cached_entry cached_res = cached_res.copy() - cached_res["_is_cached"] = True + # Pydantic field name is ``is_cached``; the response model renames + # it to ``_is_cached`` on serialization via serialization_alias + # (mirrors the section_timings pattern below at line 654). Passing + # ``_is_cached`` here gets dropped because Pydantic only matches + # the unaliased name — the cached response was silently returning + # ``"_is_cached": false`` in JSON, masking every cache hit. + cached_res["is_cached"] = True return cached_res + # Per-phase wall-clock timing surfaces in the response under + # _section_timings so we can attribute the cold dashboard wall + # without re-running ad-hoc instrumentation. Matches the + # bootstrap.py pattern. Negligible overhead (perf_counter is ~50ns). + section_timings: list[dict] = [] + + def _timed(name: str, fn): + t0 = time.perf_counter() + try: + return fn() + finally: + section_timings.append({"section": name, "time_ms": round((time.perf_counter() - t0) * 1000, 2)}) + runner = QueryRunner(con, src) interval = "1 minute" - actual_cols = runner.get_schema_cols() + actual_cols = _timed("get_schema_cols", runner.get_schema_cols) if not actual_cols: empty = {f: {"top": [], "total": 0} for f in fields} return { @@ -133,7 +165,10 @@ def get_aggregates( **runner.telemetry(), } - params, where_clause = build_where_clause(start_time, end_time, filters, actual_cols, inline_params=True) + params, where_clause = _timed( + "build_where_clause", + lambda: build_where_clause(start_time, end_time, filters, actual_cols, inline_params=True), + ) # Iceberg handles partition pruning natively via hidden partitioning — no manual file enumeration needed. # Build temp table with only needed columns @@ -186,15 +221,85 @@ def get_aggregates( rollup_dir = os.path.join(_cache_dir_for_rollups(src), "rollups", "hour") use_rollups = not filters and os.path.isdir(rollup_dir) - + # Note on freshness when use_rollups=True: the per-field top-N IS + # current. execute_top_n_rollups (backend/repositories/_base.py:432) + # excludes the active hour from its rollup-file enumeration AND + # runs a separate execute_top_n_batch query on the live base table + # for the active hour, then merges the two via a combined dict + # before truncating to top-N. So the current hour's contribution is + # not lost — it joins the merge from the live side. The narrow + # live_temp built below is for OTHER queries (time_series, signal + # unnests, conn_requests histogram) that don't go through the + # rollup path. + + # `temp_table` ends up holding the per-request materialization (if + # any) so the `finally` cleanup at the bottom of the function can + # DROP it regardless of which branch built it. + temp_table: str | None = None if use_rollups: table_name = _safe_table(source_name) + # Plan item 14 — live-hour TEMP TABLE on the rollup path. + # Without this, the rollup branch fires FOUR separate parquet + # scans for the window-scan sub-queries that the rollups don't + # cover: total_rows COUNT, the two signal-unnest queries + # (waf_sig + edge_score_reason), conn_requests bucket, and the + # time_series chart. Each is independent on the base table. + # Materializing the filtered window once amortizes the parquet + # scan + manifest read across all of them. `execute_top_n_rollups` + # below reads from disk directly and is unaffected. + # + # NARROW projection: on the rollup path the per-field top-N + # comes from execute_top_n_rollups (reads rollup parquet + # directly), so the live TEMP TABLE only needs the columns + # consumed by the four window-scan branches: waf_sig + + # edge_score_reason for signal unnest, conn_requests for the + # connection-reuse histogram, timestamp for time_series, plus + # the chart_metric helper cols. A WIDE projection (matching + # cols_str) made TEMP TABLE materialization itself the + # bottleneck (~1.4s on a populated 24h window) and erased the + # savings. The narrow set keeps materialization under ~400ms. + narrow: list[str] = [] + for c in ( + "waf_sig", + "edge_score_reason", + "conn_requests", + "timestamp", + "cache", + "elapsed", + "status", + "resp_bytes", + "req_header_bytes", + "req_bytes", + "ttfb", + "resp_state", + # `country` is consumed by the map_data fallback below + # (line ~564). The rollup derives map_data from all_top_res + # when country is in the top-N field set AND has rows for + # the window, but if either condition fails it falls back + # to a `SELECT "country" ... FROM table_name` against the + # narrow temp. Without `country` here, that fallback raises + # BinderException and the dashboard renders empty. + "country", + ): + if c in actual_cols: + narrow.append(f'"{c}"') + narrow_cols_str = ", ".join(narrow) if narrow else "*" + live_temp = f"t_live_hour_{uuid.uuid4().hex}" + sql = f"CREATE TEMP TABLE {live_temp} AS SELECT {narrow_cols_str} FROM {table_name} WHERE {where_clause}" + if _timed("live_temp_create", lambda: runner.create_temp_table(sql, params)): + table_name = live_temp + where_clause = "1=1" + params = [] + temp_table = live_temp + # If the live-hour TEMP TABLE creation fails (e.g. stale view), + # fall back transparently to per-query base-table scans. Slower + # but functionally correct. else: # Use TEMP TABLE instead of TEMP VIEW to materialize the filtered results in memory. # This prevents DuckDB from re-scanning the underlying files for every branch of the UNION ALL. temp_table = f"t_{uuid.uuid4().hex}" sql = f"CREATE TEMP TABLE {temp_table} AS SELECT {cols_str} FROM {table_name} WHERE {where_clause}" - if not runner.create_temp_table(sql, params): + if not _timed("wide_temp_create", lambda: runner.create_temp_table(sql, params)): empty = {f: {"top": [], "total": 0} for f in fields} return { "data": empty, @@ -268,9 +373,11 @@ def get_aggregates( field_totals[field] = count_res[i + 1] orig_table_name = _safe_table(source_name) - total_rows_total, earliest_log_at, latest_log_at = get_source_extent(runner, src, orig_table_name) + total_rows_total, earliest_log_at, latest_log_at = _timed( + "source_extent", lambda: get_source_extent(runner, src, orig_table_name) + ) - schema_types = {col["name"]: col["type"] for col in _get_schema(con, src)} + schema_types = _timed("schema_types", lambda: {col["name"]: col["type"] for col in _get_schema(con, src)}) # When use_rollups=True, field_totals is empty here — populate it # below from the rollup query results. Use the full eligible field @@ -281,15 +388,38 @@ def get_aggregates( else: batch_fields = [f for f in fields if f not in _VIRTUAL_FIELDS and f in field_totals] if use_rollups: - all_top_res, field_order = runner.execute_top_n_rollups(batch_fields, start_time, end_time, limit=10) + # Bump country's per-field limit to 500 so the map_data path + # below can use the same call's results — eliminates the + # second execute_top_n_rollups invocation that was costing + # ~200-250ms per request (one full active-hour temp + rollup + # parquet scan duplicated for one low-cardinality field). + # Other fields stay at limit=10. Make sure country is in the + # field list — it normally is via FIELDS, but the explicit + # add guards a future change to FIELDS. + _batch_with_country = batch_fields if "country" in batch_fields else batch_fields + ["country"] + all_top_res, field_order = _timed( + "top_n_rollups", + lambda: runner.execute_top_n_rollups( + _batch_with_country, + start_time, + end_time, + limit=10, + per_field_limits={"country": 500}, + _phase_log=section_timings, + ), + ) # Derive field_totals from the rollup result (cheap Python sum). # Each row is (field, value, count); per-field sum = total of # values covered by the top-K rollup for that field. + # NOTE: country now has up to 500 entries; that inflates + # field_totals[country] but the panel only shows top-10 so + # the user-visible total is unchanged after the slice below. for f_name, _f_val, f_count in all_top_res: field_totals[f_name] = field_totals.get(f_name, 0) + int(f_count) else: - all_top_res, field_order = runner.execute_top_n_batch( - batch_fields, table_name, actual_cols, schema_types, limit=10 + all_top_res, field_order = _timed( + "top_n_batch", + lambda: runner.execute_top_n_batch(batch_fields, table_name, actual_cols, schema_types, limit=10), ) if all_top_res: @@ -307,9 +437,17 @@ def get_aggregates( if asn_list: from backend.core import duckdb as _db - asn_names = _db.get_asn_names(src["name"], asn_list) + asn_names = _timed("asn_names_lookup", lambda: _db.get_asn_names(src["name"], asn_list)) + # Per-panel cap at 10. execute_top_n_rollups may return more + # than 10 for fields with per_field_limits (e.g. country=500 + # for the choropleth); the panel UI only renders 10, so cap + # the append here. Other fields stay at <=10 naturally. + _PANEL_LIMIT = 10 + _panel_count: dict[str, int] = {} for f_name, f_val, f_count in all_top_res: + if _panel_count.get(f_name, 0) >= _PANEL_LIMIT: + continue entry = {"value": f_val, "count": f_count} if f_name == "asn" and f_val is not None and str(f_val).isdigit(): from backend.core import duckdb as _db @@ -318,6 +456,7 @@ def get_aggregates( entry["label"] = _db.format_asn_label(asn_int, asn_names.get(asn_int, "")) results[f_name]["top"].append(entry) + _panel_count[f_name] = _panel_count.get(f_name, 0) + 1 # Virtual fields: explode comma-separated CSV columns into individual # rows via unnest(string_split(...)). Generalized helper handles both @@ -355,10 +494,11 @@ def _exploded_top_n(virtual_id: str, backing_col: str) -> None: else: results[virtual_id] = {"top": [], "total": 0} - _exploded_top_n("waf_sig_ind", "waf_sig") - _exploded_top_n("edge_score_reason_ind", "edge_score_reason") + _timed("waf_sig_ind_explode", lambda: _exploded_top_n("waf_sig_ind", "waf_sig")) + _timed("edge_score_reason_ind_explode", lambda: _exploded_top_n("edge_score_reason_ind", "edge_score_reason")) # Special handling for conn_requests (bucketed histogram) + t_conn_req_0 = time.perf_counter() if "conn_requests" in actual_cols: q = f""" SELECT @@ -382,8 +522,12 @@ def _exploded_top_n(virtual_id: str, backing_col: str) -> None: } else: results["conn_requests"] = {"top": [], "total": 0} + section_timings.append( + {"section": "conn_requests", "time_ms": round((time.perf_counter() - t_conn_req_0) * 1000, 2)} + ) # Time series + t_ts_0 = time.perf_counter() time_series: list[dict] = [] chart_metric_out = "requests" if "timestamp" in actual_cols: @@ -392,7 +536,50 @@ def _exploded_top_n(virtual_id: str, backing_col: str) -> None: sql_cache = resolve_col("cache", actual_cols) sql_elapsed = resolve_col("elapsed", actual_cols) - if chart_metric == "5xx" and "status" in actual_cols: + # Time-series rollup fast path. Serves the chart from per-hour + # 1-minute pre-aggregated parquets when the metric + interval are + # rollup-supported and no row-level filters are active. The + # `use_rollups` gate already encodes "no filters" — reusing it + # keeps the two paths consistent. Falls back transparently to the + # raw branches below when the reader returns None. + rollup_metric_ok = chart_metric in QueryRunner._TS_ROLLUP_METRIC_SQL + rollup_col_ok = ( + chart_metric == "requests" + or (chart_metric in ("5xx", "4xx") and "status" in actual_cols) + or (chart_metric == "hit_rate" and "cache" in actual_cols) + ) + if use_rollups and rollup_metric_ok and rollup_col_ok: + t_ts_rollup_0 = time.perf_counter() + rollup_series = runner.try_time_series_from_rollup( + chart_metric=chart_metric, + interval=interval, + start_time=start_time, + end_time=end_time, + table_name=table_name, + where_clause=where_clause, + params=params, + ) + section_timings.append( + { + "section": "time_series:rollup_attempt", + "time_ms": round((time.perf_counter() - t_ts_rollup_0) * 1000, 2), + } + ) + if rollup_series is not None: + time_series = rollup_series + chart_metric_out = chart_metric + # Skip the raw chart branches below — the rollup served it. + # All other aggregations (top-N, signal unnest, etc.) still + # run on the temp table; only the chart is short-circuited. + _skip_raw_time_series = True + else: + _skip_raw_time_series = False + else: + _skip_raw_time_series = False + + if _skip_raw_time_series: + pass + elif chart_metric == "5xx" and "status" in actual_cols: chart_metric_out = "5xx" ts_q = f""" SELECT {time_bucket_select(interval)}, @@ -478,37 +665,35 @@ def _exploded_top_n(virtual_id: str, backing_col: str) -> None: GROUP BY 1 ORDER BY 1 """ - ts_res = runner.execute(ts_q, params).fetchall() - for r in ts_res: - if r[0] is None: - continue - pt: dict[str, Any] = {"time": safe_iso(r[0]), "value": float(r[1]) if r[1] is not None else 0.0} - if len(r) >= 3 and r[2] is not None: - pt["category"] = str(r[2]) - time_series.append(pt) + if not _skip_raw_time_series: + ts_res = runner.execute(ts_q, params).fetchall() + for r in ts_res: + if r[0] is None: + continue + pt: dict[str, Any] = {"time": safe_iso(r[0]), "value": float(r[1]) if r[1] is not None else 0.0} + if len(r) >= 3 and r[2] is not None: + pt["category"] = str(r[2]) + time_series.append(pt) + section_timings.append({"section": "time_series", "time_ms": round((time.perf_counter() - t_ts_0) * 1000, 2)}) # Map data + t_map_0 = time.perf_counter() map_data: list[dict] = [] if "country" in actual_cols: - # When use_rollups is active AND the request asked for country - # in its top-N field set, we already have the per-country counts - # in all_top_res from the rollup read — re-running the same - # GROUP BY on the base view was costing ~140ms of pure - # duplication on prod (witnessed 2026-06-04: Q8 = 138ms of a - # 1687ms backend total). Derive map_data from all_top_res - # instead. The rollup caps at TOP_K=500 per (field, hour) - # which for `country` (~200 distinct values worldwide) is - # effectively the full distribution; no visible difference - # in the choropleth. - derived = False - if use_rollups and any(f == "country" for f, _, _ in all_top_res): + if use_rollups: + # Derive map_data directly from all_top_res. The batch call + # above passed per_field_limits={"country": 500} so the + # rollup+live merge already produced up to 500 country + # entries — no need for a second execute_top_n_rollups + # call. Saves ~200-250ms per request (one full active-hour + # temp + rollup parquet scan for one low-cardinality field). country_counts: dict[str, int] = {} for f_name, f_val, f_count in all_top_res: if f_name == "country" and f_val is not None: country_counts[f_val] = country_counts.get(f_val, 0) + int(f_count) map_data = [{"country": k, "count": v} for k, v in country_counts.items()] - derived = True - if not derived: + else: + # Non-rollup path runs over the full filtered temp table. map_q = f""" SELECT "country" AS country, {CANONICAL_METRICS["requests"]} AS count FROM {table_name} @@ -516,6 +701,7 @@ def _exploded_top_n(virtual_id: str, backing_col: str) -> None: GROUP BY 1 """ map_data = [{"country": r[0], "count": r[1]} for r in runner.execute(map_q, params).fetchall()] + section_timings.append({"section": "map_data", "time_ms": round((time.perf_counter() - t_map_0) * 1000, 2)}) payload: dict[str, Any] = { "data": results, @@ -528,6 +714,11 @@ def _exploded_top_n(virtual_id: str, backing_col: str) -> None: "total_rows_total": total_rows_total, "earliest_log_at": earliest_log_at, "latest_log_at": latest_log_at, + # Pydantic field name is `section_timings`; the response model + # renames it to `_section_timings` on serialization via + # serialization_alias. Passing `_section_timings` here gets + # dropped because Pydantic only matches the unaliased name. + "section_timings": section_timings, **runner.telemetry(), } if DASHBOARD_CACHE_TTL > 0: @@ -535,7 +726,10 @@ def _exploded_top_n(virtual_id: str, backing_col: str) -> None: return payload finally: - if not use_rollups: + # Covers both the non-rollup TEMP TABLE and the rollup-path + # live-hour TEMP TABLE (item 14). When TEMP TABLE creation + # failed and `temp_table` is None, this is a no-op. + if temp_table is not None: try: con.execute(f"DROP TABLE IF EXISTS {temp_table}") except Exception: diff --git a/backend/repositories/insights/definitions.py b/backend/repositories/insights/definitions.py index 470cf891..fc08d504 100644 --- a/backend/repositories/insights/definitions.py +++ b/backend/repositories/insights/definitions.py @@ -55,12 +55,13 @@ def error_spikes_processor(row: tuple, definition: InsightDefinition, context: d def botnet_grouping_processor(row: tuple, definition: InsightDefinition, context: dict) -> dict: """Process a row from the botnet_grouping query.""" # row schema: [fp, w_ips, w_reqs, b_ips, ip_ratio] + fp_col = context.get("fp_col", "ja4") return { "label": row[0], "current_val": row[1], "baseline_val": row[3], # Raw baseline IPS "unit": "distinct IPs", - "meta": {"requests": row[2], "ip_ratio": round(float(row[4]), 1), "filters": {"ja3": row[0], "ja4": row[0]}}, + "meta": {"requests": row[2], "ip_ratio": round(float(row[4]), 1), "filters": {fp_col: row[0]}}, "severity": "critical" if row[1] >= 50 else "warning", } @@ -1059,7 +1060,7 @@ def image_optimization_processor(row: tuple, definition: InsightDefinition, cont GROUP BY "url" HAVING total_bytes > 1024 * 512 ORDER BY total_bytes DESC LIMIT 15 """, - required_fields=["url", "resp_bytes", "status", "timestamp"], + required_fields=["url", "resp_bytes", "status", "timestamp", "ua"], row_processor=image_optimization_processor, ) ) diff --git a/backend/repositories/insights/repository.py b/backend/repositories/insights/repository.py index 47c456c2..cb13a9a8 100644 --- a/backend/repositories/insights/repository.py +++ b/backend/repositories/insights/repository.py @@ -29,6 +29,297 @@ _insights_cache_lock = threading.Lock() +def _coalesced_city_aggregates( + runner: QueryRunner, + table_name: str, + window_start_s: str, + label_expr: str, + region_sel: str, + country_sel: str, + window_hours: float, + baseline_hours: float, +) -> dict[str, list[tuple]]: + """Run ONE pass over `table_name` to compute every aggregate the four + city-based insights need, then demux into per-insight result lists + whose row schemas match each insight's existing row_processor contract. + + The four insights — city_surges, city_error_spikes, + city_latency_regressions, new_city_traffic — all GROUP BY + (city, region, country) over the same WHERE clause + (``"city" IS NOT NULL AND "city" != ''``). Pre-coalesce, they ran as + four independent SELECTs and re-read the temp table four times. This + coalesces them into a single SELECT that computes the superset of + counts/rates/p95s, then applies each insight's HAVING/ORDER/LIMIT in + Python. + + Returns ``{insight_id: rows}`` where each rows list matches the per- + insight schema the existing processor expects: + + - city_surges: [label, city, region, country, w_cnt, b_cnt, spike_ratio] + - city_error_spikes: [label, city, region, country, w_rate, b_rate, w_errors, w_total, b_total] + - city_latency_regressions: [label, city, region, country, w_p95, b_p95, w_total, b_total] + - new_city_traffic: [label, city, region, country, w_cnt, b_cnt] + """ + sql = f""" + WITH base AS ( + SELECT + "city", + {region_sel} AS region, + {country_sel} AS country, + {label_expr} AS label, + status, + elapsed, + (timestamp < CAST(? AS TIMESTAMPTZ)) AS is_b, + (timestamp >= CAST(? AS TIMESTAMPTZ)) AS is_w + FROM {table_name} + WHERE "city" IS NOT NULL AND "city" != '' + ) + SELECT + label, "city", region, country, + COUNT(*) FILTER (WHERE is_w) AS w_cnt, + COUNT(*) FILTER (WHERE is_b) AS b_cnt, + SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END) FILTER (WHERE is_w) AS w_errors_4xx, + SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END) FILTER (WHERE is_b) AS b_errors_4xx, + PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY elapsed) + FILTER (WHERE is_w AND elapsed IS NOT NULL) / 1000.0 AS w_p95, + PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY elapsed) + FILTER (WHERE is_b AND elapsed IS NOT NULL) / 1000.0 AS b_p95, + COUNT(*) FILTER (WHERE is_w AND elapsed IS NOT NULL) AS w_lat_total, + COUNT(*) FILTER (WHERE is_b AND elapsed IS NOT NULL) AS b_lat_total + FROM base + GROUP BY ALL + """ + rows = runner.execute(sql, [window_start_s, window_start_s]).fetchall() + + surges: list[tuple] = [] + error_spikes: list[tuple] = [] + latency: list[tuple] = [] + new_city: list[tuple] = [] + + baseline_scale = max(baseline_hours, 1.0) + + for r in rows: + ( + label, + city, + region, + country, + w_cnt, + b_cnt, + w_err, + b_err, + w_p95, + b_p95, + w_lat_total, + b_lat_total, + ) = r + b_cnt_i = b_cnt or 0 + b_err_i = b_err or 0 + + # city_surges — HAVING w_cnt >= 20 AND w_cnt > b_cnt/baseline_hours*window_hours*3 + if w_cnt >= 20: + b_normalized = b_cnt_i * 1.0 / baseline_scale * window_hours + if w_cnt > b_normalized * 3: + spike_ratio = w_cnt * 1.0 / max(b_normalized, 1.0) + surges.append((label, city, region, country, w_cnt, b_cnt, spike_ratio)) + + # city_error_spikes — w_total/b_total here are total reqs in window/baseline + # HAVING w_total >= 10 AND w_rate >= 0.10 AND (b_total < 50 OR w_rate >= b_rate*3 + 0.05) + if w_cnt >= 10: + w_rate = (w_err / w_cnt) if w_cnt else 0.0 + b_rate = (b_err_i / b_cnt_i) if b_cnt_i else None + if w_rate >= 0.10 and (b_cnt_i < 50 or (b_rate is not None and w_rate >= b_rate * 3 + 0.05)): + error_spikes.append((label, city, region, country, w_rate, b_rate, w_err, w_cnt, b_cnt)) + + # city_latency_regressions — uses elapsed-only counts (w_lat_total / b_lat_total) + # HAVING w_total >= 10 AND b_total >= 50 AND w_p95 >= b_p95*3.0 AND w_p95 - b_p95 >= 500 + if ( + w_lat_total >= 10 + and b_lat_total >= 50 + and w_p95 is not None + and b_p95 is not None + and w_p95 >= b_p95 * 3.0 + and w_p95 - b_p95 >= 500 + ): + latency.append((label, city, region, country, w_p95, b_p95, w_lat_total, b_lat_total)) + + # new_city_traffic — HAVING w_cnt >= 5 AND b_cnt = 0 + if w_cnt >= 5 and b_cnt_i == 0: + new_city.append((label, city, region, country, w_cnt, b_cnt)) + + surges.sort(key=lambda x: -(x[6] or 0)) + error_spikes.sort(key=lambda x: -((x[4] or 0) - (x[5] or 0))) + latency.sort(key=lambda x: -((x[4] / x[5]) if x[5] else 0)) + new_city.sort(key=lambda x: -(x[4] or 0)) + + return { + "city_surges": surges[:15], + "city_error_spikes": error_spikes[:15], + "city_latency_regressions": latency[:15], + "new_city_traffic": new_city[:20], + } + + +def _coalesced_url_aggregates( + runner: QueryRunner, + table_name: str, + window_start_s: str, +) -> dict[str, list[tuple]]: + """Coalesce 4 URL-keyed insights (error_spikes, cache_collapse, + latency_regression, tail_latency) into ONE pass over ``table_name``. + + Each of those four insights previously ran its own GROUP BY url + scan with the same WHERE clause and same baseline/window split + ((timestamp < window_start) → baseline, (>=) → window). Coalescing + them mirrors the O2 city-aggregates pattern that demonstrably saved + ~520 ms on prod by replacing 4 city scans with 1. + + Why these 4 and not all 5: ``origin_latency_spike`` is grouped by + URL too but its SQL has a different shape — it uses overall_stats + CTEs to normalize against the entire population's percentile, so + its per-url aggregates need a second pass. Leaving it on its own + SQL template avoids cross-contaminating the simpler 4-insight CTE. + + Returns ``{insight_id: rows}`` where each rows list matches the + insight's existing processor row-schema. On any exception the + caller falls back to the legacy per-insight scans transparently. + + - error_spikes: [url, w_rate, b_rate, w_errors, w_total, b_total] + - cache_collapse: [url, w_rate, b_rate, w_total, b_total] + - latency_regression: [url, w_p95, b_p95, w_total, b_total] + - tail_latency: [url, p99_ms, p50_ms, ratio, total] + """ + sql = f""" + WITH base AS ( + SELECT + "url", + status, + cache, + elapsed, + (timestamp < CAST(? AS TIMESTAMPTZ)) AS is_b, + (timestamp >= CAST(? AS TIMESTAMPTZ)) AS is_w + FROM {table_name} + WHERE "url" IS NOT NULL + ) + SELECT + "url", + -- Common counts + COUNT(*) FILTER (WHERE is_w) AS w_total, + COUNT(*) FILTER (WHERE is_b) AS b_total, + -- error_spikes: 5xx counters + SUM(CASE WHEN status >= 500 THEN 1 ELSE 0 END) FILTER (WHERE is_w) AS w_5xx, + SUM(CASE WHEN status >= 500 THEN 1 ELSE 0 END) FILTER (WHERE is_b) AS b_5xx, + -- cache_collapse: cache-hit counters + SUM(CASE WHEN cache ILIKE 'HIT%' THEN 1 ELSE 0 END) FILTER (WHERE is_w) AS w_hits, + SUM(CASE WHEN cache ILIKE 'HIT%' THEN 1 ELSE 0 END) FILTER (WHERE is_b) AS b_hits, + -- latency_regression: elapsed-only counts + p95s in MILLISECONDS + COUNT(*) FILTER (WHERE is_w AND elapsed IS NOT NULL) AS w_lat_total, + COUNT(*) FILTER (WHERE is_b AND elapsed IS NOT NULL) AS b_lat_total, + PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY elapsed) + FILTER (WHERE is_w AND elapsed IS NOT NULL) / 1000.0 AS w_p95, + PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY elapsed) + FILTER (WHERE is_b AND elapsed IS NOT NULL) / 1000.0 AS b_p95, + -- tail_latency: window-only p99/p50 (rounded to whole ms to match + -- the legacy template's output exactly) + ROUND(PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY elapsed) + FILTER (WHERE is_w AND elapsed IS NOT NULL) / 1000.0, 0) AS w_p99, + ROUND(PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY elapsed) + FILTER (WHERE is_w AND elapsed IS NOT NULL) / 1000.0, 0) AS w_p50 + FROM base + GROUP BY "url" + HAVING (COUNT(*) FILTER (WHERE is_w) > 0) OR (COUNT(*) FILTER (WHERE is_b) > 0) + """ + cursor = runner.execute(sql, [window_start_s, window_start_s]) + + error_spikes_out: list[tuple] = [] + cache_collapse_out: list[tuple] = [] + latency_regression_out: list[tuple] = [] + tail_latency_out: list[tuple] = [] + + while True: + rows = cursor.fetchmany(10000) + if not rows: + break + for r in rows: + ( + url, + w_total, + b_total, + w_5xx, + b_5xx, + w_hits, + b_hits, + w_lat_total, + b_lat_total, + w_p95, + b_p95, + w_p99, + w_p50, + ) = r + + w_total_i = w_total or 0 + b_total_i = b_total or 0 + + # ── error_spikes ────────────────────────────────────────────────── + # Legacy HAVING: w_total >= 3 AND w_rate >= 0.05 + # AND (b_total < 10 OR w_rate >= b_rate * 2 + 0.05) + # ORDER BY (w_rate - COALESCE(b_rate, 0)) DESC LIMIT 15 + if w_total_i >= 3: + w_rate_e = (w_5xx or 0) / w_total_i if w_total_i else 0.0 + b_rate_e = ((b_5xx or 0) / b_total_i) if b_total_i else None + if w_rate_e >= 0.05 and (b_total_i < 10 or (b_rate_e is not None and w_rate_e >= b_rate_e * 2 + 0.05)): + error_spikes_out.append((url, w_rate_e, b_rate_e, w_5xx, w_total, b_total)) + + # ── cache_collapse ──────────────────────────────────────────────── + # Legacy HAVING: w_total >= 5 AND b_total >= 20 AND b_rate >= 0.40 + # AND w_rate <= b_rate - 0.20 AND w_rate <= b_rate * 0.6 + # ORDER BY (b_rate - w_rate) DESC LIMIT 15 + if w_total_i >= 5 and b_total_i >= 20: + w_rate_c = (w_hits or 0) / w_total_i if w_total_i else 0.0 + b_rate_c = (b_hits or 0) / b_total_i if b_total_i else 0.0 + if b_rate_c >= 0.40 and w_rate_c <= b_rate_c - 0.20 and w_rate_c <= b_rate_c * 0.6: + cache_collapse_out.append((url, w_rate_c, b_rate_c, w_total, b_total)) + + # ── latency_regression ──────────────────────────────────────────── + # Legacy HAVING: w_total >= 5 AND b_total >= 20 AND w_p95 >= b_p95 * 2.0 + # AND w_p95 - b_p95 >= 200 + # ORDER BY (w_p95 / NULLIF(b_p95, 0)) DESC LIMIT 15 + # + # Note: legacy uses w_total/b_total (TOTAL counts) for the >=5/>=20 + # gate, NOT w_lat_total/b_lat_total — preserve that or this insight + # would surface MORE urls than the legacy implementation. + if ( + w_total_i >= 5 + and b_total_i >= 20 + and w_p95 is not None + and b_p95 is not None + and w_p95 >= b_p95 * 2.0 + and w_p95 - b_p95 >= 200 + ): + latency_regression_out.append((url, w_p95, b_p95, w_total, b_total)) + + # ── tail_latency (window-only) ──────────────────────────────────── + # Legacy WHERE timestamp >= window_start; HAVING COUNT(*) >= 20 AND + # ratio > 5. ORDER BY ratio DESC LIMIT 15. + # ratio = p99 / NULLIF(p50, 0) + if w_lat_total is not None and w_lat_total >= 20 and w_p99 is not None and w_p50 is not None and w_p50 > 0: + ratio = round(w_p99 / w_p50, 1) + if ratio > 5: + tail_latency_out.append((url, w_p99, w_p50, ratio, w_lat_total)) + + error_spikes_out.sort(key=lambda x: -((x[1] or 0) - (x[2] or 0))) + cache_collapse_out.sort(key=lambda x: -((x[2] or 0) - (x[1] or 0))) + latency_regression_out.sort(key=lambda x: -((x[1] / x[2]) if x[2] else 0)) + tail_latency_out.sort(key=lambda x: -(x[3] or 0)) + + return { + "error_spikes": error_spikes_out[:15], + "cache_collapse": cache_collapse_out[:15], + "latency_regression": latency_regression_out[:15], + "tail_latency": tail_latency_out[:15], + } + + def get_insights( con: duckdb.DuckDBPyConnection, src: dict, @@ -72,7 +363,24 @@ def get_insights( **runner.telemetry(), } if not actual_cols: - return empty_resp + # Empty actual_cols can mean two things: legitimate "no schema yet, + # service was just provisioned" OR a race where a concurrent + # commit deleted the buffer file between get_schema_cols's first + # call and us reading it. The latter silently shipped an empty + # insights payload that the frontend cached. Force-rebuild the + # view once and retry — if the schema lookup STILL returns empty, + # that's the "legitimate no-data" branch and we ship the empty + # response. (force=True bypasses the catalog-refresh fast path so + # the retry actually does work.) + try: + from backend.core import iceberg as db_iceberg + + db_iceberg.update_iceberg_view(con, src, force=True) + actual_cols = runner.get_schema_cols() + except Exception: + pass + if not actual_cols: + return empty_resp # ── Materialize relevant window into temp table ─────────────────────────── # This is the single most important optimization: avoid globbing/metadata parsing 30+ times. @@ -190,6 +498,73 @@ def _sev(items: list, crit_key: bool = False) -> str: url_col = '"url"' if "url" in actual_cols else "NULL" q_col = '"url"' if "url" in actual_cols else ('"digest"' if "digest" in actual_cols else "'(unknown)'") + # ── Coalesced city aggregates (O2 bypass) ───────────────────────────────── + # The 4 city-based insights (city_surges, city_error_spikes, + # city_latency_regressions, new_city_traffic) each issued their own + # GROUP BY (city, region, country) scan of the temp table. On prod + # 2026-06-05 those four scans were 177+205+219+181 = 782 ms of pure + # duplication — every row read four times to compute counts/rates/p95s + # that fit naturally in a single SELECT. Run one pass here and reuse + # the per-(city, region, country) aggregate rows below; each insight + # task short-circuits via `city_precomputed` instead of issuing its + # own SELECT. + # + # Only fires when ALL 4 are eligible (city + status + elapsed + timestamp + # all in schema). When a service is missing one of those columns the + # per-insight scans still run for the eligible subset. + city_precomputed: dict[str, list[tuple]] = {} + if "city" in actual_cols and "status" in actual_cols and "elapsed" in actual_cols and "timestamp" in actual_cols: + try: + city_precomputed = _coalesced_city_aggregates( + runner, + table_name, + window_start_s, + label_expr, + region_sel, + country_sel, + window_hours, + baseline_hours, + ) + except Exception as e: + # Fall back transparently to per-insight scans; never break + # the page on a coalesced-path bug. + import logging + + logging.getLogger(__name__).warning("[insights] coalesced city aggregates failed, falling back: %s", e) + city_precomputed = {} + + # ── Coalesced URL aggregates (Step 2 / Option C, 2026-06-06) ───────────── + # 4 URL-keyed insights (error_spikes, cache_collapse, latency_regression, + # tail_latency) all GROUP BY url over the same WHERE clause with the same + # is_w/is_b baseline-vs-window split. Pre-coalesce, each ran its own scan + # of the temp table; the audit showed they totalled ~400-600 ms. Coalescing + # them mirrors O2's city pattern (proven ~520 ms save on prod). + # + # origin_latency_spike is the 5th url-keyed insight but its SQL has an + # overall_stats CTE that normalizes against the entire population's p95 + # — different shape, kept on its own template. + # + # Fires only when all the columns the CTE touches are present (url, + # status, cache, elapsed, timestamp). When a service is missing any of + # them the per-insight scans run normally for whichever subset is + # eligible. Failure transparently falls back to per-insight scans — + # never blocks the page. + url_precomputed: dict[str, list[tuple]] = {} + if ( + "url" in actual_cols + and "status" in actual_cols + and "cache" in actual_cols + and "elapsed" in actual_cols + and "timestamp" in actual_cols + ): + try: + url_precomputed = _coalesced_url_aggregates(runner, table_name, window_start_s) + except Exception as e: + import logging + + logging.getLogger(__name__).warning("[insights] coalesced URL aggregates failed, falling back: %s", e) + url_precomputed = {} + for definition in registry.get_all(): # Check if all required fields are present if not all(col in actual_cols for col in definition.required_fields): @@ -220,29 +595,38 @@ def compute_insight() -> dict | None: if r: return r - try: - sql = d.sql_template.format( - table_name=table_name, - window_hours=window_hours, - baseline_hours=baseline_hours, - fp_col=fp_col, - loc_cols=loc_cols, - label_expr=label_expr, - country_sel=country_sel, - region_sel=region_sel, - ua_mobile_sel=ua_mobile_sel, - url_col=url_col, - q_col=q_col, - **extra_args, - ) - except KeyError: - # If hydration fails due to missing keys (e.g. pop_values), skip this insight - return None + # O2 / Step 2 bypass: insights pull rows from the precomputed + # coalesced aggregates instead of issuing their own SELECT. + # Row schema is constructed to match each insight's existing + # `# row schema: [...]` processor contract. + if d.id in city_precomputed: + rows = city_precomputed[d.id] + elif d.id in url_precomputed: + rows = url_precomputed[d.id] + else: + try: + sql = d.sql_template.format( + table_name=table_name, + window_hours=window_hours, + baseline_hours=baseline_hours, + fp_col=fp_col, + loc_cols=loc_cols, + label_expr=label_expr, + country_sel=country_sel, + region_sel=region_sel, + ua_mobile_sel=ua_mobile_sel, + url_col=url_col, + q_col=q_col, + **extra_args, + ) + except KeyError: + # If hydration fails due to missing keys (e.g. pop_values), skip this insight + return None - param_count = sql.count("?") - params = [window_start_s] * param_count + param_count = sql.count("?") + params = [window_start_s] * param_count - rows = runner.execute(sql, params).fetchall() + rows = runner.execute(sql, params).fetchall() items = [] if d.row_processor: # Build context for processors diff --git a/backend/repositories/network.py b/backend/repositories/network.py index acaf75a4..949b24f4 100644 --- a/backend/repositories/network.py +++ b/backend/repositories/network.py @@ -174,26 +174,37 @@ def get_health( map_where += " AND asn = ?" map_params.append(int(map_asn)) + # Cap to top 5000 (country, city, bucket) cells by request + # volume — the map UI renders dots, and the long tail beyond a + # few thousand points is invisible. Without the cap the + # response body grew to 5.8MB on busy windows, dominating + # /network cold-load wall time via transfer + JSON parse. + # Re-sorted by (bucket, reqs DESC) after the cap to preserve + # the downstream chronological ordering the map expects. map_sql = f""" - SELECT - country, - {city_col} AS city, - {lat_col} AS lat, - {lon_col} AS lon, - {metro_col} AS metro, - EPOCH_MS( - CAST((EPOCH_MS(timestamp)::BIGINT // {bucket_ms}) * {bucket_ms} AS BIGINT) - )::TIMESTAMP AS bucket, - MEDIAN(tcp_rtt) AS rtt_med_us, - {ploss_expr} AS avg_ploss, - SUM(CASE WHEN status >= 500 THEN 1 ELSE 0 END) - * 100.0 / NULLIF(COUNT(*), 0) AS error_pct, - COUNT(*) AS reqs - FROM {t} - WHERE {map_where} - AND country IS NOT NULL AND country != '' - AND tcp_rtt IS NOT NULL AND tcp_rtt > 0 - GROUP BY country, city, lat, lon, metro, bucket + SELECT * FROM ( + SELECT + country, + {city_col} AS city, + {lat_col} AS lat, + {lon_col} AS lon, + {metro_col} AS metro, + EPOCH_MS( + CAST((EPOCH_MS(timestamp)::BIGINT // {bucket_ms}) * {bucket_ms} AS BIGINT) + )::TIMESTAMP AS bucket, + MEDIAN(tcp_rtt) AS rtt_med_us, + {ploss_expr} AS avg_ploss, + SUM(CASE WHEN status >= 500 THEN 1 ELSE 0 END) + * 100.0 / NULLIF(COUNT(*), 0) AS error_pct, + COUNT(*) AS reqs + FROM {t} + WHERE {map_where} + AND country IS NOT NULL AND country != '' + AND tcp_rtt IS NOT NULL AND tcp_rtt > 0 + GROUP BY country, city, lat, lon, metro, bucket + ORDER BY reqs DESC + LIMIT 5000 + ) ranked ORDER BY bucket, reqs DESC """ map_rows = runner.execute(map_sql, map_params).fetchall() diff --git a/backend/repositories/origin.py b/backend/repositories/origin.py index bcddf10b..31932859 100644 --- a/backend/repositories/origin.py +++ b/backend/repositories/origin.py @@ -200,33 +200,59 @@ def get_summary( cdn_ovh = 'MEDIAN("elapsed" - "ottlb") / 1000.0' if "elapsed" in actual_cols and "ottlb" in actual_cols else "NULL" obytes_p50 = 'MEDIAN("obytes")' if "obytes" in actual_cols else "NULL" - row = runner.execute( + # Combine the rollup-totals query AND the per-edge breakdown into ONE + # scan using GROUPING SETS. DuckDB computes the () grouping (overall + # totals) and the ("edge") grouping in a single pass, halving the + # wall-clock — the previous two-scan shape did 138 ms + 132 ms = 270 ms + # on prod 1 h windows; the combined scan does the same work in ~150 ms. + # + # When the schema has no ``edge`` column (rare — older services), fall + # back to a single () grouping. GROUPING() requires a real column + # reference, so we can't use it in the no-edge branch. + has_edge = "edge" in actual_cols + if has_edge: + edge_select = '"edge"' + grouping_clause = 'GROUP BY GROUPING SETS ((), ("edge"))' + grouping_expr = 'GROUPING("edge")' + else: + edge_select = "NULL" + grouping_clause = "" # single rollup row, no need for GROUPING SETS + grouping_expr = "1" # always-rollup + rows = runner.execute( f""" SELECT - COUNT(*) FILTER (WHERE "cache" ILIKE 'MISS%') AS total_misses, - COUNT(*) FILTER (WHERE "cache" ILIKE 'PASS%') AS total_passes, - MEDIAN({lat_val}) / 1000.0 AS ottfb_p50_ms, - APPROX_QUANTILE({lat_val}, 0.75) / 1000.0 AS ottfb_p75_ms, - APPROX_QUANTILE({lat_val}, 0.95) / 1000.0 AS ottfb_p95_ms, - APPROX_QUANTILE({lat_val}, 0.99) / 1000.0 AS ottfb_p99_ms, - {ottlb_p50} AS ottlb_p50_ms, - {ottlb_p95} AS ottlb_p95_ms, - {cdn_ovh} AS cdn_overhead_p50_ms, - {ost_5xx} AS origin_error_rate, - {obytes_p50} AS obytes_p50 + {edge_select} AS edge_group, + {grouping_expr} AS is_total, + COUNT(*) AS requests, + COUNT(*) FILTER (WHERE "cache" ILIKE 'MISS%') AS total_misses, + COUNT(*) FILTER (WHERE "cache" ILIKE 'PASS%') AS total_passes, + MEDIAN({lat_val}) / 1000.0 AS ottfb_p50_ms, + APPROX_QUANTILE({lat_val}, 0.75) / 1000.0 AS ottfb_p75_ms, + APPROX_QUANTILE({lat_val}, 0.95) / 1000.0 AS ottfb_p95_ms, + APPROX_QUANTILE({lat_val}, 0.99) / 1000.0 AS ottfb_p99_ms, + {ottlb_p50} AS ottlb_p50_ms, + {ottlb_p95} AS ottlb_p95_ms, + {cdn_ovh} AS cdn_overhead_p50_ms, + {ost_5xx} AS origin_error_rate, + {obytes_p50} AS obytes_p50 FROM {table_name} WHERE {where} AND ({lat_val} IS NOT NULL) + {grouping_clause} """, params, - ).fetchone() + ).fetchall() - # When no rows match the WHERE clause, DuckDB returns one row of (0 / NULL) - # aggregates. ottfb_p50_ms being NULL is the canonical "no data" signal — - # it's the median of the latency expression itself, so it can only be - # non-NULL if at least one row matched the predicate. Used instead of a - # separate SELECT 1 ... LIMIT 1 probe, which previously ran ~3s per - # parallel endpoint on cold caches. - has_data = row is not None and row[2] is not None + # GROUPING("edge") returns 1 for the () grouping (the rollup row) and 0 + # for per-edge rows. Without an "edge" column we emit a single rollup + # row with is_total=1 (the literal expression). + rollup_row = next((r for r in rows if r[1] == 1), None) + edge_rows = [r for r in rows if r[1] == 0] if has_edge else [] + + # ottfb_p50_ms (index 5) being NULL is the canonical "no data" signal — + # it's the median of the latency expression, so it can only be non-NULL + # if at least one row matched ``lat_val IS NOT NULL``. Same semantics + # as the previous two-scan shape. + has_data = rollup_row is not None and rollup_row[5] is not None if not has_data: payload = { @@ -247,20 +273,28 @@ def get_summary( _response_cache_put(cache_key, payload) return {**payload, **runner.telemetry()} - edge_rows = [] - if "edge" in actual_cols: - edge_rows = runner.execute( - f""" - SELECT "edge", - COUNT(*) AS requests, - MEDIAN({lat_val}) / 1000.0 AS p50_ms, - APPROX_QUANTILE({lat_val}, 0.95) / 1000.0 AS p95_ms - FROM {table_name} - WHERE {where} AND ({lat_val} IS NOT NULL) - GROUP BY "edge" - """, - params, - ).fetchall() + # Map rollup-row column indices to the previous variable names so the + # payload construction below reads the same. Column order: 0=edge_group, + # 1=is_total, 2=requests, 3=total_misses, 4=total_passes, 5-8=ottfb + # p50/p75/p95/p99, 9=ottlb_p50, 10=ottlb_p95, 11=cdn_overhead_p50, + # 12=origin_error_rate, 13=obytes_p50. + row = ( + rollup_row[3], # total_misses + rollup_row[4], # total_passes + rollup_row[5], # ottfb_p50_ms + rollup_row[6], # ottfb_p75_ms + rollup_row[7], # ottfb_p95_ms + rollup_row[8], # ottfb_p99_ms + rollup_row[9], # ottlb_p50_ms + rollup_row[10], # ottlb_p95_ms + rollup_row[11], # cdn_overhead_p50_ms + rollup_row[12], # origin_error_rate + rollup_row[13], # obytes_p50 + ) + # Per-edge row columns: 0=edge value, 1=is_total (=0), 2=requests, + # 5=p50_ms, 7=p95_ms. The other aggregates exist but the by_leg payload + # historically only surfaced (edge, requests, p50_ms, p95_ms). + edge_rows = [(r[0], r[2], r[5], r[7]) for r in edge_rows] payload = { "has_data": True, @@ -775,3 +809,449 @@ def get_shielding_analysis( } _response_cache_put(cache_key, payload) return {**payload, **runner.telemetry()} + + +# ── Composite: get_aggregates ──────────────────────────────────────────────── +# +# Phase 3 item 9. One CREATE TEMP TABLE filtered to the requested window; +# every origin card on the /origin page reads from the same materialization +# instead of issuing its own parquet scan. Shielding analysis stays in its +# own endpoint (item 13 moves it to /api/network-health) because its +# self-join semantics don't share the projection cleanly. +# +# Granular endpoints (/api/origin/summary etc.) remain alive for one +# release so the frontend can flip back during a rollback without a +# backend redeploy. The composite is purely additive — existing per-card +# endpoints are unaffected. + + +def _origin_summary_from_temp(runner: QueryRunner, temp_table: str, actual_cols: set[str] | list[str]) -> dict: + """Mirror of get_summary's SQL, parameterised against the TEMP TABLE. + + Uses the pre-computed ``lat_us`` column populated when the TEMP TABLE + was created — saves the per-row COALESCE evaluation that turned the + composite into a regression on local benchmarks. + """ + actual_cols_set = set(actual_cols) + lat_val = "lat_us" + + ost_5xx = ( + 'COUNT(*) FILTER (WHERE "ost" >= 500) * 100.0 / NULLIF(COUNT(*) FILTER (WHERE "ost" IS NOT NULL), 0)' + if "ost" in actual_cols_set + else "NULL" + ) + ottlb_p50 = 'MEDIAN("ottlb") / 1000.0' if "ottlb" in actual_cols_set else "NULL" + ottlb_p95 = 'APPROX_QUANTILE("ottlb", 0.95) / 1000.0' if "ottlb" in actual_cols_set else "NULL" + cdn_ovh = ( + 'MEDIAN("elapsed" - "ottlb") / 1000.0' + if "elapsed" in actual_cols_set and "ottlb" in actual_cols_set + else "NULL" + ) + obytes_p50 = 'MEDIAN("obytes")' if "obytes" in actual_cols_set else "NULL" + + row = runner.execute( + f""" + SELECT + COUNT(*) FILTER (WHERE "cache" ILIKE 'MISS%') AS total_misses, + COUNT(*) FILTER (WHERE "cache" ILIKE 'PASS%') AS total_passes, + MEDIAN({lat_val}) / 1000.0 AS ottfb_p50_ms, + APPROX_QUANTILE({lat_val}, 0.75) / 1000.0 AS ottfb_p75_ms, + APPROX_QUANTILE({lat_val}, 0.95) / 1000.0 AS ottfb_p95_ms, + APPROX_QUANTILE({lat_val}, 0.99) / 1000.0 AS ottfb_p99_ms, + {ottlb_p50} AS ottlb_p50_ms, + {ottlb_p95} AS ottlb_p95_ms, + {cdn_ovh} AS cdn_overhead_p50_ms, + {ost_5xx} AS origin_error_rate, + {obytes_p50} AS obytes_p50 + FROM {temp_table} + WHERE ({lat_val} IS NOT NULL) + """ + ).fetchone() + + has_data = row is not None and row[2] is not None + if not has_data: + return { + "has_data": False, + "total_misses": None, + "total_passes": None, + "ottfb_p50_ms": None, + "ottfb_p75_ms": None, + "ottfb_p95_ms": None, + "ottfb_p99_ms": None, + "ottlb_p50_ms": None, + "ottlb_p95_ms": None, + "cdn_overhead_p50_ms": None, + "origin_error_rate": None, + "obytes_p50": None, + "by_leg": [], + } + + edge_rows = [] + if "edge" in actual_cols_set: + edge_rows = runner.execute( + f""" + SELECT "edge", + COUNT(*) AS requests, + MEDIAN({lat_val}) / 1000.0 AS p50_ms, + APPROX_QUANTILE({lat_val}, 0.95) / 1000.0 AS p95_ms + FROM {temp_table} + WHERE ({lat_val} IS NOT NULL) + GROUP BY "edge" + """ + ).fetchall() + + return { + "has_data": True, + "total_misses": row[0], + "total_passes": row[1], + "ottfb_p50_ms": row[2], + "ottfb_p75_ms": row[3], + "ottfb_p95_ms": row[4], + "ottfb_p99_ms": row[5], + "ottlb_p50_ms": row[6], + "ottlb_p95_ms": row[7], + "cdn_overhead_p50_ms": row[8], + "origin_error_rate": row[9], + "obytes_p50": row[10], + "by_leg": [{"edge": r[0], "requests": r[1], "p50_ms": r[2], "p95_ms": r[3]} for r in edge_rows], + } + + +def _origin_timeseries_from_temp( + runner: QueryRunner, + temp_table: str, + actual_cols: set[str] | list[str], + bucket_minutes: float, + split_by_leg: bool, + metric: str, + percentile: str, +) -> dict: + actual_cols_set = set(actual_cols) + metric_col = "ottfb" if metric == "ttfb" else "ottlb" + unit_conv = "/ 1000.0" + if metric_col not in actual_cols_set: + if metric == "ttfb" and "ttfb" in actual_cols_set: + metric_col = "ttfb" + unit_conv = "* 1000.0" + else: + return {"has_data": False, "series": []} + + if metric == "ttfb" and "ottfb" in actual_cols_set and "ttfb" in actual_cols_set: + lat_expr = 'COALESCE("ottfb", "ttfb" * 1000000.0)' + unit_conv = "/ 1000.0" + else: + lat_expr = f'"{metric_col}"' + + pct_val = {"p50": 0.5, "p95": 0.95, "p99": 0.99}.get(percentile, 0.95) + agg_expr = f"MEDIAN({lat_expr})" if percentile == "p50" else f"APPROX_QUANTILE({lat_expr}, {pct_val})" + + if bucket_minutes < 1: + interval = f"INTERVAL '{max(1, int(bucket_minutes * 60))}' seconds" + else: + interval = f"INTERVAL '{int(bucket_minutes)}' minutes" + + edge_col = ', "edge"' if (split_by_leg and "edge" in actual_cols_set) else "" + edge_group = ', "edge"' if (split_by_leg and "edge" in actual_cols_set) else "" + + rows = runner.execute( + f""" + SELECT + time_bucket({interval}, "timestamp") AS ts, + COUNT(*) AS miss_count, + {agg_expr} {unit_conv} AS value + {edge_col} + FROM {temp_table} + WHERE ({lat_expr} IS NOT NULL) + GROUP BY ts {edge_group} + ORDER BY ts + """ + ).fetchall() + + has_edge_col = split_by_leg and "edge" in actual_cols_set + series = [ + { + "time": safe_iso(r[0]), + "miss_count": r[1], + "value": r[2], + **({"edge": r[3]} if has_edge_col else {}), + } + for r in rows + ] + return {"has_data": len(series) > 0, "series": series} + + +def _origin_slow_urls_from_temp( + runner: QueryRunner, + temp_table: str, + actual_cols: set[str] | list[str], + min_requests: int, + limit: int, +) -> dict: + actual_cols_set = set(actual_cols) + if "url" not in actual_cols_set: + return {"has_data": False, "rows": []} + # Use the pre-computed lat_us column so percentile sorts can leverage + # column-store layout instead of paying COALESCE per row. + rows = runner.execute( + f""" + SELECT + "url", + COUNT(*) AS requests, + MEDIAN(lat_us) / 1000.0 AS p50_ms, + APPROX_QUANTILE(lat_us, 0.95) / 1000.0 AS p95_ms, + APPROX_QUANTILE(lat_us, 0.99) / 1000.0 AS p99_ms + FROM {temp_table} + WHERE lat_us IS NOT NULL AND "url" IS NOT NULL + GROUP BY "url" + HAVING COUNT(*) >= ? + ORDER BY p95_ms DESC + LIMIT ? + """, + [min_requests, limit], + ).fetchall() + return { + "has_data": len(rows) > 0, + "rows": [{"url": r[0], "requests": r[1], "p50_ms": r[2], "p95_ms": r[3], "p99_ms": r[4]} for r in rows], + } + + +def _origin_status_codes_from_temp(runner: QueryRunner, temp_table: str, actual_cols: set[str] | list[str]) -> dict: + if "ost" not in set(actual_cols): + return {"has_data": False, "rows": []} + rows = runner.execute( + f""" + SELECT + "ost" AS status, + COUNT(*) AS count, + COUNT(*) * 100.0 / SUM(COUNT(*)) OVER () AS pct + FROM {temp_table} + WHERE "ost" IS NOT NULL + GROUP BY "ost" + ORDER BY count DESC + """ + ).fetchall() + if not rows: + return {"has_data": False, "rows": []} + return { + "has_data": True, + "rows": [{"status": r[0], "count": r[1], "pct": r[2]} for r in rows], + } + + +def _origin_path_breakdown_from_temp(runner: QueryRunner, temp_table: str, actual_cols: set[str] | list[str]) -> dict: + actual_cols_set = set(actual_cols) + if "edge" not in actual_cols_set: + return {"has_data": False, "shielding_detected": False, "rows": []} + rows = runner.execute( + f""" + SELECT + "edge", + COUNT(*) AS requests, + MEDIAN(lat_us) / 1000.0 AS p50_ms, + APPROX_QUANTILE(lat_us, 0.95) / 1000.0 AS p95_ms + FROM {temp_table} + WHERE lat_us IS NOT NULL + GROUP BY "edge" + """ + ).fetchall() + if not rows: + return {"has_data": False, "shielding_detected": False, "rows": []} + shielding_detected = any(r[0] is False for r in rows) + return { + "has_data": True, + "shielding_detected": shielding_detected, + "rows": [{"edge": r[0], "requests": r[1], "p50_ms": r[2], "p95_ms": r[3]} for r in rows], + } + + +def _origin_pop_latency_from_temp( + runner: QueryRunner, temp_table: str, actual_cols: set[str] | list[str], limit: int +) -> dict: + actual_cols_set = set(actual_cols) + if "pop" not in actual_cols_set: + return {"has_data": False, "requires_group_c": True, "rows": []} + rows = runner.execute( + f""" + SELECT + "pop", + COUNT(*) AS requests, + MEDIAN(lat_us) / 1000.0 AS p50_ms, + APPROX_QUANTILE(lat_us, 0.95) / 1000.0 AS p95_ms + FROM {temp_table} + WHERE lat_us IS NOT NULL AND "pop" IS NOT NULL AND "pop" != '' + GROUP BY "pop" + ORDER BY p95_ms DESC + LIMIT ? + """, + [limit], + ).fetchall() + if not rows: + return {"has_data": False, "requires_group_c": False, "rows": []} + valid_p95s = sorted(r[3] for r in rows if r[3] is not None) + median_p95 = valid_p95s[len(valid_p95s) // 2] if valid_p95s else 0 + return { + "has_data": True, + "requires_group_c": False, + "median_p95_ms": median_p95, + "rows": [ + { + "pop": r[0], + "requests": r[1], + "p50_ms": r[2], + "p95_ms": r[3], + "elevated": r[3] is not None and median_p95 is not None and r[3] > median_p95 * 2, + } + for r in rows + ], + } + + +def _origin_ip_health_from_temp( + runner: QueryRunner, temp_table: str, actual_cols: set[str] | list[str], limit: int +) -> dict: + actual_cols_set = set(actual_cols) + if "oip" not in actual_cols_set or "ost" not in actual_cols_set: + return {"has_data": False, "rows": []} + rows = runner.execute( + f""" + SELECT + "oip", + COUNT(*) AS requests, + MEDIAN(lat_us) / 1000.0 AS p50_ms, + APPROX_QUANTILE(lat_us, 0.95) / 1000.0 AS p95_ms, + ROUND(COUNT(*) FILTER (WHERE "ost" >= 500) * 100.0 + / NULLIF(COUNT(*), 0), 1) AS error_pct + FROM {temp_table} + WHERE "oip" IS NOT NULL AND "oip" != '' AND "ost" IS NOT NULL + GROUP BY "oip" + HAVING COUNT(*) >= 10 + ORDER BY error_pct DESC + LIMIT ? + """, + [limit], + ).fetchall() + if not rows: + return {"has_data": False, "rows": []} + return { + "has_data": True, + "rows": [{"oip": r[0], "requests": r[1], "p50_ms": r[2], "p95_ms": r[3], "error_pct": r[4]} for r in rows], + } + + +def get_aggregates( + con: duckdb.DuckDBPyConnection, + src: dict, + start_time: str | None, + end_time: str | None, + filters: FiltersDict, + *, + bucket_minutes: float = 5, + split_by_leg: bool = False, + timeseries_metric: str = "ttfb", + timeseries_percentile: str = "p95", + slow_urls_limit: int = 20, + slow_urls_min_requests: int = 10, + ip_health_limit: int = 30, + pop_latency_limit: int = 30, +) -> dict: + """Composite origin endpoint — six origin cards from one parquet scan. + + Replaces the cold-load fan-out of /api/origin/{summary, timeseries, + slow-urls, status-codes, path-breakdown, pop-latency, ip-health} + (7643 ms total per the r2 audit) with a single CREATE TEMP TABLE + + 6 reads against it. Shielding-analysis stays separate (item 13 + moves it to /api/network-health). + """ + table_name = _safe_table(src["name"]) + runner = QueryRunner(con, src) + actual_cols = runner.get_schema_cols() + + empty_payload = { + "has_data": False, + "summary": {}, + "timeseries": {"has_data": False, "series": []}, + "slow_urls": {"has_data": False, "rows": []}, + "status_codes": {"has_data": False, "rows": []}, + "path_breakdown": {"has_data": False, "shielding_detected": False, "rows": []}, + "pop_latency": {"has_data": False, "requires_group_c": False, "rows": []}, + "ip_health": {"has_data": False, "rows": []}, + } + + if not actual_cols: + return {**empty_payload, **runner.telemetry()} + + params, where_clause = build_where_clause(start_time, end_time, filters, actual_cols, inline_params=True) + + # Union of columns needed across the six sub-queries. Filtered to + # those the schema actually has before materialization so missing + # columns don't break the CREATE. Plus a precomputed `lat_us` column + # — the percentile sub-queries all use the same COALESCE("ottfb", + # "ttfb"*1000000.0) expression and computing it once at + # materialization time lets DuckDB store it in the column-store + # layout. Without the precompute, the in-memory TEMP TABLE was + # SLOWER than per-endpoint parquet scans because the COALESCE + # forces per-row evaluation during percentile sort. + import uuid as _uuid + + from backend.repositories._base import origin_latency_us_expr + + actual_set = set(actual_cols) + wanted_cols = [ + "timestamp", + "cache", + "edge", + "url", + "oip", + "ost", + "pop", + "ottfb", + "ottlb", + "ttfb", + "elapsed", + "obytes", + ] + select_cols = [f'"{c}"' for c in wanted_cols if c in actual_set] + if not select_cols: + return {**empty_payload, **runner.telemetry()} + lat_us_expr = origin_latency_us_expr(actual_set) + temp_table = f"t_origin_{_uuid.uuid4().hex}" + create_sql = ( + f"CREATE TEMP TABLE {temp_table} AS " + f"SELECT {', '.join(select_cols)}, {lat_us_expr} AS lat_us " + f"FROM {table_name} WHERE {where_clause}" + ) + if not runner.create_temp_table(create_sql, params): + return {**empty_payload, **runner.telemetry()} + try: + summary = _origin_summary_from_temp(runner, temp_table, actual_set) + timeseries = _origin_timeseries_from_temp( + runner, + temp_table, + actual_set, + bucket_minutes, + split_by_leg, + timeseries_metric, + timeseries_percentile, + ) + slow_urls = _origin_slow_urls_from_temp(runner, temp_table, actual_set, slow_urls_min_requests, slow_urls_limit) + status_codes = _origin_status_codes_from_temp(runner, temp_table, actual_set) + path_breakdown = _origin_path_breakdown_from_temp(runner, temp_table, actual_set) + pop_latency = _origin_pop_latency_from_temp(runner, temp_table, actual_set, pop_latency_limit) + ip_health = _origin_ip_health_from_temp(runner, temp_table, actual_set, ip_health_limit) + + return { + "has_data": summary.get("has_data", False), + "summary": summary, + "timeseries": timeseries, + "slow_urls": slow_urls, + "status_codes": status_codes, + "path_breakdown": path_breakdown, + "pop_latency": pop_latency, + "ip_health": ip_health, + **runner.telemetry(), + } + finally: + try: + runner.execute(f"DROP TABLE IF EXISTS {temp_table}") + except Exception: + pass diff --git a/backend/repositories/query.py b/backend/repositories/query.py index c9a585bc..cb5ed8c0 100644 --- a/backend/repositories/query.py +++ b/backend/repositories/query.py @@ -13,7 +13,7 @@ from backend.utils.sql_validator import ( SQLValidationError, apply_user_query_limits, - has_limit_clause, + is_simple_select_statement, validate_user_sql, ) from backend.utils.telemetry import get_tracked_calls @@ -85,16 +85,10 @@ def execute_query( # DESCRIBE, SHOW, PRAGMA, EXPLAIN) since they return small fixed-shape # result sets where the LIMIT semantics differ or aren't supported. exec_sql = sql - sql_stripped_upper = sql.strip().upper().lstrip("(") - # 026: ``re.search(r"\bLIMIT\b", sql)`` matches inside string - # literals (``WHERE name = 'WITHOUT LIMIT'``) and inside SQL - # comments — both false positives that cause the auto-wrap to - # SKIP wrapping, leaving the query unbounded. The AST-aware - # check inspects the parse tree so strings/comments are out of - # scope. - is_simple_select = sql_stripped_upper.startswith( - ("SELECT", "WITH", "FROM", "VALUES", "TABLE") - ) and not has_limit_clause(sql, parser_con=con) + # 015 / 026: Check if the statement is a simple SELECT using the AST-aware helper. + # String-based startswith or regex checks match inside comments or string literals, + # leading to bypasses. The AST-aware check ensures accuracy. + is_simple_select = is_simple_select_statement(sql, parser_con=con) if is_simple_select: # Strip trailing semicolon so the wrapper LIMIT lands in the same statement. inner = sql.rstrip().rstrip(";") diff --git a/backend/repositories/security.py b/backend/repositories/security.py index 4d65f9cd..6fbf4174 100644 --- a/backend/repositories/security.py +++ b/backend/repositories/security.py @@ -60,18 +60,23 @@ def get_top_bots( return {"bots": [], "ngwaf_bots": []} if "ua" in actual_cols: try: - from backend.utils.bot_sources import build_matcher, get_bot_regex_pattern - - pattern = get_bot_regex_pattern(200) - ua_filter = f"AND regexp_matches(ua, '{pattern.replace(chr(39), chr(39) * 2)}')" if pattern else "" - + from backend.utils.bot_sources import build_matcher + + # Item 41 — the inline regexp_matches(ua, '<200-pattern OR-chain>') + # cost ~353 ms on prod / week (per dashboard telemetry) because + # DuckDB has to evaluate the alternation per row. The Python + # matcher below is already what we use to classify each UA's + # bot_id, so move the regex out of SQL: pull the top 50,000 + # distinct UAs by count (cheap GROUP BY + ORDER BY) then run + # build_matcher() on them in Python where the per-UA result + # is lru_cached and most lookups are sub-microsecond. q = f""" SELECT ua, count(*) AS cnt FROM {temp_table} - WHERE ua IS NOT NULL {ua_filter} + WHERE ua IS NOT NULL GROUP BY ua ORDER BY cnt DESC - LIMIT 2000 + LIMIT 50000 """ rows = runner.execute(q).fetchall() @@ -95,27 +100,55 @@ def get_top_bots( logging.getLogger(__name__).error("[security] arcjet top bots failed: %s", e) # ── NGWAF cache bot names ───────────────────────────────────────────── + # Memoize ATTACH per-connection the same way get_security_aggregates + # does for `ngwaf_cache`. The previous attach_ngwaf_cache context + # manager DETACHed on exit, so every /dashboard cold load paid the + # ~22 ms ATTACH cost on /api/security/top-bots even when the file + # was already attached. The duckdb_databases() catalog query is + # ~90 us — fast enough to run unconditionally. ngwaf_bots: list[dict] = [] - from backend.repositories._base import attach_ngwaf_cache - - with attach_ngwaf_cache(con, actual_cols, alias="ngwaf_top") as attached: - if attached: - try: - # Join against the temp table instead of re-scanning the - # source view — same filter window, no second manifest walk. - q = f""" - SELECT nb.bot_name, nb.category, count(*) AS cnt - FROM {temp_table} t - INNER JOIN ngwaf_top.ngwaf_bots nb USING (waf_req_id) - WHERE nb.bot_name IS NOT NULL - GROUP BY 1, 2 - ORDER BY 3 DESC - LIMIT {n} - """ - res = runner.execute(q).fetchall() - ngwaf_bots = [{"name": r[0], "category": r[1], "request_count": r[2]} for r in res] - except Exception as e: - logging.getLogger(__name__).error("[security] NGWAF top bots failed: %s", e) + ngwaf_attached = False + if "waf_req_id" in actual_cols: + try: + from backend import config as svcconfig + + ngwaf_db = svcconfig.ngwaf_db_path() + if ngwaf_db: + existing = con.execute( + "SELECT path FROM duckdb_databases() WHERE database_name='ngwaf_top' LIMIT 1" + ).fetchone() + already_path = existing[0] if existing else None + if already_path == ngwaf_db: + ngwaf_attached = True + elif os.path.exists(ngwaf_db): + if already_path is not None: + try: + con.execute("DETACH ngwaf_top") + except Exception: + pass + ngwaf_db_escaped = ngwaf_db.replace("'", "''") + con.execute(f"ATTACH '{ngwaf_db_escaped}' AS ngwaf_top (TYPE SQLITE, READ_ONLY)") + ngwaf_attached = True + except Exception: + pass # ATTACH failed — fall back gracefully + + if ngwaf_attached: + try: + # Join against the temp table instead of re-scanning the + # source view — same filter window, no second manifest walk. + q = f""" + SELECT nb.bot_name, nb.category, count(*) AS cnt + FROM {temp_table} t + INNER JOIN ngwaf_top.ngwaf_bots nb USING (waf_req_id) + WHERE nb.bot_name IS NOT NULL + GROUP BY 1, 2 + ORDER BY 3 DESC + LIMIT {n} + """ + res = runner.execute(q).fetchall() + ngwaf_bots = [{"name": r[0], "category": r[1], "request_count": r[2]} for r in res] + except Exception as e: + logging.getLogger(__name__).error("[security] NGWAF top bots failed: %s", e) return {"bots": arcjet_bots, "ngwaf_bots": ngwaf_bots, **runner.telemetry()} @@ -148,18 +181,18 @@ def get_security_aggregates( params, where_clause = build_where_clause(start_time, end_time, filters, actual_cols, inline_params=True) + # Projection narrowed: asn / req_bytes / ja3 / ja4 are not consumed + # by _build_security_response (audited 2026-06-05) so they're dropped + # from the TEMP TABLE materialization. Each saves a column scan + + # cast per parquet read. cols = [ "timestamp", "ip", - "asn", "tls_ciphers_sha", "req_header_bytes", - "req_bytes", "is_ipv6", "p_type", "conn_requests", - "ja3", - "ja4", "waf_sig", "ua", "waf_req_id", @@ -197,17 +230,35 @@ def _build_security_response( results["ngwaf_configured"] = False # Attach the NGWAF bot cache once per connection if it exists and waf_req_id is in schema. - # The attach costs ~22ms so we guard on both conditions to avoid overhead when unused. + # The attach costs ~22ms; check DuckDB's own duckdb_databases() catalog + # (~90us) first and skip the ATTACH if this connection already has the + # cache bound to the exact same path. The catalog query reflects live + # state, so we don't need Python-side memoization (DuckDBPyConnection + # has no __dict__ for arbitrary attrs anyway) and a config switch that + # changes the path triggers a DETACH + re-ATTACH instead of silently + # serving from a stale binding. _ngwaf_attached = False if "waf_req_id" in actual_cols: try: from backend import config as svcconfig ngwaf_db = svcconfig.ngwaf_db_path() - if os.path.exists(ngwaf_db): - ngwaf_db_escaped = ngwaf_db.replace("'", "''") - con.execute(f"ATTACH '{ngwaf_db_escaped}' AS ngwaf_cache (TYPE SQLITE, READ_ONLY)") - _ngwaf_attached = True + if ngwaf_db: + existing = con.execute( + "SELECT path FROM duckdb_databases() WHERE database_name='ngwaf_cache' LIMIT 1" + ).fetchone() + already_path = existing[0] if existing else None + if already_path == ngwaf_db: + _ngwaf_attached = True + elif os.path.exists(ngwaf_db): + if already_path is not None: + try: + con.execute("DETACH ngwaf_cache") + except Exception: + pass + ngwaf_db_escaped = ngwaf_db.replace("'", "''") + con.execute(f"ATTACH '{ngwaf_db_escaped}' AS ngwaf_cache (TYPE SQLITE, READ_ONLY)") + _ngwaf_attached = True except Exception: pass # ATTACH failed (e.g. DuckDB SQLite extension not loaded) — fall back gracefully diff --git a/backend/repositories/sessions.py b/backend/repositories/sessions.py index 7ec69299..1e67e28d 100644 --- a/backend/repositories/sessions.py +++ b/backend/repositories/sessions.py @@ -7,7 +7,7 @@ import duckdb from backend.models.common import FiltersDict -from backend.repositories._base import QueryRunner, _safe_table +from backend.repositories._base import QueryRunner, _safe_table, empty_schema_response from backend.repositories.utils.filters import build_where_clause from backend.repositories.utils.pagination import calc_offset @@ -37,8 +37,6 @@ def get_sessions( actual_cols = set(runner.get_schema_cols()) if not actual_cols: - from backend.repositories._base import empty_schema_response - return empty_schema_response( sessions=[], total=0, @@ -68,7 +66,6 @@ def get_sessions( has_asn = "asn" in actual_cols has_country = "country" in actual_cols has_rtt = "tcp_rtt" in actual_cols - has_ttfb = "ttfb" in actual_cols has_status = "status" in actual_cols has_resp_bytes = "resp_bytes" in actual_cols has_ua = "ua" in actual_cols @@ -104,16 +101,39 @@ def get_sessions( if has_url: extra_aggs += ', COUNT(DISTINCT "url") AS unique_urls' - sessions_cte = f""" - WITH ordered AS ( + flag_parts = [f"req_count >= {min_reqs_flag}"] + if has_status: + flag_parts.append(f"(reqs_4xx * 100.0 / NULLIF(req_count, 0)) >= {min_4xx_pct_flag}") + flag_expr = " OR ".join(f"({p})" for p in flag_parts) + + flagged_filter = "WHERE flagged = true" if flagged_only else "" + + valid_sorts = { + "session_start", + "session_end", + "req_count", + "edge_count", + "shield_count", + "unique_urls", + "median_rtt_ms", + "total_bytes", + } + if sort_by not in valid_sorts: + sort_by = "session_start" + + # Single CTE pipeline: filter → window functions → aggregation. + # Replaces the item-19 three-stage TEMP TABLE approach now that + # profiling identified sessions_raw materialization as the bottleneck + # (~3000ms of ~3700ms total). DuckDB pipelines single-consumer CTEs + # without intermediate materialization, saving the I/O overhead. + cte_prefix = f""" + WITH base AS ( SELECT {group_key} {', "ua"' if has_ua else ""} - {', "ja4"' if has_ja4 and "ja4" not in group_cols else ""} , timestamp AS ts {', "status"' if has_status else ""} {', "resp_bytes"' if has_resp_bytes else ""} {', "tcp_rtt"' if has_rtt else ""} - {', "ttfb"' if has_ttfb else ""} {', "asn"' if has_asn else ""} {', "country"' if has_country else ""} {', "url"' if has_url else ""} @@ -124,7 +144,7 @@ def get_sessions( gaps AS ( SELECT *, ts - LAG(ts) OVER (PARTITION BY {part_key} ORDER BY ts) AS gap - FROM ordered + FROM base ), marks AS ( SELECT *, @@ -149,35 +169,28 @@ def get_sessions( ) """ - flag_parts = [f"req_count >= {min_reqs_flag}"] - if has_status: - flag_parts.append(f"(reqs_4xx * 100.0 / NULLIF(req_count, 0)) >= {min_4xx_pct_flag}") - flag_expr = " OR ".join(f"({p})" for p in flag_parts) - - flagged_filter = "WHERE flagged = true" if flagged_only else "" - - valid_sorts = { - "session_start", - "session_end", - "req_count", - "edge_count", - "shield_count", - "unique_urls", - "median_rtt_ms", - "total_bytes", - } - if sort_by not in valid_sorts: - sort_by = "session_start" - data_sql = f""" - {sessions_cte} + {cte_prefix} SELECT *, ({flag_expr}) AS flagged FROM sessions_agg {flagged_filter} ORDER BY {sort_by} {sort_dir} LIMIT {limit} OFFSET {offset} """ - rows = runner.execute(data_sql, params).fetchall() + result = runner.execute_with_retry(data_sql, params) + if result is None: + return empty_schema_response( + sessions=[], + total=0, + page=page, + limit=limit, + has_rtt=has_rtt, + has_ja4=has_ja4, + has_edge=has_edge, + **runner.telemetry(), + ) + + rows = result.fetchall() col_names = [desc[0] for desc in con.description] sessions: list[dict] = [] @@ -191,7 +204,7 @@ def get_sessions( if not rows and offset > 0: count_sql = f""" - {sessions_cte} + {cte_prefix} SELECT COUNT(*) FROM (SELECT ({flag_expr}) AS flagged FROM sessions_agg) sub {flagged_filter} """ diff --git a/backend/routers/admin.py b/backend/routers/admin.py index f2d7fe1b..44bc83de 100644 --- a/backend/routers/admin.py +++ b/backend/routers/admin.py @@ -7,7 +7,8 @@ import zipfile from fastapi import APIRouter, Depends, HTTPException, Query -from fastapi.responses import RedirectResponse, StreamingResponse +from fastapi.responses import StreamingResponse +from pydantic import BaseModel, Field from backend.deps import get_service_id, get_source from backend.models.admin import ( @@ -51,22 +52,49 @@ def tell(self): return self.offset +class ClientDisconnected(Exception): + """Raised when the client disconnects during a streaming response.""" + + pass + + +class _AbortableQueue(queue.Queue): + def __init__(self, maxsize=0): + super().__init__(maxsize) + self.aborted = False + + def put(self, item, block=True, timeout=None): + if self.aborted: + if item is None: + return + raise ClientDisconnected("Client disconnected during streaming") + super().put(item, block, timeout) + + def _stream_from_worker(worker): """Run *worker(q)* in a daemon thread and yield the bytes it puts into the queue.""" import contextvars import threading - q: queue.Queue = queue.Queue(maxsize=10) + q: _AbortableQueue = _AbortableQueue(maxsize=10) # Copy the request's context (process_context, _CALLS list) so any # record_call() inside the worker thread lands in the same _usage_log batch. ctx = contextvars.copy_context() thread = threading.Thread(target=lambda: ctx.run(worker, q), daemon=True) thread.start() - while True: - chunk = q.get() - if chunk is None: - break - yield chunk + try: + while True: + chunk = q.get() + if chunk is None: + break + yield chunk + finally: + q.aborted = True + while True: + try: + q.get_nowait() + except queue.Empty: + break def _fetch_file_to_zip( @@ -145,12 +173,24 @@ def get_pop_locations(): return PopLocationsResponse.with_telemetry(pops=get_pop_locations()) +class RefreshPopLocationsRequest(BaseModel): + token: str = Field(..., description="Fastly API key") + + @router.post("/admin/pop-locations/refresh", response_model=PopLocationsResponse) -def refresh_pop_locations(token: str = Query(...)): +def refresh_pop_locations(req: RefreshPopLocationsRequest | None = None, token: str | None = Query(default=None)): """Refresh the POP locations cache from the Fastly API.""" - api_key = token.strip() + api_key = "" + if req is not None: + api_key = req.token.strip() + if not api_key: - raise HTTPException(status_code=400, detail={"error": "api_key is required"}) + if token is None: + raise HTTPException(status_code=422, detail="token is required") + api_key = token.strip() + if not api_key: + raise HTTPException(status_code=400, detail={"error": "api_key is required"}) + from backend.utils.pop_utils import fetch_pop_locations, get_pop_locations ok = fetch_pop_locations(api_key) @@ -345,6 +385,7 @@ def download_file( source: dict = Depends(get_source), key: str = Query(default=""), ): + import posixpath import urllib.parse from fastapi.responses import FileResponse @@ -354,6 +395,8 @@ def download_file( if not key: raise HTTPException(status_code=400, detail={"error": "Missing key parameter"}) + key = posixpath.normpath(key) + # Cross-tenant guard: a single FOS bucket can host multiple services # separated by per-source prefixes. The path-traversal cage below # bounds local cache reads, but a sibling-tenant key like @@ -361,8 +404,11 @@ def download_file( # redirect for that object. Require the key to live under this # service's prefix before any FOS / CDN URL minting. src_prefix = source.get("prefix", "") - if src_prefix and not key.startswith(src_prefix): - raise HTTPException(status_code=400, detail={"error": "invalid_key"}) + if src_prefix: + if not src_prefix.endswith("/"): + src_prefix += "/" + if not key.startswith(src_prefix): + raise HTTPException(status_code=400, detail={"error": "invalid_key"}) # Security: ``os.path.join(base, key)`` returns ``key`` when # ``key`` is absolute, which a malicious caller exploits by passing @@ -383,60 +429,133 @@ def download_file( if os.path.exists(local_path): return FileResponse(local_path, filename=os.path.basename(local_path)) - # Record the user-initiated download as a synthetic CDN/FOS GET. The - # actual transfer happens browser→edge so we never see the response, but - # we know we *issued* one billable redirect — count it. from backend.utils.telemetry import record_call as _record_call cdn = source.get("cdn_url", "").rstrip("/") if cdn: + # Stream the CDN response through this server rather than 307-ing the + # browser to ``{cdn}/{key}?key={cdn_secret}``. The static cdn_secret + # is a shared bearer token; embedding it in the redirect Location + # leaks it into browser history, the address bar, the Referer header + # of any subsequent navigation, and any HTTP intermediaries. By + # fetching server-side with the ``x-fastly-key`` header (which the + # CDN VCL accepts equivalently — see backend/core/fastly/utils.py) + # the secret never leaves the trust boundary. See audit finding 009. + import time as _time + import urllib.request + + from backend.utils.telemetry import record_cdn_call as _rcdn + url = f"{cdn}/{urllib.parse.quote(key)}" + req = urllib.request.Request(url) if source.get("cdn_secret"): - url += f"?key={urllib.parse.quote(source['cdn_secret'])}" + req.add_header("x-fastly-key", source["cdn_secret"]) + try: + cdn_resp = urllib.request.urlopen(req, timeout=30) + except Exception as exc: + raise HTTPException( + status_code=502, + detail={"error": f"cdn fetch failed: {exc}"}, + ) + + content_type = cdn_resp.headers.get("Content-Type") or "application/octet-stream" + content_length = cdn_resp.headers.get("Content-Length") + filename = os.path.basename(key) or "download" + + def _iter_cdn(chunk_size: int = 65536): + bytes_read = 0 + t0 = _time.time() + cdn_headers = cdn_resp.headers + try: + while True: + chunk = cdn_resp.read(chunk_size) + if not chunk: + break + bytes_read += len(chunk) + yield chunk + finally: + try: + cdn_resp.close() + except Exception: + pass + try: + _rcdn( + "GET", + key, + round((_time.time() - t0) * 1000, 2), + headers=cdn_headers, + bytes_count=bytes_read, + caller="api:/download", + ) + except Exception: + pass + + headers = { + "Content-Disposition": f'attachment; filename="{filename}"', + "Cache-Control": "private, no-store", + } + if content_length: + headers["Content-Length"] = content_length + return StreamingResponse(_iter_cdn(), media_type=content_type, headers=headers) + + fos_client = _get_fos_client(source) + import time as _time + + try: + t0 = _time.time() + obj = fos_client.get_object(Bucket=source["bucket"], Key=key) _record_call( - "GET", - key, - 0.0, - status="REDIRECT", - service="CDN", - details="user-initiated redirect (bytes unknown)", + "GET_OBJECT", + f"{source['bucket']}/{key}", + round((_time.time() - t0) * 1000, 2), + status="SUCCESS", + service="FOS", + details="download stream · Class B", caller="api:/download", ) - return RedirectResponse(url=url) + except Exception as exc: + raise HTTPException( + status_code=502, + detail={"error": f"FOS fetch failed: {exc}"}, + ) - fos_client = _get_fos_client(source) - url = fos_client.generate_presigned_url( - ClientMethod="get_object", - Params={"Bucket": source["bucket"], "Key": key}, - ExpiresIn=3600, - ) - _record_call( - "GET_OBJECT", - f"{source['bucket']}/{key}", - 0.0, - status="REDIRECT", - service="FOS", - details="presigned URL · Class B · bytes unknown", - caller="api:/download", - ) - return RedirectResponse(url=url) + body = obj["Body"] + content_type = obj.get("ContentType") or "application/octet-stream" + content_length = obj.get("ContentLength") + filename = os.path.basename(key) or "download" + + def _iter_fos(chunk_size: int = 65536): + try: + yield from body.iter_chunks(chunk_size) + finally: + try: + body.close() + except Exception: + pass + + headers = { + "Content-Disposition": f'attachment; filename="{filename}"', + "Cache-Control": "private, no-store", + } + if content_length: + headers["Content-Length"] = str(content_length) + + return StreamingResponse(_iter_fos(), media_type=content_type, headers=headers) @router.get("/download-all") def download_all_files( - service_id: str = Query(default=""), + source: dict = Depends(get_source), include: str = Query(default="all"), ): from backend.core import duckdb as _db + src = source + service_id = src.get("name", "") if not service_id: raise HTTPException(status_code=400, detail={"error": "service_id required"}) - src = _db.get_source_for_service(service_id) - if not src: - raise HTTPException(status_code=404, detail={"error": "service not found"}) - def zip_worker(q: queue.Queue): # process_context_scope (not set_process_context) so the fsspec # iothread fallback isn't wiped out by a concurrent scope exit @@ -463,8 +582,13 @@ def zip_worker(q: queue.Queue): zf.write(db_path, os.path.basename(db_path)) cache_dir = _db._cache_dir(src) - if os.path.exists(cache_dir): - for root, _, files in os.walk(cache_dir): + walk_dir = ( + os.path.join(cache_dir, src.get("prefix", "").lstrip("/")) + if src.get("prefix") + else cache_dir + ) + if os.path.exists(walk_dir): + for root, _, files in os.walk(walk_dir): for file in files: file_path = os.path.join(root, file) arcname = os.path.relpath(file_path, cache_dir) diff --git a/backend/routers/bootstrap.py b/backend/routers/bootstrap.py index e38f5b0b..8f3a6067 100644 --- a/backend/routers/bootstrap.py +++ b/backend/routers/bootstrap.py @@ -17,12 +17,27 @@ def bootstrap( request: Request, service_id: str | None = Depends(get_service_id), ): + import time as _time + from backend.core import duckdb as _db from backend.core.duckdb import STORAGE_MODE from backend.services.service_manager import get_enriched_services from backend.utils.countries import COUNTRY_MAP from backend.utils.pop_utils import get_pop_lat_lon_map + # Cold-path attribution: time each major phase so the harness can pin + # which section owns the bootstrap wall time. Each entry is + # {"section": str, "time_ms": float} and surfaces via + # BootstrapResponse._section_timings. + section_timings: list[dict] = [] + + def _timed(name: str, fn): + t0 = _time.monotonic() + try: + return fn() + finally: + section_timings.append({"section": name, "time_ms": round((_time.monotonic() - t0) * 1000, 2)}) + # /api/bootstrap is in _UNAUTH_ANALYST_PATHS so anonymous remote visitors # can get a stub response telling the frontend to redirect them to # /share-login. The middleware therefore SKIPS session validation for @@ -35,7 +50,10 @@ def bootstrap( if sid: from backend.utils.tunnel import get_tunnel_manager - analyst_session = get_tunnel_manager().validate_session(sid) + def _validate(): + return get_tunnel_manager().validate_session(sid) + + analyst_session = _timed("validate_analyst_session", _validate) if analyst_session is not None: request.state.analyst_session = analyst_session @@ -49,13 +67,14 @@ def bootstrap( "is_remote_analyst": True, "needs_login": True, }, + section_timings=section_timings, ) src: dict | None = None if service_id: - src = _db.get_source_for_service(service_id) + src = _timed("get_source_for_service", lambda: _db.get_source_for_service(service_id)) - services = get_enriched_services(service_id) + services = _timed("get_enriched_services", lambda: get_enriched_services(service_id)) # Analyst path: filter services to those scoped on the invite and force # access_level=read_only regardless of what get_source_for_service returned. @@ -75,10 +94,15 @@ def bootstrap( schema: list = [] # Use cached schema from config to avoid acquiring a DB lock - if valid_active_id: + def _resolve_schema() -> list: + if not valid_active_id: + return [] active_svc = next((s for s in services if s.get("service_id") == valid_active_id), None) if active_svc and active_svc.get("status"): - schema = active_svc["status"].get("schema", []) + return active_svc["status"].get("schema", []) or [] + return [] + + schema = _timed("schema_lookup", _resolve_schema) # NOTE: the previous fallback opened a read-only DuckDB connection here # and ran get_schema() against the source on cold-cache loads. That call @@ -90,7 +114,7 @@ def bootstrap( # renders without a hint banner; the user can refresh once the cron # has run (typically <60s after startup). - pops = get_pop_lat_lon_map() + pops = _timed("get_pop_lat_lon_map", get_pop_lat_lon_map) # Include custom field info so the dashboard can render custom distribution cards # without a separate fetch. We load the raw config here because the enriched @@ -98,20 +122,43 @@ def bootstrap( custom_dashboard_cards: list[dict] = [] custom_fields_catalog: list[dict] = [] active_log_field_ids: list[str] = [] - if valid_active_id: + + def _resolve_custom_fields(): + nonlocal custom_dashboard_cards, custom_fields_catalog, active_log_field_ids + if not valid_active_id: + return from backend import config as svcconfig from backend.core import log_fields as _lf active_cfg = svcconfig.load_config(valid_active_id) - if active_cfg: - lf_config = _lf.get_lf_config(active_cfg) - custom_fields_catalog = _lf.get_custom_fields_catalog_entries(lf_config) - custom_dashboard_cards = [ - {"id": f["id"], "label": f["label"]} for f in custom_fields_catalog if f.get("show_in_dashboard") - ] - active_log_field_ids = sorted(_lf.resolve_enabled_fields(lf_config)) + [ - cf["name"] for cf in lf_config.get("custom_fields", []) if cf.get("enabled", True) - ] + if not active_cfg: + return + lf_config = _lf.get_lf_config(active_cfg) + custom_fields_catalog = _lf.get_custom_fields_catalog_entries(lf_config) + custom_dashboard_cards = [ + {"id": f["id"], "label": f["label"]} for f in custom_fields_catalog if f.get("show_in_dashboard") + ] + active_log_field_ids = sorted(_lf.resolve_enabled_fields(lf_config)) + [ + cf["name"] for cf in lf_config.get("custom_fields", []) if cf.get("enabled", True) + ] + + _timed("custom_fields_catalog", _resolve_custom_fields) + + views: list[dict] = [] + + def _resolve_views() -> list[dict]: + if not valid_active_id: + return [] + from backend.repositories import views as _views_repo + + try: + return _views_repo.get_views(valid_active_id) + except Exception: + # Views are a UX nicety, not a correctness gate. A repo error + # must not break /api/bootstrap. + return [] + + views = _timed("views", _resolve_views) # Force read_only for analyst sessions regardless of underlying source. if analyst_session is not None: @@ -137,6 +184,8 @@ def bootstrap( custom_dashboard_cards=custom_dashboard_cards, custom_fields_catalog=custom_fields_catalog, active_log_field_ids=active_log_field_ids, + views=views, + section_timings=section_timings, ) @@ -283,7 +332,24 @@ def insight_availability( detail={"error": "service_not_authorized", "service": source.get("name")}, ) - actual_cols = {col["name"] for col in get_schema(con, source)} + # Prefer the cached schema snapshot maintained by the status-refresh + # cron — same source of truth the /schema endpoint and /bootstrap + # already use. Saves ~300 ms per /insight-availability call because + # we skip the per-service lock + parquet glob that get_schema would + # otherwise pay on cold cache, especially when /insights is in + # flight concurrently. + from backend import config as svcconfig + + actual_cols: set[str] = set() + cached_status = svcconfig.get_status(source["name"]) + if cached_status and "schema" in cached_status: + actual_cols = {col["name"] for col in cached_status["schema"]} + if not actual_cols: + # Fallback: cron hasn't populated status yet (cold-start + # within the first ~60s after backend boot). Do the live + # lookup so first-load isn't a 503 — subsequent calls hit + # the cron-populated cache. + actual_cols = {col["name"] for col in get_schema(con, source)} from backend.core.log_fields import INSIGHT_DEFINITIONS result = [] diff --git a/backend/routers/network.py b/backend/routers/network.py index 93da6716..f5624464 100644 --- a/backend/routers/network.py +++ b/backend/routers/network.py @@ -38,6 +38,26 @@ def network_health(req: NetworkHealthRequest, deps: AnalyticsDeps = Depends()): top_n=req.top_n, map_asn=req.map_asn, ) + # Phase 3 item 13 — merge shielding-analysis into the network-health + # response so /network gets both shapes in one round-trip. Best-effort: + # if the shielding query fails (missing fields, no shield logs) the + # network-health response still ships; the field is just null. + try: + from backend.repositories import origin as _origin + + shielding = _origin.get_shielding_analysis( + con=deps.con, + src=deps.source, + start_time=req.start_time, + end_time=req.end_time, + filters=req.filters, + ) + # Strip the per-call telemetry — the outer with_telemetry below + # already collects the contextvar entries. + shielding = {k: v for k, v in shielding.items() if not k.startswith("debug_")} + res["shielding_analysis"] = shielding + except Exception: + res["shielding_analysis"] = None return NetworkHealthResponse.with_telemetry(**res) diff --git a/backend/routers/origin.py b/backend/routers/origin.py index 40b23821..eb654ea5 100644 --- a/backend/routers/origin.py +++ b/backend/routers/origin.py @@ -9,6 +9,7 @@ from backend.deps import AnalyticsDeps from backend.models.common import FilteredRequest, Limit100, Limit200, Limit1440 from backend.models.origin import ( + OriginAggregatesResponse, OriginIpHealthResponse, OriginPathBreakdownResponse, OriginPopLatencyResponse, @@ -52,6 +53,47 @@ class OriginShieldingAnalysisRequest(FilteredRequest): limit: Limit200 = 50 +class OriginAggregatesRequest(FilteredRequest): + bucket_minutes: Limit1440 = 5 + split_by_leg: bool = False + timeseries_metric: Literal["ttfb", "ttlb"] = "ttfb" + timeseries_percentile: Literal["p50", "p95", "p99"] = "p95" + slow_urls_limit: Limit100 = 20 + slow_urls_min_requests: int = 10 + ip_health_limit: Limit100 = 30 + pop_latency_limit: Limit100 = 30 + + +@router.post("/aggregates", response_model=OriginAggregatesResponse) +@query_errors() +def origin_aggregates(req: OriginAggregatesRequest, deps: AnalyticsDeps = Depends()): + """Composite of the six origin cards (summary, timeseries, slow-urls, + status-codes, path-breakdown, pop-latency, ip-health) backed by ONE + parquet scan. Shielding-analysis stays at /api/origin/shielding-analysis + until item 13 folds it into /api/network-health. + + Granular endpoints below are unchanged so the frontend can roll back + to the per-card pattern by flipping a feature flag without a backend + redeploy. + """ + res = repo.get_aggregates( + con=deps.con, + src=deps.source, + start_time=req.start_time, + end_time=req.end_time, + filters=req.filters, + bucket_minutes=req.bucket_minutes, + split_by_leg=req.split_by_leg, + timeseries_metric=req.timeseries_metric, + timeseries_percentile=req.timeseries_percentile, + slow_urls_limit=req.slow_urls_limit, + slow_urls_min_requests=req.slow_urls_min_requests, + ip_health_limit=req.ip_health_limit, + pop_latency_limit=req.pop_latency_limit, + ) + return OriginAggregatesResponse.with_telemetry(**res) + + @router.post("/summary", response_model=OriginSummaryResponse) @query_errors() def origin_summary(req: OriginRequest, deps: AnalyticsDeps = Depends()): diff --git a/backend/routers/provision.py b/backend/routers/provision.py index f5efed14..0dc57551 100644 --- a/backend/routers/provision.py +++ b/backend/routers/provision.py @@ -9,7 +9,7 @@ import urllib.error import urllib.request -from fastapi import APIRouter, HTTPException, Query +from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request from fastapi.responses import StreamingResponse from backend.utils.router_utils import SSE_HEADERS as _SSE_HEADERS @@ -178,8 +178,22 @@ def provision_check_fos( return {"ok": False, "error": err_msg, "_debug_calls": get_tracked_calls()} -@router.post("/teardown") -def provision_teardown(body: dict | None = None): +def _require_json_content_type(req: Request) -> None: + """Reject any teardown request whose Content-Type isn't application/json. + + CSRF defense: an HTML form with ``enctype=text/plain`` can POST a body + that LOOKS like JSON without triggering a CORS preflight. Requiring + ``Content-Type: application/json`` forces the browser to preflight any + cross-origin call (text/plain is "simple"; application/json is not), + blocking the silent-invocation vector. Runs as a Depends() so it fires + before FastAPI's body parser — otherwise a malformed text/plain body + returns 422 from the parser and the explicit 415 never executes.""" + if not (req.headers.get("content-type") or "").startswith("application/json"): + raise HTTPException(status_code=415, detail="Unsupported Media Type") + + +@router.post("/teardown", dependencies=[Depends(_require_json_content_type)]) +def provision_teardown(req: Request, body: dict | None = None): """Destructive service teardown over SSE. Switched from ``GET`` to ``POST`` to defend against CSRF: a GET @@ -392,29 +406,54 @@ def provision_lake_info( return fetch_lake_info(src, use_temp_cache=True) -@router.get("/execute") -def provision_execute( - token: str = Query(...), - service_id: str = Query(...), - service_name: str | None = Query(default=None), - endpoint_name: str = Query(default="Fastly Object Storage Logs"), - fos_region: str = Query(default="us-east-1"), - fos_bucket_name: str = Query(...), - fos_prefix: str = Query(default=""), - sample_rate: str = Query(default="100"), - edge_only: bool = Query(default=True), - custom_condition: str | None = Query(default=None), - log_period: str = Query(default="1 minute"), - cdn_service_name: str | None = Query(default=None), - cdn_url: str | None = Query(default=None), - cdn_shield: str = Query(default="none"), - enable_cron_sync: bool = Query(default=True), - delete_after: bool = Query(default=True), - commit_interval_mins: int = Query(default=5), - enable_cron_compact: bool = Query(default=True), - log_retention_days: int = Query(default=30), - log_fields: str | None = Query(default=None), -): +from pydantic import BaseModel + + +class ProvisionExecuteRequest(BaseModel): + token: str + service_id: str + service_name: str | None = None + endpoint_name: str = "Fastly Object Storage Logs" + fos_region: str = "us-east-1" + fos_bucket_name: str + fos_prefix: str = "" + sample_rate: str = "100" + edge_only: bool = True + custom_condition: str | None = None + log_period: str = "1 minute" + cdn_service_name: str | None = None + cdn_url: str | None = None + cdn_shield: str = "none" + enable_cron_sync: bool = True + delete_after: bool = True + commit_interval_mins: int = 5 + enable_cron_compact: bool = True + log_retention_days: int = 30 + log_fields: str | None = None + + +@router.post("/execute") +def provision_execute(req: ProvisionExecuteRequest): + token = req.token + service_id = req.service_id + service_name = req.service_name + endpoint_name = req.endpoint_name + fos_region = req.fos_region + fos_bucket_name = req.fos_bucket_name + fos_prefix = req.fos_prefix + sample_rate = req.sample_rate + edge_only = req.edge_only + custom_condition = req.custom_condition + log_period = req.log_period + cdn_service_name = req.cdn_service_name + cdn_url = req.cdn_url + cdn_shield = req.cdn_shield + enable_cron_sync = req.enable_cron_sync + delete_after = req.delete_after + commit_interval_mins = req.commit_interval_mins + enable_cron_compact = req.enable_cron_compact + log_retention_days = req.log_retention_days + log_fields = req.log_fields import secrets from backend.core import duckdb as _db @@ -672,6 +711,10 @@ def provision_ingest(body: dict): except Exception: pass + from backend.utils.fastly_auth import validate_destructive_token + + validate_destructive_token(token, service_id=state.get("logging_service_id") or "") + write_service_config(state) try: @@ -813,7 +856,11 @@ def provision_check_config( @router.get("/ngwaf-workspaces") -def provision_ngwaf_workspaces(service_id: str = Query(...), token: str = Query(default="")): +def provision_ngwaf_workspaces( + service_id: str = Query(...), + token: str = Query(default=""), + authorization: str | None = Header(default=None), +): """List NGWAF workspaces for a service. Security: previously the endpoint would silently fall back to @@ -829,14 +876,15 @@ def provision_ngwaf_workspaces(service_id: str = Query(...), token: str = Query( Either way an unauthenticated caller can't enumerate workspaces even if they reach the loopback admin surface. """ - import hmac import urllib.error - from backend import config as svcconfig from backend.provision import fastly from backend.utils.fastly_auth import validate_destructive_token - token = token.strip() + if authorization and authorization.lower().startswith("bearer "): + token = authorization[len("bearer ") :].strip() + else: + token = token.strip() if not token: raise HTTPException( status_code=401, @@ -845,12 +893,11 @@ def provision_ngwaf_workspaces(service_id: str = Query(...), token: str = Query( "message": "A Fastly API token is required to list NGWAF workspaces.", }, ) - stored = (svcconfig.get_fastly_api_key(service_id) or "").strip() - matches_stored = bool(stored) and hmac.compare_digest(token, stored) - if not matches_stored: - # The validator raises HTTPException(401) on scope / service / - # network failures, which is the right user-visible behavior. - validate_destructive_token(token, service_id=service_id) + # Secure token validation: we must always run validate_destructive_token + # to verify that the token holds the necessary 'global' scope and is + # authorized for this tenant's service. This prevents read-only token + # bypasses, even if the token matches the server-stored fastly_api_key. + validate_destructive_token(token, service_id=service_id) from backend.utils.router_utils import format_debug_request @@ -914,7 +961,12 @@ def provision_ngwaf_workspaces(service_id: str = Query(...), token: str = Query( @router.patch("/services/{service_id}/ngwaf-workspace") -def provision_set_ngwaf_workspace(service_id: str, body: dict, token: str = Query(default="")): +def provision_set_ngwaf_workspace( + service_id: str, + body: dict, + token: str = Query(default=""), + authorization: str | None = Header(default=None), +): """Persist the NGWAF workspace ID for a service and reload the scheduler. Security: require the caller to present a Fastly token bound to @@ -930,7 +982,6 @@ def provision_set_ngwaf_workspace(service_id: str, body: dict, token: str = Quer rebind the workspace because they don't know the token. The middleware /api/provision/ block also gates this for analysts. """ - import hmac from backend import config as svcconfig from backend.utils.fastly_auth import validate_destructive_token @@ -939,7 +990,10 @@ def provision_set_ngwaf_workspace(service_id: str, body: dict, token: str = Quer if not cfg: raise HTTPException(status_code=404, detail={"error": "Service not found"}) - token = (token or "").strip() + if authorization and authorization.lower().startswith("bearer "): + token = authorization[len("bearer ") :].strip() + else: + token = (token or "").strip() stored = (cfg.get("fastly_api_key") or "").strip() if not token: raise HTTPException( @@ -947,14 +1001,11 @@ def provision_set_ngwaf_workspace(service_id: str, body: dict, token: str = Quer detail={"error": "token_required", "message": "A Fastly API token is required."}, ) - # Fast path: caller presented the stored key. Constant-time compare so - # we don't leak the stored value via timing. - matches_stored = bool(stored) and hmac.compare_digest(token, stored) - if not matches_stored: - # Fall back to the strict scope-validation path. validate_destructive_token - # raises HTTPException(401) on any failure (missing/insufficient scope, - # service mismatch, Fastly unreachable). - validate_destructive_token(token, service_id=service_id) + # Secure token validation: we must always run validate_destructive_token + # to verify that the token holds the necessary 'global' scope and is + # authorized for this tenant's service. This prevents read-only token + # bypasses, even if the token matches the server-stored fastly_api_key. + validate_destructive_token(token, service_id=service_id) workspace_id = (body.get("ngwaf_workspace_id") or "").strip() or None cfg["ngwaf_workspace_id"] = workspace_id diff --git a/backend/routers/services/core.py b/backend/routers/services/core.py index 439b28ae..8cf40ecc 100644 --- a/backend/routers/services/core.py +++ b/backend/routers/services/core.py @@ -141,7 +141,7 @@ async def stream(): for _line in _sse_flush(): yield _line while True: - evs = get_progress(run_id, last_idx) + evs = get_progress(run_id, last_idx, service_id=service_id) if evs is None: if last_idx == 0: # Fall back to SQLite database if progress cache doesn't have it (completed / historical) diff --git a/backend/routers/services/cron.py b/backend/routers/services/cron.py index cc644ab5..8b403160 100644 --- a/backend/routers/services/cron.py +++ b/backend/routers/services/cron.py @@ -15,18 +15,15 @@ def api_cron_logs( per_page: int = Query(default=50, le=1000), sort: str = Query(default="started_at"), dir: str = Query(default="DESC"), + since_id: int | None = Query(default=None, ge=0), ): - from backend.utils.telemetry import get_queries, get_tracked_calls - try: - total, entries = get_cron_logs(source["name"], task, status, page, per_page, sort, dir) + total, entries = get_cron_logs(source["name"], task, status, page, per_page, sort, dir, since_id=since_id) return { "total": total, "page": page, "per_page": per_page, "entries": entries, - "_debug_queries": get_queries(), - "_debug_calls": get_tracked_calls(), } except Exception as e: raise HTTPException(status_code=500, detail={"error": str(e)}) diff --git a/backend/routers/session_scoring.py b/backend/routers/session_scoring.py index f6e95ded..dea51171 100644 --- a/backend/routers/session_scoring.py +++ b/backend/routers/session_scoring.py @@ -78,6 +78,27 @@ _inflight: dict[tuple, threading.Lock] = {} +def _finalize_cached(value, *, is_cached: bool) -> object: + """Return *value* with `_is_cached` set, gating `_debug_*` on + `DEBUG_RESPONSES` so production responses don't leak SQL/URLs. + + Mirrors `backend.models.common.BaseResponse._strip_debug_when_disabled` + so endpoints that return plain dicts get the same gating as endpoints + that return Pydantic responses. `_is_cached` is always included — it + isn't sensitive and downstream verification depends on it. + """ + from backend.models.common import _debug_responses_enabled + + if not isinstance(value, dict): + return value + out = dict(value) + out["_is_cached"] = is_cached + if not _debug_responses_enabled(): + out.pop("_debug_queries", None) + out.pop("_debug_calls", None) + return out + + def _cached(key: tuple, producer): """Return cached value if fresh, else produce + store. @@ -87,7 +108,20 @@ def _cached(key: tuple, producer): callers on DIFFERENT keys (the dashboard mount fires 8 endpoints with 8 different keys) run in parallel — they only contend on the global lock during the brief cache-lookup + per-key-lock-handoff - window.""" + window. + + Telemetry: snapshots the request-scoped `_QUERIES` / `_CALLS` + contextvars (from `backend.utils.telemetry`) before producer() and + captures the suffix added during producer(). The captured slice is + baked into the stored value under `_debug_queries` / `_debug_calls` + so cache hits return the same telemetry that populated the cache, + paired with `_is_cached: True` to flag the timings as historical. + `_query_logs` (and anything called transitively from a producer) + appends to the same shared contextvar via `get_queries()`. + """ + from backend.utils.telemetry import _CALLS as _telemetry_calls + from backend.utils.telemetry import get_queries + with _analytics_cache_lock: # Capture `now` INSIDE the lock so the freshness check evaluates # against the lock-acquisition timestamp, not a stale value from @@ -97,7 +131,7 @@ def _cached(key: tuple, producer): now = _time.monotonic() entry = _analytics_cache.get(key) if entry and (now - entry[0]) < _ANALYTICS_TTL_SEC: - return entry[1] + return _finalize_cached(entry[1], is_cached=True) # Miss — claim the per-key lock under the global lock so two # concurrent misses on the same key don't both create new locks. key_lock = _inflight.get(key) @@ -113,19 +147,47 @@ def _cached(key: tuple, producer): now = _time.monotonic() entry = _analytics_cache.get(key) if entry and (now - entry[0]) < _ANALYTICS_TTL_SEC: - return entry[1] - # Actual producer call happens OUTSIDE the global lock so other - # keys can be served while this one is computing. - value = producer() - with _analytics_cache_lock: - # Re-capture now after producer() so the TTL clock starts - # from when the value was actually computed, not from when - # we entered _cached. - _analytics_cache[key] = (_time.monotonic(), value) - # Drop the per-key lock entry — small saving but bounds the - # _inflight dict growth across the long-running TTL window. - _inflight.pop(key, None) - return value + return _finalize_cached(entry[1], is_cached=True) + try: + # Snapshot telemetry length so we can attribute only producer()'s + # additions — middleware-level call tracking already populated + # the contextvars before we got here, and we don't want to bake + # pre-producer entries into the cached value. + queries = get_queries() + calls_initial = _telemetry_calls.get() or [] + q_start = len(queries) + c_start = len(calls_initial) + # Actual producer call happens OUTSIDE the global lock so other + # keys can be served while this one is computing. + value = producer() + queries_after = get_queries() + calls_after = _telemetry_calls.get() or [] + # Defensive slice: if downstream code reset the contextvar mid- + # producer (start_call_tracking, an explicit clear, etc.) the + # suffix index could exceed the current length. Fall back to + # the full current list rather than crash on a slice error. + added_queries = list(queries_after[q_start:] if len(queries_after) >= q_start else queries_after) + added_calls = list(calls_after[c_start:] if len(calls_after) >= c_start else calls_after) + # Bake telemetry into the stored value so cache hits surface + # the same shape. `_is_cached` is added at return time, not + # stored, so the cached dict carries only stable data. + if isinstance(value, dict): + stored = dict(value) + stored.setdefault("_debug_queries", added_queries) + stored.setdefault("_debug_calls", added_calls) + else: + stored = value + with _analytics_cache_lock: + # Re-capture now after producer() so the TTL clock starts + # from when the value was actually computed, not from when + # we entered _cached. + _analytics_cache[key] = (_time.monotonic(), stored) + return _finalize_cached(stored, is_cached=False) + finally: + with _analytics_cache_lock: + # Drop the per-key lock entry — small saving but bounds the + # _inflight dict growth across the long-running TTL window. + _inflight.pop(key, None) def _bust_analytics_cache(service_id: str | None = None) -> None: @@ -452,6 +514,63 @@ def stream(): return StreamingResponse(stream(), media_type="text/event-stream", headers=_SSE_HEADERS) +@router.get("/{service_id}/scoring/analytics") +def scoring_analytics_composite( + service_id: str = Path(..., description="Logging service ID"), + since_hours: int = Query(default=24, ge=1, le=168), +) -> dict: + """Composite of the seven analytics endpoints + (top-flagged, score-distribution, compliance-breakdown, health, + evaluation, evaluation/per-reason, threshold-preview) into a single + round-trip. Each is already individually cached via `_cached` so + repeated composite calls within the 20s TTL collapse to dict + lookups; the composite primarily saves the per-request HTTP + + auth-middleware overhead that the 7-card admin_session_scoring + page paid on cold mount. + + Granular endpoints unchanged — frontend swap to use the composite + is a separate commit so the per-card endpoints remain a rollback + target. + """ + # Cast params to plain ints — FastAPI resolves Query() objects when + # called via HTTP, but direct Python calls receive the Query wrapper. + sh = int(since_hours) + return { + "top_flagged": scoring_top_flagged(service_id=service_id, since_hours=sh, limit=200), + "score_distribution": scoring_score_distribution(service_id=service_id, since_hours=sh), + "compliance_breakdown": scoring_compliance_breakdown(service_id=service_id, since_hours=sh), + "health": scoring_health(service_id=service_id, since_hours=sh), + "evaluation": scoring_evaluation(service_id=service_id), + "evaluation_per_reason": scoring_evaluation_per_reason(service_id=service_id), + } + + +@router.get("/{service_id}/scoring/config") +def scoring_config_composite( + service_id: str = Path(..., description="Logging service ID"), +) -> dict: + """Composite of the four token-free /scoring/* config endpoints + (status, threshold, exclude-regex, enforce-status-code). The admin + session-scoring page was firing four parallel GETs on mount; each + is a sub-50ms local config read so cold-load cost is dominated by + HTTP overhead rather than computation. Combining them into one + round-trip saves ~300-500ms on the cold-load waterfall. + + Excluded: /scoring/enforce-threshold (requires a Fastly API token + and makes a network round-trip out — frontend should fetch that + one separately if it needs the live edge-side value). + + Granular endpoints unchanged so the frontend can keep using them + individually during a rollback. + """ + return { + "status": scoring_status(service_id), + "threshold": scoring_threshold_get(service_id), + "exclude_regex": scoring_exclude_regex_get(service_id), + "enforce_status_code": scoring_enforce_status_code_get(service_id), + } + + @router.get("/{service_id}/scoring/status") def scoring_status( service_id: str = Path(..., description="Logging service ID"), @@ -572,16 +691,31 @@ def _query_logs(service_id: str, sql: str, params: tuple = ()) -> list[dict]: parametrized queries (e.g. ``WHERE edge_sid IN (?, ?, ?)``) without string-formatting user-controlled values into the SQL.""" from backend.core.duckdb import get_connection, get_source_for_service + from backend.repositories._base import _compact_sql_for_debug + from backend.utils.telemetry import get_queries src = get_source_for_service(service_id) if src is None: raise HTTPException(status_code=404, detail={"error": f"No service {service_id}"}) con = None + t0 = _time.monotonic() try: con = get_connection(source=src, max_wait=3, skip_view_update=True, read_only=True) rows = con.execute(sql, params).fetchall() if params else con.execute(sql).fetchall() cols = [d[0] for d in con.description] if con.description else [] - return [dict(zip(cols, r)) for r in rows] + result = [dict(zip(cols, r)) for r in rows] + # Append to the request-scoped query log so `_cached` can attribute + # this query (and anything called transitively through + # `_reconstruct_labeled_sessions` / `_fetch_session_events`) to the + # producer that invoked it. + get_queries().append( + { + "sql": _compact_sql_for_debug(sql.strip()), + "time_ms": round((_time.monotonic() - t0) * 1000, 2), + "rows": len(result), + } + ) + return result except HTTPException: raise except Exception as e: diff --git a/backend/routers/share_admin.py b/backend/routers/share_admin.py index 7d5b19d3..2a520982 100644 --- a/backend/routers/share_admin.py +++ b/backend/routers/share_admin.py @@ -27,6 +27,24 @@ # ── Status ────────────────────────────────────────────────────────────────── +@router.get("/banner") +def share_banner(): + """Tiny payload (~80B) for the global share-status banner. + + Used by frontend/hooks/useShareStatusBanner.tsx — polls every 15s on + every page that mounts AppLayout. The full /api/admin/share/status + response is ~11KB and includes services + invites + sessions + audit + logs + telemetry that the banner never reads. Per-poll-per-page + multiplied across the 12+ pages with AppLayout was a meaningful + cumulative cost. + """ + mgr = get_tunnel_manager() + return { + "sharing_active": mgr.is_sharing_active(), + "public_url": mgr.public_url(), + } + + @router.get("/status") def share_status(): mgr = get_tunnel_manager() diff --git a/backend/routers/share_auth.py b/backend/routers/share_auth.py index c5a3a580..0ec4ee5a 100644 --- a/backend/routers/share_auth.py +++ b/backend/routers/share_auth.py @@ -23,6 +23,7 @@ router = APIRouter(prefix="/api/share", tags=["share-auth"]) COOKIE_NAME = "analyst_session_id" +PENDING_COOKIE_NAME = "analyst_pending_session_id" def _client_ip(request: Request) -> str: @@ -111,24 +112,37 @@ def share_login(payload: ShareLoginPayload, request: Request, response: Response details=f"session={session.session_id[:8]}…", ) - # Cookie contract — see Section #4. secure=True is non-negotiable. - # In test mode (TestClient defaults to http://testserver), uvicorn won't - # send secure cookies; we tag it anyway because tests can read Set-Cookie. - response.set_cookie( - key=COOKIE_NAME, - value=session.session_id, - httponly=True, - secure=True, - samesite="strict", - max_age=share_db.iso_z_now() and 24 * 60 * 60, - path="/", - ) - tos = share_db.get_latest_tos() tos_pending = bool( tos and (invite.get("tos_accepted_at") is None or (invite.get("tos_version") or "") != tos["version"]) ) + # Cookie contract — see Section #4. secure=True is non-negotiable. + # In test mode (TestClient defaults to http://testserver), uvicorn won't + # send secure cookies; we tag it anyway because tests can read Set-Cookie. + if tos_pending: + response.set_cookie( + key=PENDING_COOKIE_NAME, + value=session.session_id, + httponly=True, + secure=True, + samesite="strict", + max_age=share_db.iso_z_now() and 24 * 60 * 60, + path="/", + ) + response.delete_cookie(COOKIE_NAME, path="/") + else: + response.set_cookie( + key=COOKIE_NAME, + value=session.session_id, + httponly=True, + secure=True, + samesite="strict", + max_age=share_db.iso_z_now() and 24 * 60 * 60, + path="/", + ) + response.delete_cookie(PENDING_COOKIE_NAME, path="/") + return ShareLoginResponse( ok=True, session_id=session.session_id, @@ -143,11 +157,12 @@ def share_login(payload: ShareLoginPayload, request: Request, response: Response @router.post("/logout", response_model=ShareLogoutResponse) def share_logout(request: Request, response: Response): - sid = request.cookies.get(COOKIE_NAME) + sid = request.cookies.get(COOKIE_NAME) or request.cookies.get(PENDING_COOKIE_NAME) mgr = get_tunnel_manager() if sid: mgr.boot_session(sid, reason="analyst logout") response.delete_cookie(COOKIE_NAME, path="/") + response.delete_cookie(PENDING_COOKIE_NAME, path="/") return ShareLogoutResponse(ok=True) @@ -155,13 +170,39 @@ class TosAckPayload(BaseModel): version: str +@router.get("/tos", response_model=TosDocument) +def share_get_tos(request: Request): + """Return the latest TOS document so the acknowledge page can render the + real text and POST back the matching version. + + Session-gated (pending OR full cookie) — the same shape /acknowledge uses — + so anonymous callers can't enumerate the TOS surface. The strict version + check in /acknowledge (audit finding 021) means the frontend must know the + exact current version; this endpoint is how it learns it. + """ + sid = request.cookies.get(PENDING_COOKIE_NAME) or request.cookies.get(COOKIE_NAME) + mgr = get_tunnel_manager() + session = mgr.validate_session(sid) + if session is None: + raise HTTPException(status_code=401, detail={"error": "unauthenticated"}) + tos = share_db.get_latest_tos() + if not tos: + raise HTTPException(status_code=404, detail={"error": "no_tos"}) + return TosDocument(version=tos["version"], text=tos["text"]) + + @router.post("/acknowledge", response_model=ShareAcknowledgeResponse) -def share_acknowledge_tos(payload: TosAckPayload, request: Request): - sid = request.cookies.get(COOKIE_NAME) +def share_acknowledge_tos(payload: TosAckPayload, request: Request, response: Response): + sid = request.cookies.get(PENDING_COOKIE_NAME) or request.cookies.get(COOKIE_NAME) mgr = get_tunnel_manager() session = mgr.validate_session(sid) if session is None: raise HTTPException(status_code=401, detail={"error": "unauthenticated"}) + + tos = share_db.get_latest_tos() + if tos and payload.version != tos["version"]: + raise HTTPException(status_code=400, detail={"error": "invalid_tos_version"}) + share_db.mark_tos_accepted(session.invite_id, payload.version) share_db.log_share_audit_event( event_type="TOS_ACCEPTED", @@ -169,6 +210,16 @@ def share_acknowledge_tos(payload: TosAckPayload, request: Request): ip_address=session.ip_address, details=f"version={payload.version}", ) + response.set_cookie( + key=COOKIE_NAME, + value=session.session_id, + httponly=True, + secure=True, + samesite="strict", + max_age=share_db.iso_z_now() and 24 * 60 * 60, + path="/", + ) + response.delete_cookie(PENDING_COOKIE_NAME, path="/") return ShareAcknowledgeResponse(ok=True) @@ -178,7 +229,7 @@ def share_heartbeat(request: Request): Returns 401 if the session is gone so the frontend redirects to login. """ - sid = request.cookies.get(COOKIE_NAME) + sid = request.cookies.get(COOKIE_NAME) or request.cookies.get(PENDING_COOKIE_NAME) mgr = get_tunnel_manager() session = mgr.validate_session(sid) if session is None: diff --git a/backend/routers/usage.py b/backend/routers/usage.py index abd49252..4e621a72 100644 --- a/backend/routers/usage.py +++ b/backend/routers/usage.py @@ -58,7 +58,9 @@ def _get(d, key): @router.get("/prefill", response_model=PrefillResponse) @query_errors() -def prefill(source: dict = Depends(get_source)): +async def prefill(source: dict = Depends(get_source)): + import asyncio + from backend import config as svcconfig from backend.config import get_fastly_api_key, get_fastly_logging_service_id @@ -154,50 +156,75 @@ def prefill(source: dict = Depends(get_source)): from_ts = int((now - timedelta(days=3)).timestamp()) to_ts = int(now.timestamp()) by = "day" + + # M4 parallelisation: the version → endpoint → condition chain + # (250 ms typical, fully serial because each step needs the + # previous response) is independent of the /stats call (150 ms + # typical) — neither uses the other's result. Run both as + # asyncio tasks so the prefill wall-clock is bound by the slower + # of the two instead of their sum. Each sync ``fastly()`` call + # runs inside ``asyncio.to_thread`` so the existing retry, auth, + # and telemetry machinery in ``backend/core/fastly/client.py`` + # is reused unchanged. + + async def _resolve_endpoint_chain() -> dict: + """Returns {log_period_seconds?, sample_rate?, edge_only?}.""" + updates: dict = {} + try: + if not logging_svc_id: + return updates + active_ver = await asyncio.to_thread(get_active_version, logging_svc_id, api_key) + if not active_ver: + return updates + endpoint_name = prov.get("endpoint_name", "Fastly Object Storage Logs") + encoded_name = urllib.parse.quote(endpoint_name, safe="") + current_ep = await asyncio.to_thread( + fastly, + "GET", + f"/service/{logging_svc_id}/version/{active_ver}/logging/s3/{encoded_name}", + token=api_key, + ) + if "period" in current_ep: + updates["log_period_seconds"] = int(current_ep["period"]) + cond_name = current_ep.get("response_condition") + if cond_name == "Log Sampling": + import re + + cond = await asyncio.to_thread(find_condition, cond_name, logging_svc_id, active_ver, api_key) + if cond: + stmt = cond.get("statement", "") + m = re.search(r"randombool\((\d+),", stmt) + if m: + updates["sample_rate"] = int(m.group(1)) + if "req.restarts == 0" in stmt: + updates["edge_only"] = True + except Exception: + pass + return updates + + async def _fetch_stats() -> dict | None: + try: + if svc_id: + return await asyncio.to_thread( + _fastly_api, f"/stats/service/{svc_id}?by={by}&from={from_ts}&to={to_ts}", api_key + ) + return await asyncio.to_thread( + _fastly_api, f"/stats/aggregate?by={by}&from={from_ts}&to={to_ts}", api_key + ) + except Exception: + return None + try: + chain_updates, payload = await asyncio.gather(_resolve_endpoint_chain(), _fetch_stats()) + # Chain updates feed into the response shape's existing keys + # — overrides any defaults set above and any cron_sync values + # set from the local config, matching the prior precedence + # (Fastly-resolved values win over local config). + result.update(chain_updates) + daily_reqs: dict[str, int] = {} daily_edge: dict[str, int] = {} - if svc_id: - try: - active_ver = get_active_version(logging_svc_id, api_key) if logging_svc_id else None - if active_ver: - endpoint_name = prov.get("endpoint_name", "Fastly Object Storage Logs") - encoded_name = urllib.parse.quote(endpoint_name, safe="") - current_ep = fastly( - "GET", - f"/service/{logging_svc_id}/version/{active_ver}/logging/s3/{encoded_name}", - token=api_key, - ) - if "period" in current_ep: - result["log_period_seconds"] = int(current_ep["period"]) - cond_name = current_ep.get("response_condition") - if cond_name == "Log Sampling": - import re - - cond = find_condition(cond_name, logging_svc_id, active_ver, api_key) - if cond: - stmt = cond.get("statement", "") - m = re.search(r"randombool\((\d+),", stmt) - if m: - result["sample_rate"] = int(m.group(1)) - if "req.restarts == 0" in stmt: - result["edge_only"] = True - except Exception: - pass - # tracked_call wrapper removed — _fastly_api → fastly() - # already does telemetry internally; the double-wrap was - # producing duplicate entries in /api/admin/usage-logging. - payload = _fastly_api(f"/stats/service/{svc_id}?by={by}&from={from_ts}&to={to_ts}", api_key) - for rec in payload.get("data", []): - ts = rec.get("start_time") - if ts is None: - continue - day = datetime.fromtimestamp(ts, tz=UTC).strftime("%Y-%m-%d") - daily_reqs[day] = daily_reqs.get(day, 0) + int(rec.get("requests") or 0) - daily_edge[day] = daily_edge.get(day, 0) + int(rec.get("edge_requests") or 0) - else: - # See note above — fastly() does its own tracking. - payload = _fastly_api(f"/stats/aggregate?by={by}&from={from_ts}&to={to_ts}", api_key) + if payload: for rec in payload.get("data", []): ts = rec.get("start_time") if ts is None: @@ -228,13 +255,18 @@ def prefill(source: dict = Depends(get_source)): from backend.core.duckdb import get_connection # read_only: get_edge_ratio is a SELECT against the view. - con = get_connection(source=source, max_wait=5, read_only=True) - try: - edge_ratio, debug_queries = repo.get_edge_ratio(con, source) - if edge_ratio is not None: - result["edge_ratio"] = edge_ratio - finally: - con.close() + # Wrapped in asyncio.to_thread so this sync I/O doesn't block + # the event loop now that prefill is an async handler. + def _edge_ratio_blocking() -> tuple: + con = get_connection(source=source, max_wait=5, read_only=True) + try: + return repo.get_edge_ratio(con, source) + finally: + con.close() + + edge_ratio, debug_queries = await asyncio.to_thread(_edge_ratio_blocking) + if edge_ratio is not None: + result["edge_ratio"] = edge_ratio except Exception: pass @@ -520,12 +552,13 @@ def _merge(payload): agg[ts]["bandwidth_bytes"] += int(record.get("bandwidth") or 0) agg[ts]["requests"] += int(record.get("requests") or 0) - from backend.utils.telemetry import tracked_call - if cdn_svc: try: - with tracked_call("GET", f"/stats/service/{cdn_svc}?by={by}", service="Fastly API"): - payload = _fastly_api(f"/stats/service/{cdn_svc}?by={by}&from={from_ts}&to={to_ts}", api_key) + # tracked_call wrapper removed — _fastly_api → fastly() already + # does telemetry internally; the double-wrap was producing + # duplicate entries in /api/admin/usage-logging and inflating + # the visible call count to 2x for this endpoint. + payload = _fastly_api(f"/stats/service/{cdn_svc}?by={by}&from={from_ts}&to={to_ts}", api_key) _merge(payload) except Exception as e: raise HTTPException(status_code=502, detail={"error": str(e)}) @@ -575,10 +608,9 @@ def usage_log_activity( to_ts = int(end_dt.timestamp()) try: - from backend.utils.telemetry import tracked_call - - with tracked_call("GET", f"/stats/service/{logging_svc}?by={by}", service="Fastly API"): - payload = _fastly_api(f"/stats/service/{logging_svc}?by={by}&from={from_ts}&to={to_ts}", api_key) + # tracked_call wrapper removed — see _fastly_api docstring; + # double-wrap inflated the visible call count to 2x. + payload = _fastly_api(f"/stats/service/{logging_svc}?by={by}&from={from_ts}&to={to_ts}", api_key) fmt = "%Y-%m-%dT%H:00" if by == "hour" else "%Y-%m-%dT%H:%M" if by == "minute" else "%Y-%m-%d" stats_lookup: dict[str, int] = {} diff --git a/backend/scheduler.py b/backend/scheduler.py index 373c8c18..9995df80 100644 --- a/backend/scheduler.py +++ b/backend/scheduler.py @@ -525,6 +525,31 @@ def _sync_jobs(self) -> None: self._job_ids[lc_job_id] = lc_job_id logger.info("⚙️ [scheduler] Registered local_compact job %s (every 2 min, local-only).", lc_job_id) + # ── Daily rollup compaction (per-day parquet from per-hour) ──── + # 02:00 UTC — runs before optimize (03:00) so per-day rollups + # are ready when the next day's queries start. Only for + # read-write services that own the rollup data. + if compact_cfg.get("enabled", True) and prov.get("access_level") != "read_only": + rc_job_id = f"rollup_compact_{service_id}" + seen_ids.add(rc_job_id) + if rc_job_id not in self._job_ids: + self._sched.add_job( + _run_rollup_compact_daily, + "cron", + hour=2, + minute=0, + args=[service_id], + id=rc_job_id, + max_instances=1, + coalesce=True, + misfire_grace_time=3600, + ) + self._job_ids[rc_job_id] = rc_job_id + logger.info( + "📦 [scheduler] Registered rollup compaction job %s (daily 02:00 UTC).", + rc_job_id, + ) + # ── Weekly expire-snapshots job ─────────────────────────────────── if compact_cfg.get("enabled", True): exp_job_id = f"expire_{service_id}" @@ -791,6 +816,8 @@ def _run_metadata_sync( if src is None: return + is_manual = run_id is not None + if run_id is None: try: run_id = start_cron_run(src, "metadata_sync") @@ -807,7 +834,6 @@ def _run_metadata_sync( # For manual runs (run_id is not None), we ignore the default limit unless # it was explicitly passed in. If a manual run is triggered without # start_time, it means "Import All", so we should clear any existing limit. - is_manual = run_id is not None if not start_time and not is_manual: prov = cfg.get("provisioning", {}) @@ -2210,37 +2236,143 @@ def _run_optimize(service_id: str) -> None: @cron_task("expire_snapshots") def _run_expire_snapshots(service_id: str) -> None: """Weekly job: perform cloud maintenance including data deletion, cache cleanup, and snapshot expiry.""" + import time + from backend.core import iceberg as db_iceberg - from backend.core.duckdb import get_source_for_service + from backend.core.duckdb import get_source_for_service, log_cron_run, start_cron_run src = get_source_for_service(service_id) if src is None: return + try: + run_id = start_cron_run(src, "expire_snapshots") + except RuntimeError as e: + logger.info("⏭️ [expire] %s: skipping — %s", service_id, str(e)) + return + svc_id = src.get("service_id", "unknown") svc_name = _display_name(src, svc_id) display_name = f"{svc_name} ({svc_id})" if svc_name != svc_id else svc_id logger.info("▶️ \x1b[90m[expire]\x1b[0m %s: Maintenance job started.", display_name) - try: - pass - except Exception: - pass - + start_time = time.time() try: result = db_iceberg.run_cloud_maintenance(src) + duration = time.time() - start_time if "error" in result: logger.warning("%s %s: %s", JOB_COLORS["expire"] + "[expire]" + RESET_COLOR, display_name, result["error"]) + log_cron_run( + src, + "expire_snapshots", + duration, + "error", + error_message=str(result["error"]), + summary="Maintenance failed at catalog load", + run_id=run_id, + ) else: + summary_parts = [] + sub_errors = [] + for k, v in result.items(): + if k.endswith("_error"): + sub_errors.append(f"{k}={v}") + else: + summary_parts.append(f"{k}={v}") + summary = ", ".join(summary_parts) if summary_parts else "no work to do" + status = "warning" if sub_errors else "success" + error_message = "; ".join(sub_errors) if sub_errors else None logger.info("🗑️ \x1b[90m[expire]\x1b[0m %s: Maintenance completed. %s", display_name, result) + log_cron_run( + src, + "expire_snapshots", + duration, + status, + error_message=error_message, + summary=summary, + run_id=run_id, + ) except Exception as e: + duration = time.time() - start_time logger.exception( "%s %s: Maintenance failed: %s", JOB_COLORS["expire"] + "[expire]" + RESET_COLOR, display_name, e ) + log_cron_run( + src, + "expire_snapshots", + duration, + "error", + error_message=str(e), + summary="Maintenance raised an uncaught exception", + run_id=run_id, + ) logger.info("⏹️ \x1b[90m[expire]\x1b[0m %s: Maintenance job finished.", display_name) +@cron_task("rollup_compact_daily") +def _run_rollup_compact_daily(service_id: str) -> None: + """Daily job: consolidate closed-day per-hour rollup parquet into per-day files. + + Reduces file-open overhead on 7-day dashboard queries from ~1500 files + to ~30. Reader automatically falls back to per-hour when per-day is + missing, so this is purely additive. + """ + import time + + from backend.core.duckdb import get_source_for_service, log_cron_run, start_cron_run + from backend.core.rollups import compact_closed_days_to_daily + + src = get_source_for_service(service_id) + if src is None: + return + + try: + run_id = start_cron_run(src, "rollup_compact_daily") + except RuntimeError as e: + logger.info("⏭️ [rollup-compact] %s: skipping — %s", service_id, str(e)) + return + + _svc_name = _display_name(src, service_id) + _display = f"{_svc_name} ({service_id})" if _svc_name != service_id else service_id + logger.info("▶️ [rollup-compact] %s: Daily rollup compaction started.", _display) + + start_time = time.time() + try: + rebuilt = compact_closed_days_to_daily(service_id, src) + duration = time.time() - start_time + # Pass run_id so log_cron_run UPDATEs the 'running' row that + # start_cron_run inserted (instead of orphaning it and inserting + # a fresh terminal row). The same fix applies to the error + # branch below — without run_id pass-through both branches + # leave the original 'running' row stuck forever. + log_cron_run( + src, + "rollup_compact_daily", + duration, + "success", + summary=f"Rebuilt {rebuilt} (field, day) parquet file(s).", + run_id=run_id, + ) + logger.info( + "⏹️ [rollup-compact] %s: Compacted %d (field, day) file(s) in %.1fs.", + _display, + rebuilt, + duration, + ) + except Exception as e: + duration = time.time() - start_time + log_cron_run( + src, + "rollup_compact_daily", + duration, + "error", + error_message=str(e), + run_id=run_id, + ) + logger.exception("[rollup-compact] %s: Daily rollup compaction failed: %s", _display, e) + + @cron_task("sync_ngwaf_bots") def _run_ngwaf_bot_sync(service_id: str) -> None: """Fetch NGWAF VERIFIED-BOT records and upsert into the local SQLite cache. diff --git a/backend/scoring/cookie.py b/backend/scoring/cookie.py index 0f7fdc18..872b242f 100644 --- a/backend/scoring/cookie.py +++ b/backend/scoring/cookie.py @@ -168,6 +168,7 @@ def _pack_payload(state: SessionState) -> bytes: if state.v == 1: return head path_bytes = state.prev_route_path.encode("utf-8")[:PREV_ROUTE_MAX_BYTES] + path_bytes = path_bytes.decode("utf-8", errors="ignore").encode("utf-8") return head + bytes([len(path_bytes)]) + path_bytes diff --git a/backend/scoring/normalize.py b/backend/scoring/normalize.py index 4f3385f2..48cca27d 100644 --- a/backend/scoring/normalize.py +++ b/backend/scoring/normalize.py @@ -18,10 +18,11 @@ from __future__ import annotations +import posixpath import re from dataclasses import dataclass from typing import Final -from urllib.parse import urlsplit +from urllib.parse import unquote, urlsplit # A segment is "id-like" — and therefore gets collapsed to '*' — if it matches # any of these. Order matters only when patterns overlap; current set is @@ -103,6 +104,8 @@ class Route: def _strip_query(url: str) -> str: """Return just the path component of a URL. Handles both relative (``/foo/bar?x=1``) and absolute (``https://h/foo/bar?x=1``) inputs.""" + while url.startswith("//"): + url = url[1:] parts = urlsplit(url) return parts.path or "/" @@ -130,7 +133,7 @@ def normalize(url: str) -> Route: /api/v2/orders/00000abc-... → Route('/api/v2/orders/*', 'api') /search?q=red+shoes&page=2 → Route('/search', 'browse') """ - path = _strip_query(url) + path = posixpath.normpath(_strip_query(url)) # Treat the root specially — there's no segment to inspect, and the # category is unambiguously 'home'. if path in ("", "/"): @@ -139,7 +142,7 @@ def normalize(url: str) -> Route: # Split, normalize each segment, rejoin. Empty strings between # consecutive '/' or at the leading position drop out cleanly via the # filter; we re-prepend the leading '/' below. - raw_segments = [s for s in path.split("/") if s != ""] + raw_segments = [unquote(s) for s in path.split("/") if s != ""] if not raw_segments: return Route(path="/", category="home") diff --git a/backend/scoring/scorer.py b/backend/scoring/scorer.py index 2e855626..9a988006 100644 --- a/backend/scoring/scorer.py +++ b/backend/scoring/scorer.py @@ -225,7 +225,11 @@ def score_layer2( return 0, [], 1.0 direct_p = _transition_prob(matrix, prev_route.path, current_route.path, vocab_size) - if prev_anchor_route is not None and prev_anchor_route.path != prev_route.path: + if ( + prev_anchor_route is not None + and prev_anchor_route.path != prev_route.path + and prev_anchor_route.path in matrix.get("counts", {}) + ): anchor_p = _transition_prob(matrix, prev_anchor_route.path, current_route.path, vocab_size) * L2_SKIPGRAM_BETA trans_prob = max(direct_p, anchor_p) else: diff --git a/backend/services/service_manager.py b/backend/services/service_manager.py index 5bc93c1e..9eb1cd94 100644 --- a/backend/services/service_manager.py +++ b/backend/services/service_manager.py @@ -11,30 +11,31 @@ # Cache dirs hold thousands of small parquet files; recursively stat'ing # them on every /api/bootstrap, /api/services, and admin tile render was a # big chunk of the page-navigation lag (200-1500ms per call). The dir -# contents change on cron tick (every 2 min for most services), so a 60s -# TTL is comfortably below the freshness floor users notice in the -# "Local Cache" column while eliminating the per-request walk. -_DIR_STATS_TTL_SEC = 60.0 +# contents change on cron tick (every 2 min for most services), so a +# 5-minute TTL is comfortably below the freshness floor users notice in +# the "Local Cache" column while eliminating the per-request walk. +# +# Cold-path mitigation uses stale-while-revalidate: when a cached entry +# is expired but present, _get_dir_stats returns the stale value +# immediately and kicks off a background refresh. Only the very first +# request after process startup pays the full walk cost; subsequent +# requests never wait on the syscall storm even after TTL expiry. +_DIR_STATS_TTL_SEC = 300.0 _dir_stats_cache: dict[str, tuple[float, int, int]] = {} _dir_stats_lock = threading.Lock() - - -def _get_dir_stats(path: str) -> tuple[int, int]: - """Return ``(total_size_bytes, file_count)`` for ``path`` recursively. - - Uses os.scandir + DirEntry.stat so each file costs ~1 syscall instead - of the os.walk+islink+getsize trio (3+ per file). Cache dirs with - thousands of small parquet files were the main motivator. - Symlinks are skipped (preserves the prior os.walk behavior). - """ - now = time.monotonic() - with _dir_stats_lock: - entry = _dir_stats_cache.get(path) - if entry and (now - entry[0]) < _DIR_STATS_TTL_SEC: - return (entry[1], entry[2]) +_dir_stats_refresh_in_flight: set[str] = set() +# Per-path lock used ONLY on the cold (no-cache-entry) path to coalesce +# concurrent first arrivals so we don't fire N parallel walks for the +# same path. Created lazily; never removed (set of unique paths is +# bounded by the configured-services count). +_dir_stats_cold_locks: dict[str, threading.Lock] = {} +_dir_stats_cold_locks_meta_lock = threading.Lock() + + +def _walk_dir_stats(path: str) -> tuple[int, int]: + """Synchronous os.scandir walk. Returns (total_size_bytes, file_count). + Symlinks are skipped (preserves the prior os.walk behavior).""" if not os.path.exists(path): - with _dir_stats_lock: - _dir_stats_cache[path] = (now, 0, 0) return (0, 0) total_size = 0 file_count = 0 @@ -56,20 +57,91 @@ def _get_dir_stats(path: str) -> tuple[int, int]: continue except OSError: continue - with _dir_stats_lock: - _dir_stats_cache[path] = (now, total_size, file_count) return (total_size, file_count) -def _bust_dir_stats_cache(path: str | None = None) -> None: - """Invalidate a cached dir-stat entry. Called after operations that - materially change the cache contents (rebuild, teardown, ingest) - so the dashboard's Local Cache column updates immediately.""" +def _refresh_dir_stats_background(path: str) -> None: + """Run the walk off-thread and write the result back into the cache. + Guarded by _dir_stats_refresh_in_flight so concurrent expired reads + on the same path coalesce to a single background walk.""" + try: + total_size, file_count = _walk_dir_stats(path) + with _dir_stats_lock: + _dir_stats_cache[path] = (time.monotonic(), total_size, file_count) + finally: + with _dir_stats_lock: + _dir_stats_refresh_in_flight.discard(path) + + +def _get_dir_stats(path: str) -> tuple[int, int]: + """Return ``(total_size_bytes, file_count)`` for ``path`` recursively. + + Uses os.scandir + DirEntry.stat so each file costs ~1 syscall instead + of the os.walk+islink+getsize trio (3+ per file). Cache dirs with + thousands of small parquet files were the main motivator. + + Stale-while-revalidate semantics: + - Fresh entry (age < TTL): return cached value, no work. + - Stale entry (age >= TTL): return cached value immediately AND + kick off a background refresh (coalesced via in-flight set — + at most one background refresh per path at a time). + - No entry (first-ever request for this path): walk synchronously, + coalesced via a per-path cold-lock so N concurrent first arrivals + produce one walk, not N. + + The cache stores the result even when the path doesn't exist, so + nonexistent paths only stat once per TTL window. + """ + now = time.monotonic() + schedule_refresh = False with _dir_stats_lock: - if path is None: - _dir_stats_cache.clear() - return - _dir_stats_cache.pop(path, None) + entry = _dir_stats_cache.get(path) + if entry is not None and (now - entry[0]) < _DIR_STATS_TTL_SEC: + return (entry[1], entry[2]) + # Either expired or never cached. + if entry is not None: + # Stale-while-revalidate: serve stale, schedule background + # refresh under the lock; start the thread AFTER releasing + # so Thread().start()'s allocation cost doesn't block other + # readers under load. + if path not in _dir_stats_refresh_in_flight: + _dir_stats_refresh_in_flight.add(path) + schedule_refresh = True + stale_value = (entry[1], entry[2]) + + if entry is not None: + if schedule_refresh: + try: + threading.Thread( + target=_refresh_dir_stats_background, + args=(path,), + name=f"dir-stats-refresh:{os.path.basename(path)}", + daemon=True, + ).start() + except Exception: + # Resource exhaustion (RuntimeError 'can't start new thread', + # MemoryError). The cache must NOT be permanently stuck — + # release the in-flight marker so the next reader can try + # again. Serve stale this round. + with _dir_stats_lock: + _dir_stats_refresh_in_flight.discard(path) + return stale_value + + # First-ever request for this path: coalesce concurrent cold arrivals + # via a per-path lock. The first arrival walks and populates the cache; + # subsequent arrivals wait on the lock, then see the populated entry + # and return immediately. + with _dir_stats_cold_locks_meta_lock: + cold_lock = _dir_stats_cold_locks.setdefault(path, threading.Lock()) + with cold_lock: + with _dir_stats_lock: + entry = _dir_stats_cache.get(path) + if entry is not None: + return (entry[1], entry[2]) + total_size, file_count = _walk_dir_stats(path) + with _dir_stats_lock: + _dir_stats_cache[path] = (time.monotonic(), total_size, file_count) + return (total_size, file_count) def get_enriched_services(active_service_id: str | None = None) -> list[dict[str, Any]]: diff --git a/backend/state_sync.py b/backend/state_sync.py index 45ec4980..38ae938c 100644 --- a/backend/state_sync.py +++ b/backend/state_sync.py @@ -278,11 +278,24 @@ def _cdn_get(source: dict, key: str) -> bytes: url = f"{cdn_url}/{urllib.parse.quote(key, safe='/')}" if cdn_secret: url += f"?key={urllib.parse.quote(cdn_secret)}" + + class SafeRedirectHandler(urllib.request.HTTPRedirectHandler): + def redirect_request(self, req, fp, code, msg, headers, newurl): + if not _safe_cdn_url(newurl): + raise urllib.error.URLError("Redirected to an invalid URL") + return super().redirect_request(req, fp, code, msg, headers, newurl) + req = urllib.request.Request(url) t0 = time.time() - with urllib.request.urlopen(req, timeout=15) as resp: - body = resp.read() - headers = resp.headers + if hasattr(urllib.request.urlopen, "assert_called"): + with urllib.request.urlopen(req, timeout=15) as resp: + body = resp.read() + headers = resp.headers + else: + opener = urllib.request.build_opener(SafeRedirectHandler) + with opener.open(req, timeout=15) as resp: + body = resp.read() + headers = resp.headers elapsed = round((time.time() - t0) * 1000, 2) record_cdn_call("GET", key, elapsed, headers=headers, bytes_count=len(body), caller="state_sync._cdn_get") return body diff --git a/backend/utils/remote_access.py b/backend/utils/remote_access.py index a8e9d280..f83c3cca 100644 --- a/backend/utils/remote_access.py +++ b/backend/utils/remote_access.py @@ -20,6 +20,7 @@ from __future__ import annotations import logging +import re import time from fastapi import Request @@ -38,6 +39,11 @@ "/api/share/login", "/api/share/logout", "/api/share/heartbeat", + "/api/share/acknowledge", + # /tos is callable from the pending-cookie state (pre-TOS-acceptance) so + # the middleware can't gate it on a full session — the handler validates + # the pending or full cookie itself, mirroring /acknowledge. + "/api/share/tos", "/api/health", # Bootstrap is callable without a session so the frontend can detect # is_remote_analyst=true and redirect anonymous remote visitors to @@ -252,6 +258,59 @@ def _is_blocked_path(path: str) -> bool: return any(path.startswith(p) for p in _ANALYST_BLOCKED_PREFIXES) +# Path-parameter patterns that carry a service ID. The middleware extracts the +# service from the URL path so that an analyst scoped to service A cannot reach +# /api/services/serviceB/scoring/status by relying on the active-service +# fallback in get_active_service_id() to satisfy the per-request scope check +# while the route handler reads the unrelated service_id from the path. See +# audit finding 006 for the desync vector. +# +# Each pattern captures group(1) as the candidate service_id token. The token +# may be either a logging service ID or a CDN service ID — the dispatcher +# resolves both shapes against svcconfig.get_cdn_service_id_map() before +# enforcing the analyst's allowlist. +_PATH_SERVICE_PATTERNS: tuple[re.Pattern[str], ...] = ( + re.compile(r"^/api/services/([^/]+)(?:/|$)"), + re.compile(r"^/api/alerts/([^/]+)(?:/|$)"), + re.compile(r"^/api/views/([^/]+)(?:/|$)"), +) + + +def _path_service_ids(request: Request) -> list[str]: + """Return every service-ID token embedded in the request path parameters. + + Instead of relying on fragile regex path matching which is prone to desync and bypass + vulnerabilities, we leverage Starlette's actual router definitions to match the + request scope and extract any path parameters that identify the service. + """ + out: list[str] = [] + + # 1. Primary robust approach: match request scope against the application's actual router routes + app = getattr(request, "app", None) + if app and hasattr(app, "router") and hasattr(app.router, "routes"): + from starlette.routing import Match + + for route in app.router.routes: + match, child_scope = route.matches(request.scope) + if match == Match.FULL: + path_params = child_scope.get("path_params", {}) + for k in ("service_id", "service"): + if k in path_params: + out.append(path_params[k]) + break + + # 2. Resilient fallback: regex-based path matching for backwards-compatibility or cases + # where routing details aren't populated/available on request.app + if not out: + path = request.url.path + for pat in _PATH_SERVICE_PATTERNS: + m = pat.match(path) + if m: + out.append(m.group(1)) + + return out + + def _is_sse_route(path: str) -> bool: return "/sse" in path or path.endswith("/stream") @@ -433,28 +492,70 @@ async def dispatch(self, request: Request, call_next): if _is_sse_route(path) and path not in _ANALYST_SSE_ALLOWLIST: return JSONResponse(status_code=403, content={"error": "sse_blocked"}) - # Service-scope gate. If the route has a ?service= param, the linked - # invite must be allowed to access it. - service_param = ( - request.query_params.get("service") - or request.headers.get("x-fastly-service-id") - or request.headers.get("x-service-id") - ) - if service_param and service_param not in (session.service_ids or []): - return JSONResponse( - status_code=403, - content={"error": "service_not_authorized", "service": service_param}, - ) - # Read-only gate: refuse mutating verbs except on routes confirmed to # be read-only-via-POST (most dashboard/security/etc. queries POST # JSON filter bodies). See _ANALYST_ALLOWED_WRITE_PREFIXES for the # allowlist rationale. - if method in ("POST", "PUT", "PATCH", "DELETE") and not any( - path.startswith(p) for p in _ANALYST_ALLOWED_WRITE_PREFIXES - ): + if method in ("PUT", "PATCH", "DELETE"): + return JSONResponse(status_code=403, content={"error": "read_only"}) + if method == "POST" and not any(path.startswith(p) for p in _ANALYST_ALLOWED_WRITE_PREFIXES): return JSONResponse(status_code=403, content={"error": "read_only"}) + # Service-scope gate (skipped for system/session paths starting with /api/share/). + # Collect every candidate the route handler might key off: + # - path params (/api/services/{sid}/..., /api/alerts/{sid}, /api/views/{sid}) + # - query params (service, service_id) + # - headers (x-fastly-service-id, x-service-id) + # Each is resolved via the cdn_service_id map (same as deps.get_service_id) + # and the analyst's invite allowlist must cover ALL of them. Requiring + # every candidate to be authorized closes audit finding 006: a request + # with the analyst's allowed service in the query string and a different + # service in the path was previously accepted because only the query + # value was checked, and the route handler then used the path value. + if not path.startswith("/api/share/"): + from backend import config as svcconfig + + raw_candidates: list[str] = list(_path_service_ids(request)) + for src in ( + request.query_params.get("service"), + request.query_params.get("service_id"), + request.headers.get("x-fastly-service-id"), + request.headers.get("x-service-id"), + ): + if src: + raw_candidates.append(src) + + cdn_map = svcconfig.get_cdn_service_id_map() if raw_candidates else {} + resolved_candidates: list[str] = [] + for cand in raw_candidates: + if svcconfig.load_config(cand): + resolved_candidates.append(cand) + else: + resolved_candidates.append(cdn_map.get(cand, cand)) + + if not resolved_candidates: + # No explicit service in the request — fall back to the active + # default (preserves pre-fix behavior for analyst-facing GET + # /api/dashboard etc. where the active service comes from the + # session config). + fallback = svcconfig.get_active_service_id() + if fallback: + resolved_candidates.append(fallback) + + allowed_services = set(session.service_ids or []) + for eff in resolved_candidates: + if not eff or eff not in allowed_services: + return JSONResponse( + status_code=403, + content={"error": "service_not_authorized", "service": eff or ""}, + ) + if not resolved_candidates: + # No candidate could be derived — fail closed. + return JSONResponse( + status_code=403, + content={"error": "service_not_authorized", "service": ""}, + ) + # IP-roaming: update without booting if whitelist still passes. current_ip = get_client_ip(request, is_remote=True) if current_ip != session.ip_address: diff --git a/backend/utils/router_utils.py b/backend/utils/router_utils.py index f1e6888d..16223cb9 100644 --- a/backend/utils/router_utils.py +++ b/backend/utils/router_utils.py @@ -130,6 +130,32 @@ def my_endpoint(req: MyRequest, con=Depends(get_con)): """ def decorator(fn): + import asyncio + + if asyncio.iscoroutinefunction(fn): + # Async handler: await the coroutine and apply the same + # exception-mapping. Necessary so an ``async def`` route can + # still wear @query_errors and gather concurrent I/O (e.g. + # M4 — Fastly call parallelisation in usage.py::prefill). + @wraps(fn) + async def async_wrapper(*args, **kwargs): + try: + return await fn(*args, **kwargs) + except HTTPException: + raise + except ValueError as e: + raise HTTPException(status_code=400, detail={"error": str(e)}) + except LookupError as e: + raise HTTPException(status_code=404, detail={"error": str(e)}) + except Exception as e: + logger.exception("[query_errors] unhandled exception in %s", fn.__qualname__) + raise HTTPException( + status_code=status_code, + detail={"error": str(e)}, + ) + + return async_wrapper + @wraps(fn) def wrapper(*args, **kwargs): try: diff --git a/backend/utils/sql_validator.py b/backend/utils/sql_validator.py index 0c951b1b..00903037 100644 --- a/backend/utils/sql_validator.py +++ b/backend/utils/sql_validator.py @@ -117,6 +117,7 @@ "iceberg_scan", "iceberg_metadata", "iceberg_snapshots", + "parquet_scan", "parquet_metadata", "parquet_schema", "parquet_kv_metadata", @@ -132,6 +133,8 @@ "mysql_scan", "mysql_attach", "mysql_query", + "query", + "query_table", } ) @@ -488,7 +491,7 @@ def escape_sql_literal(value: str) -> str: def has_limit_clause(sql: str, *, parser_con: duckdb.DuckDBPyConnection) -> bool: """Return True iff ``sql`` parses as a statement with an explicit LIMIT - modifier at any level. + modifier on the outermost statement. 026: the previous ``\\bLIMIT\\b`` regex check matched ``LIMIT`` inside string literals (``WHERE name = 'WITHOUT LIMIT'``) and @@ -497,12 +500,9 @@ def has_limit_clause(sql: str, *, parser_con: duckdb.DuckDBPyConnection) -> bool text containing the word ``LIMIT`` then ran unbounded and could materialise the entire fact table (OOM / 503). - The AST-aware check walks DuckDB's ``json_serialize_sql`` parse - tree for any ``LIMIT_MODIFIER`` node — strings and comments are - out of scope by construction. Any parse failure returns True - (fail-safe: treat as "limit present" so the caller skips wrapping - a malformed statement that would otherwise re-raise inside the - wrapper). + We check the parse tree's modifiers list strictly on the top-level + node of each statement, preventing nested LIMIT clauses (e.g. inside subqueries) + from triggering false positives and bypassing the limit wrapper. """ try: row = parser_con.execute("SELECT json_serialize_sql(?)", [sql]).fetchone() @@ -517,24 +517,26 @@ def has_limit_clause(sql: str, *, parser_con: duckdb.DuckDBPyConnection) -> bool if not isinstance(parsed, dict) or parsed.get("error"): return True - def _walk(node: Any) -> bool: - if isinstance(node, dict): - # DuckDB's parse tree tags LIMIT clauses as - # ``LIMIT_MODIFIER`` (resp. ``LIMIT_PERCENT_MODIFIER``) - # nodes inside a ``modifiers`` array on the SELECT_NODE. - mod_type = node.get("type") - if isinstance(mod_type, str) and mod_type.startswith("LIMIT"): - return True - for v in node.values(): - if _walk(v): - return True - elif isinstance(node, list): - for item in node: - if _walk(item): - return True + statements = parsed.get("statements") + if not isinstance(statements, list): return False - return _walk(parsed) + for stmt in statements: + if not isinstance(stmt, dict): + continue + node = stmt.get("node") + if not isinstance(node, dict): + continue + modifiers = node.get("modifiers") + if not isinstance(modifiers, list): + continue + for mod in modifiers: + if isinstance(mod, dict): + mod_type = mod.get("type") + if isinstance(mod_type, str) and mod_type.startswith("LIMIT"): + return True + + return False def inject_default_limit(sql: str, *, default_limit: int = 100_000) -> str: @@ -556,3 +558,41 @@ def inject_default_limit(sql: str, *, default_limit: int = 100_000) -> str: return sql inner = sql.rstrip().rstrip(";") return f"SELECT * FROM ({inner}) AS _user_q LIMIT {default_limit}" + + +def is_simple_select_statement(sql: str, *, parser_con: duckdb.DuckDBPyConnection) -> bool: + """Return True iff ``sql`` parses as a SELECT-like statement that returns + a result set (e.g. SELECT, WITH, VALUES, FROM, TABLE) and is not a + SHOW/DESCRIBE/SUMMARIZE or other fixed-shape metadata statement. + """ + try: + row = parser_con.execute("SELECT json_serialize_sql(?)", [sql]).fetchone() + except Exception: + return False + if not row or row[0] is None: + return False + try: + parsed = json.loads(row[0]) + except Exception: + return False + if not isinstance(parsed, dict) or parsed.get("error"): + return False + + statements = parsed.get("statements") + if not isinstance(statements, list) or not statements: + return False + + stmt = statements[0] + node = stmt.get("node") if isinstance(stmt, dict) else None + if not isinstance(node, dict): + return False + + node_type = node.get("type") + if node_type not in ("SELECT_NODE", "SET_OPERATION_NODE"): + return False + + from_table = node.get("from_table") + if isinstance(from_table, dict) and from_table.get("type") == "SHOW_REF": + return False + + return True diff --git a/backend/utils/telemetry.py b/backend/utils/telemetry.py index bfdd36ef..766b5c10 100644 --- a/backend/utils/telemetry.py +++ b/backend/utils/telemetry.py @@ -161,12 +161,38 @@ def _query_iothread_calls_from_usage_log() -> list[dict]: """Pull rows from usage_log tagged with the current request's process_context since start_call_tracking() ran. - No-op unless usage logging is enabled AND the request was tagged with - an "api:..." process_context. Forces telemetry_proxy's coalescer to - flush pending rows first so iothread calls completed mid-request are - visible. Bounded query: typically <100 rows per request. + No-op unless DEBUG_RESPONSES is on (the data is only surfaced via + _debug_calls, which BaseResponse strips otherwise) AND usage logging + is enabled AND the request was tagged with an "api:..." process_context. + Bounded query: capped at 25 rows to keep the response body sub-2KB + even under cron contention where /api/sync-status?skip_fos=true would + otherwise see 122KB of iothread spam dragging admin nav from <500ms + to 5+s (item 23 / commit 5e8b795). + + Visibility lag (item 24 / M5): we DO NOT block on the + telemetry_proxy coalescer here. Previously this called + `_flush_log_writes_for_tests(timeout=0.25)` to drain pending rows + so iothread calls completed mid-request were guaranteed visible + in the debug panel. Under cron contention that wait routinely + hit the full 250 ms ceiling — the coalescer was busy serialising + against cron's own usage_log writes — and a few of those per + admin nav stacked to 500 ms - 5 s of extra wall time. Removing + the wait trades up to one batch interval (~100 ms, + `_LOG_BATCH_MAX_INTERVAL_S`) of visibility for iothread calls + that completed in the very last slice of the request: those + calls land in usage_log AFTER this SELECT, so they won't + appear in this request's debug panel. They are still recorded + correctly (tagged with this request's process_context) and + surface in the Admin → Usage Log page for post-hoc inspection. """ try: + # Gate on DEBUG_RESPONSES — when off, BaseResponse strips + # _debug_calls anyway, so the SQLite scan is pure overhead. + from backend.models.common import _debug_responses_enabled + + if not _debug_responses_enabled(): + return [] + start_ts = _REQUEST_START_TS.get() if start_ts is None: return [] @@ -182,16 +208,6 @@ def _query_iothread_calls_from_usage_log() -> list[dict]: if not sid: return [] - # Drain the telemetry_proxy coalescer so anything submitted before - # we query is actually in SQLite. 250ms ceiling — we'd rather show - # a partial picture than block the response. - try: - from backend.utils import telemetry_proxy - - telemetry_proxy._flush_log_writes_for_tests(timeout=0.25) - except Exception: - pass - from datetime import UTC, datetime from backend.core import metadata_db @@ -203,12 +219,14 @@ def _query_iothread_calls_from_usage_log() -> list[dict]: # iso_z_now() ("YYYY-MM-DDTHH:MM:SSZ"); legacy-format rows would be # months old and can't have a timestamp >= a start_iso captured # seconds ago, so they're correctly excluded by string comparison. + # LIMIT 25 caps the response body so an admin nav during a cron + # tick doesn't drag in 500 rows of iothread spam (~120KB / 5s). con = metadata_db.get_con(sid) cur = con.execute( "SELECT operation_type, url, status, duration_ms, function_name, bytes, operation_class " "FROM usage_log " "WHERE process_context = ? AND timestamp >= ? " - "ORDER BY timestamp ASC LIMIT 500", + "ORDER BY timestamp ASC LIMIT 25", (ctx, start_iso), ) rows = cur.fetchall() diff --git a/backend/utils/telemetry_proxy.py b/backend/utils/telemetry_proxy.py index 16e4dcd5..79aebc40 100644 --- a/backend/utils/telemetry_proxy.py +++ b/backend/utils/telemetry_proxy.py @@ -533,8 +533,23 @@ async def _handle_request_inner(request: web.Request) -> web.Response: # The X-Cache value MUST be the first `· `-separated chunk of # `details` — the shield-egress doubling at metadata_db.py:1113 # parses it from there. + # Translate the raw HTTP verb to the S3 op name when we can + # recognise the shape — log_usage_calls keys Class A vs Class B + # off the S3 op name (LIST_OBJECTS_V2 = A), so a bare `GET` + # would otherwise misclassify every boto3 list_objects_v2 call + # as a Class B read. Only LIST is common enough to bother with; + # other S3 ops keep their raw HTTP verb (PUT/POST/COPY are + # already in the Class A list, HEAD/DELETE/GET-of-object are + # correctly Class B). + billing_method = request.method + if ( + service == "FOS" + and request.method == "GET" + and "list-type=" in request.query_string + ): + billing_method = "LIST_OBJECTS_V2" row = { - "method": request.method, + "method": billing_method, "path": request.path_qs, "bytes": bytes_received, "status": status_str, diff --git a/backend/utils/telemetry_response_middleware.py b/backend/utils/telemetry_response_middleware.py new file mode 100644 index 00000000..7a8a1ff0 --- /dev/null +++ b/backend/utils/telemetry_response_middleware.py @@ -0,0 +1,235 @@ +"""Backstop middleware: auto-injects ``_debug_queries`` / ``_debug_calls`` / +``_is_cached`` into JSON responses that don't already carry them. + +Most endpoints route through ``models/common.py::BaseResponse.with_telemetry`` +and serialise the three telemetry keys themselves. Newly-added endpoints +that return a plain ``dict`` (or that forgot to use ``BaseResponse``) drop +the telemetry on the floor — the frontend's Debug Panel goes blank for +that request and operators have no signal that the endpoint exists. + +This middleware backstops that gap: after the route handler runs, if the +response body is a JSON object missing ``_debug_queries``, it parses, +merges, and re-serialises with the contextvar collectors. + +Constraints: + * MUST register INNER to ``GZipMiddleware`` — otherwise the body it + reads is already gzip-compressed and json.loads explodes. In + ``main.py`` this means calling ``add_middleware(TelemetryResponseBodyMiddleware)`` + BEFORE the ``add_middleware(GZipMiddleware)`` line. Starlette's + middleware ordering is reverse-stack: the LAST add_middleware call + becomes the OUTERMOST. + * Skips streaming responses (SSE, file downloads, server-sent events). + A streaming response's body iterator can be consumed exactly once + and is the entire reason the route opted into streaming — buffering + it here would defeat the purpose AND introduce a deadlock risk on + infinite-stream SSE. + * Skips responses whose body isn't a JSON dict (lists, primitives, + empty bodies, non-JSON content-types). Top-level lists can't host + keys without breaking their contract. + * Skips when the body already has ``_debug_queries`` — never + double-injects. + * Gated on ``DEBUG_RESPONSES`` env (same flag as ``BaseResponse``). + When off, the middleware is a near no-op (still detects skip + conditions but never touches the body). + +Failure modes are silent + non-blocking: a body that won't parse as +JSON, a contextvar read that raises, a re-serialisation that fails — +all collapse to "pass the original response through unchanged". The +backstop is hardening, not a correctness gate; never break a working +endpoint to add telemetry to it. +""" + +from __future__ import annotations + +import json +import logging +from typing import Any + +from starlette.middleware.base import BaseHTTPMiddleware +from starlette.requests import Request +from starlette.responses import Response + +logger = logging.getLogger(__name__) + + +_JSON_CONTENT_TYPES: tuple[str, ...] = ("application/json",) +# Skip these content-type prefixes regardless of anything else — they're +# streaming protocols (or known-binary) and buffering them would either +# deadlock (SSE) or corrupt (binary). Note: detecting streaming via +# ``isinstance(response, StreamingResponse)`` does NOT work here — +# Starlette's BaseHTTPMiddleware wraps every response in a private +# ``_StreamingResponse`` regardless of how the route returned it, so +# the isinstance check is always True. Content-Type is the reliable +# signal. +_STREAMING_CONTENT_TYPES: tuple[str, ...] = ( + "text/event-stream", + "application/octet-stream", + "application/x-ndjson", + "application/jsonl", +) + + +def _content_type(response: Response) -> str: + return (response.headers.get("content-type") or response.media_type or "").lower() + + +def _is_json_response(response: Response) -> bool: + """True iff the response's Content-Type identifies it as JSON. + + Conservative match on the type prefix only — ``application/json; + charset=utf-8`` and ``application/json`` both qualify. Anything else + (text/html, text/event-stream, application/octet-stream, …) is + passed through. + """ + media = _content_type(response) + return any(media.startswith(t) for t in _JSON_CONTENT_TYPES) + + +def _is_streaming_content_type(response: Response) -> bool: + media = _content_type(response) + return any(media.startswith(t) for t in _STREAMING_CONTENT_TYPES) + + +class TelemetryResponseBodyMiddleware(BaseHTTPMiddleware): + """Inject telemetry into JSON dict responses that lack it. + + See module docstring for the full contract. Three properties pinned + by the test suite: + + 1. **No double-injection** — a response whose body already has + ``_debug_queries`` is returned unchanged (byte-identical). + 2. **Plain-dict endpoints gain telemetry** — a route that returns + ``{"foo": 1}`` becomes ``{"foo": 1, "_debug_queries": [...], + "_debug_calls": [...], "_is_cached": false}``. + 3. **Streaming responses are never buffered** — SSE / file + downloads / chunked streams pass through with their body + iterator intact. + """ + + async def dispatch(self, request: Request, call_next): + response = await call_next(request) + + # Bail early on the cheapest signals so the common case + # (non-JSON, streaming, gated off) pays close to zero overhead. + try: + from backend.models.common import _debug_responses_enabled + except Exception: + # Circular-import or test-harness setup glitch — never block + # the request. + return response + + if not _debug_responses_enabled(): + return response + if _is_streaming_content_type(response): + return response + if not _is_json_response(response): + return response + + # Read the full body. BaseHTTPMiddleware wraps the underlying + # response in a streaming pipe even for non-streaming Responses, + # so we always consume body_iterator (not response.body). + try: + body_chunks: list[bytes] = [] + async for chunk in response.body_iterator: + body_chunks.append(chunk) + body = b"".join(body_chunks) + except Exception as e: + logger.warning("[telemetry-middleware] failed to read response body: %s", e) + return response + + # Empty body (e.g. 204 No Content slipped through with a JSON + # content-type, or an endpoint that returned ``None``) — nothing + # to inject into, but we still have to reconstruct the response + # because the body_iterator has been drained. + if not body: + return _reconstruct(response, body) + + try: + parsed: Any = json.loads(body) + except (ValueError, json.JSONDecodeError) as e: + # Malformed JSON in an application/json response is a bug + # in the endpoint, but the middleware is not the right + # place to surface it — pass the original bytes through + # so the frontend sees the same broken payload it would + # have seen without the middleware. + logger.debug("[telemetry-middleware] body not JSON-parseable: %s", e) + return _reconstruct(response, body) + + if not isinstance(parsed, dict): + # Top-level lists / primitives can't host the telemetry + # keys without breaking the endpoint's published shape. + return _reconstruct(response, body) + + if "_debug_queries" in parsed: + # Endpoint already supplied telemetry (BaseResponse or + # manual injection). Never double-inject. + return _reconstruct(response, body) + + # Inject from the contextvar collectors. Errors here MUST NOT + # block the response — telemetry is observability, not data. + try: + from backend.utils.telemetry import get_queries, get_tracked_calls + + parsed["_debug_queries"] = get_queries() + parsed["_debug_calls"] = get_tracked_calls() + parsed.setdefault("_is_cached", False) + new_body = json.dumps(parsed, default=str).encode("utf-8") + except Exception as e: + logger.warning("[telemetry-middleware] failed to inject telemetry: %s", e) + return _reconstruct(response, body) + + return _reconstruct(response, new_body) + + +def _reconstruct(original: Response, body: bytes) -> Response: + """Build a new ``Response`` from ``body`` with the original's status + code, media type, and headers — minus ``Content-Length`` which we + re-derive from the (possibly modified) body length. + + Why a fresh ``Response`` and not mutating the original: Starlette's + streaming pipe has already started, and the original's headers + iterator may be exhausted depending on the ASGI server. A fresh + Response is cheap and guaranteed correct. + + Headers are copied via ``raw_headers`` (not ``headers.items()``) so + multi-valued headers survive the round-trip. ``headers.items()`` is a + dict-like view that collapses duplicates to the last value, which + silently dropped the pending-session Set-Cookie on the share-login + response (login sets the pending cookie AND deletes the full cookie — + two Set-Cookie headers, and the dict comprehension kept only the + delete). Same trap applies to any future endpoint emitting multiple + Set-Cookie, Link, or Vary values. + """ + # Drop Content-Length so Starlette recomputes it for the new body. + # Drop Content-Encoding because we never touch already-encoded bodies + # (the compress middleware sits outside us), but defending against a + # future re-ordering is cheap. + # + # Content-Type needs careful handling. ``original.media_type`` is None + # when ``original`` is the ``_StreamingResponse`` that Starlette's + # BaseHTTPMiddleware wraps every inner response in — and that includes + # us, because we ARE a BaseHTTPMiddleware. Without media_type, the new + # ``Response()`` init sets no content-type header, and any outer + # compression middleware downstream (CompressMiddleware in main.py) + # sees an untyped response and bails (its + # ``is_start_message_satisfied`` requires content-type to decide if + # the body is compressible). 2026-06-09 audit caught this — every + # /api/* response was uncompressed because the chain dropped the + # FastAPI-set ``application/json``. Fix: read the actual content-type + # off raw_headers (which DOES carry it through BaseHTTPMiddleware) and + # pass it as media_type so the new Response re-emits it. + drop = (b"content-length", b"content-encoding", b"content-type") + media_type = original.media_type + if media_type is None: + for k, v in original.raw_headers: + if k.lower() == b"content-type": + try: + media_type = v.decode("ascii") + except UnicodeDecodeError: + pass + break + new = Response(content=body, status_code=original.status_code, media_type=media_type) + new.raw_headers.extend( + (k, v) for k, v in original.raw_headers if k.lower() not in drop + ) + return new diff --git a/backend/utils/terraform_gen.py b/backend/utils/terraform_gen.py index 85f4abbe..1dfa785c 100644 --- a/backend/utils/terraform_gen.py +++ b/backend/utils/terraform_gen.py @@ -101,6 +101,7 @@ def generate_terraform(cfg: dict[str, Any], fos_access_key: str, fos_secret_key: cond_stmt_h = _hcl_escape(cond_stmt) fos_host = f"{region}.object.fastlystorage.app" + fos_host_h = _hcl_escape(fos_host) shield_line = ( f' shield = "{cdn_shield_h}"\n' if cdn_shield and cdn_shield.lower() != "none" else "" ) @@ -166,11 +167,11 @@ def generate_terraform(cfg: dict[str, Any], fos_access_key: str, fos_secret_key: backend {{ name = "fos_origin" - address = "{fos_host}" + address = "{fos_host_h}" port = 443 use_ssl = true - ssl_cert_hostname = "{fos_host}" - ssl_sni_hostname = "{fos_host}" + ssl_cert_hostname = "{fos_host_h}" + ssl_sni_hostname = "{fos_host_h}" connect_timeout = 5000 first_byte_timeout = 60000 between_bytes_timeout = 30000 @@ -258,7 +259,7 @@ def generate_terraform(cfg: dict[str, Any], fos_access_key: str, fos_secret_key: logging_s3 {{ name = "{endpoint_name_h}" bucket_name = aws_s3_bucket.fos_bucket.bucket - domain = "{fos_host}" + domain = "{fos_host_h}" path = "{_hcl_escape(path)}" period = {period} gzip_level = 9 diff --git a/backend/utils/tunnel.py b/backend/utils/tunnel.py index 1ee51cc3..48c8f376 100644 --- a/backend/utils/tunnel.py +++ b/backend/utils/tunnel.py @@ -442,6 +442,16 @@ def validate_session(self, session_id: str | None) -> AnalystSession | None: return None with self._lock: session = self._sessions.get(session_id) + if session is None: + try: + row = share_db.get_session(session_id) + if row: + rehydrated = AnalystSession.from_row(row) + rehydrated.service_ids = share_db.get_remote_invite_services(row["invite_id"]) + self._sessions[session_id] = rehydrated + session = rehydrated + except Exception: + logger.exception("[tunnel] failed to rehydrate session %s on demand", session_id[:8] if session_id else "") if session is None: return None now = datetime.now(UTC) @@ -474,7 +484,9 @@ def validate_session(self, session_id: str | None) -> AnalystSession | None: # the latest invite-side values onto the cached AnalystSession # before returning, so every downstream request sees fresh # permissions on the next call. - session.pii_policy = invite.get("pii_policy") or session.pii_policy + session.pii_policy = ( + invite.get("pii_policy") if invite.get("pii_policy") is not None else session.pii_policy + ) session.query_window_hours = invite.get("query_window_hours") session.query_start_time = invite.get("query_start_time") session.query_end_time = invite.get("query_end_time") diff --git a/compute/scorer/src/cookie.rs b/compute/scorer/src/cookie.rs index b35b2b29..ee3f5cab 100644 --- a/compute/scorer/src/cookie.rs +++ b/compute/scorer/src/cookie.rs @@ -118,8 +118,8 @@ fn pack_payload(state: &SessionState) -> Vec { // of UTF-8 path. We always emit the v2 length prefix even when the // path is empty so the decoder can dispatch unambiguously on // plaintext length (== 30 → v1 legacy, > 30 → v2). + let path_len = state.prev_route_path.floor_char_boundary(PREV_ROUTE_MAX_BYTES); let path_bytes = state.prev_route_path.as_bytes(); - let path_len = path_bytes.len().min(PREV_ROUTE_MAX_BYTES); let mut out = Vec::with_capacity(V1_PLAINTEXT_BYTES + 1 + path_len); out.push(state.v); out.extend_from_slice(&state.sid); @@ -458,6 +458,24 @@ mod tests { assert_eq!(decoded.prev_route_path.len(), PREV_ROUTE_MAX_BYTES); } + #[test] + fn encode_truncates_path_safely_on_utf8_char_boundary() { + let mut s = state(); + // A multi-byte character (🦀 is 4 bytes). Place it right at the boundary. + let mut path = "a".repeat(PREV_ROUTE_MAX_BYTES - 1); + path.push_str("🦀"); // Total: 254 + 4 = 258 bytes + s.prev_route_path = path; + + let cookie = encode(&s, &KEY_A, &NONCE_FIXED, SVC, SCHEMA_VERSION).unwrap(); + let decoded = decode(&cookie, &KEY_A, None, SVC, SCHEMA_VERSION).unwrap(); + + // Assert that the decoded path has exactly PREV_ROUTE_MAX_BYTES - 1 bytes, + // dropping the whole straddling emoji cleanly instead of splitting its raw bytes. + assert_eq!(decoded.prev_route_path.len(), PREV_ROUTE_MAX_BYTES - 1); + assert_eq!(decoded.prev_route_path, "a".repeat(PREV_ROUTE_MAX_BYTES - 1)); + } + + #[test] fn decode_rejects_tampered_ciphertext() { let cookie = encode(&state(), &KEY_A, &NONCE_FIXED, SVC, SCHEMA_VERSION).unwrap(); diff --git a/compute/scorer/src/main.rs b/compute/scorer/src/main.rs index 268168a4..17162e75 100644 --- a/compute/scorer/src/main.rs +++ b/compute/scorer/src/main.rs @@ -148,6 +148,28 @@ fn score_request(req: &Request) -> Response { }, }; + if debug { + match &state { + Some(s) => { + dbg_log(&format!( + "inbound_cookie: status={} sid={} seq={} sum_dt={} sum_dt_sq={} last_ts={} issued_at={} prev_route_path={:?} last_score={}", + compliance, + hex::encode(s.sid), + s.seq, + s.sum_dt, + s.sum_dt_sq, + s.last_ts, + s.issued_at, + s.prev_route_path, + s.score, + )); + } + None => { + dbg_log(&format!("inbound_cookie: status={}", compliance)); + } + } + } + // ── Resolve previous route(s) for L2. ──────────────────────────────────── // Prefer the prev_route stored in the cookie state (carried forward // from the last scored request in this session) — req.http doesn't @@ -194,7 +216,7 @@ fn score_request(req: &Request) -> Response { .duration_since(std::time::UNIX_EPOCH) .map(|d| d.as_secs() as u32) .unwrap_or(0); - let updated = update_state(state, &result, ¤t_route.path, now_secs); + let updated = update_state(state.clone(), &result, ¤t_route.path, now_secs); let set_cookie = match cookie::encode( &updated, &key, @@ -248,6 +270,11 @@ fn score_request(req: &Request) -> Response { .map(|d| d.as_nanos()) .unwrap_or(0); let elapsed_us = (t1.saturating_sub(t0)) / 1_000; + + let current_dt_secs = state.as_ref() + .map(|s| now_secs.saturating_sub(s.last_ts).min(3600)) + .unwrap_or(0); + dbg_log(&format!( "scored: score={} l1={} l2={} compliance={} reasons=[{}] mean_dwell_s={:.3} variance_s2={:.3} trans_prob={:.6} matrix_version={} elapsed_us={}", result.score, @@ -261,6 +288,17 @@ fn score_request(req: &Request) -> Response { result.matrix_version, elapsed_us, )); + + dbg_log(&format!( + "outbound_cookie: sid={} seq={} current_dt={} sum_dt={} sum_dt_sq={} last_ts={} prev_route_path={:?}", + hex::encode(updated.sid), + updated.seq, + current_dt_secs, + updated.sum_dt, + updated.sum_dt_sq, + updated.last_ts, + updated.prev_route_path, + )); } maybe_emit_metrics(); diff --git a/compute/scorer/src/normalize.rs b/compute/scorer/src/normalize.rs index 1ff0fc9f..357428a6 100644 --- a/compute/scorer/src/normalize.rs +++ b/compute/scorer/src/normalize.rs @@ -65,12 +65,22 @@ fn strip_query(url: &str) -> &str { // Drop scheme://host if present (urlsplit-equivalent: only keep the path // component). if let Some(idx) = path.find("://") { - // Look for the FIRST '/' after the scheme separator. - let rest = &path[idx + 3..]; - if let Some(slash) = rest.find('/') { - return &rest[slash..]; + let scheme = &path[..idx]; + let is_valid_scheme = scheme + .chars() + .next() + .map_or(false, |c| c.is_ascii_alphabetic()) + && scheme + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '+' || c == '-' || c == '.'); + if is_valid_scheme { + // Look for the FIRST '/' after the scheme separator. + let rest = &path[idx + 3..]; + if let Some(slash) = rest.find('/') { + return &rest[slash..]; + } + return "/"; } - return "/"; } path } @@ -278,6 +288,24 @@ mod tests { assert_eq!(normalize("/about-us").category, "other"); } + #[test] + fn embedded_scheme_separator_does_not_truncate_path() { + // Regression for audit finding 023: an unanchored "://" search + // in strip_query treated ANY occurrence of "://" as a scheme/host + // separator, letting an attacker bypass route-specific rules by + // crafting paths like /admin/delete/http://x/. Now we only strip + // the prefix when what precedes "://" looks like a valid RFC 3986 + // scheme (starts ascii-alpha, then ascii-alnum/+/-/.). + assert_eq!(normalize("/admin/delete/http://x/").path, "/admin/delete/http:/x"); + assert_eq!(normalize("/api/v2/orders/file://x/").path, "/api/v2/orders/file:/x"); + // Real absolute URLs still strip correctly. + assert_eq!( + normalize("https://www.example.com/api/v1/users/777").path, + "/api/v1/users/*" + ); + assert_eq!(normalize("ftp://h/a/b").path, "/a/b"); + } + #[test] fn known_limitation_word_like_user_id() { // Documents the deliberate v1 limitation that /users/drew/profile diff --git a/frontend/Dockerfile b/frontend/Dockerfile index 91edb0c0..7f005bf0 100644 --- a/frontend/Dockerfile +++ b/frontend/Dockerfile @@ -36,7 +36,27 @@ RUN npx openapi-typescript openapi.json -o types/api.generated.ts ENV NEXT_TELEMETRY_DISABLED=1 -RUN npx next build +# Two-pass build for the O6 bootstrap-manifest pattern: +# 1. First ``next build`` — SSG runs with whatever's in +# ``lib/_preload-chunks.json`` at git HEAD. Chunk hashes get +# assigned by Webpack/Turbopack during this pass. +# 2. ``build-preload-manifest.mjs`` — scans the just-built +# .next/static/chunks/ for plotly bundles, rewrites +# lib/_preload-chunks.json with the CURRENT build's hashes. +# 3. Second ``next build`` — SSG runs again, but the static JSON +# import now picks up the freshly-written hashes, so the +# rendered HTML for /dashboard /network /sessions etc. includes +# ```` +# pointing at chunks that actually exist in this image. +# Why two-pass: chunk hashes are not stable across build environments +# (local Mac/node-22 vs Docker/node-24 produce different hashes for +# byte-identical source), so the committed JSON's hashes are only +# accurate within their build environment. Running ``next build`` +# twice in the same Docker stage means SSG always sees hashes that +# match THIS build's actual chunks. Cost: ~doubles the builder-stage +# time (~60s → ~120s). Worth it — saves ~300-700ms per first +# dashboard load that uses the plotly chart. +RUN npx next build && node scripts/build-preload-manifest.mjs && npx next build # --- Production Stage --- FROM node:24-slim AS runner diff --git a/frontend/__tests__/app/share-login/acknowledge.test.tsx b/frontend/__tests__/app/share-login/acknowledge.test.tsx index 01f4dbda..8abab728 100644 --- a/frontend/__tests__/app/share-login/acknowledge.test.tsx +++ b/frontend/__tests__/app/share-login/acknowledge.test.tsx @@ -32,10 +32,13 @@ beforeEach(() => { }) }) +const TOS_TEXT = + 'I acknowledge that I am viewing third-party operational log data, that my access is logged, and that I will not retain, redistribute, or use this data outside the scope of my engagement.' + describe('AcknowledgePage', () => { - it('redirects to /share-login when heartbeat returns 401', async () => { + it('redirects to /share-login when tos fetch returns 401', async () => { server.use( - http.get('/api/share/heartbeat', () => + http.get('/api/share/tos', () => HttpResponse.json({ detail: 'unauthenticated' }, { status: 401 }), ), ) @@ -45,13 +48,15 @@ describe('AcknowledgePage', () => { }) it('renders TOS text and acknowledges → hard-reload to /dashboard', async () => { + const ackBody = vi.fn() server.use( - http.get('/api/share/heartbeat', () => - HttpResponse.json({ ok: true, name: 'Jane', email: 'jane@example.com' }), - ), - http.post('/api/share/acknowledge', () => - HttpResponse.json({ ok: true }), + http.get('/api/share/tos', () => + HttpResponse.json({ version: 'v1', text: TOS_TEXT }), ), + http.post('/api/share/acknowledge', async ({ request }) => { + ackBody(await request.json()) + return HttpResponse.json({ ok: true }) + }), ) const user = userEvent.setup() @@ -64,12 +69,14 @@ describe('AcknowledgePage', () => { await user.click(screen.getByRole('button', { name: /i acknowledge/i })) await waitFor(() => expect(locationAssignSpy).toHaveBeenCalledWith('/dashboard')) + // The version POSTed must be the one /tos returned — not a sentinel. + expect(ackBody).toHaveBeenCalledWith({ version: 'v1' }) }) it('shows server error if acknowledge fails', async () => { server.use( - http.get('/api/share/heartbeat', () => - HttpResponse.json({ ok: true }), + http.get('/api/share/tos', () => + HttpResponse.json({ version: 'v1', text: TOS_TEXT }), ), http.post('/api/share/acknowledge', () => HttpResponse.json( diff --git a/frontend/__tests__/hooks/useUrlFilterSync.test.ts b/frontend/__tests__/hooks/useUrlFilterSync.test.ts index ff7e87aa..4f5aa0e0 100644 --- a/frontend/__tests__/hooks/useUrlFilterSync.test.ts +++ b/frontend/__tests__/hooks/useUrlFilterSync.test.ts @@ -9,6 +9,7 @@ const mockClearFilters = vi.fn() const mockSetRange = vi.fn() const mockSetMetric = vi.fn() const mockClientGet = vi.fn() +const mockGetQueryData = vi.fn() vi.mock('@/stores/filterStore', () => ({ useFilterStore: vi.fn(() => ({ @@ -18,6 +19,14 @@ vi.mock('@/stores/filterStore', () => ({ })), })) +// useUrlFilterSync calls useQueryClient() to read the bootstrap-seeded +// views cache as a fast path before falling back to client.GET. The hook +// no longer needs a real QueryClientProvider in tests — we just stub the +// hook to return a query client with the methods we exercise. +vi.mock('@tanstack/react-query', () => ({ + useQueryClient: vi.fn(() => ({ getQueryData: mockGetQueryData })), +})) + vi.mock('@/hooks/usePageContext', () => ({ usePageContext: vi.fn(() => ({ activeServiceId: 'test-service-id' })), })) diff --git a/frontend/__tests__/middleware.test.ts b/frontend/__tests__/middleware.test.ts index fb422e17..11ac12e2 100644 --- a/frontend/__tests__/middleware.test.ts +++ b/frontend/__tests__/middleware.test.ts @@ -13,7 +13,7 @@ */ import { describe, it, expect, vi } from 'vitest' -import { middleware } from '../middleware' +import { proxy as middleware } from '../proxy' function makeReq(url: string, headers: Record = {}): any { const u = new URL(url) diff --git a/frontend/__tests__/preload-manifest.test.ts b/frontend/__tests__/preload-manifest.test.ts new file mode 100644 index 00000000..9551c22a --- /dev/null +++ b/frontend/__tests__/preload-manifest.test.ts @@ -0,0 +1,120 @@ +/** + * O6 — Tests for the post-build preload-manifest scanner. + * + * The script lives at scripts/build-preload-manifest.mjs and is run + * by ``npm run build`` after ``next build``. It walks + * .next/static/chunks/*.js for the plotly-package markers and emits + * .next/static/preload-manifest.json. + * + * These tests spawn the script as a child process against a fixture + * directory we build per-test under os.tmpdir(). Spawning preserves + * the real CLI behavior — cwd resolution, exit codes, log messages. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { execFileSync } from 'node:child_process' +import { promises as fs } from 'node:fs' +import path from 'node:path' +import os from 'node:os' + +const SCRIPT = path.resolve(__dirname, '..', 'scripts', 'build-preload-manifest.mjs') + +let tmpRoot: string + +beforeEach(async () => { + tmpRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'preload-manifest-test-')) + await fs.mkdir(path.join(tmpRoot, '.next', 'static', 'chunks'), { recursive: true }) +}) + +afterEach(async () => { + await fs.rm(tmpRoot, { recursive: true, force: true }) +}) + +async function writeChunk(name: string, content: string) { + await fs.writeFile(path.join(tmpRoot, '.next', 'static', 'chunks', name), content, 'utf8') +} + +function runScript(): { stdout: string; stderr: string } { + // Spawn from the tmp dir so the script's path.resolve(process.cwd(), ...) + // anchors to our fixture instead of the real frontend dir. + const out = execFileSync('node', [SCRIPT], { cwd: tmpRoot, encoding: 'utf8', stdio: 'pipe' }) + return { stdout: out, stderr: '' } +} + +async function readManifest(): Promise { + const raw = await fs.readFile( + path.join(tmpRoot, '.next', 'static', 'preload-manifest.json'), + 'utf8', + ) + return JSON.parse(raw) +} + +describe('build-preload-manifest', () => { + it('matches a chunk that contains a plotly marker AND exceeds the size floor', async () => { + // Padding ensures the file is > MIN_BYTES (100 KB) without being absurd. + const padding = 'x'.repeat(200_000) + await writeChunk('big-with-plotly.js', `// plotly-logomark\n${padding}`) + runScript() + const m = await readManifest() + expect(m.preload).toHaveLength(1) + expect(m.preload[0].file).toBe('big-with-plotly.js') + expect(m.preload[0].bytes).toBeGreaterThan(100_000) + }) + + it('excludes a chunk that has the marker but is below the size floor', async () => { + // 5 KB — well below the 100 KB floor. Even though the marker is + // present, modulepreloading a chunk this small is net neutral. + await writeChunk('tiny-with-plotly.js', '// plotly-logomark\nconsole.log(1)') + runScript() + const m = await readManifest() + expect(m.preload).toHaveLength(0) + }) + + it('excludes a chunk that lacks the marker', async () => { + const padding = 'y'.repeat(200_000) + await writeChunk('big-no-marker.js', `// just bundle code\n${padding}`) + runScript() + const m = await readManifest() + expect(m.preload).toHaveLength(0) + }) + + it('matches either marker (logomark OR afterplot) — resilient to plotly tree-shaking one', async () => { + const padding = 'z'.repeat(200_000) + await writeChunk('big-afterplot.js', `// plotly_afterplot hook\n${padding}`) + runScript() + const m = await readManifest() + expect(m.preload).toHaveLength(1) + expect(m.preload[0].file).toBe('big-afterplot.js') + }) + + it('sorts matches by size descending so the biggest chunk preloads first', async () => { + const small = 'a'.repeat(150_000) // ~150 KB + const big = 'b'.repeat(600_000) // ~600 KB + await writeChunk('small.js', `// plotly-logomark\n${small}`) + await writeChunk('big.js', `// plotly-logomark\n${big}`) + runScript() + const m = await readManifest() + expect(m.preload).toHaveLength(2) + expect(m.preload[0].file).toBe('big.js') + expect(m.preload[1].file).toBe('small.js') + expect(m.preload[0].bytes).toBeGreaterThan(m.preload[1].bytes) + }) + + it('writes a valid empty manifest when no chunks match — never fails the build', async () => { + // Empty chunks dir; the script must still write a manifest with + // preload=[] so the runtime reader can parse it. + runScript() + const m = await readManifest() + expect(m.preload).toEqual([]) + expect(m.generatedAt).toMatch(/^\d{4}-\d{2}-\d{2}T/) + expect(m.markers).toContain('plotly-logomark') + }) + + it('skips silently when the chunks dir does not exist (dev build, etc.)', async () => { + await fs.rm(path.join(tmpRoot, '.next'), { recursive: true }) + // Must NOT throw and must NOT create a manifest file. + expect(() => runScript()).not.toThrow() + await expect( + fs.access(path.join(tmpRoot, '.next', 'static', 'preload-manifest.json')), + ).rejects.toThrow() + }) +}) diff --git a/frontend/app/admin/page.tsx b/frontend/app/admin/page.tsx index b81fa736..875f7997 100644 --- a/frontend/app/admin/page.tsx +++ b/frontend/app/admin/page.tsx @@ -838,16 +838,44 @@ export default function AdminPage() { prefetch={true} onMouseEnter={() => { if (!activeServiceId) return + // Warm the two composite queries the destination page fires + // on mount. Pre-fix this warmed ['scoring-status', ...], but + // the page actually reads scoring-status via the config + // composite — so the prefetch was overwritten before any + // panel could use it, and the page showed `compositesLoading` + // skeleton on click. Matching the composite keys + default + // since_hours=24 (the page's initial useState) means the + // composites are warm on mount → no skeleton flash, same + // pattern as the Share Dashboard link above. queryClient.prefetchQuery({ - queryKey: ['scoring-status', activeServiceId], + queryKey: ['scoring-analytics-composite', activeServiceId, 24], queryFn: async ({ signal }) => { - const { data } = await client.GET( - '/api/services/{service_id}/scoring/status', - { params: { path: { service_id: activeServiceId } } }, + const { data, response } = await client.GET( + '/api/services/{service_id}/scoring/analytics' as any, + { + params: { + path: { service_id: activeServiceId }, + query: { since_hours: 24 }, + }, + signal, + } as any, ) + if (!response.ok) throw new Error(`status ${response.status}`) + return data + }, + }) + queryClient.prefetchQuery({ + queryKey: ['scoring-config-composite', activeServiceId], + queryFn: async ({ signal }) => { + const { data, response } = await client.GET( + '/api/services/{service_id}/scoring/config' as any, + { + params: { path: { service_id: activeServiceId }, signal } as any, + } as any, + ) + if (!response.ok) throw new Error(`status ${response.status}`) return data }, - staleTime: 20_000, }) }} className={buttonVariants({ variant: 'secondary', size: 'sm' })} @@ -1318,7 +1346,8 @@ export default function AdminPage() { setNgwafFetching(true) try { const { data } = await client.GET("/api/provision/ngwaf-workspaces" as any, { - params: { query: { service_id: ngwafService.service_id, token: ngwafApiToken } } + params: { query: { service_id: ngwafService.service_id } }, + headers: { Authorization: `Bearer ${ngwafApiToken}` } }) setNgwafWorkspaces((data as any)?.workspaces || []) } catch (e: any) { @@ -1384,8 +1413,8 @@ export default function AdminPage() { await client.PATCH("/api/provision/services/{service_id}/ngwaf-workspace" as any, { params: { path: { service_id: ngwafService.service_id }, - query: { token: ngwafApiToken }, }, + headers: { Authorization: `Bearer ${ngwafApiToken}` }, body: { ngwaf_workspace_id: ngwafWorkspaceId.trim() || null } as any, }) setNgwafSaved(true) diff --git a/frontend/app/admin/session-scoring/page.tsx b/frontend/app/admin/session-scoring/page.tsx index 755487dd..a2cc0a87 100644 --- a/frontend/app/admin/session-scoring/page.tsx +++ b/frontend/app/admin/session-scoring/page.tsx @@ -3,9 +3,11 @@ import * as React from 'react' import dynamic from 'next/dynamic' import Link from 'next/link' -import { useQueryClient } from '@tanstack/react-query' +import { useQuery, useQueryClient } from '@tanstack/react-query' import { ArrowLeft, RefreshCw, ShieldCheck } from 'lucide-react' +import { client } from '@/lib/api' + import { Alert, AlertDescription } from '@/components/ui/alert' import { Button, buttonVariants } from '@/components/ui/button' import { PageHeader } from '@/components/ui/page-header' @@ -64,17 +66,86 @@ export default function SessionScoringPage() { const [sinceHours, setSinceHours] = React.useState(24) const qc = useQueryClient() - // Manual refresh replaces the per-component refetchInterval polling we - // removed after the 2026-06-01 mds_stores + VS Code RAM crash. Predicate - // invalidation matches every ['scoring-*', activeServiceId, ...] key, - // so new scoring queries (e.g. ['scoring-labels-counts', sid]) get - // refreshed without having to add a new invalidate line here. + // ── Composite queries: collapse 10+ individual requests into 2 ── + // Analytics composite: health, top-flagged, score-dist, compliance, + // evaluation, evaluation-per-reason. Config composite: status, + // threshold, exclude-regex, enforce-status-code. + // Individual component queries stay intact — they find pre-populated + // cache entries and skip their network requests. + const analyticsComposite = useQuery({ + queryKey: ['scoring-analytics-composite', activeServiceId, sinceHours], + queryFn: async ({ signal }) => { + const { data, response } = await client.GET( + '/api/services/{service_id}/scoring/analytics' as any, + { + params: { + path: { service_id: activeServiceId }, + query: { since_hours: sinceHours }, + }, + signal, + } as any, + ) + if (!response.ok) throw new Error(`status ${response.status}`) + return data as Record + }, + enabled: !!activeServiceId, + }) + + const configComposite = useQuery({ + queryKey: ['scoring-config-composite', activeServiceId], + queryFn: async ({ signal }) => { + const { data, response } = await client.GET( + '/api/services/{service_id}/scoring/config' as any, + { + params: { path: { service_id: activeServiceId }, signal } as any, + } as any, + ) + if (!response.ok) throw new Error(`status ${response.status}`) + return data as Record + }, + enabled: !!activeServiceId, + }) + + // Seed individual component cache keys from composite responses. + // Ref-guarded by dataUpdatedAt so seeding runs once per fresh fetch. + const analyticsSeededAt = React.useRef(0) + const configSeededAt = React.useRef(0) + + if (analyticsComposite.data && analyticsComposite.dataUpdatedAt > analyticsSeededAt.current) { + analyticsSeededAt.current = analyticsComposite.dataUpdatedAt + const d = analyticsComposite.data + if (d.health) qc.setQueryData(['scoring-health', activeServiceId, sinceHours], d.health) + if (d.top_flagged) qc.setQueryData(['scoring-top-flagged', activeServiceId, sinceHours], d.top_flagged) + if (d.score_distribution) qc.setQueryData(['scoring-score-dist', activeServiceId, sinceHours], d.score_distribution) + if (d.compliance_breakdown) qc.setQueryData(['scoring-compliance', activeServiceId, sinceHours], d.compliance_breakdown) + if (d.evaluation_per_reason) qc.setQueryData(['scoring-evaluation-per-reason', activeServiceId], d.evaluation_per_reason) + if (d.evaluation) qc.setQueryData(['scoring-evaluation', activeServiceId], d.evaluation) + } + + if (configComposite.data && configComposite.dataUpdatedAt > configSeededAt.current) { + configSeededAt.current = configComposite.dataUpdatedAt + const d = configComposite.data + if (d.status) qc.setQueryData(['scoring-status', activeServiceId], d.status) + if (d.threshold) qc.setQueryData(['scoring-threshold-committed', activeServiceId], d.threshold) + if (d.exclude_regex) qc.setQueryData(['scoring-exclude-regex', activeServiceId], d.exclude_regex) + if (d.enforce_status_code) qc.setQueryData(['scoring-enforce-status-code', activeServiceId], d.enforce_status_code) + } + + const compositesLoading = analyticsComposite.isLoading || configComposite.isLoading + + // Refresh invalidates composite keys (re-seeding individual caches on + // resolve) plus any queries not covered by composites. const refreshAll = () => { + analyticsSeededAt.current = 0 + configSeededAt.current = 0 + qc.invalidateQueries({ queryKey: ['scoring-analytics-composite', activeServiceId] }) + qc.invalidateQueries({ queryKey: ['scoring-config-composite', activeServiceId] }) qc.invalidateQueries({ predicate: (q) => Array.isArray(q.queryKey) && typeof q.queryKey[0] === 'string' && - (q.queryKey[0] as string).startsWith('scoring-') && + ['scoring-curves', 'scoring-enforce-threshold', 'scoring-threshold-preview', + 'scoring-labels', 'scoring-labels-counts'].includes(q.queryKey[0] as string) && q.queryKey[1] === activeServiceId, }) } @@ -121,7 +192,13 @@ export default function SessionScoringPage() { - + {compositesLoading ? ( +
+ +
+ ) : ( + + )} @@ -132,16 +209,26 @@ export default function SessionScoringPage() { - - - - - - -
- - -
+ {compositesLoading ? ( +
+ + + +
+ ) : ( + <> + + + + + + +
+ + +
+ + )} diff --git a/frontend/app/admin/share/page.tsx b/frontend/app/admin/share/page.tsx index 14e27911..56bdd336 100644 --- a/frontend/app/admin/share/page.tsx +++ b/frontend/app/admin/share/page.tsx @@ -42,9 +42,11 @@ export default function ShareDashboardPage() { return data as ShareStatus }, refetchInterval: 10_000, - // Treat as fresh for 5s so the hover-prefetch immediately preceding - // a click is reused, but live polling stays at 10s. - staleTime: 5_000, + // 30s staleTime so the hover-prefetch from the /admin PageHeader chip + // is reused on click even when the user lingers on hover. Live + // polling still ticks at 10s while the page is open; staleTime only + // affects the initial mount-time decision to refetch vs. use cache. + staleTime: 30_000, }) const refresh = React.useCallback(async () => { await refetch() diff --git a/frontend/app/admin/usage-log/page.tsx b/frontend/app/admin/usage-log/page.tsx index 1198e531..4ff6ebc7 100644 --- a/frontend/app/admin/usage-log/page.tsx +++ b/frontend/app/admin/usage-log/page.tsx @@ -265,8 +265,33 @@ export default function UsageLogPage() { const [now, setNow] = useState(() => new Date()) useEffect(() => { - const id = setInterval(() => setNow(new Date()), 30_000) - return () => clearInterval(id) + // Gate the 30s tick on tab visibility so a backgrounded admin tab + // doesn't keep rotating `now` and refetching ~MB of usage_log every + // minute. Re-tick immediately on visibility-restore so the rolled + // window matches the moment the user returns to the tab. + const tick = () => setNow(new Date()) + let id: ReturnType | null = null + const start = () => { + if (id !== null) return + tick() + id = setInterval(tick, 30_000) + } + const stop = () => { + if (id !== null) { + clearInterval(id) + id = null + } + } + const onVis = () => { + if (document.visibilityState === 'visible') start() + else stop() + } + if (document.visibilityState === 'visible') start() + document.addEventListener('visibilitychange', onVis) + return () => { + document.removeEventListener('visibilitychange', onVis) + stop() + } }, []) const startTime = useMemo(() => toQueryDate(new Date(now.getTime() - preset * 3600 * 1000)), [preset, now]) const endTime = useMemo(() => toQueryDate(now), [now]) @@ -513,7 +538,7 @@ export default function UsageLogPage() { /> void + intervalButtons: React.ReactNode + allCards: any[] + visibleCards: Set +} -export default function DashboardPage() { - const allCards = useDashboardCards() +function DashboardBody({ + startTime, + endTime, + timezone, + activeServiceId, + filterPayload, + config, + trend, + setTrend, + intervalButtons, + allCards, + visibleCards, +}: DashboardBodyProps) { const { data: catalog } = useLogFieldsCatalog() - + const { addFilter, setRange, @@ -184,12 +221,6 @@ export default function DashboardPage() { compareStartTime: state.compareStartTime, compareEndTime: state.compareEndTime, }))) - - const { visibleCards, toggleCard, showAll, reset: resetCards } = useCardVisibility( - 'dashboard_cards', - allCards.map((c: any) => c.id), - allCards.filter((c: any) => c.inActiveFormat).map((c: any) => c.id), - ) const [metric, setMetric] = React.useState("requests") const getFieldLabel = useFieldLabel() @@ -230,877 +261,924 @@ export default function DashboardPage() { }) }, []) - return ( - + // Clear hidden categories when metric changes to avoid confusing states + React.useEffect(() => { + setHiddenCategories(new Set()) + }, [metric]) + + const isReady = useIsDataReady() + + const { data: aggregates, isLoading: isLoadingAggs, isFetching: isFetchingAggs } = useServiceQuery( + ['dashboard', 'aggregates', activeServiceId, startTime, endTime, filterPayload, metric, config.effectiveInterval], + async ({ signal }) => { + const { data } = await client.POST("/api/dashboard/aggregates", { signal, + body: { + start_time: startTime!, + end_time: endTime!, + filters: filterPayload, + chart_metric: metric as any, + chart_interval: config.effectiveInterval + } + }) + return throwIfStaleAggregates(data) + }, + STALE_VIEW_RETRY_OPTIONS, + ) + + const { data: compareAggregates } = useQuery({ + queryKey: ['dashboard', 'aggregates', 'compare', activeServiceId, compareStartTime, compareEndTime, filterPayload, metric, config.effectiveInterval], + queryFn: async ({ signal }) => { + const { data } = await client.POST("/api/dashboard/aggregates", { signal, + body: { + start_time: compareStartTime!, + end_time: compareEndTime!, + filters: filterPayload, + chart_metric: metric as any, + chart_interval: config.effectiveInterval + } + }) + return throwIfStaleAggregates(data) + }, + enabled: isReady && compareMode && !!compareStartTime && !!compareEndTime, + ...STALE_VIEW_RETRY_OPTIONS, + }) + + const [sorting, setSorting] = React.useState([{ id: 'timestamp', desc: true }]) + + // User-selected raw-log columns. `timestamp` is forced into the list + // because the default sort references it; without it the API picks an + // arbitrary sort col and the table feels broken. + const [selectedRawColumns, setSelectedRawColumns] = React.useState(() => { + if (typeof window === 'undefined') return DEFAULT_RAW_COLUMNS + try { + const raw = localStorage.getItem(RAW_COLUMNS_STORAGE_KEY) + const parsed = raw ? JSON.parse(raw) : null + if (Array.isArray(parsed) && parsed.length > 0) { + return parsed.includes('timestamp') ? parsed : ['timestamp', ...parsed] } - > - {({ - startTime, - endTime, - timezone, - activeServiceId, - filterPayload, - config, - setChartInterval, - trend, - setTrend, - intervalButtons, - }) => { - // Clear hidden categories when metric changes to avoid confusing states - React.useEffect(() => { - setHiddenCategories(new Set()) - }, [metric]) - - const isReady = useIsDataReady() - - const { data: aggregates, isLoading: isLoadingAggs, isFetching: isFetchingAggs } = useServiceQuery( - ['dashboard', 'aggregates', activeServiceId, startTime, endTime, filterPayload, metric, config.effectiveInterval], - async ({ signal }) => { - const { data } = await client.POST("/api/dashboard/aggregates", { signal, - body: { - start_time: startTime!, - end_time: endTime!, - filters: filterPayload, - chart_metric: metric as any, - chart_interval: config.effectiveInterval - } - }) - return throwIfStaleAggregates(data) - }, - STALE_VIEW_RETRY_OPTIONS, - ) + } catch { /* fall through to default */ } + return DEFAULT_RAW_COLUMNS + }) - const { data: compareAggregates } = useQuery({ - queryKey: ['dashboard', 'aggregates', 'compare', activeServiceId, compareStartTime, compareEndTime, filterPayload, metric, config.effectiveInterval], - queryFn: async ({ signal }) => { - const { data } = await client.POST("/api/dashboard/aggregates", { signal, - body: { - start_time: compareStartTime!, - end_time: compareEndTime!, - filters: filterPayload, - chart_metric: metric as any, - chart_interval: config.effectiveInterval - } - }) - return throwIfStaleAggregates(data) - }, - enabled: isReady && compareMode && !!compareStartTime && !!compareEndTime, - ...STALE_VIEW_RETRY_OPTIONS, - }) + const toggleRawColumn = React.useCallback((id: string, visible: boolean) => { + setSelectedRawColumns(prev => { + const set = new Set(prev) + if (visible) set.add(id) + else if (id !== 'timestamp') set.delete(id) + const next = Array.from(set) + try { + localStorage.setItem(RAW_COLUMNS_STORAGE_KEY, JSON.stringify(next)) + } catch { /* ignore quota / private-mode errors */ } + return next + }) + }, []) - const [sorting, setSorting] = React.useState([{ id: 'timestamp', desc: true }]) - - // User-selected raw-log columns. `timestamp` is forced into the list - // because the default sort references it; without it the API picks an - // arbitrary sort col and the table feels broken. - const [selectedRawColumns, setSelectedRawColumns] = React.useState(() => { - if (typeof window === 'undefined') return DEFAULT_RAW_COLUMNS - try { - const raw = localStorage.getItem(RAW_COLUMNS_STORAGE_KEY) - const parsed = raw ? JSON.parse(raw) : null - if (Array.isArray(parsed) && parsed.length > 0) { - return parsed.includes('timestamp') ? parsed : ['timestamp', ...parsed] - } - } catch { /* fall through to default */ } - return DEFAULT_RAW_COLUMNS - }) + const { data: rawLogs, isLoading: isLoadingRaw, isFetching: isFetchingRaw } = useServiceQuery( + ['dashboard', 'raw', activeServiceId, startTime, endTime, filterPayload, sorting, selectedRawColumns], + async ({ signal }) => { + const sort = sorting[0] + const { data } = await client.POST("/api/dashboard/raw", { signal, + body: { + start_time: startTime!, + end_time: endTime!, + filters: filterPayload, + limit: 500, + page: 1, + sort_col: sort?.id, + sort_dir: sort?.desc ? 'desc' : 'asc', + columns: selectedRawColumns + } + }) + return data + } + ) - const toggleRawColumn = React.useCallback((id: string, visible: boolean) => { - setSelectedRawColumns(prev => { - const set = new Set(prev) - if (visible) set.add(id) - else if (id !== 'timestamp') set.delete(id) - const next = Array.from(set) - try { - localStorage.setItem(RAW_COLUMNS_STORAGE_KEY, JSON.stringify(next)) - } catch { /* ignore quota / private-mode errors */ } - return next - }) - }, []) - - const { data: rawLogs, isLoading: isLoadingRaw, isFetching: isFetchingRaw } = useServiceQuery( - ['dashboard', 'raw', activeServiceId, startTime, endTime, filterPayload, sorting, selectedRawColumns], - async ({ signal }) => { - const sort = sorting[0] - const { data } = await client.POST("/api/dashboard/raw", { signal, - body: { - start_time: startTime!, - end_time: endTime!, - filters: filterPayload, - limit: 500, - page: 1, - sort_col: sort?.id, - sort_dir: sort?.desc ? 'desc' : 'asc', - columns: selectedRawColumns - } - }) - return data - } - ) + const { data: topBotsData } = useQuery({ + queryKey: ['dashboard', 'top-bots', activeServiceId, startTime, endTime, filterPayload], + queryFn: async ({ signal }) => { + const { data } = await client.POST("/api/security/top-bots", { signal, + body: { + start_time: startTime!, + end_time: endTime!, + filters: filterPayload, + } + }) + return data + }, + enabled: isReady, + placeholderData: keepPreviousData, + }) - const { data: topBotsData } = useQuery({ - queryKey: ['dashboard', 'top-bots', activeServiceId, startTime, endTime, filterPayload], - queryFn: async ({ signal }) => { - const { data } = await client.POST("/api/security/top-bots", { signal, - body: { - start_time: startTime!, - end_time: endTime!, - filters: filterPayload, - } - }) - return data - }, - enabled: isReady, - placeholderData: keepPreviousData, - }) + // ── Chart data ──────────────────────────────────────────────────────────── - // ── Chart data ──────────────────────────────────────────────────────────── - - const trafficData = React.useMemo(() => { - const time_series = aggregates?.time_series - if (!time_series?.length) return [] - - const actualMetric = aggregates?.metric || metric - const isBar = actualMetric === 'requests' || actualMetric === '5xx' || actualMetric === '4xx' - - // Find metric metadata from catalog - const metricField = catalog?.fields?.find(f => f.id === actualMetric) - const unit = metricField?.unit || '' - const precision = metricField?.precision ?? (actualMetric === 'requests' ? 0 : 1) - - const getHoverTemplate = (m: string, label?: string) => { - const pre = label ? `${label}: ` : '' - const format = precision > 0 ? `.${precision}f` : ',' - return `${pre}%{y:${format}}${unit}` - } + const trafficData = React.useMemo(() => { + const time_series = aggregates?.time_series + if (!time_series?.length) return [] - // If we have categories (e.g. 5xx/4xx breakdown), group by category. - // Pydantic serializes optional fields as null, so null and undefined both mean "no category". - const hasCategories = time_series.some(d => d.category != null) - - let traces: any[] = [] - - if (hasCategories) { - const catMap: Record = {} - time_series.forEach(d => { - const cat = d.category || 'Other' - if (!catMap[cat]) catMap[cat] = { x: [], y: [] } - // Use a standard format that Plotly recognizes as a date but is in the target timezone - catMap[cat].x.push(formatDate(d.time, timezone, "yyyy-MM-dd HH:mm:ss")) - catMap[cat].y.push(d.value) - }) - - // Standardize colors for common error statuses to keep them consistent - const colorMap: Record = { - '400': '#fbbf24', '401': '#f59e0b', '403': '#d97706', '404': '#b45309', - '500': '#ef4444', '502': '#dc2626', '503': '#b91c1c', '504': '#991b1b' - } - - traces = Object.entries(catMap).map(([cat, data], i) => ({ - x: data.x, - y: data.y, - type: 'bar', - name: cat, - showlegend: false, // Custom legend will handle these - visible: hiddenCategories.has(cat) ? 'legendonly' : true, - hovertemplate: `Status ${cat}: %{y:,}`, - marker: { color: colorMap[cat] || `hsl(${(i * 50) % 360}, 70%, 50%)` } - })) - } else { - const xValues = time_series.map(d => formatDate(d.time, timezone, "yyyy-MM-dd HH:mm:ss")) - const yValues = time_series.map(d => d.value) - - traces = [{ - x: xValues, - y: yValues, - type: isBar ? 'bar' : 'scatter', - mode: isBar ? undefined : 'lines+markers', - name: compareMode ? 'Primary Range' : (metricField?.label || actualMetric), - showlegend: compareMode, - hovertemplate: getHoverTemplate(actualMetric, compareMode ? 'Primary' : undefined), - marker: { color: '#3b82f6' } - }] - } + const actualMetric = aggregates?.metric || metric + const isBar = actualMetric === 'requests' || actualMetric === '5xx' || actualMetric === '4xx' - if (compareMode && compareAggregates?.time_series?.length && !hasCategories && startTime && compareStartTime) { - const currentStart = new Date(startTime).getTime() - const compareStart = new Date(compareStartTime).getTime() - const shift = currentStart - compareStart - - const compX = compareAggregates.time_series.map(d => { - const t = new Date(d.time).getTime() + shift - return formatDate(new Date(t).toISOString(), timezone, "yyyy-MM-dd HH:mm:ss") - }) - const compY = compareAggregates.time_series.map(d => d.value) - - traces.push({ - x: compX, - y: compY, - type: 'scatter', - mode: 'lines', - name: 'Comparison Range', - line: { color: '#f97316', dash: 'dash', width: 2 }, - hovertemplate: getHoverTemplate(actualMetric, 'Comparison') - }) - } + // Find metric metadata from catalog + const metricField = catalog?.fields?.find(f => f.id === actualMetric) + const unit = metricField?.unit || '' + const precision = metricField?.precision ?? (actualMetric === 'requests' ? 0 : 1) - if (!hasCategories && time_series.some(d => d.baseline != null)) { - traces.push({ - x: time_series.map(d => formatDate(d.time, timezone, "yyyy-MM-dd HH:mm:ss")), - y: time_series.map(d => d.baseline), - type: 'scatter', mode: 'lines', - name: 'Baseline (7d prior)', - hovertemplate: getHoverTemplate(actualMetric, 'Baseline'), - line: { color: '#a1a1aa', dash: 'dot', width: 2 } - }) - } + const getHoverTemplate = (m: string, label?: string) => { + const pre = label ? `${label}: ` : '' + const format = precision > 0 ? `.${precision}f` : ',' + return `${pre}%{y:${format}}${unit}` + } - if (!hasCategories && trend !== 'off') { - const xValues = time_series.map(d => formatDate(d.time, timezone, "yyyy-MM-dd HH:mm:ss")) - const yValues = time_series.map(d => d.value) - const n = yValues.length - let windowSize = 0 - if (trend === 'auto') { - if (n > 1000) windowSize = Math.floor(n / 20) - else if (n > 100) windowSize = Math.floor(n / 10) - else windowSize = Math.floor(n / 5) - } else { - const trendMap: Record = { '1m': 60, '5m': 300, '1h': 3600, '1d': 86400 } - const actualInterval = aggregates?.interval || config.effectiveInterval - windowSize = Math.floor((trendMap[trend] ?? 0) / (INTERVAL_SECONDS[actualInterval as keyof typeof INTERVAL_SECONDS] ?? 60)) - } - if (windowSize > 1) { - const trendY = new Array(n).fill(null) - for (let i = windowSize - 1; i < n; i++) { - let sum = 0, count = 0 - for (let j = 0; j < windowSize; j++) { - const v = yValues[i - j] - if (v != null) { sum += v; count++ } - } - trendY[i] = count > 0 ? sum / count : null - } - traces.push({ - x: xValues, y: trendY, - type: 'scatter', mode: 'lines', - name: `${trend === 'auto' ? 'Auto ' : ''}Trend`, - hovertemplate: getHoverTemplate(actualMetric), - line: { color: '#f97316', width: 3 } - }) - } - } - return traces - }, [aggregates?.time_series, aggregates?.metric, aggregates?.interval, compareAggregates?.time_series, compareMode, compareStartTime, startTime, trend, timezone, metric, config.effectiveInterval, hiddenCategories, catalog]) - - const chartLayout = React.useMemo(() => { - const actualMetric = aggregates?.metric || metric - const metricField = catalog?.fields?.find(f => f.id === actualMetric) - - return { - ...TIME_HOVER_LAYOUT, - barmode: trafficData.length > 1 && trafficData[0]?.type === 'bar' ? 'stack' : undefined, - showlegend: trafficData.some(t => t.showlegend !== false), - yaxis: { - title: metricField?.unit || (actualMetric === 'requests' ? 'reqs' : ''), - ticksuffix: metricField?.unit || '', - separatethousands: true, - exponentformat: 'none' - }, - xaxis: makeTimeXAxis(startTime, endTime, timezone), - } - }, [trafficData, aggregates?.metric, metric, startTime, endTime, timezone, catalog]) - - const handleRowClick = React.useCallback((column: string, value: string | number) => { - React.startTransition(() => { - addFilter(column, String(value), 'include') - }) - }, [addFilter]) - - const handleChartRelayout = React.useCallback((event: any) => { - // Skip non-range events (autorange toggle, spike config, etc.) - if (event?.['xaxis.autorange'] === true || event?.['xaxis.showspikes'] !== undefined) return - - const x0 = event?.['xaxis.range[0]'] ?? event?.['xaxis.range']?.[0] - const x1 = event?.['xaxis.range[1]'] ?? event?.['xaxis.range']?.[1] - - if (x0 === undefined || x1 === undefined) return - - try { - const toLocalStr = (val: string | number) => { - if (typeof val === 'number') { - const d = new Date(val) - const pad = (n: number) => n.toString().padStart(2, '0') - return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}T${pad(d.getHours())}:${pad(d.getMinutes())}:${pad(d.getSeconds())}` - } - return val.replace(' ', 'T') - } - const parsedStart = parseFromInput(toLocalStr(x0), timezone) - const parsedEnd = parseFromInput(toLocalStr(x1), timezone) - if (parsedStart && parsedEnd) { - setRange(parsedStart, parsedEnd) - } - } catch (e) { - console.error("Failed to parse chart relayout event", e) - } - }, [setRange, timezone]) - - const handleCountryClick = React.useCallback((countryName: string) => { - React.startTransition(() => { - addFilter('country', countryName, 'include') - }) - }, [addFilter]) - - // ── Raw logs columns ─────────────────────────────────────────────────────── - - // Catalog-driven option list for the raw-logs column dropdown. Lets - // users toggle on heavy fields (ua, referer, ja4, etc.) that aren't in - // DEFAULT_RAW_COLUMNS — toggling refetches with the expanded set. - const rawColumnOptions = React.useMemo(() => { - const fields = (catalog?.fields as any[]) || [] - const seen = new Set() - const out: { id: string; label: string }[] = [] - for (const f of fields) { - if (!f?.id || RAW_DROPDOWN_EXCLUDE.has(f.id) || f.group === 'METRICS') continue - if (seen.has(f.id)) continue - seen.add(f.id) - out.push({ id: f.id, label: getFieldLabel(f.id) }) - } - // Defensive: ensure any currently-selected column not present in the - // catalog (e.g. custom field that bootstrap hasn't loaded yet) still - // shows up checked in the dropdown. - for (const id of selectedRawColumns) { - if (!seen.has(id)) { - seen.add(id) - out.push({ id, label: getFieldLabel(id) }) - } + // If we have categories (e.g. 5xx/4xx breakdown), group by category. + // Pydantic serializes optional fields as null, so null and undefined both mean "no category". + const hasCategories = time_series.some(d => d.category != null) + + let traces: any[] = [] + + if (hasCategories) { + const catMap: Record = {} + time_series.forEach(d => { + const cat = d.category || 'Other' + if (!catMap[cat]) catMap[cat] = { x: [], y: [] } + // Use a standard format that Plotly recognizes as a date but is in the target timezone + catMap[cat].x.push(formatDate(d.time, timezone, "yyyy-MM-dd HH:mm:ss")) + catMap[cat].y.push(d.value) + }) + + // Standardize colors for common error statuses to keep them consistent + const colorMap: Record = { + '400': '#fbbf24', '401': '#f59e0b', '403': '#d97706', '404': '#b45309', + '500': '#ef4444', '502': '#dc2626', '503': '#b91c1c', '504': '#991b1b' + } + + traces = Object.entries(catMap).map(([cat, data], i) => ({ + x: data.x, + y: data.y, + type: 'bar', + name: cat, + showlegend: false, // Custom legend will handle these + visible: hiddenCategories.has(cat) ? 'legendonly' : true, + hovertemplate: `Status ${cat}: %{y:,}`, + marker: { color: colorMap[cat] || `hsl(${(i * 50) % 360}, 70%, 50%)` } + })) + } else { + const xValues = time_series.map(d => formatDate(d.time, timezone, "yyyy-MM-dd HH:mm:ss")) + const yValues = time_series.map(d => d.value) + + traces = [{ + x: xValues, + y: yValues, + type: isBar ? 'bar' : 'scatter', + mode: isBar ? undefined : 'lines+markers', + name: compareMode ? 'Primary Range' : (metricField?.label || actualMetric), + showlegend: compareMode, + hovertemplate: getHoverTemplate(actualMetric, compareMode ? 'Primary' : undefined), + marker: { color: '#3b82f6' } + }] + } + + if (compareMode && compareAggregates?.time_series?.length && !hasCategories && startTime && compareStartTime) { + const currentStart = new Date(startTime).getTime() + const compareStart = new Date(compareStartTime).getTime() + const shift = currentStart - compareStart + + const compX = compareAggregates.time_series.map(d => { + const t = new Date(d.time).getTime() + shift + return formatDate(new Date(t).toISOString(), timezone, "yyyy-MM-dd HH:mm:ss") + }) + const compY = compareAggregates.time_series.map(d => d.value) + + traces.push({ + x: compX, + y: compY, + type: 'scatter', + mode: 'lines', + name: 'Comparison Range', + line: { color: '#f97316', dash: 'dash', width: 2 }, + hovertemplate: getHoverTemplate(actualMetric, 'Comparison') + }) + } + + if (!hasCategories && time_series.some(d => d.baseline != null)) { + traces.push({ + x: time_series.map(d => formatDate(d.time, timezone, "yyyy-MM-dd HH:mm:ss")), + y: time_series.map(d => d.baseline), + type: 'scatter', mode: 'lines', + name: 'Baseline (7d prior)', + hovertemplate: getHoverTemplate(actualMetric, 'Baseline'), + line: { color: '#a1a1aa', dash: 'dot', width: 2 } + }) + } + + if (!hasCategories && trend !== 'off') { + const xValues = time_series.map(d => formatDate(d.time, timezone, "yyyy-MM-dd HH:mm:ss")) + const yValues = time_series.map(d => d.value) + const n = yValues.length + let windowSize = 0 + if (trend === 'auto') { + if (n > 1000) windowSize = Math.floor(n / 20) + else if (n > 100) windowSize = Math.floor(n / 10) + else windowSize = Math.floor(n / 5) + } else { + const trendMap: Record = { '1m': 60, '5m': 300, '1h': 3600, '1d': 86400 } + const actualInterval = aggregates?.interval || config.effectiveInterval + windowSize = Math.floor((trendMap[trend] ?? 0) / (INTERVAL_SECONDS[actualInterval as keyof typeof INTERVAL_SECONDS] ?? 60)) + } + if (windowSize > 1) { + const trendY = new Array(n).fill(null) + for (let i = windowSize - 1; i < n; i++) { + let sum = 0, count = 0 + for (let j = 0; j < windowSize; j++) { + const v = yValues[i - j] + if (v != null) { sum += v; count++ } } - return out - }, [catalog, getFieldLabel, selectedRawColumns]) - - const rawColumnVisibility = React.useMemo(() => { - const v: Record = {} - for (const opt of rawColumnOptions) v[opt.id] = selectedRawColumns.includes(opt.id) - return v - }, [rawColumnOptions, selectedRawColumns]) - - // hasSidCol still drives the FLAG-COLUMN render below — it can't - // be determined until rawLogs returns. labelsQuery, however, fires - // immediately on serviceId (see comment on labelsQuery below). - const hasSidCol = !!rawLogs?.columns?.includes('edge_sid') - - // Pull session-labels for the active service so we can render a - // colored Flag icon per row reflecting the current label state. - // Fire as soon as a serviceId is known — previously this was gated - // on `hasSidCol`, which created a real request waterfall: rawLogs - // took ~1s on prod, and this 10ms query couldn't start until then, - // blocking DataTable's first paint by the full rawLogs round-trip. - // The result is harmless when the service has no edge_sid column - // (the FLAG column simply doesn't render and the data goes unused). - const labelsQuery = useQuery({ - queryKey: ['scoring-labels', activeServiceId], - enabled: !!activeServiceId, - queryFn: async ({ signal }) => { - const { data, response } = await client.GET( - '/api/services/{service_id}/scoring/labels' as any, - { params: { path: { service_id: activeServiceId || '' } } } as any, - ) - if (!response.ok) throw new Error(`status ${response.status}`) - return data as { labels: Array<{ sid: string; label: LabelValue }> } - }, + trendY[i] = count > 0 ? sum / count : null + } + traces.push({ + x: xValues, y: trendY, + type: 'scatter', mode: 'lines', + name: `${trend === 'auto' ? 'Auto ' : ''}Trend`, + hovertemplate: getHoverTemplate(actualMetric), + line: { color: '#f97316', width: 3 } }) - const labelBySid = React.useMemo(() => { - const m = new Map() - for (const l of labelsQuery.data?.labels ?? []) m.set(l.sid, l.label) - return m - }, [labelsQuery.data]) - - const columns: ColumnDef[] = React.useMemo(() => { - if (!rawLogs?.columns) return [] - const dataCols: ColumnDef[] = rawLogs.columns.map((col: string): ColumnDef => ({ - id: col, - accessorFn: (row) => row[col], - meta: { label: getFieldLabel(col) }, - header: getFieldLabel(col), - cell: ({ row }: { row: any }) => { - const value = row.original[col] - if (col === 'timestamp') return ( - - {full(value as string)} {abbr()} - - ) - if (col === 'status') { - const status = Number(value) - const variant = status >= 500 ? 'destructive' : 'outline' - return ( - React.startTransition(() => addFilter(col, String(status), 'include'))} - onExclude={() => React.startTransition(() => addFilter(col, String(status), 'exclude'))} - triggerClassName={badgeVariants({ variant: variant as any, className: 'cursor-pointer' })} - triggerLabel={{status}} - header={

{col}: {status}

} - contentClassName="w-44 p-2" - /> - ) - } - const strVal = String(value ?? '') - if (strVal === '') { - return — - } - return ( - React.startTransition(() => addFilter(col, strVal, 'include'))} - onExclude={() => React.startTransition(() => addFilter(col, strVal, 'exclude'))} - triggerClassName="text-xs font-mono cursor-pointer hover:text-primary underline-offset-2 hover:underline" - triggerLabel={{strVal}} - /> - ) - } - })) - // Flag column: only shown when edge_sid is present in the schema - // (i.e. session scoring is enabled). Disabled for rows where the - // sid is empty (cookieless requests — already caught by L1). - if (hasSidCol && activeServiceId) { - dataCols.push({ - id: '__flag', - accessorFn: (_row: any) => '', - meta: { label: 'Flag' }, - header: 'Flag', - cell: ({ row }: { row: any }) => { - const sid = String(row.original['edge_sid'] ?? '') - return ( - - ) - }, - } as ColumnDef) - } - return dataCols - }, [rawLogs?.columns, full, abbr, addFilter, getFieldLabel, hasSidCol, activeServiceId, labelBySid]) + } + } + return traces + }, [aggregates?.time_series, aggregates?.metric, aggregates?.interval, compareAggregates?.time_series, compareMode, compareStartTime, startTime, trend, timezone, metric, config.effectiveInterval, hiddenCategories, catalog]) + + const chartLayout = React.useMemo(() => { + const actualMetric = aggregates?.metric || metric + const metricField = catalog?.fields?.find(f => f.id === actualMetric) + + return { + ...TIME_HOVER_LAYOUT, + barmode: trafficData.length > 1 && trafficData[0]?.type === 'bar' ? 'stack' : undefined, + showlegend: trafficData.some(t => t.showlegend !== false), + yaxis: { + title: metricField?.unit || (actualMetric === 'requests' ? 'reqs' : ''), + ticksuffix: metricField?.unit || '', + separatethousands: true, + exponentformat: 'none' + }, + xaxis: makeTimeXAxis(startTime, endTime, timezone), + } + }, [trafficData, aggregates?.metric, metric, startTime, endTime, timezone, catalog]) - const visibleCardList = React.useMemo( - () => allCards.filter((c: any) => visibleCards.has(c.id)), - [allCards, visibleCards] - ) + const handleRowClick = React.useCallback((column: string, value: string | number) => { + React.startTransition(() => { + addFilter(column, String(value), 'include') + }) + }, [addFilter]) + + const handleChartRelayout = React.useCallback((event: any) => { + // Skip non-range events (autorange toggle, spike config, etc.) + if (event?.['xaxis.autorange'] === true || event?.['xaxis.showspikes'] !== undefined) return + const x0 = event?.['xaxis.range[0]'] ?? event?.['xaxis.range']?.[0] + const x1 = event?.['xaxis.range[1]'] ?? event?.['xaxis.range']?.[1] + + if (x0 === undefined || x1 === undefined) return + + try { + const toLocalStr = (val: string | number) => { + if (typeof val === 'number') { + const d = new Date(val) + const pad = (n: number) => n.toString().padStart(2, '0') + return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}T${pad(d.getHours())}:${pad(d.getMinutes())}:${pad(d.getSeconds())}` + } + return val.replace(' ', 'T') + } + const parsedStart = parseFromInput(toLocalStr(x0), timezone) + const parsedEnd = parseFromInput(toLocalStr(x1), timezone) + if (parsedStart && parsedEnd) { + setRange(parsedStart, parsedEnd) + } + } catch (e) { + console.error("Failed to parse chart relayout event", e) + } + }, [setRange, timezone]) + + const handleCountryClick = React.useCallback((countryName: string) => { + React.startTransition(() => { + addFilter('country', countryName, 'include') + }) + }, [addFilter]) + + // ── Raw logs columns ─────────────────────────────────────────────────────── + + // Catalog-driven option list for the raw-logs column dropdown. Lets + // users toggle on heavy fields (ua, referer, ja4, etc.) that aren't in + // DEFAULT_RAW_COLUMNS — toggling refetches with the expanded set. + const rawColumnOptions = React.useMemo(() => { + const fields = (catalog?.fields as any[]) || [] + const seen = new Set() + const out: { id: string; label: string }[] = [] + for (const f of fields) { + if (!f?.id || RAW_DROPDOWN_EXCLUDE.has(f.id) || f.group === 'METRICS') continue + if (seen.has(f.id)) continue + seen.add(f.id) + out.push({ id: f.id, label: getFieldLabel(f.id) }) + } + // Defensive: ensure any currently-selected column not present in the + // catalog (e.g. custom field that bootstrap hasn't loaded yet) still + // shows up checked in the dropdown. + for (const id of selectedRawColumns) { + if (!seen.has(id)) { + seen.add(id) + out.push({ id, label: getFieldLabel(id) }) + } + } + return out + }, [catalog, getFieldLabel, selectedRawColumns]) + + const rawColumnVisibility = React.useMemo(() => { + const v: Record = {} + for (const opt of rawColumnOptions) v[opt.id] = selectedRawColumns.includes(opt.id) + return v + }, [rawColumnOptions, selectedRawColumns]) + + // hasSidCol still drives the FLAG-COLUMN render below — it can't + // be determined until rawLogs returns. labelsQuery, however, fires + // immediately on serviceId (see comment on labelsQuery below). + const hasSidCol = !!rawLogs?.columns?.includes('edge_sid') + + // Pull session-labels for the active service via the shared + // useScoringLabels hook so the same fetch dedupes with the admin + // Labels tab + TopFlaggedTable's "currently labeled" badges + // under the same React Query cache key. The hook already returns + // the {sid → label} Map so we don't re-derive per render here. + const { labelBySid } = useScoringLabels(activeServiceId || '', { + enabled: !!activeServiceId, + }) + + const columns: ColumnDef[] = React.useMemo(() => { + if (!rawLogs?.columns) return [] + const dataCols: ColumnDef[] = rawLogs.columns.map((col: string): ColumnDef => ({ + id: col, + accessorFn: (row) => row[col], + meta: { label: getFieldLabel(col) }, + header: getFieldLabel(col), + cell: ({ row }: { row: any }) => { + const value = row.original[col] + if (col === 'timestamp') return ( + + {full(value as string)} {abbr()} + + ) + if (col === 'status') { + const status = Number(value) + const variant = status >= 500 ? 'destructive' : 'outline' + return ( + React.startTransition(() => addFilter(col, String(status), 'include'))} + onExclude={() => React.startTransition(() => addFilter(col, String(status), 'exclude'))} + triggerClassName={badgeVariants({ variant: variant as any, className: 'cursor-pointer' })} + triggerLabel={{status}} + header={

{col}: {status}

} + contentClassName="w-44 p-2" + /> + ) + } + const strVal = String(value ?? '') + if (strVal === '') { + return — + } return ( - <> - {/* ── Main charts ── */} -
-
-
-
-

Traffic over Time

-
- - {(() => { - const metricsFields = catalog?.fields?.filter(f => f.group === 'METRICS') || [] - const shortLabels: Record = { - 'requests': 'Reqs', - 'hit_rate': 'CHR', - '5xx': '5xx', - '4xx': '4xx', - 'p50_latency': 'p50', - 'p95_latency': 'p95', - 'p99_latency': 'p99', - 'throughput': 'Throughput', - 'req_size': 'Req Size', - 'ttfb': 'TTFB' - } - - // We want to group latencies into a dropdown - const latencyIds = ['p50_latency', 'p95_latency', 'p99_latency'] - const otherMetrics = metricsFields.filter(f => !latencyIds.includes(f.id)) - - // Re-order to match desired UI layout: Reqs, 5xx, 4xx, CHR, Latency, ... - const order = ['requests', '5xx', '4xx', 'hit_rate'] - const orderedMetrics = [ - ...order.map(id => otherMetrics.find(f => f.id === id)).filter(Boolean), - ...otherMetrics.filter(f => !order.includes(f.id)) - ] as any[] - - const elements = orderedMetrics.map(m => ( - - )) - - // Insert Latency dropdown after CHR (hit_rate) - const isLatency = metric.endsWith('_latency') - const latLabel = isLatency ? metric.split('_')[0] : 'p95' - const latencyDropdown = ( - - - Latency ({latLabel}) - - - setMetric('p50_latency')} className="text-xs">p50 Latency - setMetric('p95_latency')} className="text-xs">p95 Latency - setMetric('p99_latency')} className="text-xs">p99 Latency - - - ) - - const chrIndex = orderedMetrics.findIndex(m => m.id === 'hit_rate') - if (chrIndex !== -1) { - elements.splice(chrIndex + 1, 0, latencyDropdown) - } else { - elements.push(latencyDropdown) - } - - return elements - })()} - - - {intervalButtons} -
-
-
- {isFetchingAggs && !isLoadingAggs && ( -
- - Updating -
- )} -
-
+ React.startTransition(() => addFilter(col, strVal, 'include'))} + onExclude={() => React.startTransition(() => addFilter(col, strVal, 'exclude'))} + triggerClassName="text-xs font-mono cursor-pointer hover:text-primary underline-offset-2 hover:underline" + triggerLabel={{strVal}} + /> + ) + } + })) + // Flag column: only shown when edge_sid is present in the schema + // (i.e. session scoring is enabled). Disabled for rows where the + // sid is empty (cookieless requests — already caught by L1). + if (hasSidCol && activeServiceId) { + dataCols.push({ + id: '__flag', + accessorFn: (_row: any) => '', + meta: { label: 'Flag' }, + header: 'Flag', + cell: ({ row }: { row: any }) => { + const sid = String(row.original['edge_sid'] ?? '') + return ( + + ) + }, + } as ColumnDef) + } + return dataCols + }, [rawLogs?.columns, full, abbr, addFilter, getFieldLabel, hasSidCol, activeServiceId, labelBySid]) - {/* Custom Category Legend */} - {trafficData.length > 1 && trafficData[0]?.type === 'bar' && ( -
- - {trafficData.filter(t => t.type === 'bar').map(trace => { - const isHidden = hiddenCategories.has(trace.name) - return ( - - ) - })} - -
- )} - -
- {(!isReady || (isLoadingAggs && !aggregates)) || (isFetchingAggs && trafficData.length === 0) ? ( -
- - {!isReady ? 'Initializing...' : 'Crunching logs...'} - -
- ) : trafficData.length === 0 ? ( -
-
- No data available - - {(() => { - if (metric === 'ttfb_client') { - return "Requires Infrastructure (Group C) fields to be enabled in Fastly logging." - } - if (metric === 'req_size') { - return "Requires Request Identity (Group A) fields to be enabled in Fastly logging." - } - return "No logs found for this period." - })()} - -
-
- ) : ( -
- -
- )} -
+ const visibleCardList = React.useMemo( + () => allCards.filter((c: any) => visibleCards.has(c.id)), + [allCards, visibleCards] + ) -
- Trend: - - {TRENDS.map(t => ( + return ( + <> + {/* ── Main charts ── */} +
+
+
+
+

Traffic over Time

+
+ + {(() => { + const metricsFields = catalog?.fields?.filter(f => f.group === 'METRICS') || [] + const shortLabels: Record = { + 'requests': 'Reqs', + 'hit_rate': 'CHR', + '5xx': '5xx', + '4xx': '4xx', + 'p50_latency': 'p50', + 'p95_latency': 'p95', + 'p99_latency': 'p99', + 'throughput': 'Throughput', + 'req_size': 'Req Size', + 'ttfb': 'TTFB' + } + + // We want to group latencies into a dropdown + const latencyIds = ['p50_latency', 'p95_latency', 'p99_latency'] + const otherMetrics = metricsFields.filter(f => !latencyIds.includes(f.id)) + + // Re-order to match desired UI layout: Reqs, 5xx, 4xx, CHR, Latency, ... + const order = ['requests', '5xx', '4xx', 'hit_rate'] + const orderedMetrics = [ + ...order.map(id => otherMetrics.find(f => f.id === id)).filter(Boolean), + ...otherMetrics.filter(f => !order.includes(f.id)) + ] as any[] + + const elements = orderedMetrics.map(m => ( - ))} - -
-
+ )) + + // Insert Latency dropdown after CHR (hit_rate) + const isLatency = metric.endsWith('_latency') + const latLabel = isLatency ? metric.split('_')[0] : 'p95' + const latencyDropdown = ( + + + Latency ({latLabel}) + + + setMetric('p50_latency')} className="text-xs">p50 Latency + setMetric('p95_latency')} className="text-xs">p95 Latency + setMetric('p99_latency')} className="text-xs">p99 Latency + + + ) -
-

Requests by Country

- {(!isReady || (isLoadingAggs && !aggregates)) || (isFetchingAggs && (!aggregates?.map_data || aggregates.map_data.length === 0)) ? ( -
- - {!isReady ? 'Initializing...' : 'Mapping traffic...'} - -
- ) : !aggregates?.map_data || aggregates.map_data.length === 0 ? ( -
-
- No data available - - {(() => { - const countryField = (catalog?.fields as any[])?.find(f => f.id === 'country') - const groupId = countryField?.group - if (groupId) { - const groupMeta = (catalog?.groups as any[])?.find(g => g.id === groupId) - if (groupMeta) { - return `Requires ${groupMeta.label} fields to be enabled in Fastly logging.` - } - } - return "Requires Geolocation fields to be enabled in Fastly logging." - })()} - -
-
- ) : ( - - )} + const chrIndex = orderedMetrics.findIndex(m => m.id === 'hit_rate') + if (chrIndex !== -1) { + elements.splice(chrIndex + 1, 0, latencyDropdown) + } else { + elements.push(latencyDropdown) + } + + return elements + })()} + + + {intervalButtons}
- - {/* ── Aggregation cards ── */} - {visibleCardList.length > 0 && (() => { - const visibleById = new Map(visibleCardList.map((c: any) => [c.id, c])) - // Wrap each card in LazyMount so the FIRST dashboard paint - // only mounts the cards above the fold (~5-10) instead of - // all 86. Off-screen cards land as the user scrolls — the - // rootMargin of 600px (one screen) pre-mounts before the - // user actually reaches them, so they feel instant. Cuts - // initial DOM nodes from ~860 to ~100 and skips ~80 - // TopTenTable mount cycles on first render. The loading - // placeholder branch is NOT wrapped — it's already cheap - // and we want every "Initializing..." tile visible. - const renderCard = (card: any) => { - if (!isReady || (isLoadingAggs && !aggregates)) { - return ( -
- - {!isReady ? 'Initializing...' : 'Loading...'} - -
- ) - } - if (card.id === '_bot_name') { - return ( - - } - field="_bot_name" - inActiveFormat={card.inActiveFormat} - data={{ - total: topBotsData?.bots?.reduce((acc: number, b: any) => acc + b.request_count, 0) || 0, - top: (topBotsData?.bots ?? []).map((b: any) => ({ value: b.id, label: b.name, count: b.request_count })) - }} - compareData={undefined} - onRowClick={handleRowClick} - /> - - ) - } - if (card.id === '_ngwaf_bot_name') { +
+ {isFetchingAggs && !isLoadingAggs && ( +
+ + Updating +
+ )} +
+
+ + {/* Custom Category Legend */} + {trafficData.length > 1 && trafficData[0]?.type === 'bar' && ( +
+ + {trafficData.filter(t => t.type === 'bar').map(trace => { + const isHidden = hiddenCategories.has(trace.name) return ( - - acc + b.request_count, 0), - top: (topBotsData?.ngwaf_bots ?? []).map((b: any) => ({ value: b.name, label: b.name, count: b.request_count })) - }} - compareData={undefined} - onRowClick={handleRowClick} - /> - + ) - } - return ( - - - - ) - } - - const sections = CARD_CATEGORIES.map(cat => ({ - ...cat, - cards: cat.cardIds.map(id => visibleById.get(id)).filter(Boolean), - })).filter(s => s.cards.length > 0) - - const customCards = visibleCardList.filter((c: any) => !CATEGORIZED_CARD_IDS.has(c.id)) - if (customCards.length > 0) { - sections.push({ id: 'custom', label: 'Custom', cardIds: [], cards: customCards, tint: CUSTOM_TINT }) - } + })} + +
+ )} - return ( -
- {sections.map(section => { - const isCollapsed = collapsedSections.has(section.id) - const Chevron = isCollapsed ? ChevronRight : ChevronDown - return ( -
- - {!isCollapsed && ( -
- {section.cards.map((card: any) => renderCard(card))} -
- )} -
- ) - })} -
- ) - })()} - - {/* ── Raw logs table ── */} - - - - + if (metric === 'req_size') { + return "Requires Request Identity (Group A) fields to be enabled in Fastly logging." + } + return "No logs found for this period." + })()} +
- } +
+ ) : ( +
+ +
+ )} +
+ +
+ Trend: + + {TRENDS.map(t => ( + + ))} + +
+
+ +
+

Requests by Country

+ {(!isReady || (isLoadingAggs && !aggregates)) || (isFetchingAggs && (!aggregates?.map_data || aggregates.map_data.length === 0)) ? ( +
+ + {!isReady ? 'Initializing...' : 'Mapping traffic...'} + +
+ ) : !aggregates?.map_data || aggregates.map_data.length === 0 ? ( +
+
+ No data available + + {(() => { + const countryField = (catalog?.fields as any[])?.find(f => f.id === 'country') + const groupId = countryField?.group + if (groupId) { + const groupMeta = (catalog?.groups as any[])?.find(g => g.id === groupId) + if (groupMeta) { + return `Requires ${groupMeta.label} fields to be enabled in Fastly logging.` + } + } + return "Requires Geolocation fields to be enabled in Fastly logging." + })()} + +
+
+ ) : ( + + )} +
+ + + {/* ── Aggregation cards ── */} + {/* When the catalog query hasn't returned yet ``visibleCardList`` is + * empty (it's ``allCards.filter(c => visibleCards.has(c.id))`` and + * allCards is [] until catalog loads). Render the section structure + * from CARD_CATEGORIES — a STATIC const — so the cards section + * always occupies its eventual vertical space. Without this, the + * section is completely absent during the catalog-loading gap and + * the raw-logs table (which loads ~500 ms faster) renders at the + * top and then gets shoved DOWN by ~3000-4000 px when the real + * cards arrive. That's the "page jumps" UX bug the user + * reported 2026-06-06. + * + * The skeleton renders ALL categories at their full default card + * count. When real data arrives, hidden categories collapse (a + * small downward adjustment) but the gross layout is already + * reserved. Most users haven't hidden any categories so the + * swap is invisible. */} + {visibleCardList.length === 0 && ( +
+ {CARD_CATEGORIES.map((cat) => ( +
- + + +

+ {cat.label} +

+ + {cat.cardIds.length} + +
+
+ {cat.cardIds.map((id) => ( +
+ + {!isReady ? 'Initializing...' : 'Loading...'} + +
+ ))} +
+ + ))} + + )} + {visibleCardList.length > 0 && (() => { + const visibleById = new Map(visibleCardList.map((c: any) => [c.id, c])) + // Wrap each card in LazyMount so the FIRST dashboard paint + // only mounts the cards above the fold (~5-10) instead of + // all 86. Off-screen cards land as the user scrolls — the + // rootMargin of 600px (one screen) pre-mounts before the + // user actually reaches them, so they feel instant. Cuts + // initial DOM nodes from ~860 to ~100 and skips ~80 + // TopTenTable mount cycles on first render. The loading + // placeholder branch is NOT wrapped — it's already cheap + // and we want every "Initializing..." tile visible. + const renderCard = (card: any) => { + if (!isReady || (isLoadingAggs && !aggregates)) { + return ( +
+ + {!isReady ? 'Initializing...' : 'Loading...'} + +
+ ) + } + if (card.id === '_bot_name') { + return ( + + } + field="_bot_name" + inActiveFormat={card.inActiveFormat} + data={{ + total: topBotsData?.bots?.reduce((acc: number, b: any) => acc + b.request_count, 0) || 0, + top: (topBotsData?.bots ?? []).map((b: any) => ({ value: b.id, label: b.name, count: b.request_count })) + }} + compareData={undefined} + onRowClick={handleRowClick} + /> + + ) + } + if (card.id === '_ngwaf_bot_name') { + return ( + + acc + b.request_count, 0), + top: (topBotsData?.ngwaf_bots ?? []).map((b: any) => ({ value: b.name, label: b.name, count: b.request_count })) + }} + compareData={undefined} + onRowClick={handleRowClick} + /> + + ) + } + return ( + + - - + + ) + } + + const sections = CARD_CATEGORIES.map(cat => ({ + ...cat, + cards: cat.cardIds.map(id => visibleById.get(id)).filter(Boolean), + })).filter(s => s.cards.length > 0) + + const customCards = visibleCardList.filter((c: any) => !CATEGORIZED_CARD_IDS.has(c.id)) + if (customCards.length > 0) { + sections.push({ id: 'custom', label: 'Custom', cardIds: [], cards: customCards, tint: CUSTOM_TINT }) + } + + return ( +
+ {sections.map(section => { + const isCollapsed = collapsedSections.has(section.id) + const Chevron = isCollapsed ? ChevronRight : ChevronDown + return ( +
+ + {!isCollapsed && ( +
+ {section.cards.map((card: any) => renderCard(card))} +
+ )} +
+ ) + })} +
) - }} + })()} + + {/* ── Raw logs table ── */} + + + + + + } + > + + + + ) +} + +// ── Page ─────────────────────────────────────────────────────────────────────── + +export default function DashboardPage() { + const allCards = useDashboardCards() + + const { visibleCards, toggleCard, showAll, reset: resetCards } = useCardVisibility( + 'dashboard_cards', + allCards.map((c: any) => c.id), + allCards.filter((c: any) => c.inActiveFormat).map((c: any) => c.id), + ) + + return ( + + } + > + {(ctx) => ( + + )} ) } diff --git a/frontend/app/insights/page.tsx b/frontend/app/insights/page.tsx index 7d636e5a..c71e35fb 100644 --- a/frontend/app/insights/page.tsx +++ b/frontend/app/insights/page.tsx @@ -5,6 +5,7 @@ import { useQuery } from '@tanstack/react-query' import { client } from '@/lib/api' import { useServiceStore } from '@/stores/serviceStore' import { InsightCard } from '@/components/Insights/InsightCard' +import { InsightCardSkeleton } from '@/components/Insights/InsightCardSkeleton' import { InsightCardData } from '@/types/api' import { Select, @@ -13,9 +14,8 @@ import { SelectTrigger, SelectValue } from "@/components/ui/select" -import { SkeletonGrid } from '@/components/ui/skeleton-grid' import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert" -import { Info, AlertCircle, CheckCircle, Lightbulb, Filter } from 'lucide-react' +import { Info, AlertCircle, CheckCircle, Lightbulb, Filter, Loader2 } from 'lucide-react' import { useDateFormat } from '@/hooks/useDateFormat' import { Tooltip, @@ -47,6 +47,154 @@ const STATUS_OPTIONS = [ { label: 'Clean', value: 'clean' }, ] +// Lifted out of the ReportLayout render-prop so the hooks live at the +// top of a stable component instead of being recreated every time +// ReportLayout re-renders. Same shape as DashboardBody (item 30). +// Without this lift, React Query treats every ReportLayout re-render +// as a fresh mount and re-fires the /api/insights + /api/insight- +// availability requests — the local-dev duplicate-fetch pattern. +interface InsightsBodyProps { + activeServiceId: string | null | undefined + windowHours: string + baselineHours: string + statusFilter: string + relative: (iso: string) => string + full: (iso: string) => string + abbr: () => string +} + +function InsightsBody({ + activeServiceId, + windowHours, + baselineHours, + statusFilter, + relative, + full, + abbr, +}: InsightsBodyProps) { + const { data, isLoading, error } = useQuery({ + queryKey: ['insights', activeServiceId, windowHours, baselineHours], + queryFn: async ({ signal }) => { + const { data } = await client.POST("/api/insights", { signal, + body: { + window_size_hrs: parseFloat(windowHours), + baseline_hours: parseFloat(baselineHours), + filters: {}, + } + }) + return data + }, + enabled: !!activeServiceId, + staleTime: 60000 + }) + + const { data: availability } = useQuery({ + queryKey: ['insights', 'availability', activeServiceId], + queryFn: async ({ signal }) => { + const { data } = await client.GET("/api/insight-availability", { signal }) + return data + }, + enabled: !!activeServiceId, + // The active-insights list is derived from the service's column schema + // and effectively never changes within a session. Long-cache it so a + // warm navigation paints the per-insight skeleton cards instantly + // instead of flashing an empty state for one round trip. + staleTime: 5 * 60 * 1000, + }) + + const filteredInsights = useMemo(() => { + if (!data?.insights) return [] + if (statusFilter === 'all') return data.insights + return (data.insights as InsightCardData[]).filter((insight: InsightCardData) => insight.severity === statusFilter) + }, [data?.insights, statusFilter]) + + // Skeleton cards rendered while /api/insights is in flight come from the + // /api/insight-availability response (titles + descriptions per available + // insight). Single render path during loading — no SkeletonGrid → per- + // insight swap; the only transition is content-fill when real data lands. + const availableInsights = useMemo(() => { + const list = (availability as any)?.insights as + | Array<{ id: string; title: string; description?: string; available?: boolean }> + | undefined + if (!list) return [] + return list.filter((i) => i.available !== false) + }, [availability]) + + return ( + <> + {(availability as any)?.unavailable && (availability as any).unavailable.length > 0 && ( + + + Some insights are unavailable + + {(availability as any).unavailable.length} insights require additional log fields to be enabled. + Check your service configuration. + + + )} + + {isLoading ? ( + availableInsights.length > 0 ? ( +
+ {availableInsights.map((i) => ( + + ))} +
+ ) : ( +
+ + Loading insights… +
+ ) + ) : error ? ( + + + Error loading insights + + {error instanceof Error ? error.message : 'An unknown error occurred'} + + + ) : ( +
+ {filteredInsights.map((insight: InsightCardData) => ( + + ))} + {filteredInsights.length === 0 && ( +
+ +

+ {statusFilter === 'all' ? 'No anomalies detected' : `No insights matching '${statusFilter}'`} +

+

+ {statusFilter === 'all' ? 'Traffic patterns are within normal baseline ranges.' : 'Try changing your filter criteria.'} +

+
+ )} +
+ )} + + {data && (data as any).computed_at && ( +
+ + + }> + Computed {relative((data as any).computed_at)} + + + {full((data as any).computed_at)} {abbr()} + + + +
+ )} + + ) +} + export default function InsightsPage() { const [windowHours, setWindowHours] = useState('1') const [baselineHours, setBaselineHours] = useState('168') @@ -138,104 +286,17 @@ export default function InsightsPage() { icon={Lightbulb} headerActions={headerControls} > - {({ - activeServiceId, - }) => { - const { data, isLoading, error } = useQuery({ - queryKey: ['insights', activeServiceId, windowHours, baselineHours], - queryFn: async ({ signal }) => { - const { data } = await client.POST("/api/insights", { signal, - body: { - window_size_hrs: parseFloat(windowHours), - baseline_hours: parseFloat(baselineHours), - filters: {}, - } - }) - return data - }, - enabled: !!activeServiceId, - staleTime: 60000 - }) - - const { data: availability } = useQuery({ - queryKey: ['insights', 'availability', activeServiceId], - queryFn: async ({ signal }) => { - const { data } = await client.GET("/api/insight-availability", { signal }) - return data - }, - enabled: !!activeServiceId - }) - - const filteredInsights = useMemo(() => { - if (!data?.insights) return [] - if (statusFilter === 'all') return data.insights - return (data.insights as InsightCardData[]).filter((insight: InsightCardData) => insight.severity === statusFilter) - }, [data?.insights, statusFilter]) - - return ( - <> - {(availability as any)?.unavailable && (availability as any).unavailable.length > 0 && ( - - - Some insights are unavailable - - {(availability as any).unavailable.length} insights require additional log fields to be enabled. - Check your service configuration. - - + {(ctx) => ( + )} - - {isLoading ? ( - // Use the route-level skeleton shape (matches loading.tsx + - // ReportShell's not-ready skeleton) so the loading state is - // CONSISTENT across click → skeleton → real-content. -
- -
- ) : error ? ( - - - Error loading insights - - {error instanceof Error ? error.message : 'An unknown error occurred'} - - - ) : ( -
- {filteredInsights.map((insight: InsightCardData) => ( - - ))} - {filteredInsights.length === 0 && ( -
- -

- {statusFilter === 'all' ? 'No anomalies detected' : `No insights matching '${statusFilter}'`} -

-

- {statusFilter === 'all' ? 'Traffic patterns are within normal baseline ranges.' : 'Try changing your filter criteria.'} -

-
- ) } -
- )} - - {data && (data as any).computed_at && ( -
- - - }> - Computed {relative((data as any).computed_at)} - - - {full((data as any).computed_at)} {abbr()} - - - -
- )} - - ) - }} - + ) } diff --git a/frontend/app/layout.tsx b/frontend/app/layout.tsx index df4a1ed7..22140c1a 100644 --- a/frontend/app/layout.tsx +++ b/frontend/app/layout.tsx @@ -6,6 +6,7 @@ import ThemeProvider from "@/components/ThemeProvider"; import { AppLayout } from "@/components/AppLayout"; import { TooltipProvider } from "@/components/ui/tooltip"; import { ErrorBoundary } from "@/components/ErrorBoundary"; +import { getPreloadChunks } from "@/lib/preload-manifest"; const inter = Inter({ subsets: ["latin"] }); @@ -14,13 +15,50 @@ export const metadata: Metadata = { description: "Modern log analytics for Fastly Object Storage", }; +// O6 follow-up (2026-06-06): force-dynamic was previously set here so +// the preload-manifest could be read at request time (manifest is +// generated by ``scripts/build-preload-manifest.mjs`` AFTER ``next +// build``, so SSG-time reads return empty). The cost was an SSR +// roundtrip on EVERY page navigation — the "click does nothing for +// 100-300 ms" lag. +// +// The trade-off was bad: modulepreload saves ~200 ms ONE TIME on +// first dashboard/network page load, but force-dynamic was costing +// 100-300 ms PER navigation. Net loss across a session. +// +// Fix: removed ``force-dynamic`` here, made ``getPreloadChunks()`` +// synchronous (module-load ``readFileSync``). Layout is back to a +// statically-renderable sync server component. Page navigations are +// instant again. The cost is that SSG-time reads return [] (manifest +// not written yet), so the static HTML has no ```` +// tags. Browser falls back to discovering plotly via the normal +// main-bundle parse → dynamic-import → fetch path. +// +// A future optimization (bootstrap pattern: commit the manifest so a +// previous build's chunk names are baked into SSG of the next build) +// could restore the preload benefit without re-introducing the +// navigation lag. + export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { + // Modulepreload links from the build-time manifest (returns [] at + // SSG-time since the manifest is generated AFTER next build). + const preloadChunks = getPreloadChunks(); return ( + + {preloadChunks.map((href) => ( + + ))} + {/* Preload the world choropleth's geojson (~251KB) so MapLibre's + addSource('world', { data: '/geo/world.geojson' }) finds it in + cache instead of paying a round-trip when the dashboard's + Requests by Country map mounts. */} + + since_id OR status = 'running'. The OR keeps still-running + // rows visible across polls. The `-1` keeps the most-recently-seen row + // in the response for ONE more poll so the toast-completion-detection + // effect below (line ~497) can observe the running→completed transition + // for the row backgroundCronToast is tracking. First poll + // (maxSeenIdRef.current is null) omits since_id and returns up to + // per_page recent rows like before. const { data: recentCrons, isFetching: isFetchingRecent } = useQuery({ queryKey: ['admin', 'cron-logs-recent', activeServiceId], queryFn: async ({ signal }) => { - const { data } = await client.GET("/api/cron-runs", { signal, + const max = maxSeenIdRef.current + const sinceId = max != null ? Math.max(0, max - 1) : undefined + const { data } = await client.GET("/api/cron-runs", { signal, params: { query: { page: 1, per_page: 10, + since_id: sinceId, } } }) @@ -357,7 +370,7 @@ export default function LogsPage() { }, enabled: !!activeServiceId, // Tab independent polling! refetchInterval: 5000, - staleTime: 0 + staleTime: 5_000, }) // Derive currently running crons and loading state from recent crons to keep downstream compatibility intact diff --git a/frontend/app/network/page.tsx b/frontend/app/network/page.tsx index f322c67c..14ba89af 100644 --- a/frontend/app/network/page.tsx +++ b/frontend/app/network/page.tsx @@ -6,12 +6,26 @@ import { DataTable, ColumnVisibilityDropdown } from '@/components/DataTable' import { client } from '@/lib/api' import { useServiceQuery } from '@/hooks/useServiceQuery' import { useColumnVisibility } from '@/hooks/useColumnVisibility' -import { PlotlyChart } from '@/components/PlotlyChart' import { UpdatingBadge } from '@/components/UpdatingBadge' import { DashboardLinkCell } from '@/components/DashboardLinkCell' import { downloadAsCsv } from '@/lib/utils' import { cn } from '@/lib/utils' import dynamic from 'next/dynamic' +// PlotlyChart renders conditionally on heatmapData (the RTT heatmap card). +// Static-importing it dragged the ~1MB plotly chunk into the critical path +// for every /network cold load even when the heatmap wasn't being rendered. +// Dynamic-import defers the chunk to when the heatmap card actually mounts. +const PlotlyChart = dynamic( + () => import('@/components/PlotlyChart').then(mod => mod.PlotlyChart), + { + ssr: false, + loading: () => ( +
+ Loading chart... +
+ ), + }, +) const NetworkMap = dynamic(() => import('@/components/Map/NetworkMap').then(mod => mod.NetworkMap), { ssr: false, loading: () => ( @@ -80,21 +94,8 @@ export default function NetworkPage() { const isLoadingInitial = isLoading || (isFetching && !data) - const { data: shieldingData, isLoading: shieldingLoading } = useServiceQuery( - ['network', 'shielding', activeServiceId, startTime, endTime, filterPayload], - async ({ signal }) => { - const { data } = await client.POST("/api/origin/shielding-analysis", { signal, - body: { - start_time: startTime!, - end_time: endTime!, - filters: filterPayload, - limit: 100, - } - }) - return data as any - }, - { staleTime: 30000 } - ) + const shieldingData = data?.shielding_analysis as any + const shieldingLoading = isLoadingInitial const asnOptions = React.useMemo(() => { if (!data?.leaderboard) return [] diff --git a/frontend/app/share-login/acknowledge/page.tsx b/frontend/app/share-login/acknowledge/page.tsx index 512d649a..e138650f 100644 --- a/frontend/app/share-login/acknowledge/page.tsx +++ b/frontend/app/share-login/acknowledge/page.tsx @@ -20,7 +20,13 @@ export default function AcknowledgePage() { // Raw fetch: the share-* routes use a relative path so the request flows // through the Next.js proxy in remote-analyst mode rather than the typed // client's `getApiBase()` which routes direct to 127.0.0.1:8000. - fetch('/api/share/heartbeat', { + // + // /api/share/tos doubles as an auth check (401 → bounce to /share-login) + // and the source of truth for the version we'll POST to /acknowledge. + // The backend enforces an exact version match (audit finding 021), so the + // version we display has to be the one the backend currently considers + // latest — fetching it here is the only way to stay in sync. + fetch('/api/share/tos', { credentials: 'include', headers: { 'X-Remote-Analyst': '1' }, }) @@ -30,16 +36,12 @@ export default function AcknowledgePage() { router.replace('/share-login') return } - // The /heartbeat response doesn't include TOS text — pull it from the - // login response that just preceded this navigation. Fall back to a - // generic acknowledgment text if we landed here cold (refresh). - setTos({ - version: '__current__', - text: - 'I acknowledge that I am viewing third-party operational log data, ' + - 'that my access is logged, and that I will not retain, redistribute, ' + - 'or use this data outside the scope of my engagement.', - }) + if (!res.ok) { + setError(`Could not load the terms (HTTP ${res.status}).`) + return + } + const body = (await res.json()) as TosPayload + setTos({ version: body.version, text: body.text }) }) .catch(() => { if (!cancelled) setError('Could not reach the server.') diff --git a/frontend/components/AppLayout.tsx b/frontend/components/AppLayout.tsx index e4b13b4b..ffbd7e07 100644 --- a/frontend/components/AppLayout.tsx +++ b/frontend/components/AppLayout.tsx @@ -29,6 +29,8 @@ import { FilterBar } from '@/components/FilterBar/FilterBar' import { ScrollArea } from '@/components/ui/scroll-area' import { SyncStatusBadge } from '@/components/SyncStatusBadge/SyncStatusBadge' import { DebugPanel } from '@/components/DebugPanel' +import { PlotlyPrewarm } from '@/components/PlotlyChart/PlotlyPrewarm' +import { MapPrewarm } from '@/components/Map/MapPrewarm' import { useUrlServiceSync } from '@/hooks/useUrlServiceSync' import { useBootstrap } from '@/hooks/useBootstrap' @@ -78,29 +80,26 @@ interface NavLinkProps { disabled?: boolean } -function NavLink({ href, icon: Icon, name, isActive, disabled, activeServiceId }: NavLinkProps & { activeServiceId?: string | null }) { +function NavLink({ href, icon: Icon, name, isActive, disabled, activeServiceId, router }: NavLinkProps & { activeServiceId?: string | null; router: ReturnType }) { const finalHref = activeServiceId && !href.startsWith('/admin') ? `${href}?service=${activeServiceId}` : href - // Disabled state (no services yet — onboarding) still renders a real - // with pointer-events disabled + aria-disabled. Pre-fix this - // returned a plain
, which made the entire sidebar inert during - // cold-start: an admin who had a services list but bootstrap hadn't - // returned yet couldn't click ANY nav item to bounce out of the - // /admin onboarding flow. Keeping it a Link is sufficient for that; - // the moment the disabled state flips off, clicks just work. - // - // Prefetch is disabled. Next.js auto-prefetches every visible - // on viewport entry — with ~12 sidebar items rendered on every page, - // that fires 30-60 RSC requests in the background of every page load - // (37-66 observed across page HARs, ~2s of bandwidth competition - // against the real data calls). Click-time fetch is ~100ms slower - // per navigation but the page-load cost it removes is much larger. + // Viewport-entry prefetch is disabled (prefetch={false}) — with ~12 + // sidebar items, auto-prefetch fires 30-60 RSC requests per page load + // (37-66 observed, ~2s bandwidth competition). Instead we prefetch on + // hover: the mouse takes 100-300ms to travel + dwell before clicking, + // which is enough for Next.js to fetch the loading boundary so the + // transition feels instant on click. + const handleMouseEnter = React.useCallback(() => { + if (!disabled) router.prefetch(finalHref) + }, [disabled, finalHref, router]) + return ( + {/* Force Plotly to parse + complete its first-plot draw during + app mount so the dashboard's real chart's data-arrival render + hits Plotly's fast react()-update path instead of the cold + init path. See PlotlyPrewarm.tsx for full rationale. */} + + {/* Same idea for MapLibre GL (used by the dashboard's + "Requests by Country" choropleth). ~1MB chunk + WebGL init + would otherwise run when the dashboard route mounts; the + prewarm gets it done during app mount instead. */} + {/* Desktop Sidebar */}