Skip to content

Address migration issues#105

Merged
casibbald merged 31 commits into
mainfrom
address-migration-issues
May 4, 2026
Merged

Address migration issues#105
casibbald merged 31 commits into
mainfrom
address-migration-issues

Conversation

@casibbald
Copy link
Copy Markdown
Contributor

@casibbald casibbald commented May 4, 2026

Note

Medium Risk
Mostly documentation and workflow/navigation changes, but it also bumps optional OpenTelemetry dependencies to a git-pinned 0.31 which could affect build reproducibility and any consumers enabling metrics/tracing features.

Overview
Introduces a new docs/llmwiki/ knowledge base (schema, index, logs, topic/entity pages) and replaces the root AGENT.md with AGENTS.md focused on strict agent rules and wiki-first navigation.

Aligns observability guidance and dependencies: updates docs/OBSERVABILITY_APP_INTEGRATION.md to reference microscaler-observability as the telemetry owner, and bumps optional OpenTelemetry crates to 0.31 with git [patch.crates-io] pins (plus corresponding Cargo.lock updates). Also tightens project hygiene with JSF-aligned clippy.toml thresholds, clarifies GraphQL as a legacy/frozen feature in docs, and adds Kubernetes test-infrastructure manifests for a namespaced Postgres deployment (config/k8s/test-infrastructure/*).

Reviewed by Cursor Bugbot for commit a0e703f. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • New Features

    • Added Kubernetes test infrastructure for PostgreSQL deployment with persistent storage and health checks.
    • Implemented per-table migration file writing with view-aware dependency tracking and seed ordering.
    • Enhanced metrics with local provider lifecycle management.
  • Bug Fixes

    • Enforced strict seed filename validation (YYYYMMDDHHMMSS format) with warnings for non-compliant files.
    • Added auto-inference of zero defaults for NOT NULL columns lacking explicit defaults.
    • Improved SQL dependency extraction to handle views and foreign key relationships.
  • Documentation

    • Established comprehensive LLM wiki with 30+ knowledge pages covering architecture, observability, coding standards, and operational guides.
    • Clarified platform positioning: OpenAPI/REST-first (BRRTRouter) with GraphQL as legacy/frozen optional feature.
    • Added JSF compliance documentation and Pragmatic Rust guidelines reference.
  • Chores

    • Upgraded OpenTelemetry dependencies from 0.29.x to 0.31.x with Git revision overrides.
    • Updated Tilt development setup for LAN-accessible testing.
    • Added Clippy configuration with JSF-aligned thresholds.

Session log for the three-layer defense-in-depth work on the hauliage
fleet Vehicle entity. Cross-references the committed artifacts:

- BRRTRouter multipart 415 fix (bd12de7)
- Fleet three-layer defense (hauliage 68ac8e6)
- F5 scaffolding + x-brrtrouter-impl backfill + PRDs (hauliage b1b44a9)
- playwright-bdd regenerated specs (hauliage b807fe1)

Includes:
- BRRTRouter request validation probe results (test 1-6 all reject
  bad input with HTTP 400/415/409 as appropriate).
- Fleet vehicle Type enum drift root cause and 3-layer fix.
- F5 audit across all 15 hauliage microservices — 35 F5 bugs patched,
  1 original + 3 additional orphans documented, 3 identity Z1 zombie
  impls surfaced and routed to product backlog.
- x-brrtrouter-impl marker coverage — 152 ops, 144 true, 8 false, 0 gaps.
- Compliance lifecycle for Vehicle — OUT_OF_SERVICE default at creation,
  ACTIVE transition requires MOT + insurance + tax expiry dates, 409
  with missing_fields array if not.
- Add docs/llmwiki: SCHEMA, README, index, log, docs-catalog, three seed topics
- Rewrite AGENT.md to rules-only; link wiki from README and memory bank
- Add reference pages: workspace/module map, planning docs index
- Add entity pages: LifeModel/Record, executor/pool, migrate/compare
- Add topic pages: query, relations, session, reflector, scalars,
  observability, migrate CLI, derive, integration testing
- Extend docs-catalog synthesis table and index; append log
- Point AGENT.md at full index; update memory bank
- Add entities/transaction-boundaries, topics/graphql-optional-feature
- Extend docs-catalog synthesis table and index; append log
- Import verbatim references under docs/references/ (JSF writeup, audit,
  compliance; Microsoft rust-guidelines)
- Add llmwiki topics for Lifeguard-specific synthesis; link from AGENT.md
- Align clippy.toml numeric thresholds with BRRTRouter / microscaler-observability
- Update docs-catalog, index, log; cross-link microscaler-observability wiki
Prepend a pointer to cylon-local-infra/docs/desktop-dev-environment.md
at the top of this repo's AGENTS.md. The canonical doc explains the
Mac + ms02 + DGX Spark topology, where commands execute (Mac for
Ansible, ms02 via SSH for builds/tests/Tilt, Nvidia cluster for
inference via OpenAI-compatible API), the ~/Workspace NFS mount layout,
and the Starlink-driven SSH tunneling.

Keeping topology in one canonical file prevents drift across the ~16
Microscaler repos that have their own AGENTS.md. Do not duplicate its
contents here; link to it and fix the canonical doc if reality drifts.
@gitguardian
Copy link
Copy Markdown

gitguardian Bot commented May 4, 2026

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
21245408 Triggered Generic Password a0e703f config/k8s/test-infrastructure/postgres-deployment.yaml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@casibbald casibbald merged commit 668d7e1 into main May 4, 2026
6 of 8 checks passed
@casibbald casibbald deleted the address-migration-issues branch May 4, 2026 09:04
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 4, 2026

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8963a073-2a75-4cab-98cb-9961c8098508

📥 Commits

Reviewing files that changed from the base of the PR and between 0024194 and a0e703f.

⛔ Files ignored due to path filters (4)
  • .agent/session.log is excluded by !**/*.log
  • Cargo.lock is excluded by !**/*.lock
  • migrations/generated/inventory/20260127141142_generated_from_entities.sql is excluded by !**/generated/**
  • migrations/generated/inventory/20260127143606_generated_from_entities.sql is excluded by !**/generated/**
📒 Files selected for processing (58)
  • .agent/memory-bank/activeContext.md
  • .agent/memory-bank/progress.md
  • .idea/vcs.xml
  • AGENT.md
  • AGENTS.md
  • Cargo.toml
  • LIFEGUARD_GAP_ANALYSIS.md
  • README.md
  • SECURITY_PROMPT.md
  • clippy.toml
  • config/k8s/test-infrastructure/postgres-deployment.yaml
  • config/k8s/test-infrastructure/postgres-pvc.yaml
  • config/k8s/test-infrastructure/postgres-service.yaml
  • docs/OBSERVABILITY_APP_INTEGRATION.md
  • docs/llmwiki/README.md
  • docs/llmwiki/SCHEMA.md
  • docs/llmwiki/docs-catalog.md
  • docs/llmwiki/entities/life-executor-pool-and-routing.md
  • docs/llmwiki/entities/life-model-and-life-record.md
  • docs/llmwiki/entities/migrate-compare-and-sql-generation.md
  • docs/llmwiki/entities/transaction-boundaries.md
  • docs/llmwiki/index.md
  • docs/llmwiki/log.md
  • docs/llmwiki/reference/planning-docs-index.md
  • docs/llmwiki/reference/workspace-and-module-map.md
  • docs/llmwiki/topics/brrtrouter-integration-pitfalls.md
  • docs/llmwiki/topics/coding-standards-jsf-inspired.md
  • docs/llmwiki/topics/derive-macros-and-attributes.md
  • docs/llmwiki/topics/documentation-landscape.md
  • docs/llmwiki/topics/graphql-optional-feature.md
  • docs/llmwiki/topics/index-and-derive-constraints.md
  • docs/llmwiki/topics/integration-testing-and-ci.md
  • docs/llmwiki/topics/migrate-cli-integration.md
  • docs/llmwiki/topics/observability-and-logging.md
  • docs/llmwiki/topics/postgres-scalars-uuid-chrono.md
  • docs/llmwiki/topics/pragmatic-rust-guidelines.md
  • docs/llmwiki/topics/query-select-and-active-model.md
  • docs/llmwiki/topics/raw-sql-vs-selectquery-policy.md
  • docs/llmwiki/topics/reflector-cache-and-coherence.md
  • docs/llmwiki/topics/relations-loaders-scopes.md
  • docs/llmwiki/topics/session-identity-map.md
  • docs/references/jsf-audit-opinion.md
  • docs/references/jsf-compliance.md
  • docs/references/jsf-writeup.md
  • docs/references/rust-guidelines.md
  • docs/rls-integration-design.md
  • justfile
  • lifeguard-migrate/README.md
  • lifeguard-migrate/src/generated_migration_diff.rs
  • lifeguard-migrate/src/lib.rs
  • lifeguard-migrate/src/migration_writer.rs
  • lifeguard-migrate/src/sql_dependency_order.rs
  • lifeguard-migrate/src/sql_generator.rs
  • scripts/dev_up.py
  • src/lib.rs
  • src/metrics.rs
  • tests/integration/active_model_crud.rs
  • tests/integration/related_trait.rs

📝 Walkthrough

Walkthrough

This PR introduces a comprehensive knowledge-management overhaul, migration-ordering enhancements, observability stack upgrade, and development-infrastructure updates. It establishes a persistent docs/llmwiki/ wiki scaffold, implements per-table migration files with view-aware and seed-order dependencies, upgrades OpenTelemetry to 0.31.x, refactors metrics to use local providers, and adds Kubernetes test-infrastructure templates.

Changes

Migration & Seed Ordering Framework

Layer / File(s) Summary
Public API & Module Exports
lifeguard-migrate/src/lib.rs
Exports new pub mod migration_writer; making per-table migration logic available to consumers.
Data Structures & Emission Logic
lifeguard-migrate/src/migration_writer.rs
Introduces EmissionOutcome, MigrationHeader, and write_per_table_migration_file() to handle per-table migration file generation, baseline accumulation, diff detection (fast-skip when schemas match), and view rewrite semantics; ~590 lines with comprehensive tests.
View & Seed Dependency Detection
lifeguard-migrate/src/sql_dependency_order.rs
Extends extract_created_tables_from_migration_sql() to recognize views, adds extract_view_source_tables_from_migration_sql() to parse view dependencies, implements extract_inserted_tables_from_sql() and extract_table_level_fk_edges_from_migration_sql() for seed FK mapping, and adds write_seed_order_file() with topological sort and timestamp-aware tie-breaking; ~1150 lines.
Auto-Inferred Defaults
lifeguard-migrate/src/sql_generator.rs
Adds infer_zero_default_for_sql_type() to infer 0/false defaults for NOT NULL numeric/boolean columns, emits configurable stderr warnings via warn_auto_inferred_default(), and integrates into generate_create_table_sql() with LIFEGUARD_SILENCE_INFERRED_DEFAULTS escape hatch; ~156 lines added.
Merged Baseline Identity & Skip Optimization
lifeguard-migrate/src/generated_migration_diff.rs
Updates accumulate_table_baselines_from_dir() to treat CREATE OR REPLACE VIEW sections as full baselines and adds early-exit in build_service_migration_body_from_accumulated() via normalized SQL equality check to skip delta emission when baseline matches fresh output.
Documentation & README
lifeguard-migrate/README.md
Documents FK-aware apply/seed ordering helpers, enforces timestamped seed filename grammar (YYYYMMDDHHMMSS_<slug>.sql) with skip/warning behavior, and records auto-inferred default warning behavior and env var silencing.
Integration Tests
tests/integration/related_trait.rs
Comprehensive test suite for Related and FindRelated traits with test entities (TestUser, TestPost, TestTenant, TestResource), one-to-many and composite-key validation, and regression tests for WHERE-clause construction and relationship direction; ~698 lines.

Knowledge Base & Documentation Refactor

Layer / File(s) Summary
Wiki Structure & Schema
docs/llmwiki/README.md, docs/llmwiki/SCHEMA.md
Establishes wiki purpose, navigation checklist, page layout conventions, and operational workflows (Ingest/Query/Lint, Agent session start/end).
Wiki Navigation & Catalog
docs/llmwiki/index.md, docs/llmwiki/docs-catalog.md, docs/llmwiki/log.md
Provides comprehensive wiki index linking to 20+ stable topics, an inventory of source documents mapped to wiki coverage, and a chronological append-only log of wiki updates (2026-04-17 to 2026-04-18).
Reference Navigation
docs/llmwiki/reference/workspace-and-module-map.md, docs/llmwiki/reference/planning-docs-index.md
Maps workspace crates to module concerns and planning/design clusters for cross-file discovery.
Core Entity Pages
docs/llmwiki/entities/life-model-and-life-record.md, docs/llmwiki/entities/life-executor-pool-and-routing.md, docs/llmwiki/entities/migrate-compare-and-sql-generation.md, docs/llmwiki/entities/transaction-boundaries.md
Documents primary Lifeguard ORM and infrastructure concepts with status, source docs, code anchors, and cross-references.
Cross-Cutting Topics
docs/llmwiki/topics/* (19 new files)
Covers query/model selection, raw-SQL policy, relation loaders, session identity, reflector cache, observability, migration CLI, derive macros, indexing, BRRTRouter pitfalls, GraphQL optionality, coding standards, pragmatic Rust, and integration testing.
Standards & Guidelines
docs/references/jsf-compliance.md, docs/references/jsf-writeup.md, docs/references/jsf-audit-opinion.md, docs/references/rust-guidelines.md
Establishes Lifeguard's stance on JSF AV rules (allocation discipline, complexity bounds, panic-free error handling, type safety) and pragmatic Rust guidelines; ~2400 lines for Rust guidelines alone.
Agent Rules & Policy
AGENTS.md (replaces AGENT.md)
Navigation rules for agents, required pre-work (docs/llmwiki/), UUID/index/derive correctness requirements, raw-SQL escalation policy, and cross-repo pointers.
Progress Tracking & Architecture Notes
.agent/memory-bank/progress.md, .agent/memory-bank/activeContext.md
Dated progress log entries covering fleet compliance, docs reorganization, ADRs (0013–0015), seed enforcement, default-expr adoption, and migration framework refinements; tracks cross-repo observability, OTLP flushing, and BRRTRouter performance work.
RLS Integration Design
docs/rls-integration-design.md
Draft design for PostgreSQL Row Level Security (RLS) integration via stateful session variables, proposing 4 phases (SessionContext, RlsExecutor wrapper, pooled job context, integration tests) with consuming-app ownership for SQL helpers and RLS policies.
Documentation Index & Standards
README.md, LIFEGUARD_GAP_ANALYSIS.md, SECURITY_PROMPT.md
Clarify optional GraphQL feature (legacy, not OpenAPI-first), link to docs/llmwiki/ wiki, update security assessment scope, and document platform direction (OpenAPI/BRRTRouter instead of GraphQL).

Observability & Metrics Updates

Layer / File(s) Summary
Dependency Versions & Patch Registry
Cargo.toml
Upgrade OpenTelemetry 0.29.x0.31.x (opentelemetry, opentelemetry-prometheus, opentelemetry_sdk) and add [patch.crates-io] section pinning both to Git rev = 965078315b58ae14725721735f1c8e2bc2d3b445; retain feature flags and optionality.
Metrics Provider Refactoring
src/metrics.rs
Replace global set_meter_provider() wiring with local SdkMeterProvider stored in LifeguardMetrics struct to keep instruments alive without installing global provider; switch METRICS singleton from once_cell::Lazy to std::sync::LazyLock with updated init strategy.
Feature Documentation
src/lib.rs, docs/OBSERVABILITY_APP_INTEGRATION.md
Update GraphQL feature documentation and clarify host/OTLP responsibility, forbid Lifeguard from setting globals, emphasize local meter provider for Prometheus scrape, and refactor BRRTRouter reference table with shutdown/flush and test strategy details.
Clippy Thresholds
clippy.toml
Add JSF-aligned configuration block with explicit numeric thresholds for stack size, enum variant size, cognitive/type complexity, and too-many-* limits; update header/comments.

Test Infrastructure & Development Environment

Layer / File(s) Summary
Kubernetes Deployment & Storage
config/k8s/test-infrastructure/postgres-deployment.yaml, config/k8s/test-infrastructure/postgres-pvc.yaml, config/k8s/test-infrastructure/postgres-service.yaml
Define postgres:15-alpine Deployment with environment variables, port 5432, persistent volume mount at /var/lib/postgresql/data, liveness/readiness probes; PersistentVolumeClaim postgres-data requesting 1Gi ReadWriteOnce; Service exposing TCP 5432 via ClusterIP.
Development Environment & Tilt
justfile, scripts/dev_up.py, .idea/vcs.xml
Update Tilt startup to bind to 0.0.0.0:10350 with LAN-accessible URL output; simplify IntelliJ IDEA VCS mapping from specific may_postgres to root directory Git mapping.

Sequence Diagram(s)

The changes do not warrant sequence diagrams. While they introduce new infrastructure and patterns (migration ordering, RLS design), they are predominantly:

  • Documentation and knowledge-base scaffolding (non-functional references)
  • Refactoring of existing observability wiring (metrics singleton, no new interactions)
  • Framework enhancements to migration SQL generation (internal batch processing, not runtime interactions)
  • Configuration and testing infrastructure (static Kubernetes manifests and CLI tweaks)

None introduce new, multi-component runtime flows that would benefit from visualization.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

The PR combines high heterogeneity (documentation, configuration, multiple crate logic changes, schema/dependency ordering, test suites) with significant substance in several areas (migration_writer.rs with ~590 lines including tests, sql_dependency_order.rs with ~1150 lines, metrics refactoring with behavioral changes, comprehensive wiki scaffold). While individual files follow established patterns, the breadth requires careful verification of dependency-ordering correctness, backward compatibility of metrics initialization, seed/view-aware topological sort, and cross-file alignment in documentation/policy. The Cargo.toml patch section and OTEL version bump merit scrutiny. Homogeneous doc changes (many wiki topic files) reduce per-file effort, but the migration logic and metrics refactoring demand dense reasoning.

Possibly related PRs

  • Fix/tests #52: Modifies lifeguard-migrate's migration ordering and generator logic (sql_dependency_order.rs, migration-writing code) and related docs/CI around apply/seed ordering—directly related at the code level.
  • Feat/schema validators session and scopes 2 #57: Extends lifeguard-migrate's public modules (schema_migration_compare) and modifies generated_migration_diff.rs baseline/comparison logic, aligning with this PR's diff detection and module export changes.
  • Feat/schema validators session and scopes 4 #58: Touches the same lifeguard-migrate codepaths (sql_generator.rs DDL/view generation, generated_migration_diff.rs baseline logic, sql_dependency_order.rs view/seed ordering) and observability/metrics wiring, indicating overlapping migration and observability concerns.

Poem

A rabbit hops through the llmwiki glen,
Planting seeds of order—one migration, then ten.
Views and timestamps dance in topological grace,
Metrics locally thrive in their own safe space. 🐰✨
Infrastructure roots grow, knowledge takes flight—
The warren is mapped, and the path becomes bright.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch address-migration-issues

@gemini-code-assist
Copy link
Copy Markdown

Warning

Gemini is experiencing higher than usual traffic and was unable to create the review. Please try again in a few hours by commenting /gemini review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant