diff --git a/.claude/skills/api-design-guardian/SKILL.md b/.claude/skills/api-design-guardian/SKILL.md new file mode 100644 index 00000000..9a54dccc --- /dev/null +++ b/.claude/skills/api-design-guardian/SKILL.md @@ -0,0 +1,39 @@ +--- +name: api-design-guardian +description: Use when writing or changing a public API or SDK surface — GraphQL schemas (.graphql/.graphqls), REST/OpenAPI endpoints, SDK request/response types, or any client-facing contract. Enforces b.well's API-design bar (both the positive "what good looks like" and the antipatterns) before the change reaches PR, and treats external contract changes as breaking by default. +--- + +# API Design Guardian + +You enforce b.well's API/SDK design standard. APIs are contracts; a bad one costs months of client updates. Catch problems **before** the PR. Be direct — a senior architect who explains *why* and shows the fix. + +The standard is in the substrate — read and cite it, don't restate it: +- **What good looks like:** `reference-architectures/good-api-sdk.md` (grounded in the real bwell-sdk review bar). +- **Gradeable criteria:** `rubrics/api-design-rubric.md` — cite the criterion anchor (e.g. `#rub-api-breaking`) in every finding. +(These live in `icanbwell/.github`; read them there if not in the current repo.) + +## Lead with the positive bar + +Before hunting smells, hold the change to the good-API bar (`reference-architectures/good-api-sdk.md`): **layer purity** (no Apollo/generated/vendor types in domain/request objects — map at the boundary), **vendor neutrality** (no vendor error text/codes leak through the surface), **consistency** across request families and across the Kotlin/TS/Swift SDKs (diff a new object against `ConsentRequest`; use the shared `BWellResult` wrapper), **fail-fast typed errors** (validate in the builder with `require`; a closed neutral error enum with an `unknown`/default branch), **DI over singletons**, **interfaces over abstract classes**. + +## Highest-priority: contract stability + +**Treat any change to an externally- or downstream-consumed field, type, enum, or identifier as breaking by default** (`#rub-api-breaking`). Require deprecate-and-add or a new version, plus a consumer-driven contract test in the same PR. Real cost: an external id changed slug→UUID and silently broke a client for 34 days (INC-329). + +**Enum resilience is a hard check** (`#rub-api-enum-default`): flag any exhaustive `switch`/`when` over a server-driven enum with **no `default`/`unknown` branch** — that exact bug broke the Swift SDK build *twice* (EA-2401 / DCON-4580). Additive-only evolution otherwise. + +## Antipatterns to catch (with the fix) + +- **Boolean-flag params** (`skipDeleted: Boolean`) → a status/filter **enum** with a default; flags don't compose (`#rub-api-boolean-flags`). +- **Internal/generated types as responses** (`*Entity`/`*Model`/`*DTO`/Apollo types) → a purpose-built contract type; ties to layer purity (`#rub-api-internal-types`). +- **Unbounded lists** → cursor pagination (Relay `Connection`) or offset for simple cases; never return unbounded arrays (`#rub-api-pagination`). +- **Naming drift** — non-singular request names, `I`-prefix/`Impl` suffix, inconsistent booleans → match the family template and cross-SDK contract (`#rub-api-consistency`). +- **Gateway bypass** — client-facing access not through the federated graph; local type names colliding with federated canonicals (INE-1001) (`#rub-api-gateway`). + +## Verify against reality + +Behavior is confirmed against the live resolver/dev, the federated graph is the source of truth, and SDK releases are gated on the backing change being in prod (`#rub-api-verify`). Public API changes carry a Tech Design Review. + +## When to intervene / not + +Intervene on new/changed `.graphql(s)`, OpenAPI, SDK request/response types, or query params. Don't intervene when the user is only reading schemas, writing tests, or the change is internal-only (not client-facing). When you flag something, name the rubric anchor and show the corrected snippet. diff --git a/.claude/skills/cleanup-descope-e2e-data/SKILL.md b/.claude/skills/cleanup-descope-e2e-data/SKILL.md new file mode 100644 index 00000000..7a1043a9 --- /dev/null +++ b/.claude/skills/cleanup-descope-e2e-data/SKILL.md @@ -0,0 +1,102 @@ +--- +name: cleanup-descope-e2e-data +description: Use when asked to clean up, delete, or purge leftover Descope E2E test data — the "E2E Test Clinic" tenants and their users that ui-tests-descope leaves behind (no auto-cleanup; the test stack's management key is IP-restricted, error E023017). Drives the Descope MCP tools. +--- + +# Clean up Descope E2E test data + +## Overview + +`make ui-tests-descope` signs up real tenants + users in the live Descope project +(`dev-kill-the-clipboard-scanner`) and **cannot clean up after itself** — the test stack's +management key is IP-restricted (`E023017`). This skill deletes that residue through the +**Descope MCP** tools (a different, working credential path). + +E2E data is named with a stable prefix: +- **Tenants:** `E2E Test Clinic ` +- **Users:** owner `E2E Test User ` + invitee, both with `*@inbox.testmail.app` login IDs, + and (for fresh test data) belonging **only** to their E2E tenant. + +**Core mechanism:** `DeleteTenant` with `cascade: true` deletes users attached *only* to that +tenant and merely detaches shared ones — so deleting an `E2E` tenant removes its users in one +call without ever touching a user who also belongs to a protected tenant. + +## NEVER delete (hard guardrails) + +- **Wrong project.** Only ever run against project `dev-kill-the-clipboard-scanner` + (id `P3EgKEuFdQFLSuhXYk7KNsvlroTd`). If `whoami` shows any other project — **especially + production** — STOP and tell the user. Never `selectProject` to a prod project to delete. +- **`test-tenant-001` and `test-tenant-002`** — the seeded local multi-tenant dev fixtures. +- **Any tenant whose name does not start with `E2E`** (case-sensitive prefix). When in doubt, skip it. + +## Workflow + +This is a **destructive, confirmation-gated** operation. Do the read/plan steps, show the user +exactly what will be deleted, get an explicit "yes", then elevate and delete. + +### 1. Confirm the project (read-only) +``` +session({action: "whoami"}) +``` +Verify `project` / `projectId` is `dev-kill-the-clipboard-scanner` / `P3EgKEuFdQFLSuhXYk7KNsvlroTd`. +If not → STOP, report to user. + +### 2. Find E2E tenants (read-only, no elevation) +``` +tenants_read({operation: "LoadAllTenants"}) +``` +Filter `data.tenants` to those whose `name` starts with `E2E`. There is no server-side prefix +filter — filter client-side. Exclude everything in the guardrails list above. + +### 3. Enumerate each tenant's members (read-only) — for the report + leftover sweep +``` +users_read({operation: "SearchUsers", args: {tenantIds: [""], limit: 100}}) +``` +Collect `userId`, `email`, and `userTenants` for each. (`userTenants` with a single entry ⇒ +`cascade` will delete that user; multiple entries ⇒ it will only be detached.) + +### 4. Present the plan and get explicit confirmation +List each tenant (`name` + `id`) and its users (`email` + `userId`). Ask the user to confirm, +citing the exact targets. **Wait for an affirmative reply.** Do not elevate before this. + +### 5. Elevate write mode (only after confirmation) +``` +session({action: "elevate", args: {reason: ""}}) +``` + +### 6. Delete each tenant (cascades to single-tenant users) +``` +tenants_write({operation: "DeleteTenant", args: {id: "", cascade: true}}) +``` + +### 7. Sweep leftover E2E users (only if step 3 found shared/multi-tenant users `cascade` left behind) +Re-run `SearchUsers` (e.g. `args: {text: "E2E", limit: 100}`), keep only users still present whose +`name`/`email` is unmistakably E2E test data, then: +``` +users_write({operation: "DeleteUsers", args: {userIds: ["", ...]}}) +``` + +### 8. Verify +Re-run `tenants_read({operation: "LoadAllTenants"})` and confirm no `E2E`-prefixed tenants remain +(and `test-tenant-001/002` are untouched). + +## Quick reference + +| Step | MCP call | Elevation? | +|------|----------|------------| +| Check project | `session(whoami)` | no | +| List tenants | `tenants_read(LoadAllTenants)` | no | +| List a tenant's users | `users_read(SearchUsers, {tenantIds:[id]})` | no | +| Unlock writes | `session(elevate, {reason})` | — | +| Delete tenant + its users | `tenants_write(DeleteTenant, {id, cascade:true})` | yes | +| Delete specific users | `users_write(DeleteUsers, {userIds:[...]})` | yes | + +## Common mistakes + +- **Skipping the project check.** `whoami` first, every time. Deleting from the wrong project is unrecoverable. +- **Substring instead of prefix match.** Match names that *start with* `E2E`, not just contain it. +- **Elevating before confirmation.** The MCP elevation contract requires discovery → prepare → + ask → elevate. Never elevate autonomously. +- **Expecting `DeleteTenant` to delete every user.** It only deletes users attached solely to that + tenant; shared users are detached. Use the step-7 sweep for any genuinely-E2E shared/orphaned users. +- **Forgetting verification.** Re-list tenants at the end to prove the cleanup worked. \ No newline at end of file diff --git a/.claude/skills/codeable-concept-mapping/SKILL.md b/.claude/skills/codeable-concept-mapping/SKILL.md new file mode 100644 index 00000000..54288afa --- /dev/null +++ b/.claude/skills/codeable-concept-mapping/SKILL.md @@ -0,0 +1,136 @@ +--- +name: codeable-concept-mapping +description: > + Update the CodeableConcept field mapping generator script in the sensitive-data-tagger package. + Use when modifying the list of target FHIR resources, adjusting depth, adding/removing field types, + or changing the output format. Also use when the FHIR spec XSD files are updated. + Triggers on: "update mapping", "add resource to mapping", "regenerate codeable concept", + "update sensitive fields", "change mapping script". +--- + +# CodeableConcept Field Mapping Generator + +## What It Does + +`packages/sensitive-data-tagger/generatorScripts/generate_codeable_concept_mapping.py` generates a TypeScript file that maps FHIR resource types to their CodeableConcept and Coding field paths. These are fields that can carry sensitive clinical codes (CPT, HCPCS, ICD, SNOMED, LOINC, NDC, RxNorm). + +## How It Works + +1. Uses `fhir_xml_schema_parser.py` to parse the FHIR R4B (v4.3.0) XML spec (`xsd/definitions.xml/`) +2. For each target resource, recursively traverses properties up to `MAX_DEPTH` levels +3. Collects fields where type is `CodeableConcept` or `Coding` +4. Resolves reusable complex types (e.g., `Dosage.route` inside `MedicationRequest.dosageInstruction`) +5. Outputs a TypeScript file with `export const SensitiveCodeableConceptFieldMapping` + +## Key Files + +| File | Purpose | +| -------------------------------------------------------------------------------------- | ------------------------------------------------------- | +| `packages/sensitive-data-tagger/generatorScripts/generate_codeable_concept_mapping.py` | The generator script | +| `packages/sensitive-data-tagger/generatorScripts/fhir_xml_schema_parser.py` | FHIR XML schema parser (from fhir-server, do not edit) | +| `packages/sensitive-data-tagger/generatorScripts/xsd/` | FHIR R4B XSD spec files (from fhir-server, do not edit) | +| `packages/sensitive-data-tagger/src/sensitiveCodeableConceptFieldMapping.generated.ts` | Generated output (do not hand-edit) | + +## Regenerating + +```bash +cd packages/sensitive-data-tagger +make generate-codeable-concept-mapping +``` + +Requires: Docker (the Makefile runs the Python script inside a `python:3.12-alpine` container that auto-installs `lxml`). No local Python setup needed. + +## Script Configuration + +These are at the top of `generate_codeable_concept_mapping.py`: + +### Target Resources + +The `TARGET_RESOURCES` list defines which FHIR resources to include. To add a new resource, simply append it to the list. All resource names must match the FHIR R4 spec exactly (PascalCase). + +### Depth + +`MAX_DEPTH = 4` — controls how deep to traverse nested fields. Depth is counted from the resource root: + +- Depth 1: `Condition.code` +- Depth 2: `Condition.stage.summary` +- Depth 3: `Claim.item.detail.productOrService` +- Depth 4: `Claim.item.detail.subDetail.productOrService` + +### Skipped Types + +The script skips: `Extension`, `ModifierExtension`, `Identifier`, `Reference`, `Resource`, `ResourceContainer`, and the `meta` field. These either cause infinite recursion or don't carry clinical codes. + +### Coded Types + +`CODED_TYPES = {'CodeableConcept', 'Coding'}` — fields matching these types are collected. CodeableConcept contains Coding inside it, so when a CodeableConcept is found, the script stops recursing (doesn't duplicate the inner `.coding` field). + +## Output Format + +```typescript +export const SensitiveCodeableConceptFieldMapping: Record< + string, + { CodeableConcept?: string[]; Coding?: string[] } +> = { + Condition: { + CodeableConcept: [ + 'bodySite', + 'category', + 'clinicalStatus', + 'code', + 'evidence.code', + 'severity', + 'stage.summary', + 'stage.type', + 'verificationStatus', + ], + }, + Encounter: { + CodeableConcept: ['diagnosis.use', 'type' /* ... */], + Coding: ['class', 'classHistory.class'], + }, +}; +``` + +- **CodeableConcept fields**: contain `coding[]` array + `text` field +- **Coding fields**: standalone single codes (e.g., `Encounter.class`) +- Field paths are dot-separated relative to the resource (e.g., `dosageInstruction.route`) + +## Reused Infrastructure + +The script reuses patterns from `fhir-server/generatorScripts/generate_everything_operation_data.py`: + +- `get_field_type_property()` — looks up a type's entity definition by name +- `handle_nested_fields()` — recursive traversal with skip patterns and cycle detection +- `primitive_types_dict` — from `FhirXmlSchemaParser.get_fhir_primitive_types()` +- `all_classes` — from `FhirXmlSchemaParser.generate_classes()` + +## FHIR Binding Strength (Not in Output, but Useful Context) + +Each CodeableConcept field has a binding strength in the FHIR spec: + +- **required**: locked to a FHIR value set — CANNOT contain sensitive codes (e.g., `clinicalStatus`) +- **extensible**: should use value set, but CAN use other systems +- **preferred**: recommended codes, any system valid +- **example**: any code system — where sensitive codes primarily live (e.g., `Condition.code`) +- **unbound**: no binding defined — completely open + +Only 6 fields across all 31 resources have `required` binding. The rest can potentially carry sensitive codes. + +## Common Tasks + +### Add a new FHIR resource + +1. Add the resource name to `TARGET_RESOURCES` list (alphabetical order) +2. Run `make generate-codeable-concept-mapping` + +### Update FHIR spec version + +1. Copy updated XSD files from `fhir-server/generatorScripts/xsd/` to `generatorScripts/xsd/` +2. Copy updated `fhir_xml_schema_parser.py` from `fhir-server/generatorScripts/` +3. Run `make generate-codeable-concept-mapping` + +### Change traversal depth + +1. Edit `MAX_DEPTH` in the script +2. Run `make generate-codeable-concept-mapping` diff --git a/.claude/skills/connection-observability/SKILL.md b/.claude/skills/connection-observability/SKILL.md new file mode 100644 index 00000000..b9fa7634 --- /dev/null +++ b/.claude/skills/connection-observability/SKILL.md @@ -0,0 +1,350 @@ +--- +name: connection-observability +description: > + Troubleshoot and investigate healthcare connection issues using groundcover observability tools. Use this skill + whenever someone asks about ATS errors, log investigation, token refresh failures, OAuth flow problems, GraphQL + errors, or any question related to the aperture-token-service. Also use when someone mentions connection statuses + (EXPIRED, DISCONNECTED, ACCESS_ENDED), error codes, token refresh, or wants to investigate ATS logs in + groundcover. Trigger even if the user just says "why is my connection expired" or "what does this ATS error + mean" or "check ATS logs" or "what's happening with this connection". +--- + +# ATS Troubleshooting Guide + +You are a troubleshooting expert for the **Aperture Token Service (ATS)** — a Flask-based Python microservice that manages OAuth tokens and healthcare provider connections for the b.well platform. + +## How to Use This Skill + +When a user asks about an ATS error, connection issue, or behavior: + +1. **Understand the question** — What flow is involved? (OAuth callback, token refresh, API call, connection creation) +2. **Know the logs** — Read `references/logging-patterns.md` to know exactly which log statements exist for that flow, their `funcName`, content patterns, and structured attributes +3. **Determine the workload** — API issues → `aperture-token-service`, refresh/pipeline issues → `aperture-token-service-pipelines` +4. **Query groundcover using MCP tools** — Use the appropriate tool (see below) with structured attribute filters when possible +5. **Default time range: P7D** — Always use 7 days unless told otherwise +6. **Provide actionable guidance** — explain what happened (with log evidence), why, and how to fix + +--- + +## Groundcover MCP Tools — Which to Use When + +ATS investigation uses these groundcover MCP tools: + +| Tool | Use When | +|------|----------| +| `mcp__groundcover__query_logs` | **Primary tool.** All log investigation — errors, refresh failures, OAuth issues, user activity | +| `mcp__groundcover__query_traces` | Latency issues, 5xx errors, verifying provider callbacks, finding full URLs | +| `mcp__groundcover__query_events` | Pod crashes, OOMKills, restarts, Kubernetes-level issues | +| `mcp__groundcover__query_monitors` | Checking what alerts/monitors exist for ATS | +| `mcp__groundcover__query_issues` | Active alert firings, historical alert instances | + +### Tool Parameters + +All query tools accept: +- **`query`** (required) — gcQL query string. Must start with `*` or a filter. Always include `| limit N`. +- **`period`** — ISO 8601 duration. Default `PT1H`. **Always set to `P7D`** for ATS investigation unless user specifies otherwise. +- **`start`** / **`end`** — RFC3339 timestamps for specific time ranges (alternative to `period`) + +### Example Tool Calls + +**Logs — find refresh errors for a user:** +``` +tool: mcp__groundcover__query_logs +query: "* | filter client_fhir_person_id: service_slug: | sort by (_time desc) | limit 20" +period: "P7D" +``` + +**Traces — find slow requests:** +``` +tool: mcp__groundcover__query_traces +query: "* | filter service.name:aperture-token-service duration_seconds>5 | sort by (duration_seconds desc) | limit 20" +period: "P7D" +``` + +**Events — check for pod crashes:** +``` +tool: mcp__groundcover__query_events +query: "* | filter involved_object.name:*aperture-token* type:Warning | sort by (_time desc) | limit 30" +period: "P7D" +``` + +**Monitors — check ATS alerts:** +``` +tool: mcp__groundcover__query_monitors +query: "monitor_name:*aperture*" +``` + +--- + +## ATS Workloads and Log Sources + +ATS has **two workloads** in groundcover: + +| Workload | What It Handles | When to Query | +|----------|----------------|---------------| +| `aperture-token-service` | Main API (GraphQL, OAuth callbacks, REST endpoints) | Connection creation, OAuth flows, API errors, user-facing issues | +| `aperture-token-service-pipelines` | Batch pipelines (token refresh jobs) | Token refresh failures, scheduled job issues, EXPIRED status investigations | + +--- + +## Log Categories — What Exists and When to Use Each + +### 1. Request Lifecycle Logs (API workload) + +| funcName | Content Pattern | Use When | +|----------|----------------|----------| +| `before_request` | `[BEFORE_REQUEST] Started: {method} {path} {request_id} {body}` | Seeing what API calls a user made, GraphQL operation names | +| `log_request` | `[AFTER_REQUEST] Completed: {method} {path} Status: {code} Elapsed: {time}s` | Finding slow/failed requests, response status codes | +| `wrapper` | `{"function": "...", "kwargs": {...}, "return": {...}}` | REST endpoint calls and their return values | + +### 2. OAuth Callback Logs (API workload) + +| funcName | Content Pattern | Level | Use When | +|----------|----------------|-------|----------| +| `handle_callback_generic` | `[HANDLE_CALLBACK_GENERIC] Error in query params` | critical | Provider rejected the OAuth request (has `query_params.error`) | +| `handle_callback_generic` | `[HANDLE_CALLBACK_GENERIC] No stored data found` | critical | State mismatch — OAuthCallback not found in MongoDB | +| `handle_callback_generic` | `[HANDLE_CALLBACK_GENERIC] Unable to authenticate request passed to callback` | critical | Token exchange failed (check `error` attribute for exception message) | +| `handle_callback_generic` | `[HANDLE_CALLBACK_GENERIC] Received request at...` | info | Successful OAuth callback | + +### 3. Token Refresh Logs (Pipelines workload) + +| funcName | Content Pattern | Level | Use When | +|----------|----------------|-------|----------| +| `handle_token_refresh_response` | `[REFRESH_TOKENS] Token refreshed successfully` | info | Confirming successful refreshes | +| `handle_token_refresh_error` | `[REFRESH_TOKENS] Error refreshing token, setting status to EXPIRED` | error | Token marked EXPIRED — terminal refresh failure | +| `handle_token_refresh_error` | `[REFRESH_TOKENS] Error refreshing token` | error | Transient refresh error (status unchanged, will retry) | +| `handle_token_refresh_error` | `[REFRESH_TOKENS] Token already expired` | info | Token was already EXPIRED before refresh attempt | +| `process_refresh_tokens_results` | `[REFRESH_TOKENS] Refresh token results:` | info | Summary of refresh outcome (has `code`, `status`, `prev_status`) | +| `handle_token_refresh_response` | `access_token not found in successful token refresh response` | error | Provider returned success but no access_token | +| `refresh_oauth_token` | `[REFRESH_TOKENS] Empty refresh_token but token is not disconnected` | error | Token record has no refresh_token | + +### 4. Connection Hub / Config Logs (API workload) + +| funcName | Content Pattern | Use When | +|----------|----------------|----------| +| `make_request_with_retries` | `Response for url={endpoint}, params={params} is status_code: {code}` | Verifying provider config fetches from Connection Hub | + +### 5. Auth/JWT Logs (API workload) + +| Content Pattern | Level | Use When | +|----------------|-------|----------| +| `Error in validating JWT token: Invalid key id: {kid}` | error | JWT validation failures | +| `Required items not found in auth token. Falling back to using id_token` | warning | Missing claims in JWT | + +### 6. FHIR / Subscription Logs (API workload) + +| funcName | Use When | +|----------|----------| +| `fetch_batch` | SubscriptionStatus lookups — checking sync state | + +### 7. Kafka Event Logs (API workload) + +| Content Pattern | Level | Use When | +|----------------|-------|----------| +| `Error occurred while publishing kafka event` | exception | Kafka publish failures | +| `Failed to emit kafka event on token status update` | error | Token status event failures | + +--- + +## Structured Attributes for Direct Filtering + +These fields are directly filterable in gcQL (no wildcards needed): + +| Attribute | Logged By | Example Values | +|-----------|-----------|----------------| +| `client_fhir_person_id` | refresh results, callback errors | UUID | +| `service_slug` | refresh results, callback errors | provider identifier string | +| `member_id` | refresh results | UUID | +| `bwell_fhir_person_id` | refresh results | UUID | +| `funcName` | all structured logs | function name | +| `code` | process_refresh_tokens_results | Refreshed, RefreshError, ClientRefreshError, etc. | +| `status` | refresh results | New, Data Retrieved, Expired, etc. | +| `prev_status` | refresh results | same values | +| `connection_version` | refresh/callback errors | integer | +| `interop_type` | refresh errors | hapi, oauth, etc. | +| `query_params.error` | callback errors | access_denied, server_error | +| `query_params.error_description` | callback errors | Provider-specific message | + +--- + +## Groundcover Query Patterns + +### Query Rules (learned from production use) + +- **Always set `period: "P7D"`** in tool calls (7 days) unless user specifies otherwise +- Use structured attribute filters — more reliable than content wildcards +- Use `level:error` field filter, NOT `content:*error*` +- Keep to 1-2 `content:*...*` filters max — 3+ often cause query failures +- If a query fails, simplify and filter results programmatically +- Refresh logs come from `aperture-token-service-pipelines`, NOT the main API workload +- When using attribute filters like `client_fhir_person_id:`, omit the workload filter (attributes work across both) + +### Essential Queries by Scenario (all use `mcp__groundcover__query_logs` with `period: "P7D"`) + +**All activity for a user (structured attributes — PREFERRED):** +``` +* | filter client_fhir_person_id: | sort by (_time desc) | limit 30 +``` + +**All activity for a user (API-level — catches GraphQL/callbacks):** +``` +* | filter workload:aperture-token-service content:** | sort by (_time desc) | limit 30 +``` + +**Token refresh results for a user:** +``` +* | filter client_fhir_person_id: service_slug: | sort by (_time desc) | limit 20 +``` + +**Refresh failures (terminal — token marked EXPIRED):** +``` +* | filter service_slug: funcName:handle_token_refresh_error | sort by (_time desc) | limit 20 +``` + +**OAuth callback errors:** +``` +* | filter workload:aperture-token-service content:*callback* level:error | sort by (_time desc) | limit 30 +``` + +**Recent API errors:** +``` +* | filter workload:aperture-token-service level:error | sort by (_time desc) | limit 50 +``` + +**Recent pipeline errors:** +``` +* | filter workload:aperture-token-service-pipelines level:error | sort by (_time desc) | limit 50 +``` + +**Error rate overview:** +``` +* | filter workload:aperture-token-service | stats by (level) count() | limit 10 +``` + +### Trace Queries (use `mcp__groundcover__query_traces` with `period: "P7D"`) + +Service name: `aperture-token-service` + +``` +# 5xx errors +* | filter service.name:aperture-token-service http.status_code:5* | sort by (_time desc) | limit 50 + +# Slow requests +* | filter service.name:aperture-token-service duration_seconds>5 | sort by (duration_seconds desc) | limit 20 + +# OAuth callbacks (verify provider called back) +* | filter service.name:aperture-token-service http.target:*callback* | sort by (_time desc) | limit 20 + +# Generate URL calls +* | filter service.name:aperture-token-service http.target:*generate_url* | sort by (_time desc) | limit 20 +``` + +Key trace field: `resource_name` contains the **full URL with query parameters** — critical for OAuth debugging (shows client_id, redirect_uri, scopes, PKCE params). + +### Event Queries (use `mcp__groundcover__query_events` with `period: "P7D"`) + +``` +# Pod crashes, OOMs, restarts +* | filter involved_object.name:*aperture-token* type:Warning | sort by (_time desc) | limit 30 +``` + +### Tracing Request Origins (Who Called This Endpoint?) + +Use `otelTraceID` from logs → query traces with `* | filter trace_id: is_root_span:true | limit 5` → root span has `client.address` (caller IP), `user_agent.original`, and `url.path`. + +--- + +## Architecture Quick Reference + +| Layer | Key Files | +|-------|-----------| +| Entry points | `autoapp.py`, `wsgi.py` | +| App factory | `aperture_token_service/app.py` | +| Settings | `aperture_token_service/settings.py` | +| OAuth flow | `aperture_token_service/oauth/views.py`, `interactor.py`, `utils.py` | +| Token CRUD | `aperture_token_service/token/views.py`, `interactor.py` | +| GraphQL | `aperture_token_service/graphql/schema.py`, `token/graphql/resolvers.py` | +| Error handling | `aperture_token_service/graphql/error_handler.py`, `token_service_exception.py` | +| Connection logic | `aperture_token_service/token/interactor.py`, `oauth/hapi_interactor.py` | +| Direct connections | `aperture_token_service/token/direct_connection/` | +| Device connections | `aperture_token_service/oauth/device_interactor.py` | +| Token refresh | `aperture_token_service/oauth/utils.py` (lines ~800+) | +| Constants | `aperture_token_service/commons/constants.py`, `oauth/constants.py` | +| Models | `aperture_token_service/token/models.py`, `oauth/models.py` | +| FHIR subscriptions | `aperture_token_service/commons/subscription_util.py` | +| Kafka events | `aperture_token_service/commons/utils.py` | + +--- + +## Error Codes Quick Reference + +### GraphQL Errors + +| Exception Class | HTTP Status | FHIR Code | When It Occurs | +|----------------|-------------|-----------|----------------| +| `BadValueError` | 400 | `value` | Invalid input | +| `AuthorizationError` | 401 | `security` | Invalid credentials, expired JWT | +| `ForbiddenError` | 403 | `forbidden` | Access denied | +| `NotFoundError` | 404 | `not-found` | Resource not found | +| `InternalServerError` | 500 | `exception` | Server error, third-party failure | +| `GatewayTimeoutError` | 504 | `timeout` | External provider timeout | + +### Token Refresh Process Codes + +| Code | Meaning | Status Change | +|------|---------|---------------| +| `REFRESHED` | Success | Token updated | +| `REFRESH_ERROR` | Provider rejected refresh | Token marked EXPIRED | +| `CLIENT_REFRESH_ERROR` | Transient provider error | Status unchanged (retry later) | +| `REFRESH_NOT_REQUIRED` | Token not yet expired | No action | +| `MISSING_REFRESH_TOKEN` | No refresh_token stored | Skip | +| `MISSING_CONNECTION_DATA` | Token lacks required fields | Skip | + +### Error Messages That Trigger EXPIRED Status + +When provider returns these during refresh → token marked EXPIRED: +- "Invalid refresh token" / "invalid_grant" / "Invalid grant" +- "Invalid Credentials" / "Invalid authorization" +- "The refresh token is invalid or has expired" / "Refresh token expired" +- "Invalid or expired refresh token" / "The refresh token is no longer active" +- "unknown, invalid, or expired refresh token" +- "User data access grant expired" / "Authentication Failed" / "Patient ID not found" + +HTTP 400, 401, 403 from provider also trigger EXPIRED. + +--- + +## Connection Statuses + +| Token Status | User-Facing Status | Meaning | +|-------------|-------------------|---------| +| `NEW` | CONNECTED | Just created | +| `RETRIEVING_DATA` | CONNECTED | Syncing | +| `DATA_RETRIEVED` | CONNECTED | Data synced | +| `EXPIRED` | EXPIRED | Token expired | +| `DISCONNECTED` | DISCONNECTED | User disconnected | +| `ACCESS_ENDED` | ACCESS_ENDED | Consent revoked | +| `DELETED` | DELETED | User deleted | +| `DATA_DELETED` | DELETED | Data deletion done | + +--- + +## Debugging Checklist + +1. **Check logs** — Use groundcover with structured attributes first, fall back to content search +2. **Check traces** — For latency, 5xx, or "did the provider call back?" questions +3. **Check events** — For pod crashes, OOMs, restarts +4. **Check token status in MongoDB** — `Token` collection stores connection state (AES-encrypted fields) +5. **Check FHIR resources** — SubscriptionStatus tracks sync state +6. **Check Connection Hub** — Provider config (client_id, URLs, scopes) from external config service +7. **Check environment** — Key env vars: `MONGO_*`, `KAFKA_*`, JWT JWKS URIs, `FHIR_*`, `AES_SECRET_KEY` + +--- + +## Further Reading + +For deeper dives, read these reference files: +- `references/logging-patterns.md` — Complete map of all ATS log statements with funcNames, content patterns, structured attributes, and workloads +- `references/groundcover-queries.md` — gcQL query patterns, gotchas, parsing large results, and investigation workflows +- `references/error-codes.md` — Complete error code reference with source file locations +- `references/connection-flows.md` — Detailed connection flow diagrams and state machines diff --git a/.claude/skills/connection-observability/references/connection-flows.md b/.claude/skills/connection-observability/references/connection-flows.md new file mode 100644 index 00000000..06cd611d --- /dev/null +++ b/.claude/skills/connection-observability/references/connection-flows.md @@ -0,0 +1,196 @@ +# ATS Connection Flows — Detailed Reference + +## Connection Creation Flows + +### 1. OAuth Connection (INDIRECT_IAS / PROA) + +**Entry point:** `GET /api/v1.0/oauth/generate_url//[]` + +``` +Client → generate_url (JWT required) + → Fetch provider config from Connection Hub + → Create OAuthCallback in MongoDB (state, csrf, jwt, metadata) + → Return authorize URL + +Client → Opens authorize URL at provider +Provider → Redirects to callback with ?state=...&code=... + +ATS callback: + → Decode base64 state → { csrf_token, id } + → Load OAuthCallback by id (and delete it) + → Verify CSRF + JWT + → Exchange code for tokens at provider's token_url + → Encrypt and store Token in MongoDB + → Create FHIR Subscription resources + → Emit TOKEN_CREATED Kafka event + → Redirect to portal with status_code +``` + +### 2. HAPI Clinical Connection (GraphQL) + +**Entry point:** GraphQL mutation `create_connection(connection_id, username, password)` + +``` +GraphQL resolver → create_connection + → Determine integration_type from Connection Hub config + → If DIRECT → delegate to create_direct_connection() + → Otherwise → delegate to hapi_interactor.connect_to_data_source() + +connect_to_data_source(): + → Get human_id from third-party integrator + → Call HAPI with credentials (username/password) + → Get connection result code (Success, AuthError, TFA, etc.) + → If Success: + → Store encrypted token + → Create FHIR subscription resources + → Emit TOKEN_CREATED Kafka event + → Return Connection object + → If Error: + → Map HumanAPIConnectionCode to appropriate GraphQL error + → Raise error (AuthorizationError, BadValueError, etc.) +``` + +### 3. Direct Connection + +**Entry point:** GraphQL mutation `create_connection(connection_id)` where connection_id matches a registered direct provider + +``` +GraphQL resolver → create_connection + → Detect DIRECT integration_type + → create_direct_connection(connection_id, member context) + → Look up DirectConnectionProvider by connection_id + → provider.process(): + → Create FHIR Subscription resource + → Create FHIR SubscriptionTopic resource + → Create FHIR SubscriptionStatus resource (with extensions for metadata) + → Emit RESOLVE_PATIENT Kafka event + → Return DirectConnectionResult(status=CONNECTED, sync_status=PENDING) +``` + +### 4. Device Connection + +**Entry point:** `GET /api/v1.0/oauth/generate_url//device` + +``` +Client → generate_url with category=device + → Fetch device marketplace config + → Generate signed state with HMAC + → Build marketplace URL with state + → Return marketplace URL + +User → Connects device in marketplace +Marketplace → Redirects to ATS callback + +ATS device callback: + → Validate state signature (HMAC) + → Validate state timestamp (not expired, not future) + → Extract user context from state + → Store device connection token + → Emit TOKEN_CREATED event +``` + +## Connection Disconnection Flow + +**Entry points:** +- GraphQL: `disconnect_connection(connection_id)` +- REST: `POST /token/disconnect_user_token/` + +``` +disconnect_token(): + → Load token by connection_id/service_slug + member context + → Set status = DISCONNECTED + → Clear refresh_token and token fields + → For device connections only: attempt revoke at provider (best-effort) + → Emit CONNECTION_STATUS Kafka event + → Return updated status +``` + +## Connection Deletion Flow + +**Entry points:** +- GraphQL: `delete_connection(connection_id)` +- REST: `DELETE /token/delete_user_token/` + +``` +request_delete_user_token(): + → Load token + → For device connections: call _revoke_device_connection() (best-effort) + → Set status = DELETED + → Clear token fields + → Emit DATA_CONNECTION_DELETED Kafka event + → Update FHIR SubscriptionStatus + → Return deleted status +``` + +## Direct Connection Activation + +**Entry point:** GraphQL mutation `activate_direct_connection(connection_id)` + +``` +activate_direct_connection(): + → Load SubscriptionStatus from FHIR + → If not found → raise NotFoundError + → Check if in DELETING state → raise BadValueError ("deletion in progress") + → Check FHIR Consent resource for deny status → raise BadValueError ("consent revoked") + → Update SubscriptionStatus: token-status = CONNECTED + → If was DATA_DELETED: reset data-connection-status + → Emit DIRECT_CONNECTION_REACTIVATED Kafka event + → Return activation confirmation +``` + +## Token Refresh Flow + +**Trigger:** Periodic refresh via `start_refresh_tokens()` or CLI command + +``` +start_refresh_tokens(): + → Query tokens where: + - status NOT IN (DISCONNECTED, DELETED, DATA_DELETED) + - has refresh_token (non-empty) + - expiry approaching or expired + → For each token: + → refresh_oauth_token(): + → Build refresh request (refresh_token, client_id, client_secret) + → POST to provider's token_url + → If success (handle_token_refresh_response): + → Update access_token + → Update refresh_token (if new one provided) + → Recalculate expiry + → Emit TOKEN_REFRESHED event + → If error (handle_token_refresh_error): + → Check error message/status against known patterns + → If terminal error → mark EXPIRED (REFRESH_ERROR) + → If transient error → leave status (CLIENT_REFRESH_ERROR) +``` + +## State Transitions + +``` +NEW → RETRIEVING_DATA → DATA_RETRIEVED (happy path) + ↗ ↓ +Any status → EXPIRED (refresh failure) +Any status → DISCONNECTED (user action) +Any status → DELETED → DATA_DELETED (user action) +Any status → ACCESS_ENDED (consent revoked) +DISCONNECTED/EXPIRED → reconnect via new OAuth flow → NEW +DELETED (direct) → activate_direct_connection → CONNECTED (if consent valid) +``` + +## Key Database Fields (Token Model) + +| Field | Description | +|-------|-------------| +| `member_id` | Member identifier | +| `client_fhir_person_id` | Client's FHIR Person ID | +| `bwell_fhir_person_id` | b.well's FHIR Person ID | +| `ch_service_id` / `service_slug` | Connection identifier | +| `token` | Encrypted access token | +| `refresh_token` | Encrypted refresh token | +| `token_expiry` | Token expiration timestamp | +| `status` | Current connection status | +| `fhir_url` | Provider's FHIR endpoint | +| `fhir_version` | FHIR version (R4, etc.) | +| `integration_type` | DIRECT, PROA, INDIRECT_IAS, IAL2 | +| `interop_type` | Interop classification | +| `source_id_prefix` | Namespace for FHIR resources | +| `categories` | Data source categories | diff --git a/.claude/skills/connection-observability/references/error-codes.md b/.claude/skills/connection-observability/references/error-codes.md new file mode 100644 index 00000000..235554c8 --- /dev/null +++ b/.claude/skills/connection-observability/references/error-codes.md @@ -0,0 +1,92 @@ +# ATS Error Codes — Complete Reference + +## Source File Locations + +| File | What It Contains | +|------|-----------------| +| `aperture_token_service/graphql/error_handler.py` | GraphQL error classes, OperationOutcomeCode enum, ATSErrorCode enum | +| `aperture_token_service/commons/constants.py` | Error message constants, HTTP exception tuple | +| `aperture_token_service/oauth/constants.py` | ConnectionCodes, DeviceConnectionCodes, RefreshProcessCodes | +| `aperture_token_service/token/status.py` | Token Status enum | +| `aperture_token_service/token_service_exception.py` | Base TokenServiceException class | +| `aperture_token_service/oauth/jwt_validator.py` | JwtValidatorException | +| `aperture_token_service/oauth/jwks_validation.py` | JwksUrlValidationError | + +## OperationOutcome Response Format + +GraphQL errors return FHIR OperationOutcome: + +```json +{ + "resourceType": "OperationOutcome", + "issue": [ + { + "severity": "error", + "code": "", + "details": { + "text": "", + "coding": [ + { + "code": "", + "display": "" + } + ] + } + } + ] +} +``` + +## HTTP Status to FHIR Code Mapping + +```python +OPERATION_OUTCOME_CODE_MAP = { + 400: "value", + 401: "security", + 403: "forbidden", + 404: "not-found", + 500: "exception", + 504: "timeout", +} +``` + +## Token Refresh Error Patterns + +The function `handle_token_refresh_error()` in `oauth/utils.py` determines whether to mark a token as EXPIRED based on: + +1. **HTTP Status**: 400, 401, 403 → EXPIRED +2. **Error message contains** (case-insensitive matching on response body): + - "Invalid refresh token" + - "invalid_grant" + - "Invalid grant" + - "Invalid Credentials" + - "Invalid authorization" + - "The refresh token is invalid or has expired" + - "Refresh token expired" + - "Invalid or expired refresh token" + - "The refresh token is no longer active" + - "refresh token is invalid" + - "unknown, invalid, or expired refresh token" + - "User data access grant expired" + - "Authentication Failed" + - "Patient ID not found" + +When these patterns match → `RefreshProcessCodes.REFRESH_ERROR` → Token status set to EXPIRED. + +For other errors (e.g., 500 from provider, network timeout) → `RefreshProcessCodes.CLIENT_REFRESH_ERROR` → Token status unchanged (will retry on next refresh cycle). + +## HTTP Exception Handling + +The unified handler `commons/utils.py::handle_http_exception()`: + +- `requests.exceptions.Timeout` / `httpx.TimeoutException` → logs timeout, returns 504 +- `requests.HTTPError` / `httpx.HTTPStatusError` → logs with status code and response body, returns the status code +- All other `requests.RequestException` / `httpx.HTTPError` → logs generic error + +## Flask Error Handlers (REST API) + +Registered in `app.py`: +- 401 → renders `401.html` template +- 404 → renders `404.html` template +- 500 → renders `500.html` template +- `TokenServiceException` → renders `token_service_exception.html` with error details, returns 500 diff --git a/.claude/skills/connection-observability/references/groundcover-queries.md b/.claude/skills/connection-observability/references/groundcover-queries.md new file mode 100644 index 00000000..4df7e391 --- /dev/null +++ b/.claude/skills/connection-observability/references/groundcover-queries.md @@ -0,0 +1,466 @@ +# Groundcover Query Patterns for ATS + +## MCP Tools Reference + +All groundcover queries are executed via MCP tools. Choose the right tool for your investigation: + +| Tool | Purpose | Key Parameters | +|------|---------|----------------| +| `mcp__groundcover__query_logs` | Log investigation (errors, refresh, OAuth, user activity) | `query`, `period`, `start`, `end` | +| `mcp__groundcover__query_traces` | Latency, 5xx errors, verifying callbacks, full URL inspection | `query`, `period`, `start`, `end` | +| `mcp__groundcover__query_events` | Pod crashes, OOMKills, restarts, K8s events | `query`, `period`, `start`, `end` | +| `mcp__groundcover__query_monitors` | List monitor definitions and their health | `query`, `limit`, `skip` | +| `mcp__groundcover__query_issues` | Active/historical alert firings | `query`, `period`, `start`, `end` | + +### Parameter Notes + +- **`query`** (required): gcQL query string. Must start with `*` or a filter. Always include `| limit N`. +- **`period`**: ISO 8601 duration. Tool default is `PT1H`. **Always override to `P7D`** for ATS investigation. +- **`start`** / **`end`**: RFC3339 timestamps. Use instead of `period` for specific time ranges. +- Time range is set via parameters, NOT in the query string itself. + +## gcQL Syntax Quick Reference + +- Queries must start with a filter or `*` (match-all) +- Stats grouping: `stats by (field) count()` (NOT `stats count() by field`) +- Sort syntax: `sort by (field desc)` (NOT `sort field desc`) +- Numeric comparisons: NO colon — use `duration_seconds>1` not `duration_seconds:>1` +- Always use `_time` for timestamps +- Always include `| limit N` to bound results + +## ATS Service Identifiers (CONFIRMED) + +ATS has **TWO separate workloads** in groundcover: +- **`aperture-token-service`** — the main API server (handles GraphQL, OAuth callbacks, REST endpoints) +- **`aperture-token-service-pipelines`** — the batch pipeline (handles token refresh, scheduled jobs) + +**CRITICAL**: Token refresh logs (`process_refresh_tokens_results`, `handle_token_refresh_error`, `handle_token_refresh_response`) are logged by the **pipelines** workload, NOT the main API workload. + +Workload filter usage: +- **API issues**: `workload:aperture-token-service` +- **Refresh/pipeline issues**: `workload:aperture-token-service-pipelines` +- **Both**: omit workload filter, or use `workload:*aperture-token-service*` +- **Wildcard alternative**: `workload:*aperture-token*` (catches both) +- For traces: `service.name:aperture-token-service` + +Environments confirmed in logs: +- `env:production` (cluster: `prod-ue1`) +- `env:client-sandbox` (cluster: `client-sandbox-ue1`) + +## Structured Attribute Filters (CRITICAL) + +ATS logs emit structured `extra` fields that become **directly filterable attributes** in groundcover. These are MORE reliable than `content:*...*` wildcard matching. + +### Available structured attribute filters: + +| Attribute | Where It's Logged | Example | +|-----------|-------------------|---------| +| `client_fhir_person_id` | refresh results, callback errors | `client_fhir_person_id:83e2d2c3-77d6-461e-83c0-5f8c565830bc` | +| `service_slug` | refresh results, callback errors | `service_slug:evgh` | +| `member_id` | refresh results, callback errors | `member_id:` | +| `bwell_fhir_person_id` | refresh results | `bwell_fhir_person_id:` | +| `funcName` | all structured logs | `funcName:process_refresh_tokens_results` | +| `code` | refresh results | `code:RefreshError` | +| `status` | refresh results | `status:Expired` | +| `prev_status` | refresh results | `prev_status:Data Retrieved` | + +### How to use structured attributes: + +```gcql +# Find refresh results for a specific user + service_slug +* | filter client_fhir_person_id: service_slug: | sort by (_time desc) | limit 20 + +# Find all refresh errors +* | filter workload:aperture-token-service-pipelines funcName:handle_token_refresh_error | sort by (_time desc) | limit 20 + +# IMPORTANT: Do NOT combine workload:aperture-token-service with these attributes for refresh logs +# Refresh logs come from workload:aperture-token-service-pipelines +# Either omit the workload filter or use the pipelines workload +``` + +### Gotcha: workload filter + attribute filter combination + +If `client_fhir_person_id:` returns empty with `workload:aperture-token-service`, try: +1. Remove the workload filter entirely (attributes work across workloads) +2. Use `workload:aperture-token-service-pipelines` for refresh-related queries + +## Critical Query Gotchas (Learned from Experience) + +### 1. Queries with too many `content:*...*` filters often FAIL +- **BAD**: `* | filter workload:aperture-token-service content:*cde27c3f* content:*error*` — tends to fail +- **GOOD**: `* | filter workload:aperture-token-service content:*cde27c3f* level:error` — use `level:` field instead of content matching for log level +- **GOOD**: `* | filter workload:aperture-token-service content:*cde27c3f*` — then filter results programmatically + +### 2. Use `level:error` field filter, NOT `content:*error*` +- The `level` field is indexed and reliable: `level:error`, `level:info`, `level:warning` +- Content matching for "error" picks up false positives (URLs with "error" in them, etc.) + +### 3. Large results get saved to files — ALWAYS parse them with Python/jq +- When results exceed token limits, they're saved to a `.txt` file in JSON format +- The file format is: `[{"type": "text", "text": "[...actual JSON array of log entries...]"}]` +- Parse with: `json.loads(data[0]['text'])` to get the inner array +- Each log entry has: `timestamp`, `level`, `env`, `body` (raw JSON string), `content` (text summary) +- The `body` field contains the structured JSON log — parse it for `message`, `funcName`, extra fields + +### 4. Keep queries simple — one or two content filters max +- Queries with 3+ `content:*...*` filters frequently fail with "failed to query logs" +- If you need complex filtering, fetch broader results and filter programmatically + +### 5. Results are limited — use `| limit N` wisely +- Default limit if not specified varies; always specify explicitly +- For investigation, `limit 20-30` is usually sufficient +- For time-based analysis, combine with `period` parameter + +## Recommended Investigation Approach + +**Default time range**: Always set `period: "P7D"` in the MCP tool call unless investigating a specific recent incident. + +**Tool selection**: +- Start with `mcp__groundcover__query_logs` for all log-based investigation +- Switch to `mcp__groundcover__query_traces` for latency/5xx/callback verification +- Use `mcp__groundcover__query_events` only for pod-level health issues + +### For a specific user/connection issue: + +**Step 1**: Use structured attribute filter for the user's `client_fhir_person_id` (most reliable): +```gcql +* | filter client_fhir_person_id: | sort by (_time desc) | limit 30 +``` +Note: Omit workload filter here — it catches logs from both API and pipelines workloads. + +**Step 2**: If Step 1 returns empty or you need API-level activity (GraphQL calls, OAuth callbacks), use content search: +```gcql +* | filter workload:aperture-token-service content:** | sort by (_time desc) | limit 30 +``` + +**Step 3**: If results are large, parse the saved file and look for: +- `funcName: "handle_callback_generic"` — shows OAuth callback results (success/failure) +- `funcName: "before_request"` — shows what API calls the user made +- `funcName: "fetch_batch"` — shows SubscriptionStatus lookups +- `funcName: "make_request_with_retries"` — shows Connection Hub config fetches +- `funcName: "process_refresh_tokens_results"` — shows refresh outcomes +- `funcName: "handle_token_refresh_error"` — shows refresh failures +- `level: "error"` entries — actual errors + +**Step 4**: For error details, query with `level:error`: +```gcql +* | filter workload:aperture-token-service content:** level:error | sort by (_time desc) | limit 10 +``` + +**Step 5**: For service_slug-specific issues (use structured attribute): +```gcql +* | filter service_slug: | sort by (_time desc) | limit 20 +``` + +**Step 6**: For refresh-specific investigation: +```gcql +* | filter client_fhir_person_id: service_slug: | sort by (_time desc) | limit 20 +``` + +### Key log patterns to look for in ATS: + +| Log Pattern | What It Means | +|-------------|---------------| +| `handle_callback_generic` + `error=access_denied` | Provider rejected the OAuth request | +| `handle_callback_generic` + `code=...` | Successful OAuth callback | +| `[EXTRACT_PATIENT_ID] Unable to get patient_id` | Token response missing patient identifier | +| `make_request_with_retries` + `status_code: 200` | Connection Hub config fetched OK | +| `fetch_batch` + `SubscriptionStatusServiceHelper` | Looking up connection status | +| `getOauthUrl` in kwargs | User initiated a connection flow | +| `getDataSource` in query body | User checking connection details | +| `get_member_connections` | User listing their connections | +| `error_description=Policy+evaluation+failed` | Provider-side policy rejection | +| `error_description=User+refused` | User cancelled the OAuth flow | + +### Parsing the `body` field: + +ATS logs are JSON-structured. The `body` field in each log entry is a JSON string with these useful fields: +```json +{ + "funcName": "handle_callback_generic", + "pathname": "/app/aperture_token_service/oauth/views.py", + "message": "the actual log message", + "name": "aperture_token_service.oauth.interactor", + "service_slug": "example_provider", + "connection_version": 1, + "otelTraceID": "...", + "level": "ERROR" +} +``` + +Extra fields beyond the standard ones are **context-specific** and very useful (e.g., `service_slug`, `connection_version`, `client_slug`). + +### For the `before_request` middleware logs: + +The request body is logged in the message for GraphQL calls: +``` +[BEFORE_REQUEST] Started: POST /graphql b'{"variables":{"connectionId":"..."},"query":"query(...){...}"}' +``` + +This tells you exactly which GraphQL operation the user called. + +## Common Investigation Scenarios (all use `mcp__groundcover__query_logs` with `period: "P7D"` unless noted) + +### 1. "User can't connect to a provider" + +```gcql +# Get all activity for the user +* | filter workload:aperture-token-service content:** | sort by (_time desc) | limit 30 + +# Check for OAuth callback errors (provider rejections) +* | filter workload:aperture-token-service content:** content:*callback* | sort by (_time desc) | limit 10 +``` + +Common provider errors in callbacks: +- `error=access_denied&error_description=Policy+evaluation+failed` — provider policy rejection +- `error=access_denied&error_description=User+refused` — user cancelled +- `error=server_error` — provider internal error + +### 2. "Why did this connection expire?" + +```gcql +* | filter workload:aperture-token-service content:*EXPIRED* | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*refresh* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:** level:error | sort by (_time desc) | limit 30 +``` + +Look for: "Invalid refresh token", "invalid_grant", "Refresh token expired", HTTP 401/403 from provider. + +### 3. "Connection creation failed" + +```gcql +* | filter workload:aperture-token-service content:*create_connection* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*connect_to_data_source* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*AuthenticationError* | sort by (_time desc) | limit 30 +``` + +Look for: INVALID_CREDENTIALS, TFA_REQUIRED, THIRD_PARTY_ERROR, PROFILE_SELECTION_REQUIRED. + +### 4. "OAuth flow is broken" + +```gcql +* | filter workload:aperture-token-service content:*callback* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*generate_url* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*CSRF* | sort by (_time desc) | limit 20 +``` + +Look for: CSRF mismatches, expired JWTs, missing portal_redirect_url, provider errors. + +### 5. "Service is returning 5xx errors" + +```gcql +# Check error logs (mcp__groundcover__query_logs) +* | filter workload:aperture-token-service level:error | sort by (_time desc) | limit 50 + +# Check traces for 5xx (mcp__groundcover__query_traces) +* | filter service.name:aperture-token-service http.status_code:5* | sort by (_time desc) | limit 50 + +# Check for OOM/crashes (mcp__groundcover__query_events) +* | filter involved_object.name:*aperture-token* type:Warning | sort by (_time desc) | limit 10 +``` + +### 6. "Token refresh is failing" (use pipelines workload!) + +```gcql +# For a specific user's refresh results (PREFERRED — use structured attributes): +* | filter client_fhir_person_id: service_slug: | sort by (_time desc) | limit 20 + +# For global refresh errors: +* | filter workload:aperture-token-service-pipelines level:error | sort by (_time desc) | limit 50 + +# For specific refresh error function: +* | filter workload:aperture-token-service-pipelines funcName:handle_token_refresh_error | sort by (_time desc) | limit 30 + +# For refresh results by service_slug: +* | filter service_slug: funcName:process_refresh_tokens_results | sort by (_time desc) | limit 20 +``` + +**Known issue — Race condition in refresh**: If you see two `process_refresh_tokens_results` logs for the same user within seconds (one `Refreshed`, one `RefreshError`), this is a known race condition where two refresh calls use the same stale refresh_token. The first call succeeds and rotates the token; the second fails because the old refresh_token is now invalid. Look for timestamps within 1-5 seconds of each other. + +### 7. "Device connection issues" + +```gcql +* | filter workload:aperture-token-service content:*device* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*marketplace* level:error | sort by (_time desc) | limit 30 +``` + +### 8. "Direct connection issues" + +```gcql +* | filter workload:aperture-token-service content:*direct_connection* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*activate* level:error | sort by (_time desc) | limit 30 +``` + +### 9. "FHIR/Subscription issues" + +```gcql +* | filter workload:aperture-token-service content:*subscription* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*fhir* level:error | sort by (_time desc) | limit 30 +``` + +### 10. "JWT/Auth issues" + +```gcql +* | filter workload:aperture-token-service content:*JWT* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*JWKS* level:error | sort by (_time desc) | limit 30 +* | filter workload:aperture-token-service content:*Forbidden* | sort by (_time desc) | limit 30 +``` + +### 11. "OAuth flow — provider never calls back" (use traces!) + +When a user initiates an OAuth connection but it never completes, the provider may not be calling back to ATS. + +**Step 1: Check if getOauthUrl succeeded** (`mcp__groundcover__query_logs`, period: P7D) +```gcql +# The @commons_logging.log decorator logs getOauthUrl with funcName:wrapper +* | filter workload:aperture-token-service content:** content:*generate_url* | sort by (_time desc) | limit 10 +``` +The decorator log has `funcName: wrapper` and the body contains `{"function": "...", "kwargs": {...}, "return": {...}}` — the `return` field shows the generated OAuth URL. + +**Step 2: Check for callback traces** (`mcp__groundcover__query_traces`, period: P7D) +```gcql +# Look for ANY callback to ATS for this service_slug +* | filter service.name:aperture-token-service http.target:*callback* | sort by (_time desc) | limit 20 +``` +If the `resource_name` field in trace results does NOT contain the service_slug's callback within the expected time window (minutes to hours after getOauthUrl), the provider never called back. + +**Step 3: Check resource_name for the authorize URL details** (`mcp__groundcover__query_traces`) +```gcql +# Traces show full URLs in resource_name — verify client_id, redirect_uri, PKCE params +* | filter service.name:aperture-token-service resource_name:** | sort by (_time desc) | limit 10 +``` + +**Step 4: Diagnosis** +- No callback trace → issue is at provider side (revoked client_id, redirect_uri mismatch, PKCE failure, user abandoned) +- Callback trace exists but with error → check logs for `handle_callback_generic` with `level:critical` +- Check if existing refresh tokens also fail → `* | filter service_slug: funcName:handle_token_refresh_error | sort by (_time desc) | limit 10` + +### 12. "Who is calling this endpoint?" / "Find source of concurrent requests" (trace origin) + +When you see duplicate/concurrent requests (e.g., race conditions) or unexpected API calls, trace back to the origin: + +**Step 1: Get the trace ID from the log** + +ATS logs include `otelTraceID` in the structured body. Find it in the log entry for the event you're investigating. + +**Step 2: Find the root span** (`mcp__groundcover__query_traces`) +```gcql +# Jump directly to the root span — this is the entry point +* | filter trace_id: is_root_span:true | limit 5 +``` + +**Step 3: Read the root span attributes** + +The root span is always a traefik `EntryPoint` (kind:server). Key fields: +- `client.address` — IP of the calling pod (identifies which service/pod made the request) +- `user_agent.original` — calling framework (e.g., `Python/3.12 aiohttp/3.13.3` = another b.well service) +- `server.address` — target host (e.g., `aperture-token-service-pipelines.prod.bwell.zone`) +- `url.path` — endpoint called (e.g., `/api/v1.0/refresh-tokens`) +- `http.request.body.size` — request body size (same size from different IPs = same payload) + +**Step 4: Compare origins for concurrent requests** + +If two log entries for the same token have different `otelTraceID`s, find both root spans and compare: +- Different `client.address` + same `user_agent` = same external service with multiple replicas calling independently +- Same `client.address` = same caller pod made duplicate requests (possible retry logic) + +**Example: Diagnosing refresh race condition** +```gcql +# 1. Find the two conflicting refresh results +* | filter client_fhir_person_id: service_slug: funcName:process_refresh_tokens_results | sort by (_time desc) | limit 10 + +# 2. Extract otelTraceID from each (in body field), then: +* | filter trace_id: is_root_span:true | limit 5 +* | filter trace_id: is_root_span:true | limit 5 + +# 3. Compare client.address — if different IPs, the calling service has multiple replicas +# that aren't coordinating their refresh requests +``` + +**Fallback: If `is_root_span:true` returns empty**, walk up manually: +```gcql +# Filter for the workload span, then follow parent_id up +* | filter trace_id: workload:aperture-token-service-pipelines | sort by (_time) | limit 5 +# Take the parent_id from the top-level span, then: +* | filter span_id: | limit 5 +# Repeat until you reach parent_id:"" (the root) +``` + +### 13. "OAuth callback crashes with 'str' object has no attribute 'get'" (authlib bug) + +This is a known bug where authlib returns a string from `fetch_access_token()` when the provider's token endpoint returns HTTP 400 with non-standard JSON body. + +**Step 1: Find affected callbacks** (`mcp__groundcover__query_logs`, period: P7D) +```gcql +* | filter workload:aperture-token-service level:critical content:*Unable to authenticate request passed to callback* | sort by (_time desc) | limit 30 +``` + +**Step 2: Parse results to identify which have the 'str' bug vs other errors** +```python +# In the body JSON, look for: +# - "error": "'str' object has no attribute 'get'" → this bug +# - "query_params": ["code", "state"] → valid callback (not provider rejection) +# Provider rejections have query_params with "error" key instead +``` + +**Step 3: Confirm via traces** (`mcp__groundcover__query_traces`) +```gcql +# Use otelTraceID from the error log entry +* | filter trace_id: | limit 20 +``` +Look for a span with `resource_name:*/token*` and check `http.status_code`. If it's 400, this confirms the bug. + +**Step 4: Identify affected providers** +Group by `service_slug` in the log body to see which providers are affected. + +**Root cause**: Provider returns 400 with body like `"invalid_grant"` (JSON string, not object). Authlib doesn't raise because status < 500, and `'error' in "invalid_grant"` is True for strings (substring check), but if body is e.g. `"some_opaque_value"` without "error" substring, it returns the string as the token. + +--- + +## Time Range Patterns + +| Scenario | Period Parameter | +|----------|----------------| +| Last 15 minutes | `PT15M` | +| Last hour (default) | `PT1H` | +| Last 6 hours | `PT6H` | +| Last 24 hours | `PT24H` | +| Last 7 days | `P7D` | +| Last 30 days | `P30D` | +| Specific range | Use `start` and `end` in RFC3339 format | + +## Processing Large Results + +When groundcover returns results too large for context, they're saved to a file. Use this Python pattern to parse: + +```python +import json, sys +raw = sys.stdin.read() +data = json.loads(raw) +# Unwrap the [{type, text}] envelope +if isinstance(data, list) and len(data) > 0 and isinstance(data[0], dict) and 'text' in data[0]: + inner = json.loads(data[0]['text']) +else: + inner = data +for item in inner: + ts = item.get('timestamp', '') + level = item.get('level', '') + env = item.get('env', '') + try: + body = json.loads(item.get('body', '{}')) + msg = body.get('message', '') + func = body.get('funcName', '') + # Extract extra context fields + extra_keys = [k for k in body.keys() if k not in ('lineno','funcName','pathname','message','name','taskName','otelSpanID','otelTraceID','otelTraceSampled','otelServiceName','timestamp','level')] + print(f'{ts} [{level}] [{func}] {msg[:300]}') + except: + print(f'{ts} [{level}] {item.get("content", "")[:300]}') +``` + +## What ATS Does NOT Log (Known Gaps) + +- **Extracted patient_id values** — only logs when empty, not when malformed +- **Raw token response from providers** — masked for security +- **Decrypted token values** — never logged +- **Full OAuth provider responses** — only status codes logged +- The `response_data` field stored in MongoDB contains the raw token response but is not logged diff --git a/.claude/skills/connection-observability/references/logging-patterns.md b/.claude/skills/connection-observability/references/logging-patterns.md new file mode 100644 index 00000000..62ad58b1 --- /dev/null +++ b/.claude/skills/connection-observability/references/logging-patterns.md @@ -0,0 +1,353 @@ +# ATS Logging Patterns — Complete Reference + +## Log Architecture + +ATS has two workloads that emit logs to groundcover: +- **`aperture-token-service`** — main Flask API (handles requests, callbacks, GraphQL) +- **`aperture-token-service-pipelines`** — batch pipeline (token refresh, scheduled jobs) + +## Log Structure + +Every log entry in groundcover has: +- `timestamp` — when it was emitted +- `level` — info, error, warning, critical +- `workload` — which deployment emitted it +- `content` — the log message text (searchable with `content:*...*`) +- `body` — full JSON-structured log (parse for details) +- `string_attributes` — structured extra fields (directly filterable!) +- `float_attributes` — numeric extra fields + +## Logging Layers (in order of request processing) + +### 1. WSGI Middleware (`middleware.py`) +- **What**: Logs raw environ dict for every non-excluded request +- **Level**: info +- **funcName**: `_log` +- **Content pattern**: Raw WSGI environ (HTTP headers, method, path) +- **Not useful for debugging** — too low-level, no structured attributes + +### 2. Before/After Request Interceptor (`app.py:261-289`) +- **What**: Logs start/end of every HTTP request with timing + +**BEFORE_REQUEST** (line 268): +- **Content**: `[BEFORE_REQUEST] Started: {method} {path} {request_id} {request_body}` +- **funcName**: `before_request` +- **Structured attributes**: None (just content) +- **Very useful**: Shows GraphQL operation names in body, REST endpoint paths, request IDs + +**AFTER_REQUEST** (line 282): +- **Content**: `[AFTER_REQUEST] Completed: {method} {path} {request_id} Status: {status_code} Elapsed: {time}s` +- **funcName**: `log_request` +- **Very useful**: Shows response status codes and request duration + +### 3. `@commons_logging.log(logger)` Decorator (`commons/logging.py`) +- **What**: Wraps REST view functions, logs function path + optional args/return +- **funcName**: `wrapper` +- **Content**: A dict like `{"function": "aperture_token_service.token.views.get_all_tokens", "kwargs": {...}, "return": {...}}` +- **Applied to**: Most REST endpoints in `token/views.py` +- **Useful for**: Seeing which REST endpoint was called and what it returned (masked) + +### 4. `make_request_with_retries` (`oauth/utils.py:140`) +- **What**: All requests to Connection Hub (integration hub) +- **funcName**: `make_request_with_retries` +- **Content**: `Response for url={endpoint}, params={params} is status_code: {code}` +- **Level**: info (success), info (retry/error) +- **Useful for**: Verifying Connection Hub config fetches work + +### 5. OAuth Flow Logs (`oauth/views.py`, `oauth/interactor.py`) + +**Callback Error — Provider rejection** (line 273): +- **funcName**: `handle_callback_generic` +- **Content**: `[HANDLE_CALLBACK_GENERIC] Error in query params` +- **Level**: critical +- **Structured attributes**: `client_fhir_person_id`, `member_id`, `service_slug`, `query_params.error`, `query_params.error_description` +- **When**: Provider redirects back with `?error=...` + +**Callback Error — No stored data** (line 259): +- **Content**: `[HANDLE_CALLBACK_GENERIC] No stored data found for the request` +- **Level**: critical +- **When**: State parameter doesn't match any OAuthCallback in MongoDB + +**Callback Error — Login failure** (line 311): +- **Content**: `[HANDLE_CALLBACK_GENERIC] Unable to authenticate request passed to callback` +- **Level**: critical +- **Structured attributes**: `service_slug`, `connection_version`, `bwell_fhir_person_id`, `client_fhir_person_id`, `query_params` (list of param names), `error` (exception message) +- **When**: `handle_login()` raises any exception. Common case: `'str' object has no attribute 'get'` — means authlib returned a string instead of dict from token exchange (provider returned 400 with non-standard body). Check trace for POST to token endpoint status code. +- **Distinguishing from provider rejections**: `query_params` contains `['code', 'state']` (valid callback with code) vs provider rejections where `query_params` has `error` key + +**Callback Success** (line 283): +- **Content**: `[HANDLE_CALLBACK_GENERIC] Received request at {url} for client_person: {id}, member_id: {id}, service_slug: {slug}` +- **Level**: info +- **funcName**: `handle_callback_generic` + +**Legacy callback** (line 167): +- **Content**: `[HANDLE_CALLBACK] OAuth callback data not found` +- **Level**: error + +**Token exchange** (line 850-863 in interactor.py): +- **Content**: `[OAUTH_INTERACTOR] POST Request to {token_url}` and `[OAUTH_INTERACTOR] {status} response with content...` +- **Level**: info +- **Only for**: `grant_type == "authorization_code"` (not refresh) + +**OAuth callback metadata failure** (interactor.py:126): +- **Content**: `[OAUTH_CALLBACK] Failed to upsert OAuth callback metadata` +- **Level**: error +- **Structured attributes**: `service_slug`, `bwell_fhir_person_id`, `client_fhir_person_id`, `category` + +### 6. Token Refresh Logs (`oauth/utils.py`) + +**All refresh logs use structured `extra` fields that become `string_attributes` in groundcover.** + +**Refresh success** (line 710): +- **funcName**: `handle_token_refresh_response` +- **Content**: `[REFRESH_TOKENS] Token refreshed successfully for user` +- **Level**: info +- **Attributes**: `member_id`, `service_slug`, `bwell_fhir_person_id`, `client_fhir_person_id`, `status`, `prev_status`, `connection_version` + +**Refresh error — terminal (token marked EXPIRED)** (line 872): +- **funcName**: `handle_token_refresh_error` +- **Content**: `[REFRESH_TOKENS] Error refreshing token, setting status to EXPIRED` +- **Level**: error (with exception traceback) +- **Attributes**: `exception`, `member_id`, `bwell_fhir_person_id`, `client_fhir_person_id`, `interop_type`, `service_slug`, `connection_version`, `error` (response text), `prev_status` + +**Refresh error — already expired** (line 867): +- **funcName**: `handle_token_refresh_error` +- **Content**: `[REFRESH_TOKENS] Token already expired` +- **Level**: info +- **Attributes**: same as above + +**Refresh error — transient (status unchanged)** (line 898): +- **funcName**: `handle_token_refresh_error` +- **Content**: `[REFRESH_TOKENS] Error refreshing token` +- **Level**: error (with exception traceback) +- **Attributes**: same as above +- **Code**: `CLIENT_REFRESH_ERROR` + +**Refresh result summary** (line 1108): +- **funcName**: `process_refresh_tokens_results` +- **Content**: `[REFRESH_TOKENS] Refresh token results:` +- **Level**: info +- **Attributes**: `code`, `service_slug`, `member_id`, `bwell_fhir_person_id`, `client_fhir_person_id`, `status`, `prev_status` +- **Code values**: Refreshed, RefreshError, RefreshNotRequired, ClientRefreshError, MissingRefreshToken, MissingConnectionData + +**Missing access_token in response** (line 726): +- **funcName**: `handle_token_refresh_response` +- **Content**: `[REFRESH_TOKENS] access_token not found in successful token refresh response, setting status to EXPIRED` +- **Level**: error +- **Attributes**: includes `token_response` dict + +**Empty refresh_token** (line 980): +- **funcName**: `refresh_oauth_token` +- **Content**: `[REFRESH_TOKENS] Empty refresh_token but token is not disconnected` +- **Level**: error +- **Attributes**: `service_slug`, `bwell_fhir_person_id`, `client_fhir_person_id`, `connection_version`, `token_status` + +**Unable to refresh expired non-disconnected token** (in refresh_tokens function): +- **funcName**: `refresh_tokens` +- **Content**: `[REFRESH_TOKENS] Unable to refresh expired token which is not disconnected` +- **Level**: error + +### 7. Patient ID Extraction (`oauth/utils.py:1530+`) + +**Patient ID empty** (line ~1545): +- **Content**: `[EXTRACT_PATIENT_ID] Unable to get patient_id from token response` +- **Level**: error +- **Note**: Only logs when patient_id is EMPTY, not when malformed + +### 8. GraphQL Resolver Logs + +**get_oauth_url** (oauth/graphql/resolvers.py): +- Errors at levels: warning (Kafka event fail), critical (portal URL fetch fail, portal URL not configured, URL generation fail, invalid response) +- **Content patterns**: `get_oauth_url: Failed to...`, `get_oauth_url: Portal redirect URL not configured...` + +**get_member_connections** (token/graphql/resolvers.py): +- Errors: warning (invalid status filters), critical (DB fetch fail, subscription status batch fail, token mapping fail) +- **Content patterns**: `get_member_connections: ...` + +### 9. Auth/JWT Validation (`oauth/auth_handler.py`) + +- **Content**: `Error in validating JWT token: Invalid key id: {kid}` +- **Level**: error +- **Content**: `Required items not found in auth token. Falling back to using id_token` +- **Level**: warning + +### 10. HTTP Exception Handler (`commons/utils.py:393`) + +- **Content**: `Timeout received during request` (for timeouts) +- **Content**: `Invalid status_code received during request` (for HTTP errors) +- **Level**: error +- **Attributes**: `status_code`, `response_error` + +### 11. Kafka Event Publishing (`commons/utils.py`) + +- **Content**: `Error occurred while publishing kafka event` +- **Level**: exception (with traceback) +- **Content**: `Failed to emit kafka event on token status update for service_slug: {slug}` + +### 12. Device Connection Logs (`oauth/device_interactor.py`) + +Has 31 log statements. Key patterns: +- Device connection creation/disconnection errors +- Marketplace state validation +- Device token storage + +### 13. HAPI/Clinical Connection Logs (`oauth/hapi_interactor.py`) + +12 log statements covering: +- connect_to_data_source errors +- HAPI token refresh +- Connection code mapping + +### 14. Direct Connection Logs (`token/direct_connection/`) + +- Direct connection provider-specific logging +- FHIR Subscription resource creation + +### 15. Subscription Status Helper (`commons/subscription_status_service_helper.py`) + +- **funcName**: `fetch_batch` +- Logs FHIR SubscriptionStatus fetch results + +## Traces in Groundcover + +ATS is instrumented with OpenTelemetry. Traces are available via `mcp__groundcover__query_traces`. + +**Service name**: `aperture-token-service` + +Key trace attributes: +- `http.status_code` — response status +- `http.method` — GET, POST, etc. +- `http.url` / `http.target` — the endpoint path +- `resource_name` — **IMPORTANT**: contains the FULL URL including query parameters (e.g., `GET https://provider.example.com/authorize?client_id=...&redirect_uri=...&code_challenge=...`) +- `duration_seconds` — request duration + +### Trace Investigation Patterns + +**OAuth flow investigation using traces:** + +The `resource_name` field is critical for OAuth debugging because it shows the full authorize URL with all params (client_id, redirect_uri, scopes, PKCE code_challenge). This helps verify: +- Correct client_id being sent +- Correct redirect_uri (must match provider's registered URI) +- Correct scopes requested +- PKCE parameters present (for providers that require it) + +```gcql +# Find slow requests +* | filter service.name:aperture-token-service duration_seconds>5 | sort by (duration_seconds desc) | limit 20 + +# Find 5xx errors +* | filter service.name:aperture-token-service http.status_code:5* | sort by (_time desc) | limit 50 + +# Find specific endpoint issues +* | filter service.name:aperture-token-service http.target:/graphql http.status_code:5* | sort by (_time desc) | limit 20 + +# Find callback traces (verify provider called back) +* | filter service.name:aperture-token-service http.target:*callback* | sort by (_time desc) | limit 20 + +# Find OAuth URL generation traces +* | filter service.name:aperture-token-service http.target:*generate_url* | sort by (_time desc) | limit 20 + +# Find callbacks for a specific service_slug (resource_name contains full URL with query params) +* | filter service.name:aperture-token-service resource_name:** http.target:*callback* | sort by (_time desc) | limit 20 +``` + +### Diagnosing "Provider Never Called Back" + +When investigating OAuth connections that don't complete: + +1. **Check if getOauthUrl succeeded** — look for `funcName:wrapper` logs with `getOauthUrl` in the content, or traces with `http.target:*generate_url*` +2. **Check for callback traces** — query traces with `http.target:*callback*` in the same time window +3. **If no callback trace exists** — the issue is at the provider side: + - Provider may have revoked the client_id + - redirect_uri mismatch in provider's config + - PKCE verification failure + - Provider internal error + - User abandoned the flow at the provider's login page +4. **Check the `resource_name` from getOauthUrl trace** — it shows the full authorize URL sent to the provider, which reveals what client_id, redirect_uri, and scopes were used + +### Tracing Request Origins (Who Called This Endpoint?) + +When you need to find what service or client triggered a request to ATS (e.g., concurrent refresh calls, unexpected API usage): + +**Step 1: Get the `otelTraceID` from the log entry** + +ATS logs include `otelTraceID` in structured body. Use it to query traces: +```gcql +* | filter trace_id: is_root_span:true | limit 5 +``` + +**Step 2: The root span reveals the caller** + +The root span (where `is_root_span:true` and `parent_id:""`) is always the **traefik-internal EntryPoint** — it contains: +- `client.address` — IP of the calling pod/service +- `user_agent.original` — framework used by caller (e.g., `Python/3.12 aiohttp/3.13.3`) +- `server.address` — target hostname (e.g., `aperture-token-service-pipelines.prod.bwell.zone`) +- `url.path` — the endpoint called +- `http.request.body.size` — size of the request body +- `http.request.method` — HTTP method + +**Step 3: Identify the calling service from client IP** + +The `client.address` is the pod IP of the caller. To identify which service it belongs to: +- Different client IPs for the same endpoint = multiple replicas of the calling service +- Same body size + same endpoint from different IPs = coordinated (or uncoordinated) batch calls + +**Key insight**: If `is_root_span:true` filter returns empty, walk the parent chain manually: +```gcql +* | filter span_id: | limit 5 +``` + +**Traefik span hierarchy** (from innermost to outermost): +``` +ATS handler span (workload:aperture-token-service-pipelines) + └─ ReverseProxy (kind:client, traefik → ATS pod) + └─ Metrics (kind:internal) + └─ Service (kind:internal, has traefik.service.name) + └─ Headers (kind:internal) + └─ Router (kind:internal, has http.route with Host matcher) + └─ EntryPoint [ROOT] (kind:server, has client.address + user_agent) +``` + +**Example: Finding source of concurrent refresh requests** +```gcql +# 1. From logs, get the otelTraceID of each refresh +* | filter client_fhir_person_id: service_slug: funcName:process_refresh_tokens_results | sort by (_time desc) | limit 10 + +# 2. For each trace ID, find the root span +* | filter trace_id: is_root_span:true | limit 5 + +# 3. Compare client.address values — different IPs = different caller pods +# Same user_agent + different client.address = same service, multiple replicas +``` + +## Query Strategy by Issue Type + +| Issue Type | Primary Query | Workload | Key funcName | +|-----------|---------------|----------|--------------| +| User can't connect (OAuth) | `client_fhir_person_id:` or `content:**` | aperture-token-service | `handle_callback_generic` | +| Token refresh failing | `client_fhir_person_id: service_slug:` | aperture-token-service-pipelines | `process_refresh_tokens_results`, `handle_token_refresh_error` | +| API errors | `workload:aperture-token-service level:error` | aperture-token-service | varies | +| Slow requests | Use traces: `service.name:aperture-token-service duration_seconds>5` | — | — | +| Connection Hub config issues | `content:*make_request_with_retries*` or `funcName:make_request_with_retries` | aperture-token-service | `make_request_with_retries` | +| GraphQL errors | `content:*BEFORE_REQUEST*` + `content:*graphql*` | aperture-token-service | `before_request` | +| JWT/Auth failures | `content:*validating JWT*` or `content:*Forbidden*` | aperture-token-service | auth_handler | +| Kafka publish failures | `content:*kafka*` level:error | aperture-token-service | — | + +## Structured Attributes Available for Direct Filtering + +These can be used as `attribute_name:value` in gcQL filters (no wildcards needed): + +| Attribute | Logged By | Values | +|-----------|-----------|--------| +| `client_fhir_person_id` | refresh results, callback errors, refresh errors | UUID | +| `service_slug` | refresh results, callback errors, refresh errors | provider identifier string | +| `member_id` | refresh results, refresh errors | UUID | +| `bwell_fhir_person_id` | refresh results, refresh errors | UUID | +| `funcName` | all structured logs | function name string | +| `code` | process_refresh_tokens_results | Refreshed, RefreshError, RefreshNotRequired, etc. | +| `status` | refresh results | New, Data Retrieved, Expired, etc. | +| `prev_status` | refresh results | same values | +| `connection_version` | refresh errors, callback errors | integer | +| `interop_type` | refresh errors | hapi, oauth, etc. | +| `query_params.error` | callback errors | access_denied, server_error | +| `query_params.error_description` | callback errors | Provider-specific message | diff --git a/.claude/skills/design-doc-trigger/SKILL.md b/.claude/skills/design-doc-trigger/SKILL.md new file mode 100644 index 00000000..20366c70 --- /dev/null +++ b/.claude/skills/design-doc-trigger/SKILL.md @@ -0,0 +1,396 @@ +--- +name: design-doc-trigger +description: Triggers design doc scaffolding when changes require architecture review +triggers: + - New dependency not in approved-tech.yaml + - New service creation (Dockerfile, deployment manifests) + - New HTTP client to internal service + - GraphQL schema changes adding types/mutations + - New FHIR resource usage + - New Kafka event/topic +--- + +# Design Doc Trigger + +You detect when changes require architecture review and scaffold the appropriate document. Per b.well's process, not all changes need formal design docs - but when they do, missing one causes PR delays. + +## Tone + +Helpful but firm. You're preventing process friction by scaffolding docs **before** the user hits PR review. + +## Detection Rules + +### Rule 1: New Dependency + +**Trigger:** User adds dependency not in `approved-tech.yaml` + +**Detection:** +- New package in `package.json`, `requirements.txt`, `pom.xml`, `build.gradle`, `go.mod` +- Dependency name not found in approved-tech.yaml + +**Response:** +"You're adding `mongodb` to package.json. This isn't in approved-tech.yaml. + +**New technologies require an ADR** per b.well architecture process (see AGENTS.md line 264). + +Let me check JIRA to see if one already exists for this decision..." + +[Query JIRA via Atlassian MCP: `searchJiraIssuesUsingJql` for ADR with "mongodb"] + +**If found:** +"Found existing ADR: EA-1234 'Use MongoDB for...' - you're good. Reference that ADR in your PR description." + +**If not found:** +"No existing ADR found. Want me to scaffold one? This will create: +- ADR using MADR 4.0.0 template +- Pre-populated context from your code changes +- Decision options section for you to fill in alternatives +- Consequences section + +I can output the ADR here or help you create a JIRA Tech Design Review ticket with the template." + +### Rule 2: New Service + +**Trigger:** User creates new service indicators + +**Detection:** +- New `Dockerfile` in repo +- New directory with `src/`, `package.json`, and service-like structure +- New Kubernetes deployment manifest + +**Response:** +"You're creating a new service. **This requires a Design Doc** per b.well review process (AGENTS.md line 242). + +New services need: +- **Phase 1 Design Doc** covering: + - Service purpose and boundaries + - Data ownership + - External dependencies + - Event contracts (published/subscribed) + - API surface + - Failure scenarios + +Let me check if there's already a design doc in JIRA..." + +[Query JIRA: search for Design Doc tickets mentioning service name] + +**If not found:** +"No design doc found. Want me to scaffold Phase 1? + +I'll create: +- Two-phase design doc template (Phase 1 for concept approval) +- Service identity section (purpose, boundaries, ownership) +- Dependencies section (what you call, who calls you) +- Event contracts section (what events you publish/subscribe) +- Failure scenarios (what breaks when you're down) +- Open questions section + +Output as: +1. Markdown file for Confluence +2. JIRA Tech Design Review ticket template +3. Or just markdown here for you to paste + +Your choice." + +### Rule 3: New Synchronous Service Dependency + +**Trigger:** User adds HTTP client calling internal service + +**Detection:** +- New `axios`, `fetch`, `HttpClient`, `RestTemplate` calling internal URLs +- URLs contain `.internal`, `.svc.cluster.local`, or internal service names + +**Response:** +"You're adding a sync HTTP call to `patient-service.internal`. + +**New cross-service dependencies may need a Design Doc** (AGENTS.md line 267). Let me check the context: + +1. Is this a **new pattern** (first time this service talks to patient-service)? +2. Or **existing pattern** (this service already calls patient-service)? + +[Check git history for previous calls to patient-service] + +**If new pattern:** +"This is a **new dependency** - first time this service talks to patient-service. + +New cross-service dependencies require Design Doc per b.well process. This needs to document: +- Why sync instead of async (latency requirements) +- Failure scenarios (what happens when patient-service is down) +- Circuit breaker strategy +- Saga pattern consideration (if multi-step transaction) + +Want me to scaffold a design doc focusing on this integration?" + +**If existing pattern:** +"This service already calls patient-service (found 3 existing calls). No new Design Doc needed - this extends existing approved pattern." + +### Rule 4: Public API Changes + +**Trigger:** User modifies GraphQL schema, adds mutations/types + +**Detection:** +- Changes to `.graphql`, `.graphqls` files +- New `type`, `mutation`, `query` definitions +- Changes to existing type fields + +**Response:** +"You're changing the public GraphQL API (added `createPatient` mutation). + +**Public API changes require Tech Design Review** per b.well process. Let me check JIRA..." + +[Query JIRA for existing design doc covering this API] + +**If found:** +"Found design doc EA-2045 covering patient creation API. You're implementing approved design - reference EA-2045 in your PR." + +**If not found:** +"No design doc found for this API change. + +**You need a Design Doc before this PR can merge.** Want me to scaffold it? + +I'll create template covering: +- API contract (GraphQL schema) +- Request/response shapes +- Error handling +- Rate limiting / pagination +- Authentication/authorization requirements +- Backward compatibility strategy + +Output format: Markdown for Confluence or JIRA template?" + +### Rule 5: New FHIR Resource Usage + +**Trigger:** User references FHIR resources not previously used in this service + +**Detection:** +- Import/require of FHIR resource types not in existing code +- New references to FHIR resources (Patient, Observation, MedicationRequest, etc.) + +**Response:** +"You're using FHIR `MedicationRequest` resource - new for this service. + +**New FHIR resource usage should go through FDR process** (FHIR Design Review, see AGENTS.md line 242). + +Let me check for existing FDRs..." + +[Query Confluence via Atlassian MCP for FDR pages matching "MedicationRequest"] + +**If found:** +"Found FDR: 'MedicationRequest for Prescription Ordering'. You're implementing approved design. + +Make sure your implementation follows the Helix profile defined in that FDR. Reference FDR in your PR." + +**If not found:** +"No FDR found for MedicationRequest in this context. + +**FDR process ensures FHIR consistency across platform.** Want me to scaffold draft FDR? + +I'll create: +- FHIR resource mapping +- Profile extensions needed (if native FHIR fields insufficient) +- Relevant IGs (US Core, CARIN, DaVinci, NDH) +- ValueSet/CodeSystem references +- Update vs create semantics +- Relationship to other resources + +Output for Confluence page? This needs review by fhir_design_review@icanbwell.com before implementation." + +### Rule 6: New Kafka Event + +**Trigger:** User adds new Kafka event publishing or subscription + +**Detection:** +- New Kafka topic references +- New event schema files +- New producer/consumer code + +**Response:** +"You're creating new Kafka event `patient.updated`. + +**Event contracts should be documented** before publishing (AsyncAPI or design doc). + +Want me to scaffold the event contract? I'll generate: +- Topic naming per convention (domain.entity.action) +- CloudEvents envelope structure +- Avro or JSON Schema +- Partition key (must include tenant ID per AGENTS.md) +- Consumer idempotency guidance +- DLQ topic configuration + +Note: Once Redpanda schema registry is accessible, I can validate against existing schemas too." + +## Scaffolding Templates + +### ADR Template (MADR 4.0.0) + +```markdown +# [short title of solved problem and solution] + +* Status: [proposed | rejected | accepted | deprecated | superseded by ADR-XXXX] +* Date: [YYYY-MM-DD when the decision was made] +* Decision-makers: [list team members involved] +* Consulted: [list stakeholders consulted] +* Informed: [list those informed of decision] + +## Context and Problem Statement + +[Describe the context and problem statement. What is the architectural decision we need to make?] + +## Decision Drivers + +* [driver 1, e.g., a constraint, priority, business requirement] +* [driver 2] +* [driver 3] + +## Considered Options + +* [option 1] +* [option 2] +* [option 3] + +## Decision Outcome + +Chosen option: "[option 1]", because [justification. e.g., only option which meets k.o. criterion]. + +### Consequences + +* Good, because [positive consequence 1] +* Good, because [positive consequence 2] +* Bad, because [negative consequence 1] +* Bad, because [tradeoff accepted] + +## Pros and Cons of the Options + +### [option 1] + +[description] + +* Good, because [argument a] +* Good, because [argument b] +* Bad, because [argument c] + +### [option 2] + +[description] + +* Good, because [argument a] +* Good, because [argument b] +* Bad, because [argument c] + +## Links + +* [Link type] [Link to ADR] +* [JIRA ticket EA-XXXX] +* [approved-tech.yaml] +``` + +### Design Doc Template (Phase 1) + +```markdown +# [Service/Feature Name] - Design Document + +**Status:** Phase 1 - Awaiting Approval +**Author:** [Your name] +**JIRA:** EA-XXXX +**Date:** [YYYY-MM-DD] + +## Overview + +[1-2 sentence summary of what this service/feature does] + +## Motivation + +**Problem:** [What problem does this solve?] +**Impact:** [Who is affected? What's the business value?] + +## Service Identity + +**Purpose:** [Core responsibility - what does this service own?] +**Boundaries:** [What is explicitly NOT in scope?] +**Data Ownership:** [What domain data does this service own?] + +## Dependencies + +### Outbound (What We Call) +| Service | Why | Sync/Async | Failure Mode | +|---------|-----|------------|--------------| +| patient-service | Get patient demographics | Sync | Circuit breaker, cache | +| kafka | Publish update events | Async | DLQ, retry | + +### Inbound (Who Calls Us) +| Client | How | Contract | +|--------|-----|----------| +| api-gateway | GraphQL | patientQuery, patientMutation | +| admin-portal | REST | /api/v1/patients | + +## Event Contracts + +### Published Events +- `patient.created` - When new patient registered +- `patient.updated` - When patient data changes + +### Subscribed Events +- `insurance.verified` - Update patient insurance status + +## API Surface + +[GraphQL schema, REST endpoints, or other public API] + +## Failure Scenarios + +**What breaks when this service is down?** +- Patient registration blocked (critical path) +- Patient data updates queued (degraded) +- Read queries served from cache (acceptable) + +**What breaks this service?** +- patient-service down → circuit breaker, serve cached +- Kafka down → retry queue, eventual consistency +- Database down → failover to read replica + +## Open Questions + +1. [Question requiring architectural decision] +2. [Question requiring input from other teams] + +## Next Steps + +**Phase 2 Requirements:** +- [ ] Detailed API contracts +- [ ] Data model / schema +- [ ] Scaling strategy +- [ ] Monitoring / alerting plan +- [ ] Deployment strategy +``` + +## When to Scaffold vs When to Skip + +**Scaffold when:** +- User has made significant code changes indicating architectural decision +- No existing ADR/Design Doc found via JIRA search +- Change affects multiple services or public contracts + +**Skip when:** +- Existing ADR/Design Doc found covering this decision +- Change is implementation detail within service boundaries +- User explicitly says "I already have a design doc" + +## MCP Integration + +**Use Atlassian MCP to:** +1. Search JIRA: `mcp__plugin_atlassian_atlassian__searchJiraIssuesUsingJql(cloudId, jql, fields)` + - Search for existing ADRs, Design Docs, FDRs + - Query: `project = EA AND type = "Tech Design Review" AND text ~ "mongodb"` + +2. Search Confluence: `mcp__plugin_atlassian_atlassian__searchConfluenceUsingCql(cloudId, cql, limit)` + - Search for existing architecture docs + - Query: `type=page AND space=ENTARCH AND title~"FDR"` + +3. Get page content: `mcp__plugin_atlassian_atlassian__getConfluencePage(cloudId, pageId, contentFormat)` + - Read existing design docs to check coverage + +**CloudId:** See AGENTS.md for current icanbwell Atlassian cloudId (public instance identifier) + +--- + +**Remember:** You're saving the user from PR delays. Catching "needs design doc" at commit time beats catching it at PR review time. Be proactive, be helpful, scaffold the template. diff --git a/.claude/skills/dqm-local-testing/SKILL.md b/.claude/skills/dqm-local-testing/SKILL.md new file mode 100644 index 00000000..37a6a74d --- /dev/null +++ b/.claude/skills/dqm-local-testing/SKILL.md @@ -0,0 +1,632 @@ +--- +name: dqm-local-testing +description: Run the DQM pipeline services locally end-to-end (orchestrator :8211, bundler :8025, evaluator :8026, normalizer :8212) with changeset-aware test-plan generation, shared Kafka/Postgres/LocalStack infrastructure, and synthetic data via hp-validation-tests. +argument-hint: " — e.g. 'orch', 'bundler and evaluator', 'all'" +disable-model-invocation: false +allowed-tools: Bash, Read, Write, Agent +--- + +# DQM Pipeline Local Testing Skill + +Run the DQM pipeline services locally to test end-to-end bundling, consolidation, evaluation, and normalization flows. Includes synthetic test data creation following hp-validation-tests patterns. + +## When to Use + +- Testing new orchestrator/bundler/evaluator/normalizer features locally before PR merge +- Validating the full pipeline flow with synthetic data +- Debugging pipeline behavior with local log access +- Running multi-service integration tests that aren't covered by unit/integration test suites + +## Invocation + +When this skill is invoked, first run the **Changeset-Aware Testing** analysis below. Based on the git diff and test analysis, determine which services are needed and present the test plan for confirmation. + +If no local changes are detected (clean working tree), fall back to asking the user which services to run: + +1. **Orchestrator + Bundler** (bundling + consolidation only) +2. **Orchestrator + Bundler + Evaluator** (through CQL evaluation) +3. **All 4 services** (full pipeline through normalization) + +Parse `$ARGUMENTS` for service hints: +- `orchestrator` / `orch` — Orchestrator only +- `bundler` / `bundle` — Bundler only +- `evaluator` / `eval` — Evaluator only +- `normalizer` / `norm` — Normalizer only +- `all` — All 4 services +- Combinations: `orch + bundler`, `bundler and evaluator`, etc. + +## Service Registry + +| Service | Port | Repo | Docker Deps | S3 Needed? | +|---------|------|------|-------------|------------| +| Orchestrator | 8211 | `clinical-reasoning-orchestrator-service` | Kafka, Postgres | No | +| Bundler | 8025 | `fhir-bundler-service` | LocalStack (S3) | Yes (write bundles) | +| Evaluator | 8026 | `cql-evaluator-adapter-service` | (shared Kafka only) | Yes (read bundles, write results) | +| Normalizer | 8212 | `clinical-results-normalizer-service` | LocalStack (S3) | Yes (read results, write normalized) | + +## Service Architecture + +``` +[Orchestrator :8211] → Kafka → [Bundler :8025] → LocalStack S3 + ↑ ↓ + ├──── bundling.events ─────────┘ + │ + ├── consolidation.commands ──→ [Bundler] → consolidation.events ──→ [Orchestrator] + │ + ├── evaluation.commands ─────→ [Evaluator :8026] → evaluation.events ──→ [Orchestrator] + │ + └── normalization.commands ──→ [Normalizer :8212] → normalization.events → [Orchestrator] +``` + +**Topics:** +- `clinical_quality.bundling.commands` — Orchestrator sends BundleData +- `clinical_quality.bundling.events` — Bundler sends DataBundled/DataBundleFailed +- `clinical_quality.consolidation.commands` — Orchestrator sends ConsolidateData; Bundler sends ConsolidateChunk +- `clinical_quality.consolidation.events` — Bundler sends DataConsolidated/ConsolidationFailed +- `clinical_quality.evaluation.commands` — Orchestrator sends EvaluateData +- `clinical_quality.evaluation.events` — Evaluator sends DataEvaluated/EvaluationFailed +- `clinical_quality.normalization.commands` — Orchestrator sends NormalizeResults +- `clinical_quality.normalization.events` — Normalizer sends ResultsNormalized/NormalizationFailed + +## Changeset-Aware Testing + +Every time this skill is invoked, automatically analyze the local changeset to generate targeted test scenarios. This ensures local testing exercises the specific code paths that changed, not just generic happy-path flows. + +### Step 1: Analyze the changeset + +Run these in parallel to understand what changed: + +```bash +# What files changed (unstaged + staged + untracked) +git status +git diff --name-only +git diff --cached --name-only + +# Full diff for understanding the nature of changes +git diff +git diff --cached +``` + +### Step 2: Analyze related tests + +For each changed source file, find its corresponding tests: +- Unit tests: `src/test/java/.../Test.java` +- Integration tests: `src/itest/java/.../IT.java` or files in `src/itest/` +- Docker-compose test configs: `src/itest/resources/docker-compose-test.yml` + +Read the test files to understand: +- What scenarios/edge cases are being validated +- What assertions define "correct" behavior +- What test data patterns are used (inputs, expected outputs) +- What Kafka messages or REST calls the itests exercise + +### Step 3: Generate a test plan + +Based on the changeset analysis, determine: +1. **Which services need to run** — only start services relevant to the changed code paths +2. **What scenarios to exercise** — derive from the code changes + test assertions +3. **What to monitor** — specific log patterns, S3 outputs, Kafka messages, or DB state that prove the change works +4. **Expected outcomes** — what success looks like for each scenario + +### Step 4: Present the plan for confirmation + +Before executing, present a clear summary to the developer: + +``` +## Local Test Plan + +### Changes detected: +- [file1.java]: +- [file2.java]: + +### Services to start: +- Orchestrator (needed because: ...) +- Bundler (needed because: ...) + +### Test scenarios: +1. — Tests , expects + Why: validates +2. — Tests , expects + Why: validates + +### What I'll monitor: +- Logs: +- S3: +- DB: + +Does this look right? Should I add/remove any scenarios? +``` + +Wait for the developer to confirm or adjust before proceeding. + +### Step 5: Execute and validate + +Run the test plan: +1. Start infrastructure + services (only what's needed) +2. Execute each scenario +3. Capture evidence (log snippets, S3 file listings, curl responses, DB queries) +4. Compare actual outcomes to expected outcomes + +### Step 6: Produce a validation summary + +After all scenarios complete, produce a summary: + +``` +## Validation Summary + +### Changeset: () + +### Results: +| # | Scenario | Status | Evidence | +|---|----------|--------|----------| +| 1 | | PASS | | +| 2 | | FAIL | | + +### What this validates: +- +- + +### What this does NOT validate (out of scope for local testing): +- +``` + +This summary is intended to be copy-pasteable into a PR description or Slack message as evidence of local validation. + +--- + +## Prerequisites + +- Docker Desktop running +- Java 21+ (`java -version`) +- **Source root exported** — all service repos are assumed to live under one directory. Set it once per shell (defaults to `~/src`): + + ```bash + export DQM_SRC_ROOT="${DQM_SRC_ROOT:-$HOME/src}" + ``` + + Every command below uses `$DQM_SRC_ROOT/{repo}`. If your clones live elsewhere, export `DQM_SRC_ROOT` to that parent directory first. +- `~/kui/config.yml` file exists (can be empty — needed by Kafka UI container) +- `~/.dqm-local-env.sh` file with Cognito credentials (see Credentials section) +- Dev token accessible via the credentials in env file + +## Credentials Setup + +Create `~/.dqm-local-env.sh` (NEVER commit this): + +```bash +#!/bin/bash +# DQM Local Testing - Environment Variables +# NEVER commit. Lives in ~ outside any git repo. + +export auth_client_id="" +export auth_client_secret="" + +# Override scope — local profiles hardcode a scope not supported by all Cognito clients +export SPRING_APPLICATION_JSON='{"auth":{"client":{"scope":"access/*.* user/*.*"}}}' +``` + +**Getting credentials:** The client must support `client_credentials` grant with scopes `access/*.* user/*.*`. The token URL is `https://bwell-dev.auth.us-east-1.amazoncognito.com/oauth2/token`. + +## Infrastructure Setup + +### Step 1: Stop conflicting containers + +```bash +# Check for port conflicts +docker ps --format "{{.Names}} {{.Ports}}" | grep -E "9092|8025|8211|4566|5432" + +# Stop any conflicting containers +docker stop +``` + +### Step 2: Start shared infrastructure + +Use the orchestrator's docker-compose for Kafka + Postgres, and the bundler's for LocalStack. +All 4 services share one Kafka broker (localhost:9092) and one LocalStack S3 (localhost:4566). + +```bash +# Start Kafka + Postgres (from orchestrator) +cd ${DQM_SRC_ROOT}/clinical-reasoning-orchestrator-service +docker compose -f docker-compose-local.yml up -d + +# Start LocalStack only (from bundler — skip its Kafka to avoid port conflict) +# LocalStack auto-creates the bwell-dqm bucket via init-s3.py hook +cd ${DQM_SRC_ROOT}/fhir-bundler-service +docker compose -f docker-compose-local.yml up -d localstack +``` + +### Step 3: Verify infrastructure health + +```bash +# Kafka (binary path varies by image — try both) +docker exec cr-orchestrator-kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list 2>/dev/null \ + || docker exec cr-orchestrator-kafka kafka-topics --bootstrap-server localhost:9092 --list + +# Postgres +docker exec cr-orchestrator-postgres pg_isready + +# LocalStack + S3 bucket +curl -sf http://localhost:4566/_localstack/health | python3 -m json.tool +AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test aws --endpoint-url=http://localhost:4566 s3 ls +# Should show: bwell-dqm bucket (created by init-s3.py hook) +``` + +## Starting Services + +### Step 4: Build services + +Build whichever services you need to run: + +```bash +# Always needed: +cd ${DQM_SRC_ROOT}/clinical-reasoning-orchestrator-service && ./gradlew compileJava +cd ${DQM_SRC_ROOT}/fhir-bundler-service && ./gradlew compileJava + +# If running evaluator: +cd ${DQM_SRC_ROOT}/cql-evaluator-adapter-service && ./gradlew compileJava + +# If running normalizer: +cd ${DQM_SRC_ROOT}/clinical-results-normalizer-service && ./gradlew compileJava +``` + +### Step 5: Start services + +Start in foreground (one terminal per service): + +```bash +# Terminal 1: Orchestrator +source ~/.dqm-local-env.sh +cd ${DQM_SRC_ROOT}/clinical-reasoning-orchestrator-service +SPRING_PROFILES_ACTIVE=local AWS_CONFIG_FILE=/dev/null ./gradlew bootRun + +# Terminal 2: Bundler +source ~/.dqm-local-env.sh +cd ${DQM_SRC_ROOT}/fhir-bundler-service +SPRING_PROFILES_ACTIVE=local AWS_CONFIG_FILE=/dev/null ./gradlew bootRun + +# Terminal 3: Evaluator (if needed) +source ~/.dqm-local-env.sh +cd ${DQM_SRC_ROOT}/cql-evaluator-adapter-service +SPRING_PROFILES_ACTIVE=local AWS_CONFIG_FILE=/dev/null ./gradlew bootRun + +# Terminal 4: Normalizer (if needed) +source ~/.dqm-local-env.sh +cd ${DQM_SRC_ROOT}/clinical-results-normalizer-service +SPRING_PROFILES_ACTIVE=local AWS_CONFIG_FILE=/dev/null ./gradlew bootRun +``` + +Or run in background: +```bash +source ~/.dqm-local-env.sh + +cd ${DQM_SRC_ROOT}/clinical-reasoning-orchestrator-service +SPRING_PROFILES_ACTIVE=local AWS_CONFIG_FILE=/dev/null ./gradlew bootRun > /tmp/orchestrator.log 2>&1 & + +cd ${DQM_SRC_ROOT}/fhir-bundler-service +SPRING_PROFILES_ACTIVE=local AWS_CONFIG_FILE=/dev/null ./gradlew bootRun > /tmp/bundler.log 2>&1 & + +cd ${DQM_SRC_ROOT}/cql-evaluator-adapter-service +SPRING_PROFILES_ACTIVE=local AWS_CONFIG_FILE=/dev/null ./gradlew bootRun > /tmp/evaluator.log 2>&1 & + +cd ${DQM_SRC_ROOT}/clinical-results-normalizer-service +SPRING_PROFILES_ACTIVE=local AWS_CONFIG_FILE=/dev/null ./gradlew bootRun > /tmp/normalizer.log 2>&1 & +``` + +### Step 6: Verify services are healthy + +```bash +curl -sf http://localhost:8211/actuator/health # Orchestrator +curl -sf http://localhost:8025/actuator/health # Bundler +curl -sf http://localhost:8026/actuator/health # Evaluator +curl -sf http://localhost:8212/actuator/health # Normalizer +``` + +## Creating Synthetic Test Data + +The bundler requires the full dual-Person/Patient model (client-side + bwell-side) created through the Identity Gateway. The hp-validation-tests Karate framework automates this. + +### Option A: Use hp-validation-tests directly (Recommended) + +```bash +cd ${DQM_SRC_ROOT}/hp-validation-tests +git pull # Always use latest + +# Run the DQM group measure test (creates 3 persons + executes measure) +./gradlew test -Dkarate.env=dev \ + -Dclient.id= \ + -Dclient.secret= \ + -Dclient.key= \ + -Dkarate.options="--tags @bcse-group-measure" +``` + +This creates: +1. 3 synthetic users via BIG + Identity Gateway (each gets Person + Patient in FHIR) +2. Level 1 resources (Organization, Practitioner, Medication, Questionnaire) +3. Level 3 patient-linked resources (Encounter, Coverage, Condition, Observation, etc.) +4. BCSE-specific resources (mammography observation, mastectomy procedure) +5. Invokes the orchestrator's `$evaluate` endpoint + +### Option B: Create data manually via curl + +If you need persons created but want to invoke the orchestrator locally (not against deployed services), follow these steps: + +**Step 1: Get token** +```bash +TOKEN=$(curl -s -X POST "https://bwell-dev.auth.us-east-1.amazoncognito.com/oauth2/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -H "Authorization: Basic $(echo -n ':' | base64)" \ + -d "grant_type=client_credentials&scope=access/*.*+user/*.*" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") +``` + +**Step 2: Create users via BIG + Identity Gateway** + +The bundler requires the dual Person/Patient model. This CANNOT be created by direct FHIR PUT — it must go through the Identity Gateway which creates: +- A "client" Person + Patient (representing the data source) +- A "bwell" Person + Patient (the unified b.well record) +- Proper links between them + +```bash +# Generate JWE payload (use hp-validation-tests NewUserPayloadGenerator pattern) +# POST to BIG: https://big.dev.bwell.zone/api/admin/jwe/token/encrypt +# Exchange at Identity Gateway: https://bwell-identity-gateway.dev.bwell.zone/token + +# The response contains: clientFhirPersonId, clientFhirPatientId, bwellFhirPersonId, bwellFhirPatientId +``` + +**Step 3: Create clinical resources for each patient** + +After Identity Gateway creates Person+Patient, create clinical resources: +```bash +# Coverage (required for bundling) +curl -s -X POST "https://fhir.dev.bwell.zone/4_0_0/Coverage" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/fhir+json" \ + -d '{ + "resourceType": "Coverage", + "meta": {"source": "https://dqm-local-test", "security": [ + {"system": "https://www.icanbwell.com/owner", "code": ""}, + {"system": "https://www.icanbwell.com/access", "code": ""}, + {"system": "https://www.icanbwell.com/sourceAssigningAuthority", "code": ""} + ]}, + "status": "active", + "beneficiary": {"reference": "Patient/"}, + "payor": [{"reference": "Organization/"}], + "period": {"start": "2025-01-01", "end": "2025-12-31"} +}' +``` + +See `${DQM_SRC_ROOT}/hp-validation-tests/src/test/java/payload/dqm/` for all resource templates. + +## Invoking the Orchestrator + +### Clean the database (if re-running) + +```bash +docker exec cr-orchestrator-postgres psql -U postgres -d orchestratordb -c \ + "DELETE FROM work_unit_person; DELETE FROM work_unit_context; DELETE FROM execution_work_unit; DELETE FROM data_source; DELETE FROM orchestrator_executions;" +``` + +### Quick-start: Find test persons + +The dev FHIR server has real persons with non-UUID security labels (e.g., `walgreens`). To find usable person IDs: + +```bash +TOKEN=$(cat /tmp/dqm-token.txt) # or generate fresh +curl -s "https://fhir.dev.bwell.zone/4_0_0/Person?_security=walgreens&_count=3&_elements=id" \ + -H "Authorization: Bearer $TOKEN" | python3 -c " +import json, sys +bundle = json.loads(sys.stdin.read()) +ids = [e['resource']['id'] for e in bundle.get('entry', [])] +print(','.join(ids)) +" +``` + +Since `walgreens` is not a UUID, use a synthetic managingOrganization UUID (e.g., `aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee`). The `_security` label in the subject query is what drives person lookup, not managingOrganization. + +### Trigger $evaluate + +```bash +TOKEN=$(curl -s -X POST "https://bwell-dev.auth.us-east-1.amazoncognito.com/oauth2/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -H "Authorization: Basic $(echo -n ':' | base64)" \ + -d "grant_type=client_credentials&scope=access/*.*+user/*.*" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") + +curl -s -X POST "http://localhost:8211/fhir/4_0_0/Measure/\$evaluate" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/fhir+json" \ + -d '{ + "resourceType": "Parameters", + "parameter": [ + {"name": "measureId", "valueId": ""}, + {"name": "periodStart", "valueDate": "2025-01-01"}, + {"name": "periodEnd", "valueDate": "2025-12-31"}, + {"name": "subject", "valueString": "_security=&_elements=id&_id=,,"}, + {"name": "parameters", "resource": {"resourceType": "Parameters", "parameter": [ + {"name": "scheduleId", "valueString": "99"}, + {"name": "outputFormatGroup", "valueString": "Standard Execution Output V2"}, + {"name": "managingOrganization", "valueString": ""}, + {"name": "clientFhirPersonId", "valueString": ""}, + {"name": "bwellFhirPersonId", "valueString": ""} + ]}} + ] +}' | python3 -m json.tool +``` + +**Key requirements:** +- Person IDs MUST be UUID format (orchestrator validates against UUID regex) +- `_security` value must be a single label (no pipes or commas — e.g., `walgreens` not `https://www.icanbwell.com/owner|walgreens`) +- `managingOrganization` MUST be a valid UUID (the `ManagingOrganization` value object enforces `UUID.fromString()`). If the org's security label is not a UUID (e.g., `walgreens`), use a synthetic UUID like `aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee` — the orchestrator uses this for record ownership, not for FHIR queries +- `clientFhirPersonId` and `bwellFhirPersonId` are REQUIRED parameters — omitting them causes `ExecutionMappingException` +- Persons must have proper Client Patient links (created via Identity Gateway) for the bundler to resolve data. For orchestrator-only testing (leader/follower flow), any valid FHIR Person IDs suffice + +### Poll progress + +```bash +EXEC_ID="" +curl -s "http://localhost:8211/api/v1/executions/$EXEC_ID/progress" \ + -H "Authorization: Bearer $TOKEN" | python3 -m json.tool +``` + +**Expected state transitions:** +1. `POPULATING` — resolving persons from FHIR +2. `BUNDLING` — dispatching BundleData commands, waiting for bundles +3. `BUNDLED` — all persons bundled successfully +4. `CONSOLIDATED` (INE-734) — consolidation phase complete +5. `EVALUATED` — CQL evaluation complete (requires evaluator service) +6. `NORMALIZED` — normalization complete (requires normalizer service) + +For testing bundling + consolidation only, the flow stops at CONSOLIDATED (no evaluator/normalizer running). + +## Checking S3 (LocalStack) + +```bash +# List all files in bwell-dqm bucket +AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test \ + aws --endpoint-url=http://localhost:4566 s3 ls s3://bwell-dqm/ --recursive + +# Files follow pattern: +# bundled/////.ndjson +# consolidated////.ndjson +``` + +## Monitoring Logs + +```bash +# Follow orchestrator for state transitions +tail -f /tmp/orchestrator.log | grep -E "state|BUNDL|CONSOLIDAT|EVALUAT|NORMALIZ|FAILED|transition" + +# Follow bundler for bundling/consolidation activity +tail -f /tmp/bundler.log | grep -E "BundleData|Consolidat|ERROR|DataBundled|chunk" + +# Follow evaluator for CQL evaluation +tail -f /tmp/evaluator.log | grep -E "EvaluateData|Evaluation|DCS|NCQA|ERROR|DataEvaluated" + +# Follow normalizer for normalization +tail -f /tmp/normalizer.log | grep -E "NormalizeResults|Normalized|ERROR|MeasureReport" +``` + +## Teardown + +### Stop services +```bash +# If running in foreground: Ctrl+C in each terminal + +# If running in background: +for port in 8211 8025 8026 8212; do + lsof -i :$port -t 2>/dev/null | xargs kill -9 2>/dev/null +done +pkill -f "GradleDaemon" +``` + +### Stop Docker infrastructure +```bash +cd ${DQM_SRC_ROOT}/clinical-reasoning-orchestrator-service +docker compose -f docker-compose-local.yml down + +cd ${DQM_SRC_ROOT}/fhir-bundler-service +docker compose -f docker-compose-local.yml down +``` + +### Full cleanup (including volumes) +```bash +cd ${DQM_SRC_ROOT}/clinical-reasoning-orchestrator-service +docker compose -f docker-compose-local.yml down -v + +cd ${DQM_SRC_ROOT}/fhir-bundler-service +docker compose -f docker-compose-local.yml down -v +``` + +### Clean up synthetic FHIR data (optional) +```bash +TOKEN=$(curl -s -X POST "https://bwell-dev.auth.us-east-1.amazoncognito.com/oauth2/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -H "Authorization: Basic $(echo -n ':' | base64)" \ + -d "grant_type=client_credentials&scope=access/*.*+user/*.*" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") + +# Delete synthetic resources by source tag +for TYPE in Person Patient Coverage Observation Condition Encounter; do + curl -s -X DELETE "https://fhir.dev.bwell.zone/4_0_0/$TYPE?_source=https://dqm-local-test" \ + -H "Authorization: Bearer $TOKEN" +done +``` + +## Troubleshooting + +### `invalid_grant` from Cognito +The scope in `SPRING_APPLICATION_JSON` doesn't match what the Cognito client supports. Verify: +```bash +curl -s -X POST "https://bwell-dev.auth.us-east-1.amazoncognito.com/oauth2/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -H "Authorization: Basic $(echo -n ':' | base64)" \ + -d "grant_type=client_credentials&scope=access/*.*+user/*.*" +``` + +### Execution immediately FAILED as leader +Check orchestrator logs for auth errors. The FHIR client needs valid credentials for population resolution. + +### Execution created as FOLLOWER (AWAITING_GROUP_BUNDLE) +A stale leader exists in the database. Clean the DB: +```bash +docker exec cr-orchestrator-postgres psql -U postgres -d orchestratordb -c \ + "DELETE FROM work_unit_person; DELETE FROM work_unit_context; DELETE FROM execution_work_unit; DELETE FROM data_source; DELETE FROM orchestrator_executions;" +``` + +### `ClientPatient not found for personId` +The Person doesn't have proper Patient links via Identity Gateway. You CANNOT create persons via direct FHIR PUT for bundler testing — must use BIG + Identity Gateway (see hp-validation-tests). + +### Invalid personId format +Person IDs must be UUIDs. Non-UUID IDs (like `dqm-test-person-001`) are rejected by the orchestrator's `PersonQueryParameterValidator`. + +### Port already in use +```bash +lsof -i : | grep LISTEN +# Kill the offending process +``` + +### Kafka topics not auto-created +Local Kafka uses auto-topic-creation. If topics are missing: +```bash +# binary path varies by image — try both (cp-kafka uses `kafka-topics` on PATH) +docker exec cr-orchestrator-kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \ + --create --topic clinical_quality.bundling.commands --partitions 1 --replication-factor 1 2>/dev/null \ + || docker exec cr-orchestrator-kafka kafka-topics --bootstrap-server localhost:9092 \ + --create --topic clinical_quality.bundling.commands --partitions 1 --replication-factor 1 +``` + +## Key Configuration + +| Service | Port | Profile | FHIR URL | Auth | S3 Bucket | +|---------|------|---------|----------|------|-----------| +| Orchestrator | 8211 | local | fhir.dev.bwell.zone | Dev Cognito | — | +| Bundler | 8025 | local | fhir.dev.bwell.zone | Dev Cognito | bwell-dqm | +| Evaluator | 8026 | local | fhir.dev.bwell.zone | Dev Cognito | bwell-dqm | +| Normalizer | 8212 | local | fhir.dev.bwell.zone | Dev Cognito | bwell-dqm | + +| Infrastructure | Port | Source | +|----------------|------|--------| +| Kafka | 9092 | Orchestrator docker-compose | +| Kafka UI | 8080 | Orchestrator docker-compose | +| Postgres | 5432 | Orchestrator docker-compose | +| LocalStack | 4566 | Bundler docker-compose | + +### Evaluator-Specific Notes + +The evaluator calls the NCQA DCS external API for CQL evaluation. For local testing: +- Requires `NCQA_DCS_API_KEY` and `NCQA_DCS_URL` in env or application-local.yaml +- If these are not set, evaluations will fail with auth errors against NCQA +- The evaluator reads bundled data from S3 (written by the bundler) and writes evaluation results back to S3 + +### Normalizer-Specific Notes + +The normalizer reads evaluation results from S3 and produces normalized MeasureReport resources: +- Reads from `resultsStorageUri` (S3 path provided in the NormalizeResults command) +- Writes normalized output to `outputStoragePrefix` in the same bucket +- The `dcs_prefer_uncompressed` flag (default: false) controls whether it prefers .ndjson over .ndjson.gz + +## Related Skills +- `java-local-dev` — Single-service local dev +- `java-staging-connected` — Connecting to remote infrastructure +- `java-e2e-validation` — Full E2E testing via hp-validation-tests diff --git a/.claude/skills/duplication-detection/SKILL.md b/.claude/skills/duplication-detection/SKILL.md new file mode 100644 index 00000000..c18431b1 --- /dev/null +++ b/.claude/skills/duplication-detection/SKILL.md @@ -0,0 +1,215 @@ +--- +name: duplication-detection +description: Identifies code duplication opportunities for abstraction into shared libraries +triggers: + - Writing retry/backoff logic + - Writing tenant context propagation code + - Writing FHIR query boilerplate + - Writing HTTP client wrappers + - Writing error handling patterns +--- + +# Duplication Detection + +You identify when code being written already exists in shared libraries or should be extracted. Don't let developers reinvent wheels that already exist or create the third copy of a pattern. + +## Tone + +Helpful and specific. Point to exact library/module to use instead of reimplementing. + +## Patterns to Catch + +### 1. Retry/Backoff Logic + +**User writes:** +```javascript +async function fetchWithRetry(url, maxRetries = 3) { + for (let i = 0; i < maxRetries; i++) { + try { + return await fetch(url); + } catch (error) { + if (i === maxRetries - 1) throw error; + await sleep(Math.pow(2, i) * 1000); // Exponential backoff + } + } +} +``` + +**Response:** +"This retry-with-exponential-backoff already exists in `@bwell/common-utils`. + +Use instead: +```javascript +import { RetryHelper } from '@bwell/common-utils'; + +const data = await RetryHelper.withExponentialBackoff( + () => fetch(url), + { maxRetries: 3, baseDelay: 1000 } +); +``` + +Don't reimplement retry logic. Use the shared utility." + +### 2. Tenant Context Propagation + +**User writes:** +```javascript +function propagateTenantContext(req, next) { + const tenantId = req.headers['x-tenant-id']; + req.context = { tenantId }; + return next(); +} +``` + +**Response:** +"Tenant context propagation is standardized in `@bwell/tenant-middleware`. + +This is the **third service** implementing this exact pattern. Should be using: +```javascript +import { tenantContextMiddleware } from '@bwell/tenant-middleware'; + +app.use(tenantContextMiddleware()); +``` + +Want me to file a tech debt ticket to extract this pattern to shared library? This prevents drift in tenant isolation implementation." + +### 3. FHIR Query Boilerplate + +**User writes:** +```javascript +async function getPatient(id) { + const response = await fhirClient.read({ + resourceType: 'Patient', + id: id + }); + // Transform response + return { + id: response.id, + name: response.name[0].text, + birthDate: response.birthDate + }; +} +``` + +**Response:** +"FHIR query patterns are in `@bwell/fhir-utils`. Use: +```javascript +import { FhirClient } from '@bwell/fhir-utils'; + +const patient = await FhirClient.getResource('Patient', id); +// Transformation handled by client +``` + +Don't duplicate FHIR boilerplate." + +### 4. HTTP Client Wrappers + +**User writes:** +```javascript +class ApiClient { + constructor(baseURL) { + this.baseURL = baseURL; + } + + async get(path) { + const response = await fetch(`${this.baseURL}${path}`); + if (!response.ok) throw new Error('Request failed'); + return response.json(); + } + // More methods... +} +``` + +**Response:** +"This is the HTTP client wrapper pattern. If this is for external vendor integration, wrap it behind capability interface per AGENTS.md line 72: + +```javascript +// Capability interface +interface PaymentGateway { + charge(amount: number): Promise; + refund(transactionId: string): Promise; +} + +// Vendor adapter +class StripeAdapter implements PaymentGateway { + private client: ApiClient; + + async charge(amount: number): Promise { + // Stripe-specific implementation + } +} +``` + +Don't couple to vendor directly. Use adapter pattern." + +### 5. Error Standardization + +**User writes:** +```javascript +class ValidationError extends Error { + constructor(message, field) { + super(message); + this.field = field; + this.code = 'VALIDATION_ERROR'; + } +} +``` + +**Response:** +"Error classes should use `@bwell/errors` for consistency: +```javascript +import { ValidationError, ErrorCodes } from '@bwell/errors'; + +throw new ValidationError('Invalid email', { + field: 'email', + code: ErrorCodes.INVALID_FORMAT +}); +``` + +Standardized errors enable better error handling and monitoring." + +## When to Extract vs When to Use + +**Suggest extraction when:** +- Same pattern appears 3+ times across services +- Pattern has subtle correctness requirements (tenant isolation, retry timing) +- Pattern likely to evolve (FHIR query patterns, auth handling) + +**Use existing library when:** +- Pattern already extracted (retry, logging, tenant context) +- External vendor integration (use adapter pattern) +- Cross-cutting concern (monitoring, tracing, error handling) + +**Don't extract when:** +- Only used twice and unlikely to spread +- Highly service-specific logic +- Business domain logic (don't abstract business rules into "shared" code) + +## Service Boundaries Respect + +**Don't suggest coupling across domain boundaries:** + +❌ Bad: +"Import patient name formatting from billing-service" + +✅ Good: +"Extract shared formatting logic to `@bwell/patient-utils` that both services depend on" + +Services should share libraries, not call each other for utility functions. + +## Known Shared Libraries + +**Check for these before user implements:** +- `@bwell/common-utils`: Retry, sleep, array/object utils +- `@bwell/tenant-middleware`: Tenant context propagation +- `@bwell/fhir-utils`: FHIR query helpers +- `@bwell/errors`: Standardized error classes +- `@bwell/logger`: Structured logging +- `@bwell/auth`: Authentication/authorization +- `@bwell/kafka`: Kafka producer/consumer helpers + +**Reference approved-tech.yaml for full list.** + +--- + +**Remember:** You're preventing code drift and reducing maintenance burden. Every duplicated utility is a bug fix that needs to be applied N times. diff --git a/.claude/skills/fhir-resource-design/SKILL.md b/.claude/skills/fhir-resource-design/SKILL.md new file mode 100644 index 00000000..03420996 --- /dev/null +++ b/.claude/skills/fhir-resource-design/SKILL.md @@ -0,0 +1,382 @@ +--- +name: fhir-resource-design +description: FHIR modeling guidance with FDR process integration +triggers: + - Working with FHIR resources + - Data modeling for healthcare entities + - Creating FHIR profiles or StructureDefinitions + - Mapping data to FHIR +--- + +# FHIR Resource Design + +You provide guidance on FHIR resource mapping following b.well's FDR (FHIR Design Review) process and Helix profile conventions. + +## Tone + +Educational and specific. FHIR is complex - guide users toward standards-compliant solutions that follow existing patterns. + +## When to Trigger + +**Trigger when user:** +- Asks "how do I model [healthcare concept] in FHIR?" +- Creates new FHIR resource usage +- Writes FHIR mapping code +- References FHIR resources not used before in this service + +**Don't trigger for:** +- Reading existing FHIR resources (following established patterns) +- FHIR server administration tasks +- Non-healthcare data modeling + +## FDR Process Check + +**First step:** Check if FDR exists for this use case. + +``` +You're working with FHIR `MedicationRequest` for prescription ordering. + +Let me check if there's an existing FDR for this... + +[Search Confluence via Atlassian MCP: + cloudId: dc9c52d9-3f1d-4037-9c66-a57f1079b857 + cql: "type=page AND space=ENTARCH AND title~'FDR' AND title~'MedicationRequest'" +] +``` + +**If FDR exists:** +"Found FDR: 'FDR-MedicationRequest-Prescription-Ordering' (https://...) + +**Follow this FDR:** +- Uses US Core MedicationRequest profile +- Extensions: prescriber NPI, pharmacy preference +- Must link to Coverage resource for insurance +- Status values: draft → active → completed + +Implementation should match approved FDR. Need help implementing it?" + +**If no FDR:** +"No FDR found for MedicationRequest in prescription ordering context. + +**New FHIR resource usage should go through FDR process** (AGENTS.md line 242). + +Want me to scaffold draft FDR for you? I'll check: +1. Relevant Implementation Guides (US Core, CARIN, DaVinci) +2. Existing Helix profiles at fhir.icanbwell.com +3. Related FDRs that might inform this design + +Output will need review by fhir_design_review@icanbwell.com before implementation." + +## FHIR Design Principles + +### 1. Native Fields Over Extensions + +**Bad:** +```json +{ + "resourceType": "Patient", + "id": "patient-123", + "extension": [ + { + "url": "http://bwell.com/fhir/StructureDefinition/patient-email", + "valueString": "patient@example.com" + } + ] +} +``` + +**Good:** +```json +{ + "resourceType": "Patient", + "id": "patient-123", + "telecom": [ + { + "system": "email", + "value": "patient@example.com", + "use": "home" + } + ] +} +``` + +**Response:** +"Don't use extensions for data that has native FHIR fields. `Patient.telecom` is the standard way to represent email. + +**Use extensions only when:** +- Data truly doesn't fit any FHIR field +- IG (Implementation Guide) defines custom extension +- Representing b.well-specific metadata + +Check FHIR spec first: https://hl7.org/fhir/patient.html" + +### 2. Proper Resource Naming + +**Resource identifiers must follow convention:** + +``` +https://fhir.icanbwell.com/4_0_0/Patient/patient-uuid +https://fhir.icanbwell.com/4_0_0/Observation/observation-uuid +``` + +**Not:** +``` +http://example.com/Patient/123 +Patient/abc +patients/patient-123 +``` + +**CodeSystem/ValueSet naming:** +``` +https://fhir.icanbwell.com/4_0_0/CodeSystem/bwell-patient-status +https://fhir.icanbwell.com/4_0_0/ValueSet/us-core-race-ethnicity +``` + +### 3. Proper Use of References + +**Good:** +```json +{ + "resourceType": "MedicationRequest", + "subject": { + "reference": "Patient/patient-123", + "display": "John Doe" + }, + "requester": { + "reference": "Practitioner/practitioner-456", + "display": "Dr. Jane Smith" + } +} +``` + +**Bad:** +```json +{ + "resourceType": "MedicationRequest", + "patientId": "patient-123", // ❌ Custom field instead of reference + "doctorName": "Dr. Jane Smith" // ❌ String instead of Practitioner reference +} +``` + +### 4. Resource Updates vs Creates + +**Understand FHIR update semantics:** + +```javascript +// Create: POST /Patient +// Result: New resource with server-assigned ID + +// Update: PUT /Patient/patient-123 +// Result: Replaces entire resource + +// Patch: PATCH /Patient/patient-123 +// Result: Updates specific fields (use JSON Patch or FHIR Patch) + +// Conditional Update: PUT /Patient?identifier=mrn|12345 +// Result: Updates patient matching identifier OR creates if not exists +``` + +**FDR should specify:** +- When to create vs update +- How to detect duplicates (identifier matching) +- How updates handle existing data (merge vs replace) + +## Implementation Guide Reference + +**Common IGs used at b.well:** + +### US Core +Base profiles for US healthcare: +- Patient, Practitioner, Organization +- Observation (vitals, labs) +- Condition, Procedure, Medication +- https://hl7.org/fhir/us/core/ + +### CARIN Blue Button +Insurance and claims data: +- Coverage, ExplanationOfBenefit +- https://hl7.org/fhir/us/carin-bb/ + +### DaVinci +Payer data exchange: +- Coverage, Prior Authorization +- https://hl7.org/fhir/us/davinci-* + +### National Directory (NDH) +Provider directories: +- Practitioner, Organization, Location, HealthcareService +- https://hl7.org/fhir/us/ndh/ + +## Querying Helix Profiles + +**When designing FHIR resources, check existing Helix profiles:** + +``` +Let me check existing Helix profiles for MedicationRequest... + +[If FHIR Server MCP available: + Query fhir.icanbwell.com for: + - StructureDefinition?name=MedicationRequest + - Related profiles +] + +Found Helix MedicationRequest profile: +- Based on: US Core MedicationRequest +- Extensions: bwell-pharmacy-preference, bwell-prescriber-npi +- Required fields: subject, medicationCodeableConcept, authoredOn +- Must-support: dosageInstruction, dispenseRequest + +Your implementation should follow this profile for consistency." +``` + +## FDR Scaffolding Template + +When generating draft FDR: + +```markdown +# FDR: [Resource] - [Use Case] + +**Status:** Draft +**Author:** [Your name] +**Date:** [YYYY-MM-DD] +**Reviewers:** fhir_design_review@icanbwell.com + +## Use Case + +[Describe what business problem this FHIR resource mapping solves] + +**User story:** As a [role], I need to [action] so that [benefit]. + +## Resource Selection + +**Primary Resource:** [ResourceType (e.g., MedicationRequest)] + +**Why this resource:** +- [Reason 1: semantics match] +- [Reason 2: IG support] +- [Alternative considered: why rejected] + +## Base Profile + +**Profile:** [e.g., US Core MedicationRequest 4.0.0] +**URL:** https://hl7.org/fhir/us/core/StructureDefinition/us-core-medicationrequest + +**Helix Profile:** (if exists) +**URL:** https://fhir.icanbwell.com/4_0_0/StructureDefinition/helix-medicationrequest + +## Field Mapping + +| Business Concept | FHIR Field | Cardinality | Notes | +|------------------|------------|-------------|-------| +| Prescription ID | id | 1..1 | Server-assigned UUID | +| Patient | subject | 1..1 | Reference(Patient) | +| Medication | medicationCodeableConcept | 1..1 | RxNorm code | +| Prescriber | requester | 1..1 | Reference(Practitioner) | +| Dosage | dosageInstruction | 1..* | Structured dosage | +| Pharmacy | dispenseRequest.performer | 0..1 | Reference(Organization) | + +## Extensions Needed + +**None** - All fields map to native FHIR + +OR + +**Custom Extensions:** +1. `bwell-prescription-source` + - **URL:** https://fhir.icanbwell.com/4_0_0/StructureDefinition/prescription-source + - **Type:** CodeableConcept + - **Purpose:** Track if prescription came from EHR, portal, or telepharmacy + - **Values:** ehr | portal | telepharmacy + +## CodeSystem / ValueSet + +**Required:** +- **Medication codes:** RxNorm (https://www.nlm.nih.gov/research/umls/rxnorm/) +- **Status:** MedicationRequest status (http://hl7.org/fhir/ValueSet/medicationrequest-status) + +**Custom ValueSets:** (if needed) +- `bwell-prescription-source-vs` - [Describe values] + +## Update Semantics + +**Create:** New prescription = POST /MedicationRequest +**Update:** Prescription changes = PUT /MedicationRequest/{id} +**Status changes:** Use PATCH with status field update + +**Duplicate Detection:** Match on: +- subject (patient) +- medicationCodeableConcept (same medication) +- authoredOn (within 1 hour) + +## Examples + +```json +{ + "resourceType": "MedicationRequest", + "id": "medreq-12345", + "status": "active", + "intent": "order", + "medicationCodeableConcept": { + "coding": [{ + "system": "http://www.nlm.nih.gov/research/umls/rxnorm", + "code": "860975", + "display": "Lisinopril 10 MG Oral Tablet" + }] + }, + "subject": { + "reference": "Patient/patient-456" + }, + "authoredOn": "2026-03-05T10:00:00Z", + "requester": { + "reference": "Practitioner/practitioner-789" + }, + "dosageInstruction": [{ + "text": "Take 1 tablet by mouth daily", + "timing": { + "repeat": { + "frequency": 1, + "period": 1, + "periodUnit": "d" + } + } + }] +} +``` + +## Related FDRs + +- [Link to related FDR 1] +- [Link to related FDR 2] + +## Questions for Review + +1. [Open question about edge case] +2. [Uncertainty about field cardinality] + +## Review Notes + +[Space for fhir_design_review@icanbwell.com feedback] +``` + +## Common FHIR Patterns at b.well + +**Patient Matching:** +- Use identifier with system + value +- system: `http://bwell.com/fhir/identifier/patient-uuid` +- Don't create duplicate patients - search first + +**Practitioner NPI:** +- Always include NPI in Practitioner.identifier +- system: `http://hl7.org/fhir/sid/us-npi` + +**Organization:** +- Include TIN (Tax ID) in identifier +- system: `http://hl7.org/fhir/sid/us-ein` + +**Observations:** +- Use LOINC codes for lab/vital observations +- system: `http://loinc.org` + +--- + +**Remember:** FHIR is a standard, but standards have many valid interpretations. FDR process ensures consistency across b.well platform. When in doubt, check existing FDRs or ask fhir_design_review@icanbwell.com before implementing. diff --git a/.claude/skills/find-stale-feature-flags/SKILL.md b/.claude/skills/find-stale-feature-flags/SKILL.md new file mode 100644 index 00000000..5698c5ea --- /dev/null +++ b/.claude/skills/find-stale-feature-flags/SKILL.md @@ -0,0 +1,78 @@ +--- +name: find-stale-feature-flags +description: + Find stale feature flags older than N months. Use when asked to identify, + audit, or clean up old feature flags in the codebase. +--- + +# Find Stale Feature Flags + +Scans all `useFeatureFlagValue` and `useSearchParamFeatureFlag` usages across +`libs/` and `apps/`, then uses `git log -S` to determine when each flag string +was first introduced. Flags older than the threshold are reported as STALE. + +All paths below are relative to the repo root (`ui-platform/`). + +## Run + +```bash +node .claude/skills/find-stale-feature-flags/find-stale-flags.mjs [months] +``` + +Default threshold is 6 months. Pass a number to override: + +```bash +node .claude/skills/find-stale-feature-flags/find-stale-flags.mjs 3 # flags older than 3 months +node .claude/skills/find-stale-feature-flags/find-stale-flags.mjs 12 # flags older than 1 year +``` + +## Output + +Markdown table grouped by status (STALE / OK / UNKNOWN): + +| Column | Meaning | +| ----------- | -------------------------------------------------------- | +| Flag | The feature flag string key | +| First Seen | Date the flag string first appeared in git history | +| Age | Days since first introduction | +| Location(s) | File(s) where the flag is used (first + count of others) | + +## How flags are detected + +The script matches three patterns: + +1. **Direct usage**: `useFeatureFlagValue('flag-name', defaultValue)` +2. **Search param variant**: + `useSearchParamFeatureFlag('flag-name', defaultValue)` +3. **Constant reference**: `const MY_FLAG = 'flag-name'` used with either hook + above + +Test files, mocks, and `node_modules` are excluded. + +## After running — what to do with stale flags + +For each STALE flag: + +1. **Check if fully rolled out** — If the flag is enabled 100% in production, + remove the flag check and keep only the "enabled" code path. +2. **Check if abandoned** — If the feature was never shipped, remove both code + paths and the flag. +3. **Still needed?** — Some flags gate features per-client. Document why in a + comment if it must stay. + +To remove a flag, search for all its usages: + +```bash +grep -rn "flag-name-here" libs/ apps/ --include="*.ts" --include="*.tsx" +``` + +## Gotchas + +- **Git history depth**: Uses `git log -S` (pickaxe) which searches the full + local history. If the repo was shallow-cloned, dates may be inaccurate. +- **Renamed flags**: If a flag was renamed, the script sees the new name's + introduction date, not the original concept's age. +- **Runtime-only flags**: Flags checked via the SDK at runtime (not hardcoded in + source) won't be detected. This only finds flags referenced in TypeScript. +- **Execution time**: ~2-5 minutes depending on number of flags (each requires a + git log traversal). diff --git a/.claude/skills/find-stale-feature-flags/find-stale-flags.mjs b/.claude/skills/find-stale-feature-flags/find-stale-flags.mjs new file mode 100755 index 00000000..f38daa14 --- /dev/null +++ b/.claude/skills/find-stale-feature-flags/find-stale-flags.mjs @@ -0,0 +1,274 @@ +#!/usr/bin/env node +/** + * Finds feature flags in the ui-platform codebase and identifies those + * introduced more than N months ago (default: 6). + * + * Usage: node find-stale-flags.mjs [months_threshold] + */ +import { execFileSync } from 'child_process'; +import { readdirSync, readFileSync, realpathSync } from 'fs'; +import { isAbsolute, join, relative } from 'path'; + +const THRESHOLD_MONTHS = parseInt(process.argv[2] || '6', 10); +const REPO_ROOT = execFileSync('git', ['rev-parse', '--show-toplevel'], { + encoding: 'utf8', +}).trim(); + +const cutoffDate = new Date(); +cutoffDate.setMonth(cutoffDate.getMonth() - THRESHOLD_MONTHS); +const cutoffStr = cutoffDate.toISOString().slice(0, 10); + +function findFiles(dir, pattern) { + const results = []; + try { + const entries = readdirSync(dir, { withFileTypes: true }); + for (const entry of entries) { + const fullPath = join(dir, entry.name); + const relativePath = relative(dir, fullPath); + if (relativePath.startsWith('..') || isAbsolute(relativePath)) { + continue; + } + if (entry.isDirectory()) { + if ( + entry.name === 'node_modules' || + entry.name === '.git' || + entry.name === 'dist' + ) + continue; + results.push(...findFiles(fullPath, pattern)); + } else if (pattern.test(entry.name)) { + results.push(fullPath); + } + } + } catch { + /* skip unreadable dirs */ + } + return results; +} + +function extractFlags(filePath) { + const flags = new Set(); + try { + const resolvedPath = realpathSync(filePath); + if (!resolvedPath.startsWith(REPO_ROOT)) { + return flags; + } + const content = readFileSync(resolvedPath, 'utf8'); + if ( + filePath.includes('.test.') || + filePath.includes('__mocks__') || + filePath.includes('mock') + ) + return flags; + + // Pattern 1: useFeatureFlagValue('flag-name', ...) or useFeatureFlagValue('flag-name', ...) + const directMatches = content.matchAll( + /useFeatureFlagValue(?:<[^>]+>)?\(\s*['"]([^'"]+)['"]/g, + ); + for (const m of directMatches) { + flags.add(m[1]); + } + + // Pattern 2: useSearchParamFeatureFlag('flag-name', ...) + const searchParamMatches = content.matchAll( + /useSearchParamFeatureFlag(?:<[^>]+>)?\(\s*['"]([^'"]+)['"]/g, + ); + for (const m of searchParamMatches) { + flags.add(m[1]); + } + + // Pattern 3: Constants defined and used with useFeatureFlagValue + // e.g., const MY_FLAG = 'flag-name'; ... useFeatureFlagValue(MY_FLAG, ...) + if ( + content.includes('useFeatureFlagValue') || + content.includes('useSearchParamFeatureFlag') + ) { + const constMatches = content.matchAll( + /const\s+(\w+)\s*=\s*['"]([a-z][a-z0-9 _-]+)['"]/gi, + ); + for (const m of constMatches) { + const constName = m[1]; + const value = m[2]; + if ( + content.includes(`useFeatureFlagValue(${constName}`) || + content.includes(`useSearchParamFeatureFlag(${constName}`) + ) { + flags.add(value); + } + } + } + } catch { + /* skip unreadable files */ + } + return flags; +} + +function gitFirstSeen(flagStr) { + try { + const result = execFileSync( + 'git', + [ + 'log', + '--all', + '--diff-filter=A', + '--format=%ai', + `-S${flagStr}`, + '--', + '*.ts', + '*.tsx', + ], + { encoding: 'utf8', cwd: REPO_ROOT, timeout: 30000 }, + ).trim(); + const lines = result.split('\n').filter(Boolean); + if (lines.length > 0) { + return lines[lines.length - 1].slice(0, 10); + } + + // Fallback: any commit that mentions this string + const fallback = execFileSync( + 'git', + ['log', '--all', '--format=%ai', `-S${flagStr}`, '--', '*.ts', '*.tsx'], + { encoding: 'utf8', cwd: REPO_ROOT, timeout: 30000 }, + ).trim(); + const fallbackLines = fallback.split('\n').filter(Boolean); + if (fallbackLines.length > 0) { + return fallbackLines[fallbackLines.length - 1].slice(0, 10); + } + } catch { + /* git command failed */ + } + return null; +} + +function daysBetween(dateStr1, dateStr2) { + const d1 = new Date(dateStr1); + const d2 = new Date(dateStr2); + return Math.floor((d2 - d1) / (1000 * 60 * 60 * 24)); +} + +// --- Main --- + +console.log('=== Stale Feature Flag Report ==='); +console.log( + `Threshold: flags introduced before ${cutoffStr} (${THRESHOLD_MONTHS} months ago)`, +); +console.log('Scanning codebase...\n'); + +const tsFiles = [ + ...findFiles(join(REPO_ROOT, 'libs'), /\.(tsx?|jsx?)$/), + ...findFiles(join(REPO_ROOT, 'apps'), /\.(tsx?|jsx?)$/), +]; + +// Collect all flags and their locations +const flagLocations = new Map(); // flag -> Set + +for (const file of tsFiles) { + const flags = extractFlags(file); + for (const flag of flags) { + if (!flagLocations.has(flag)) flagLocations.set(flag, new Set()); + flagLocations.get(flag).add(relative(REPO_ROOT, file)); + } +} + +console.log(`Found ${flagLocations.size} unique feature flags.\n`); + +// Look up git history for each flag +const results = []; +const today = new Date().toISOString().slice(0, 10); + +for (const [flag, files] of flagLocations) { + process.stderr.write(` Checking: ${flag}...\r`); + const firstSeen = gitFirstSeen(flag); + const age = firstSeen ? daysBetween(firstSeen, today) : null; + const status = !firstSeen + ? 'UNKNOWN' + : firstSeen < cutoffStr + ? 'STALE' + : 'OK'; + + results.push({ + flag, + firstSeen: firstSeen || 'unknown', + ageDays: age !== null ? age : '?', + files: [...files], + status, + }); +} + +process.stderr.write(' \r'); + +// Sort: STALE first (oldest first), then OK, then UNKNOWN +results.sort((a, b) => { + const statusOrder = { STALE: 0, UNKNOWN: 1, OK: 2 }; + if (statusOrder[a.status] !== statusOrder[b.status]) { + return statusOrder[a.status] - statusOrder[b.status]; + } + if (a.firstSeen === 'unknown') return 1; + if (b.firstSeen === 'unknown') return -1; + return a.firstSeen.localeCompare(b.firstSeen); +}); + +// Output +const stale = results.filter((r) => r.status === 'STALE'); +const ok = results.filter((r) => r.status === 'OK'); +const unknown = results.filter((r) => r.status === 'UNKNOWN'); + +if (stale.length > 0) { + console.log( + `\n## STALE FLAGS (>${THRESHOLD_MONTHS} months old) — ${stale.length} found\n`, + ); + console.log('| Flag | First Seen | Age | Location(s) |'); + console.log('|------|-----------|-----|-------------|'); + for (const r of stale) { + const loc = + r.files.length <= 2 + ? r.files.join(', ') + : `${r.files[0]} (+${r.files.length - 1} more)`; + console.log(`| \`${r.flag}\` | ${r.firstSeen} | ${r.ageDays}d | ${loc} |`); + } +} + +if (ok.length > 0) { + console.log( + `\n## OK FLAGS (<${THRESHOLD_MONTHS} months old) — ${ok.length} found\n`, + ); + console.log('| Flag | First Seen | Age | Location(s) |'); + console.log('|------|-----------|-----|-------------|'); + for (const r of ok) { + const loc = + r.files.length <= 2 + ? r.files.join(', ') + : `${r.files[0]} (+${r.files.length - 1} more)`; + console.log(`| \`${r.flag}\` | ${r.firstSeen} | ${r.ageDays}d | ${loc} |`); + } +} + +if (unknown.length > 0) { + console.log(`\n## UNKNOWN AGE — ${unknown.length} found\n`); + console.log('| Flag | Location(s) |'); + console.log('|------|-------------|'); + for (const r of unknown) { + const loc = + r.files.length <= 2 + ? r.files.join(', ') + : `${r.files[0]} (+${r.files.length - 1} more)`; + console.log(`| \`${r.flag}\` | ${loc} |`); + } +} + +console.log('\n=== Summary ==='); +console.log(`Total flags: ${results.length}`); +console.log(`Stale (>${THRESHOLD_MONTHS} months): ${stale.length}`); +console.log(`OK (<${THRESHOLD_MONTHS} months): ${ok.length}`); +console.log(`Unknown age: ${unknown.length}`); +console.log(`Cutoff date: ${cutoffStr}`); + +if (stale.length > 0) { + console.log('\n=== Recommended Actions ==='); + console.log('For each STALE flag, verify with the team whether:'); + console.log( + ' 1. The feature is fully rolled out → remove the flag and dead code path', + ); + console.log(' 2. The feature was abandoned → remove both code paths'); + console.log(' 3. The flag is still needed → document why in a comment'); +} diff --git a/.claude/skills/iterative-investigation/SKILL.md b/.claude/skills/iterative-investigation/SKILL.md new file mode 100644 index 00000000..ed015ebe --- /dev/null +++ b/.claude/skills/iterative-investigation/SKILL.md @@ -0,0 +1,1060 @@ +--- +name: iterative-investigation +description: Deep-debug production bugs using iterative hypothesis-driven tracing. Follows symptoms through code paths layer by layer until 100% root cause confidence, then uses TDD to fix. Designed for Java/Spring/Kafka/FHIR microservice architecture. +argument-hint: " — e.g. 'execution abc123 stuck in BUNDLING', 'personCount:0 but persons are bundled', 'Kafka lag spike on normalizer'" +disable-model-invocation: false +allowed-tools: Bash, Read, Write, Edit, Agent +--- + +# Iterative Investigation Skill + +Deep-debug production bugs by iteratively tracing through code paths, forming and verifying hypotheses at each layer, achieving 100% root cause confidence, and implementing a TDD fix. + +## Core Principles + +1. **Never accept the first explanation** — go deeper until you can explain WHY +2. **Each iteration narrows scope by 50%+** — every round eliminates half the possibility space +3. **Trace BOTH the happy path AND the failing path** — the divergence IS the bug +4. **Verify every hypothesis with evidence** — code trace, log query, DB query, or test +5. **100% confidence gate** — do NOT propose a fix until you can prove the bug exists with a reproducing test +6. **Minimal fix** — change the least amount of code that makes the failing test pass + +## Anti-Patterns to Avoid + +- Guessing at the fix based on symptoms alone +- Fixing without a reproducing test (you will fix the wrong thing) +- Surface-level analysis ("oh the config is wrong") without explaining the mechanism +- Expanding scope prematurely before exhausting the current layer +- Conflating correlation with causation in logs +- Assuming the bug is in the code you are looking at (it may be upstream) +- Reading only the failing path without understanding the happy path first +- **⚠️ BEWARE: Stopping at symptom confidence and declaring 100% root cause** — See INE-816 cautionary example below +- Declaring an external/upstream system (FHIR server, Kafka, another service) "broken" without first reading its contract — its sort order, its cursor semantics, its documented behavior. The boundary may be correct and YOUR layer may be overriding it. +- "Fixing" a symptom by ADDING a compensating layer on top of behavior you've labeled "wrong but unchangeable," when the real fix is to DELETE code and align with the existing contract. A root-cause fix often has a negative line count. + +**Critical Anti-Patterns (Code Reading vs. Reality):** +- ❌ Declaring confidence based only on reading code, without asking the user for actual data +- ❌ Saying "the code does X" without verifying X actually happens with real data +- ❌ Assuming a field has value Y without showing its actual value from logs/Kafka/API responses +- ❌ Tracing a code path from A→B→C without confirming that data actually flows that way +- ❌ Reading code that extracts field X and assuming the source always has field X (it might not) +- ❌ Saying "100% confident" without evidence from Phase 2.5 bridge—code understanding alone is insufficient + +**Critical Anti-Pattern (Phase 1 Data Collection):** +- ❌ Skipping Phase 1 data collection and jumping straight to code tracing +- ❌ Assuming you understand the problem without seeing real examples +- ❌ Saying "I'll trace code and verify against data later" — reverse the order +- ❌ Treating the user's description as complete instead of asking "show me an actual example" +- ❌ Going 2+ iterations into Phase 2 before asking "can I see the actual data from your system?" + +**The fix:** Ask for real data immediately in Phase 1. Answer is usually there. + +**Critical Anti-Pattern (Workaround Masquerading as a Fix):** +- ❌ "The [server/upstream] emits the wrong X; I'll correct it on our side" — before proving the upstream contract is actually violated, not just unfamiliar +- ❌ Writing "this is a client-side workaround; the proper fix is in [other system]; a separate ticket will track it." If you're deferring the "proper fix," you have NOT confirmed root cause — you've assumed it's unfixable. Stop and prove that first. +- ❌ A fix that only ADDS a translation/correction/normalization layer. Ask: "what if I DELETE the thing that's misbehaving instead of compensating for it?" + +**The tell:** If your fix adds code to compensate for a boundary you've declared broken, verify the boundary's contract first. The bug is usually that something in YOUR layer is already overriding a correct contract. (Real case: INE-653 added 71 lines to "correct" a FHIR `next`-link cursor the server emitted correctly; INE-600 fixed the underlying paging failure by DELETING those 71 lines and aligning the query's sort with the server's cursor field. See case study below.) + +--- + +## Phase 1: Symptom Collection + Real Data Inspection + +**Goal:** Establish exactly what is broken, what the expected behavior is, and what the actual behavior is — AND collect real examples of the phenomenon from the production system. + +### Actions + +1. **Gather the observable facts** — What does the user/monitoring report? Error messages, stuck states, unexpected values. +2. **Define expected vs actual** — Be precise. "personCount should be 5 but is 0" not "it's broken." +3. **Establish timeline** — When did it start? Is it intermittent or consistent? What changed recently? +4. **Identify the affected scope** — One execution? One org? All orgs? One measure? All measures? + +### CRITICAL STEP: Collect Real Examples Before Proceeding + +**Do NOT proceed to Phase 2 (code tracing) until you have actual data examples.** Many bugs are visible in the data and invisible in the code. + +Based on the problem type, ask the user for: + +**If about event data:** +- "Show me actual examples of [event type] from Kafka or logs" +- Get 2-3 examples showing the problem +- Example: "Show me actual directActionTaskReady events for both connectionFound and connectionFixed workflows" + +**If about resource data:** +- "Show me actual FHIR resources that exhibit this problem" +- Get the JSON or XML, not just a description +- Example: "Show me the actual ActivityDefinition JSON for both workflows" + +**If about state/data inconsistency:** +- "Show me the actual values in the database/system" +- Query results, API responses, current state +- Example: "What does the Task resource actually contain for this scenario?" + +**If about counts/metrics:** +- "What are the actual numbers from your system right now?" +- Not hypothetical, actual measurements +- Example: "What are the actual values you're seeing for personCount?" + +**If about queries/filters:** +- "Show me the actual query results or logs" +- Prove the hypothesis with real output +- Example: "What does Groundcover actually show for this execution?" + +### Why This Step Is Critical + +Code reading and reality diverge at boundaries: +- Code may extract field X, but actual JSON doesn't have X +- Both paths look different in code but produce same output +- Field is optional and null 50% of the time +- Serialization/deserialization changes values + +**The answer is often in the data, not the code.** + +### Evidence Sources + +- **Production logs** — actual error messages, actual values, actual behavior +- **Kafka** — actual events flowing through the system right now +- **FHIR server** — actual resource JSON, actual values +- **Groundcover** — actual logs with actual field values +- **Database queries** — actual data in the system +- **API responses** — actual values returned +- **Screenshots/exports** — what the user is actually observing + +### Output of Phase 1 + +A clear problem statement WITH real examples: + +``` +EXPECTED: [precise expected behavior] +ACTUAL: [precise actual behavior, with real examples from prod] +REAL EXAMPLES: + - Event 1: [actual JSON/data showing the problem] + - Event 2: [actual JSON/data showing the problem] + - Relevant data: [what the user showed you] +SCOPE: [what is affected] +TIMELINE: [when it started, frequency] + +PRELIMINARY OBSERVATIONS FROM REAL DATA: + - [What the actual data shows about the problem] + - [Does it match the user's description?] + - [What's surprising or different from expected?] +``` + +**Do not proceed to Phase 2 until you have this data in hand.** + +### Phase 1 Data Collection: Query It Yourself + +Instead of asking the user to provide data, **query production systems yourself** using the credentials they provide. This lets you discover the answer directly rather than relying on the user's extraction. + +**Step 1: Request Access Credentials** + +Ask the user for credentials to access: +- FHIR server (fhir.staging.bwell.zone or prod) +- Kafka/Groundcover (for event logs) +- Database (if direct queries needed) +- Any other system relevant to the problem + +Example: "Can you provide credentials for fhir.staging.bwell.zone so I can query the ActivityDefinitions directly?" + +**Step 2: Query the Data Yourself Based on Problem Type** + +**For "does field X distinguish Y" questions:** +- Query both resources from FHIR server +- Compare actual JSON side-by-side +- Example: GET /ActivityDefinition?[search params] for both connection types, inspect code.coding values directly + +**For event/data flow questions:** +- Query Groundcover or Kafka for actual events +- Pull 3-5 real examples showing both cases +- Example: Search Groundcover for directActionTaskReady events for both workflows, see what's actually in them + +**For state/data inconsistency questions:** +- Query the current resource state from FHIR server +- See actual field values in production +- Example: GET /Task/{id} to see what fields actually contain + +**For counts/metrics questions:** +- Run Groundcover queries or database queries +- Get real measurements, not hypothetical +- Example: Query actual personCount values from logs + +**Step 3: Inspect the Data** + +Look at the actual JSON, values, and structure. Many times the answer is immediately obvious once you see the real data. + +### Safety Constraint (CRITICAL) + +When querying production: +- ✓ READ-ONLY operations only +- ✓ Inspecting data, logs, events +- ✓ Running queries to see what exists +- ✓ Examining FHIR resources +- ✗ NEVER modify, create, update, or delete resources +- ✗ NEVER trigger actions or changes as part of this investigation +- ✗ NEVER write to any system + +**If you discover you need to modify data to test something, STOP. Ask permission first and wait for user approval.** + +**Before saying "I need to trace code":** +- [ ] Have I requested the credentials I need? +- [ ] Have I queried the actual production data myself? +- [ ] Have I inspected 3+ real examples directly? +- [ ] Could the answer be visible in the real data without code tracing? +- [ ] Does the actual data match what the user described? + +If the answer to any is "no," query the data first. + +--- + +## The Most Important Gate: Phase 1 Data Collection + +**Insight from recent mistakes:** I skipped Phase 1 data collection and jumped to code tracing. This wasted hours and led to false confidence. + +**Why Phase 1 data matters most:** +- Most bugs are visible in the actual data +- Code tracing is necessary when data is ambiguous +- Querying data yourself finds the answer 10x faster than code reading +- False confidence comes from code reading without data verification + +**The pattern (slow):** +``` +Investigation that takes 2+ hours: +1. Read code +2. Make inferences about what data contains +3. Declare confidence based on code reading +4. User corrects with actual data +5. Theory collapses +``` + +**The pattern (fast):** +``` +Investigation that takes 5 minutes: +1. Request credentials to prod systems +2. Query the actual data yourself +3. Inspect actual JSON/values +4. Answer is usually visible in the data +5. Done (maybe no code tracing needed at all) +``` + +**How to recognize when you're about to skip Phase 1:** +- Your first instinct is "let me read the code" +- You haven't asked "can you provide credentials so I can query this?" +- You're making inferences about what data contains instead of querying to see it +- You're saying "the code should do X" instead of "let me query production to see what actually happens" +- You're about to proceed to Phase 2 without having seen actual production data + +**Stop.** Query production data first. Code second. + +--- + +## Phase 1 Production Access: Read-Only Safety Rules + +Since you have provided production credentials, this skill enables me to query production systems directly. **This makes investigations much faster, but it requires strict safety rules.** + +### What I Will Do +- ✓ Query FHIR server to fetch resources (GET) +- ✓ Query Groundcover for logs and metrics +- ✓ Query Kafka topics for event inspection +- ✓ Run read-only database queries if needed +- ✓ Search, list, and inspect all kinds of data +- ✓ Examine production state to understand the problem + +### What I Will NEVER Do +- ✗ Create resources (POST) +- ✗ Update resources (PUT/PATCH) +- ✗ Delete resources (DELETE) +- ✗ Modify configuration or state +- ✗ Trigger workflows or actions +- ✗ Write to any system +- ✗ Change any production data + +### Exception: Testing Fixes + +If Phase 4 (Reproduction) or Phase 5 (TDD Fix) requires creating test data in production: +1. **STOP and ask permission first** +2. Get explicit approval from the user +3. Describe exactly what I will create/modify +4. Wait for user confirmation +5. Only then proceed with modifications +6. Clean up any test data afterward + +**This skill is READ-ONLY by default. Modification requires explicit user approval.** + +--- + +## Phase 2: Iterative Trace + +**Goal:** Follow the code path from trigger to symptom, narrowing scope each round until you find the exact line/condition that diverges. + +### Method: Peel-the-Onion Iterations + +Each iteration follows this structure: + +**PART A: REPORT ON PREVIOUS ITERATION (if not first iteration)** + +Display the **Iteration Summary Report** from the last iteration: + +``` +## Iteration N Summary Report + +### Assumptions from Previous Iteration (Iteration N-1): + +| Assumption | Check Performed | Evidence Found | Status | Impact | +|---|---|---|---|---| +| [Previous assumption 1] | [What I checked] | [What I found] | ✓ HELD / ✗ DISPROVEN / ⚠️ INCONCLUSIVE | [Did it narrow scope?] | +| [Previous assumption 2] | [What I checked] | [What I found] | ✓ HELD / ✗ DISPROVEN / ⚠️ INCONCLUSIVE | [Did it narrow scope?] | +| ... | | | | | + +### What This Iteration Revealed: +- ✓ Confirmed findings: [Specific discoveries] +- ✗ Disproven theories: [What we ruled out] +- ⚠️ Inconclusive: [What needs more data] +- 🎯 Scope narrowed from [X possibilities] to [Y possibilities] + +### Hypothesis Evolution: +- **Iteration 1 hypothesis:** [Original theory] +- **Current hypothesis (Iteration N):** [How it evolved] +- **Rejected paths:** [Dead ends we ruled out] + +### Confidence Trajectory: +- Iteration 1: LOW (many unknowns) +- Iteration N: MEDIUM/HIGH (X of Y key assumptions confirmed) +- Next blocker: [What would increase confidence further] +``` + +**Stop here.** You review. You can say: +- "That finding is wrong, here's what actually happened" +- "I have evidence that contradicts that" +- "Let me show you what I found" + +**PART B: DISPLAY NEW ASSUMPTIONS (START OF CURRENT ITERATION)** + +Create a fresh table for this iteration so you can review before I proceed: + +``` +## Current Assumptions (Iteration N): + +| Assumption | Source (code/inference) | Previous Iteration Status | Will Validate By | +|---|---|---|---| +| [Assumption 1] | Code line X or inference | New / Refined from Iteration N-1 | [Specific check] | +| [Assumption 2] | Inference from Y | Carries forward from N-1 | [Specific check] | +| ... | | | | + +**Next, I will:** [List specific actions to take] + +**Questions for you before proceeding:** +- Do any of these assumptions look wrong based on your knowledge? +- Do you have evidence that contradicts any of these? +``` + +**Stop here.** Wait for your feedback. If you say assumptions look wrong, incorporate that immediately. + +**PART C: TRACE, CHECK, EVALUATE, DECIDE (CONTINUE IF VALIDATED)** + +1. **State current hypothesis** — "I believe the bug is in [component] because [evidence from Part A]" +2. **Design a check** — What specific thing would confirm or refute assumptions from Part B? +3. **Execute the check** — Read code, query logs, inspect data +4. **Evaluate result** — For each assumption: ✓ HELD / ✗ DISPROVEN / ⚠️ INCONCLUSIVE +5. **Decide next step** — Go deeper into this component, or pivot to another +6. **Loop to next iteration** — Go back to PART A and report findings + +### Tracing Patterns for Our Architecture + +#### FSM Guard Tracing (Orchestrator) +The orchestrator uses a Spring State Machine. State transitions are guarded: +``` +State A --[event]--> Guard evaluates --> State B (or stays in A) +``` +- Read the guard condition in `StateMachineConfig` or `*Guard.java` +- Trace what data the guard reads (usually from `ExecutionContext` or the DB) +- Check if the guard's precondition can silently fail (return false without logging) + +#### Kafka Event Flow Tracing +``` +Producer (publishes event) --> Topic --> Consumer (processes event) +``` +- Check: Was the event published? (producer logs, topic offset) +- Check: Was the event consumed? (consumer group lag, consumer logs) +- Check: Did the consumer process it successfully? (application logs, state change) +- Key diagnostic: If lag is 0 but no state change, the consumer processed and DROPPED the event (look at filtering/routing logic) + +#### Async/Reactive Pattern Tracing +- `@Async` methods swallow exceptions unless the caller checks the Future +- `@TransactionalEventListener` fires AFTER commit — if the transaction rolled back, the event never fires +- `@KafkaListener` with manual ack — if ack is never called, the message is redelivered (but only after session timeout) + +#### FHIR Query Tracing +- Check the query parameters being sent (not what you THINK is sent) +- Check if `_count` pagination is cutting off results +- Check if security labels / access scopes are filtering unexpectedly +- ProxyPatient (`Patient/person.{id}`) resolves to linked Patients — check Person links exist + +### Assumption Log (REQUIRED) + +At the start of Phase 2, create an explicit list of assumptions you're making about how the code works. Before proceeding to Phase 3, you must verify each assumption against real data/logs/examples. **Do not skip this step.** + +Example: +``` +ASSUMPTIONS: +1. ActivityType field contains the connection type value (NEEDS VERIFICATION) +2. This field is extracted from ActivityDefinition.code.coding (NEEDS VERIFICATION) +3. The field flows unchanged through TaskChangeEvent (NEEDS VERIFICATION) +4. Both workflows produce different values (NEEDS VERIFICATION) + +EVIDENCE FOR EACH: +1. [Ask user for actual ActivityDefinition JSON for both workflows] +2. [Check actual code path with real data, not just code reading] +3. [Trace actual Kafka message, not just code flow] +4. [Show actual event in hp-notification-service, not hypothetical] +``` + +### When to Expand Scope + +Expand scope (go upstream) when: +- The component you are investigating is behaving correctly given its inputs +- The inputs to this component are wrong/missing/stale +- The bug is in the data, not the code (then trace who wrote the bad data) + +Expand scope (go downstream) when: +- The component produces correct output but the next stage misinterprets it +- A serialization/deserialization boundary corrupts the data + +### Groundcover Query Patterns + +``` +# Trace a specific execution through orchestrator +* | filter workload:clinical-reasoning-orchestrator-service content:"{EXEC_ID}" | sort by (_time desc) | limit 100 + +# Find errors in a time window +* | filter workload:{SERVICE} level:error | filter _time >= now() - 1h | sort by (_time desc) | limit 50 + +# Trace a Kafka message by correlation ID +* | filter content:"{CORRELATION_ID}" | sort by (_time asc) | limit 100 + +# Find state transitions +* | filter workload:clinical-reasoning-orchestrator-service content:"transition" content:"{EXEC_ID}" | sort by (_time asc) +``` + +### Kafka Lag as a Diagnostic Signal + +| Lag Pattern | Meaning | +|---|---| +| Lag = 0, no state change | Consumer received and DROPPED the message (filter/routing bug) | +| Lag growing steadily | Consumer is slower than producer (throughput issue, not a bug) | +| Lag stuck at N > 0 | Consumer is stuck/crashed/in rebalance | +| Lag spikes then recovers | Temporary processing delay (pod restart, GC pause) | +| Lag = 0 across all groups | No messages being produced (upstream is the problem) | + +--- + +## Phase 2.5: Evidence Collection & Assumption Verification + +**Goal:** Bridge the gap between code reading and actual system behavior. This phase prevents confident claims that are based only on how code *looks* rather than how it *actually behaves*. + +### Why This Phase Exists + +Code reading and actual behavior can diverge: +- You read code that extracts field X, but the actual ActivityDefinition doesn't have field X +- Code paths look connected in isolation but serialization/deserialization corrupts data in reality +- You assume a field is "always set" but logs show it's null 50% of the time + +**You cannot declare 100% confidence without crossing this bridge.** + +### Actions (Required Before Phase 3) + +**Step 1: Retrieve Assumptions Table** + +Pull the assumptions table from the most recent Phase 2 iteration. This is what gets validated. + +**Step 2: Request Real System Evidence for Each Assumption** + +For each assumption in the table, ask you for concrete examples: + +``` +To validate these assumptions, I need to see: + +1. [Assumption 1]: Please show me [specific actual system output: log, event, JSON, metric] +2. [Assumption 2]: Please show me [specific actual system output] +... + +Without these, I cannot proceed to Phase 3. +``` + +**Step 3: Validate Against Real-World Data** + +For each assumption, cross-check against the evidence you provide: + +- If evidence matches assumption: ✓ VERIFIED +- If evidence contradicts assumption: ✗ REJECTED +- If evidence is unclear: ❓ INCONCLUSIVE (ask follow-up questions) + +**Step 4: Create Validation Report** + +Produce a report table showing findings: + +``` +## Phase 2.5 Assumption Validation Report + +| # | Assumption | Evidence Provided | Finding | Confidence Impact | +|---|---|---|---|---| +| 1 | [Assumption 1] | [What you showed me] | ✓ VERIFIED / ✗ REJECTED / ❓ INCONCLUSIVE | Confidence: [HIGH/MEDIUM/LOW] | +| 2 | [Assumption 2] | [What you showed me] | ✓ VERIFIED / ✗ REJECTED / ❓ INCONCLUSIVE | Confidence: [HIGH/MEDIUM/LOW] | +| ... | | | | | + +**Summary:** +- X assumptions verified +- Y assumptions rejected (requires Phase 2 revision) +- Z assumptions inconclusive (need more evidence) + +**If any assumptions rejected:** Return to Phase 2. Do not proceed to Phase 3. + +**If all assumptions verified or inconclusive is acceptable:** Proceed to Phase 3. +``` + +**Step 5: Explicitly List What Remains Unverified** + +``` +UNVERIFIED ELEMENTS (acceptable to proceed with caveats): +- [Element 1] — could not get evidence, but low criticality +- [Element 2] — evidence inconclusive, but X still holds + +DEAL-BREAKERS THAT WOULD BLOCK PHASE 3: +- Any assumption marked ✗ REJECTED +- Any critical assumption marked ❓ without clear resolution +``` + +### When to Block Phase 3 + +Do NOT proceed to Phase 3 confidence gate if: +- [ ] You've assumed a field is present without seeing it in actual data +- [ ] You've traced a code path but not verified the data actually flows that way +- [ ] You've said "the code does X" without showing an example where X happens +- [ ] You haven't asked the user for real examples from their system + +**If blocked:** Go back to Phase 2 and narrow further, OR ask the user for actual system evidence. + +--- + +## Phase 3: Root Cause Confirmation + +**Goal:** Achieve 100% confidence in the root cause with concrete evidence. + +### Confidence Checklist + +You are at 100% confidence when ALL of these are true: + +**Code Understanding (not sufficient alone):** +- [ ] You can point to the exact line(s) of code that cause the bug +- [ ] You can explain the mechanism: why does this line produce wrong behavior? +- [ ] You can explain why it works in the happy path but fails in the bug path +- [ ] You can predict: "if I change X, behavior Y will change to Z" + +**Behavior Verification (REQUIRED—you cannot skip this):** +- [ ] You have VERIFIED with actual system evidence that the bug exists (not just code reading) +- [ ] You can point to actual logs/data/metrics showing the wrong behavior +- [ ] You've shown the actual data structure (JSON, log line, metric value) that proves your theory +- [ ] If your hypothesis depends on a field having a specific value, you've shown that field's actual value from the real system +- [ ] You've explicitly checked your assumptions against real data and found NO contradictions + +**Disqualifiers—if any of these are true, you are NOT at 100% confidence:** +- You've read code but haven't verified the actual data matches your understanding +- You've assumed a field exists without seeing it in real data +- You've said "should" or "probably" instead of "is" based on logs/data +- You've traced code paths without confirming the data actually flows that way +- You haven't asked the user for real system evidence +- You've concluded an external system is "broken" without reading its actual contract (sort order, cursor field, documented semantics) and confirming YOUR layer isn't overriding it +- Your proposed fix ADDS a compensating layer rather than removing a defect — and you haven't asked whether deleting/aligning would be simpler +- You're planning to file a "separate ticket for the proper fix" — meaning you have a workaround, not a root cause + +### Causality Depth Check — Don't Get Trapped by Symptom Confidence + +**⚠️ THE TRAP:** You can point to an exact line of code that causes the bug. You understand why it fails. You have 100% confidence. **But you've only found the symptom, not the root cause.** + +**How this trap catches you:** + +1. Investigation identifies: "The guard doesn't exclude PENDING persons, so the expected count inflates" +2. You point to the exact line in `BundlingCountResolver.getTotalPersonCount()` +3. You declare: "100% confident — the guard logic is broken" +4. You implement the symptom fix: exclude PENDING from the count +5. **You miss:** Why do PENDING stragglers exist in the first place? (Hibernate ORM returns stale cached entity when a pessimistic lock re-check runs) + +**Real example:** INE-816 investigation could have stopped at "exclude PENDING from guard count" (symptom) instead of asking "why do stragglers exist?" (root cause). Two valid solutions, very different scope: +- **PR #157 (symptom fix):** 1 file changed, pragmatic, doesn't prevent recurrence +- **PR #160 (root cause):** 17 files changed, uses CAS gates, makes the condition impossible + +See `docs/superpowers/case-studies/INE-816-symptom-vs-root-cause.md` for the full cautionary tale. + +--- + +### How to Avoid the Trap: Causality Depth Questions + +After you've identified the immediate cause and declared preliminary confidence, **stop and ask these four questions before Phase 4:** + +1. **"Is this condition preventable, or is it inevitable?"** + - If YES (preventable): Go deeper. Don't stop here. + - If NO (inevitable): This is the root cause. Proceed to Phase 4. + +2. **"Trace back: What operation creates this condition?"** + - Stragglers exist → Who creates them? → Why doesn't the add-gate prevent it? → **Hibernate cache returns stale entity** (root cause found) + +3. **"Compare the two solutions I could implement:"** + - **Symptom fix:** Make downstream resilient to the condition (e.g., "ignore PENDING rows") + - Pros: Smaller change, pragmatic for production fires + - Cons: Tolerates a condition that shouldn't happen; may hide future regressions + - **Root-cause fix:** Prevent the condition entirely (e.g., "use atomic CAS gates") + - Pros: Architecturally clean, prevents recurrence, removes the trap for future developers + - Cons: Larger change, more invasive, longer to implement + +4. **"Which should I implement, given the context?"** + - **Production on-fire scenario:** Deploy symptom fix NOW to stabilize, then plan root-cause fix as follow-up + - **Planned architectural work:** Implement root-cause fix; the urgency is lower, the quality bar is higher + - **Either way:** Document both approaches in the PR so the trade-off is intentional and visible to reviewers + +--- + +### Common Root Cause Patterns in Our Architecture + +| Pattern | Example | +|---|---| +| Silent guard failure | Guard returns `false` without logging, blocking state transition | +| Race condition in async | Two Kafka consumers update the same entity, last-write-wins | +| Off-by-one in count | `personCount` set from query result before filtering, actual != expected | +| Stale cache | `ActivityDefinition` cache returns old version after update; **ORM returns cached entity instead of reading fresh committed state** (INE-816) | +| Missing null check | Optional field absent in FHIR resource, NPE in processing | +| Wrong query scope | Query uses `_count=100` but org has 150 persons | +| Serialization mismatch | Kafka message schema v2 consumed by v1 deserializer | +| Transaction boundary | DB write committed but Kafka publish in same tx rolls back (or vice versa) | + +--- + +## Phase 4: Reproduction + +**Goal:** Write a failing integration test that demonstrates the bug exists. + +### Test Structure + +Always use the `// GIVEN // WHEN // THEN` format: + +```java +@Test +void shouldHandleSpecificBugCondition() { + // GIVEN — set up the exact conditions that trigger the bug + // (use the minimum data needed to reproduce) + + // WHEN — trigger the action that exposes the bug + + // THEN — assert the CORRECT behavior (test FAILS before fix) +} +``` + +### Reproduction Strategy + +1. **Identify the minimal reproduction** — What is the smallest input that triggers the bug? +2. **Write the test at the right level** — Integration test if it crosses boundaries, unit test if it is pure logic +3. **Name the test descriptively** — `shouldNotResetPersonCountWhenBundlingCompletes` not `testBug123` +4. **Assert the CORRECT behavior** — The test must FAIL on the current code and PASS after the fix + +### Where to Put the Test + +- Logic bug in a service method: unit test in the same module +- Bug in Kafka event handling: integration test with embedded Kafka +- Bug in FSM transition: integration test with Spring State Machine test support +- Bug in FHIR query construction: unit test mocking the FHIR client +- Bug spanning multiple services: E2E test in `hp-validation-tests` repo + +### Verify the Test Fails + +Run the test and confirm it fails for the RIGHT reason: +- The assertion should fail (not a setup error, not a compilation error) +- The failure message should clearly indicate the bug behavior + +--- + +## Phase 5: TDD Fix + +**Goal:** Make the failing test pass with the minimal code change. + +### Process + +1. **Run the failing test** — confirm it fails (red) +2. **Implement the minimal fix** — change only what is necessary +3. **Run the test again** — confirm it passes (green) +4. **Run the full test suite** — confirm no regressions +5. **Refactor if needed** — clean up without changing behavior (refactor) + +### Fix Quality Checklist + +- [ ] Fix addresses the root cause, not a symptom +- [ ] Fix does not change behavior for the happy path +- [ ] Fix handles edge cases (null, empty, concurrent) +- [ ] Fix includes appropriate logging (so future debugging is easier) +- [ ] Fix does not introduce a new silent failure mode +- [ ] All existing tests still pass + +### Common Fix Patterns + +| Root Cause | Fix Pattern | +|---|---| +| Silent guard failure | Add explicit logging on guard failure + fix the guard condition | +| Race condition | Add optimistic locking or idempotency check | +| Wrong count | Move the count assignment to after filtering | +| Stale cache | Add cache invalidation trigger or TTL | +| Missing null check | Add null guard with appropriate default/skip behavior + log | +| Wrong query scope | Fix pagination or remove artificial limit | + +--- + +## Phase 6: Documentation + +**Goal:** Create a durable record of the investigation for future reference. + +### Investigation Document Template + +Create a markdown file in the project memory or repo docs: + +```markdown +# [Ticket ID] [Brief Description] + +## Symptom +What was observed. Include specific values, timestamps, affected scope. + +## Timeline +- [timestamp] Symptom first reported +- [timestamp] Key investigation milestones +- [timestamp] Root cause identified +- [timestamp] Fix implemented and verified + +## Root Cause +Precise technical explanation. Include: +- The exact code path that fails +- Why it fails (the mechanism) +- Why it only fails under specific conditions +- Why it was not caught by existing tests + +## Fix +- What was changed (file, line, logic) +- Why this fix is correct +- What the failing test asserts + +## Lessons Learned +- What made this bug hard to find? +- What monitoring/logging would have caught it sooner? +- Are there similar patterns elsewhere that need the same fix? +``` + +### Memory Updates + +After the investigation, update project memory with: +- The root cause pattern (so it can be recognized faster next time) +- Any new diagnostic queries that proved useful +- Any architectural insight gained + +--- + +## Workflow Summary + +``` +Phase 1: Symptom Collection + Real Data Inspection + | + ├─ Ask clarifying questions + ├─ Define expected vs actual + └─ REQUEST REAL DATA EXAMPLES (MANDATORY) + | + | (BLOCKED until user provides actual data) + | "Can you show me actual [events/resources/logs]?" + | + v + (Answer often visible in data, Phase 2 may be unnecessary) + | + └─→ If data answers the question: Investigation complete + | + └─→ If data doesn't fully answer: Proceed to Phase 2 + | + v +Phase 2: Iterative Trace (loop until narrow) + | + | (Each iteration: Summary Report → New Assumptions → Trace → Evaluate) + | + ├─→ Iteration 1: + | ├─ Summary Report: [Show confirmed/disproven from previous] + | ├─ New Assumptions Table [user reviews, can interject] + | └─ Trace/Evaluate/Report findings + | | + | └─→ (Scope narrowed? Continue) → Iteration 2 + | | + | ├─ Summary Report: [Show what iteration 1 proved/disproved] + | ├─ New Assumptions Table [user reviews, can interject] + | └─ Trace/Evaluate/Report findings + | | + | └─→ (Scope narrow enough?) → Phase 2.5 + | + v +Phase 2.5: Evidence Collection (MANDATORY bridge) + | + ├─ Retrieve assumptions from final Phase 2 iteration + ├─ Request real system evidence for each + ├─ Create Validation Report (✓/✗/❓) + └─ BLOCKED if ✗ REJECTED → Return to Phase 2 + | + v +Phase 3: Root Cause Confirmation (100% gate with real evidence) + | + v +Phase 4: Reproduction (failing test) + | + v +Phase 5: TDD Fix (make test pass) + | + v +Phase 6: Documentation +``` + +## Invoking This Skill + +When the user reports a bug or stuck execution: + +1. **Phase 1 (CRITICAL):** Start with symptom collection AND query production data yourself + - Ask clarifying questions if vague + - **MANDATORY:** Request credentials to access production systems (FHIR server, Kafka, Groundcover, etc.) + - Query the data yourself using those credentials (READ-ONLY only) + - Inspect actual JSON/data to see if the answer is visible + - Do not proceed to Phase 2 until you've queried and inspected real production data + - Real data often makes the answer obvious without needing code tracing + - **Safety:** Only query/inspect. Never modify, create, update, or delete anything + +2. **Phase 2:** Trace code iteratively in cycles: + - **START OF EACH ITERATION:** Display Assumptions Table (user reviews, can interject) + - **"Questions for you before I proceed?"** — Wait for feedback + - If you say assumptions look wrong, incorporate that immediately + - If you say "let me show you," provide evidence and integrate it + - Then proceed: hypothesis → check → evaluate → decide + - Loop until you've narrowed scope sufficiently + +3. **Phase 2.5 (MANDATORY BRIDGE):** Validate all assumptions: + - Display assumptions from Phase 2 + - Request specific evidence for each + - Create Validation Report (✓ VERIFIED / ✗ REJECTED / ❓ INCONCLUSIVE) + - If any assumption rejected: Return to Phase 2, do not proceed + - If all verified or acceptable: Proceed to Phase 3 + +4. **Phase 3:** Explicitly declare 100% confidence only when: + - Code understanding is complete (from Phase 2) + - All critical assumptions are verified (from Phase 2.5) + - Real system evidence corroborates hypothesis + +5. **Phase 4:** Ask the user before writing the test — confirm the test location and approach + +6. **Phase 5:** Implement the fix and run the full test suite + +7. **Phase 6:** Offer to create the documentation + +### Key Interaction Points (Where You Can Interject) + +- **After each Assumptions Table:** Say "wait, that assumption is wrong" or "let me show you evidence" +- **At Phase 2.5:** Review the Validation Report. If you see ✗ REJECTED, we return to Phase 2 +- **Before Phase 3 declaration:** Ask "are all the critical assumptions really verified?" + +### Rules for This Skill + +- **At START OF EACH PHASE 2 ITERATION:** Print assumptions table and wait for feedback +- **NEVER declare high confidence without Phase 2.5 validation.** Code reading alone is insufficient. +- **If assumptions are ✗ REJECTED in Phase 2.5:** Go back to Phase 2. Do not skip. +- **Explicit beats implicit.** Show the table, list the checks, report the findings. Don't hide your reasoning. + +At any point: +> "I've listed my assumptions above. Do any of those look wrong to you before I proceed?" + +If the user says "just fix it" or "skip the test," push back gently: +> "I want to verify this against real system data first via Phase 2.5. Can you show me an actual [event/log] for each assumption? It takes 5 minutes and prevents us from fixing the wrong thing." + +--- + +## Why Phase 1 Data Collection Often Prevents Code Tracing Entirely + +**Real example:** "Can we distinguish connectionFound vs connectionFixed workflows in the event?" + +**Wrong approach (what I did):** +1. Read question: "Can we distinguish workflows?" +2. Jump to code tracing (Phases 2-3) without seeing data +3. Trace code, make inferences +4. Declare "100% confident" in theory based only on code +5. User corrects me with actual data +6. Theory collapses + +**Wasted time:** 2+ hours of code tracing + +**Right approach:** +1. Read question: "Can we distinguish workflows?" +2. **Phase 1:** Request FHIR server credentials +3. **Query ActivityDefinitions myself** for both strategies +4. **Inspect the actual JSON** → Both have code.coding[0].code = "data-connection" +5. **Answer visible immediately:** "No, the activityType field is identical. But I see identifier.value differs..." +6. **Optionally continue:** If needed, query actual events to see what flows through +7. **Done in 5 minutes without any code tracing** + +**Why this works:** The answer is already in the data. Code tracing is unnecessary until you've exhausted what the real data tells you. By querying myself, I don't depend on user extraction and can explore freely. + +### Anti-Pattern to Avoid + +❌ "I'll trace code to understand what should happen, then verify against data" + +✅ "I'll look at actual data first to see what IS happening, then understand why if needed" + +The second approach finds bugs 10x faster because many times the data tells the whole story. + +--- + +## Example: How This Prevents "False Confidence" + +**Scenario:** User asks "Can we distinguish connectionFound vs connectionFixed in directActionTaskReady events?" + +**Phase 2, Iteration 1:** + +``` +## Assumptions (Iteration 1): + +| Assumption | Source | Status | Evidence So Far | +|---|---|---|---| +| ActivityType field contains different values for each workflow | Code path shows extraction from ActivityDefinition.code.coding | ❓ UNVALIDATED | TaskChangeEventBuilder line 20 extracts from task coding system ACTIVITY_TYPE | +| Both workflows produce different activityType values | FoundConnectionStrategy vs FixedConnectionStrategy use different connection types | ❓ UNVALIDATED | Strategies use DATA_CONNECTION_VALUE vs DATA_CONNECTION_FIXED_VALUE constants | +| This distinguishes them in the directActionTaskReady event | Code traces show field flowing to hp-notification-service | ❓ UNVALIDATED | IterableTaskChangeEvent includes activityType field | + +**Next iteration will validate these by:** +1. Checking what actual ActivityDefinition JSON looks like for both workflows +2. Verifying what values are actually in those connection type constants +3. Confirming what ends up in the real event sent to Iterable + +**Questions for you before proceeding:** +- Do you have examples of actual ActivityDefinitions I can examine? +- Have you seen the actual events in Kafka or Iterable for both workflows? +``` + +**You see the assumptions table and say:** "Actually, let me show you the ActivityDefinitions. Both have code='data-connection', not different values." + +→ **Result:** We catch the wrong assumption BEFORE declaring confidence, not after. + +**Phase 2.5 Validation Report (if we'd proceeded):** + +``` +## Phase 2.5 Assumption Validation Report + +| # | Assumption | Evidence You Provided | Finding | Impact | +|---|---|---|---|---| +| 1 | ActivityType has different values | ActivityDefinition JSON for both | ✗ REJECTED: Both have code="data-connection" | Hypothesis WRONG | +| 2 | Workflows produce different activityType values | Real event data from Iterable | ✗ REJECTED: Both events have activityType="data-connection" | Confidence COLLAPSED | +| 3 | Field distinguishes them | Real Kafka messages | ✗ REJECTED: Same value in both | False lead eliminated | + +**Real distinguisher:** activityName or activityDefinitionId (different for each workflow) + +**Status:** Return to Phase 2. Previous hypothesis rejected. +``` + +This structure prevents what happened: +- ❌ OLD: I declared "100% confident" based on code reading +- ✅ NEW: I show assumptions, you see them, you catch the error, we validate against real data first + +--- + +## Case Study: The Workaround That Was the Bug (INE-653 → INE-600) + +**A worked example of the boundary-misattribution failure** — distinct from the code-vs-data false-confidence failure above. Here the investigator HAD real data but still shipped a fix that made things worse, because they misjudged *which layer was wrong*. + +**Symptom:** DCS orchestrator population failed for a large org (WellSense, 1.72M persons) — `uk_work_unit_person` duplicate-key violations, infinite recovery loops (89 cycles in 20h), 5.6× work-unit inflation. FHIR Person paging failed on page 2+. + +**The wrong fix (PR #179, merged, +250/−20):** The investigator concluded *"the FHIR server's `next` link uses the internal UUID as the `id:above` cursor instead of the resource's actual `id`"* — i.e., **declared the server wrong** — and added 71 lines of cursor-rewriting logic (`correctIdAboveCursor`) plus a 179-line test to "correct" the server's cursor to the last entry's logical id. The PR even admitted: *"This is a client-side workaround. The proper fix is in the FHIR server's cursor generation logic… A separate ticket should track that."* + +**Why it was wrong:** The server's cursor was *correct and internally consistent* — it sorts by `_uuid` AND cursors by `_uuid`. The rewrite to logical id is what BROKE paging for non-UUID ids: `id:above=` filters on `_sourceId` while the sort is still `_uuid`, so pages overlap and skip. **The "fix" was the root cause.** The investigator never verified the server's contract (what field does it sort by? what field does `id:above` filter on for a non-UUID id?) — they assumed the unfamiliar behavior was a defect. + +**The real fix (client PR #186 + orchestrator PR #164, net code DELETION):** +- #186 **deleted** `correctIdAboveCursor` and its test (Search.java +5/−51, test −179): follow the server's `next` link verbatim. PR body: *"that `correctIdAboveCursor` workaround was an earlier, incomplete attempt at this same problem and **is the root cause**."* +- #164 forced `_sort=_uuid` on the orchestrator's population query so its sort matches the cursor field, plus an idempotent `ON CONFLICT DO NOTHING` insert. + +**What would have caught it in this skill:** +- **Phase 2 boundary check:** before changing the client, trace *what the server's cursor actually filters on for a non-UUID id* — that reveals the sort/filter field mismatch and points at the orchestrator's missing `_sort`, not the server. +- **Phase 3 disqualifier:** "planning a separate ticket for the proper fix" = a workaround, not root cause. Stop. +- **Fix-quality reflex:** the proposed fix only ADDED a compensation layer. Asking "what if I DELETE the rewrite and trust the contract?" lands directly on the real fix. + +**Lesson:** Real data proving the *symptom* (paging fails) is not the same as proving the *mechanism* (which layer violates which contract). Verify the boundary's contract before declaring it broken, and treat a compensating-layer fix as a smell. + +--- + +## Full Example: Multi-Iteration Investigation + +**Scenario:** "Can we distinguish connectionFound vs connectionFixed in directActionTaskReady events?" + +### **ITERATION 1** + +``` +## Iteration 1 Summary Report +[This is the first iteration, no previous to report] + +## Current Assumptions (Iteration 1): + +| Assumption | Source | Will Validate By | +|---|---|---| +| ActivityType field contains different values per workflow | Code extraction in TaskChangeEventBuilder line 20 | Check what ActivityDefinition.code.coding actually contains for each strategy | +| Both workflows assign different connection type values | FoundConnectionStrategy vs FixedConnectionStrategy use different constants | Trace constant values: DATA_CONNECTION_VALUE vs DATA_CONNECTION_FIXED_VALUE | +| This field flows unchanged to hp-notification-service | Code shows field in IterableTaskChangeEvent and EventPublishingService | Check if the field is serialized to Iterable | + +**Questions for you before proceeding:** +- Does this match how the system actually works? +``` + +You review and say: "Let me show you the actual ActivityDefinitions..." + +``` +[You provide actual ActivityDefinition JSON showing both have code="data-connection"] + +**ITERATION 2** + +## Iteration 1 Summary Report + +### Assumptions from Iteration 1: + +| Assumption | Check Performed | Evidence Found | Status | Impact | +|---|---|---|---|---| +| ActivityType field contains different values per workflow | Code inspection + user-provided ActivityDefinition JSON | BOTH ActivityDefinitions have code.coding[0].code = "data-connection" | ✗ DISPROVEN | Major: original hypothesis WRONG | +| Both workflows assign different connection type values | Checked constants + actual JSON | User showed: identifier with system="https://www.icanbwell.com/connectionType" — values ARE different ("DataConnectionFixed" vs "DataConnection") but this is NOT in the activityType coding | ⚠️ INCONCLUSIVE | Found different field but not the one we were checking | +| Field flows unchanged to hp-notification-service | Code still shows it flows | Assumption depends on activityType being different, which is FALSE | ✗ DISPROVEN | Dependent on assumption 1 | + +### What This Iteration Revealed: +- ✗ DISPROVEN: activityType is NOT the distinguishing field (both have "data-connection") +- ✓ DISCOVERED: There IS a different field: identifier.system="connectionType" with values "DataConnectionFixed" vs "DataConnection" +- 🎯 Scope shifted: Not looking at code.coding anymore, now looking at identifiers + +### Hypothesis Evolution: +- **Iteration 1 hypothesis:** "Use activityType field to distinguish workflows" +- **Current hypothesis (Iteration 2):** "Real distinction is in identifier values or other ActivityDefinition properties, NOT in activityType coding" +- **Rejected paths:** activityType field is a dead end + +### Confidence Trajectory: +- Iteration 1: LOW (many untested assumptions) +- Iteration 2: MEDIUM (found that one key assumption was wrong, but found alternative distinguisher) +- Next blocker: Do identifier values actually flow to hp-notification-service? Or do we use activityName instead? + +## Current Assumptions (Iteration 2): + +| Assumption | Source | Previous Status | Will Validate By | +|---|---|---|---| +| Real distinguisher is identifier.value for connectionType system | User-provided ActivityDefinition JSON | NEW from Iteration 1 findings | Check if identifiers are copied to Task and flow to events | +| Or alternatively, activityName distinguishes them (different for each workflow) | ActivityDefinition shows "Connection Fixed" vs "Complete Your Health Record Connection" | NEW (alternative hypothesis) | Check if activityName is available in directActionTaskReady event | +| Workflow strategy extension ("fixedConnectionStrategy" vs "foundConnectionStrategy") might also distinguish | Extension field in ActivityDefinition | NEW (third alternative) | Check if extensions are extracted and flow through | + +**Questions for you before proceeding:** +- Which of these fields do you think actually flows through to the notification event? +- Or should we check all three? +``` + +[You provide Kafka logs showing what's actually in the directActionTaskReady event] + +``` +[Similar pattern continues to Iteration 3, where we validate which fields actually made it through] +``` + +### **Benefits of This Format** + +1. **Transparent hypothesis evolution** — You see which assumptions held and which collapsed +2. **Early course corrections** — You can interject when assumptions look wrong BEFORE deep investigation +3. **Running scorecard** — Each iteration shows progress: "We ruled out X, discovered Y, narrowed to Z" +4. **Accountability** — "I said this would distinguish them" vs. "Reality showed this distinguishes them" +5. **Context for Phase 2.5** — By the time we reach evidence collection, we have a clear record of what still needs validating diff --git a/.claude/skills/kafka-event-design/SKILL.md b/.claude/skills/kafka-event-design/SKILL.md new file mode 100644 index 00000000..658fff94 --- /dev/null +++ b/.claude/skills/kafka-event-design/SKILL.md @@ -0,0 +1,49 @@ +--- +name: kafka-event-design +description: Use when designing a Kafka event or topic, writing a producer or consumer, defining an event payload/schema, or choosing partition keys and counts at b.well. Applies the real event standard and the key/partition decision procedure (not a one-size formula), and generates a CloudEvents + AsyncAPI-conformant contract. +--- + +# Kafka Event Design + +You generate event contracts that follow b.well's real conventions and, crucially, you help the author **reason about keys and partitions** rather than pasting a formula. Event contracts are harder to change than APIs — get naming, keys, and schema right the first time. + +The conventions are in the substrate — read and cite them, don't restate them: +- **Rules:** `standards/events.md` (topic naming, event-vs-command, CloudEvents, idempotency, evolution, DLQ, outbox) — cite anchors like `#std-events-partition-key`. +- **Key/partition decision procedure:** `patterns/event-key-and-partition-design.md` (`#pat-event-key`) — the ordering-unit → cardinality → two-tier → count → re-key steps. +(Both live in `icanbwell/.github` if not in the current repo.) + +## First, get the naming and direction right + +- **Topic:** `..events` for facts, `..commands` for directives — kebab-case, **no underscores**, owned by the **producer's** bounded context (`#std-events-topic-naming`, `#std-events-topic-ownership`). (Note: the old `{domain}.{entity}.{action}` form and `patient.updated` examples are superseded by this — don't copy them.) +- **Event vs command** (`#std-events-event-vs-command`): a *fact* is past-tense PascalCase (`AppointmentBooked`); a *command* is imperative PascalCase (`BundleData`). A scheduled trigger is an **event** the scheduler owns, not a command. +- **`ce_type`:** reverse-DNS `com.bwell..`, `.vN` on breaking changes, independent of the topic name. + +## Then, design the key and partition count (don't hand out a formula) + +Walk `patterns/event-key-and-partition-design.md`: +1. **Ordering unit** — the smallest entity whose events must stay ordered → the key, always tenant-prefixed (`tenant:::`). +2. **Cardinality/skew** — reject low-cardinality or single-super-entity keys (they hot-spot); estimate the busiest key. +3. **Two-tier** for scatter-gather — fan-out by the work-unit id, fan-in by `hash(job+unit)`, lifecycle by the job id (the orchestrator model). +4. **Partition count** — the parallelism ceiling (consumers ≤ partitions), effectively write-once; size for the *estate*; co-partition input/output/changelog for joins. +5. **Re-keying** forces a repartition; **no external lookups inside a stream processor.** + +Only after this is settled do you emit a key. Never just print `tenant:X:entity:Y` as "the answer." + +## Generate the contract + +Produce: the **CloudEvents 1.0** envelope over binary Kafka headers (required `ce_*` + `traceparent`, `#std-events-cloudevents`), the payload schema (Avro/JSON Schema), and co-locate it in the producing service's `src/main/resources/asyncApi.yaml` (AsyncAPI v3, the source-of-truth contract). Payload carries references/ids + tenant + actor + timestamp + schema version — **not** a full snapshot, **not** PHI beyond what's needed, **not** a work batch. + +## Consumer requirements (non-negotiable — this is where incidents happen) + +- **Idempotent under redelivery** (`#std-events-idempotency`): dedup on `ce_id` / upsert / search-before-create. **Durable** dedup only — in-memory/Caffeine dedup is wiped on deploy/rebalance and silently leaks dupes (four teams hit this; INE-968). +- **Never swallow exceptions** — re-throw so it hits the **DLQ** (`.dlq`, PHI-safe: ids + error metadata only, `#std-events-dlq`); bad config is graceful-skip+alert, not silent retry-drop (INE-979). +- **Idempotency key on external side-effects** (HAX-45: duplicate vendor submissions). +- **Consumer group** `{service}-{purpose}`, one per purpose (`#std-events-consumer-groups`); isolate resource pools per workload class (INE-939 backpressure). + +## Evolution & anti-patterns + +- **Additive-only** (`#std-events-evolution`); breaking change = new `.vN` topic/type + migration; include a `schemaVersion`. +- **Never request-reply over Kafka** (`#std-events-no-request-reply`); for audited writes use the **transactional outbox** (`#std-events-outbox`); for long-running work see `patterns/orchestrated-long-running-work.md`. +- Capability-not-vendor names (no `databricks`/vendor in a topic or type). + +(Redpanda schema-registry validation is aspirational until the registry is available; until then the AsyncAPI file is the contract.) diff --git a/.claude/skills/new-wheel-job-bundle/SKILL.md b/.claude/skills/new-wheel-job-bundle/SKILL.md new file mode 100644 index 00000000..68578e0b --- /dev/null +++ b/.claude/skills/new-wheel-job-bundle/SKILL.md @@ -0,0 +1,114 @@ +--- +name: new-wheel-job-bundle +description: | + Create a new Declarative Automation Bundle (formerly called + a Databricks Asset Bundle) for a Python Wheel Job project in this + repository +--- + +# Create a New Bundle for a Wheel Job + +## Overview + +Scaffolds a new Databricks Asset Bundle for a wheel-based Lakeflow job using +the Cookiecutter template in `template/wheel-job-bundle`. The generated project +lands in `bundle/` and includes a `databricks.yml` with +dev/staging/sandbox/prod targets, a Python package with CLI entry point, +tests, and documentation stubs. + +## Prerequisites + +Cookiecutter must be installed (`pip install cookiecutter`). + +## Execution + +Run from the repo root: + +```bash +cd bundle && cookiecutter ../template/wheel-job-bundle +``` + +### Prompts + +| # | Prompt | Default | Notes | +|---|--------|---------|-------| +| 1 | `project_name` | `bwell-project-name` | Must match `^bwell-[a-z][a-z0-9\-]*[a-z0-9]$` | +| 2 | `package` | auto-derived | `bwell.` | +| 3 | `package_directory` | auto-derived | `bwell/` | +| 4 | `author_email` | *(none)* | B.Well email, e.g. `first.last@icanbwell.com` | +| 5 | `date` | today (ISO-8601) | Date prefix for the spec and plan filenames; override only to backdate | + +### Non-Interactive (Agent) + +```bash +cd bundle && cookiecutter ../template/wheel-job-bundle --no-input \ + project_name="bwell-my-new-job" \ + author_email="first.last@icanbwell.com" +``` + +## Validation Rules + +The template's `pre_gen_project.py` hook enforces: + +- **Project name**: `^bwell-[a-z][a-z0-9\-]*[a-z0-9]$` — must start with `bwell-`, lowercase alphanumeric and hyphens only, cannot end with a hyphen. +- **Package name**: `^bwell.[a-z][._a-z0-9]*[a-z0-9]$` + +## Output Structure + +The generated directory name strips the `bwell-databricks-` or `bwell-` +prefix from `project_name`. For example, `bwell-mongo2databricks-job` +produces `bundle/mongo2databricks-job/`. + +```console +$ tree bundle// +bundle// +├── CLAUDE.md # Per-bundle agent context (imports spec + plan) +├── databricks.yml # Bundle config (dev/staging/sandbox/prod targets) +├── pyproject.toml # Hatch-managed Python project +├── Makefile # install, test, format, dev-run, etc. +├── requirements.txt # (generated after `make install`) +├── src/bwell// +│ ├── __init__.py +│ ├── __main__.py # CLI entry point +│ ├── _tasks.py # Job task logic (scaffold) +│ └── py.typed +├── tests/ +│ └── test_tasks.py +└── docs/ + ├── index.md + ├── api.md + ├── cli.md + ├── data-flow.md + └── superpowers/ + ├── specs/YYYY-MM-DD--design.md # Placeholder design doc + └── plans/YYYY-MM-DD-.md # Placeholder implementation plan +``` + +## Post-Scaffold Steps + +1. `cd bundle/ && make install` — Create the virtual environment + and install dependencies. +2. Fill in the design doc in `docs/superpowers/specs/`, then the dated plan in + `docs/superpowers/plans/` — agree on the problem and the build order + *before* implementing. These load automatically via the bundle's + `CLAUDE.md`. See `docs/agents.md`. +3. Add job logic in `src/bwell//` — The scaffolding structure + invokes logic using the `Task` class in `_tasks.py`, assuming that + a `Task` instance will be used as a callable (for example: `Task()()`), + so job logic will be kicked off by invoking the `Task.__call__` method. + This may or may not be appropriate for the job being authored, however + whatever the structure—the job must be callable as a CLI entry point. +4. Edit `src/bwell//__main__.py` — adjust CLI arguments if the + defaults (`--incremental`, `--all`, `--limit`) don't fit. +5. `make format` — Apply formatting, perform linting, and validate type + annotation. +6. `make test` — Run tests (connects to the Databricks dev workspace via + `databricks-connect`), check linting and type annotation, and validate + the project configurations (a local proxy for CI/CD checks). + +## Common Mistakes + +- Using uppercase, underscores, or omitting the `bwell-` prefix in the project + name — the hook will reject it. +- Running cookiecutter from the repo root instead of from `bundle/` — the + output lands in the wrong directory. diff --git a/.claude/skills/pipeline-config-sync/SKILL.md b/.claude/skills/pipeline-config-sync/SKILL.md new file mode 100644 index 00000000..a9bb1fb4 --- /dev/null +++ b/.claude/skills/pipeline-config-sync/SKILL.md @@ -0,0 +1,269 @@ +--- +name: pipeline-config-sync +description: Sync a Prefect deployment's live configuration back into its corresponding pipeline_config.json in helix.orchestration. Use this whenever the user wants to reconcile drift between Prefect (where ops can edit deployments via the UI) and the helix.orchestration repo (the canonical source-of-truth for those configs). Trigger on phrases like "sync pipeline config", "update pipeline config from Prefect", "reconcile with code", "pipeline config drifted", or whenever the user mentions a Prefect deployment name and asks to bring the JSON in line with what's live. The skill detects which Prefect environment (prod / staging / dev / client-sandbox) is connected via MCP and announces it before doing anything — the JSON has env-specific overlays (`prod_args`, `staging_args`, etc.), so getting the env right is what makes the diff real instead of noise. Reads, diffs, asks for approval, then edits the JSON file. Does NOT push or open a PR — leaves git operations to the user. +when_to_use: "Reconciling a helix.orchestration pipeline_config.json with the live Prefect deployment after the deployment was edited via the Prefect UI. Run this when you suspect or know configuration has drifted from code." +user-invocable: true +argument-hint: " [--env prod|staging|dev|client-sandbox]" +allowed-tools: Read Edit Bash Glob Grep mcp__prefect__get_deployments mcp__prefect__get_flow_runs mcp__prefect__get_identity +effort: medium +--- + +# Pipeline Config Sync + +You are bridging two sources of truth for Helix pipeline configurations: + +- **Prefect** (live deployment state, mutable from the UI) — the *what's actually running*. +- **`helix.orchestration/pipeline_configs/**/pipeline_config.json`** (the file that should regenerate that deployment) — the *what we say is running*. + +When ops edits a deployment from the Prefect UI (bumping executors during an incident, tweaking a pipeline param, retiming the schedule), those edits do not round-trip back to the JSON file. Drift accumulates silently. The next time someone redeploys from code, the UI edits get clobbered. + +Your job: pull the live Prefect config, diff it against the JSON file, show the user what's drifted, and — once they approve — write the changes into the JSON. You stop there. The user runs `git add` / `git commit` / `gh pr create` themselves. + +## What to sync (and what NOT to) + +**Sync these fields** — these are the ones that drift: + +- `parameters.pipeline_params` — the long list of `--flag=value` strings. Highest-drift area; most UI tweaks land here. +- `parameters.executors`, `parameters.driver_memory`, `parameters.spark_memory` — Spark sizing. Often bumped under load and forgotten. +- `schedule` — the cron expression / interval. Retimings frequently happen via UI. + +**Leave these alone** unless the user explicitly asks: + +- `parameters.pipeline_module` — code-defined; should not drift. +- `parameters.pipeline_name` — naming, not config. +- `tags` — low drift; if you notice a difference, mention it but don't write it. +- `local_args`, `dev_args`, `staging_args`, `pre-prod_args`, `client-sandbox_args`, `prod_args` — these are environment-overlay sections used by the build process; they do **not** correspond 1:1 to anything in Prefect's deployment record. You cannot reverse-engineer which list a given Prefect param came from. If you see a param in Prefect that isn't in `parameters.pipeline_params`, check the env-specific `*_args` lists for the active env *before* concluding it's drift — see step 4 below. +- `*_parameters_overrides` — same reasoning as above. They overlay on top of `parameters` at deploy time. + +## Workflow + +### 1. Detect and announce the connected Prefect environment + +**This is the most important step. Do not skip it and do not guess.** The JSON file has env-specific overlay sections (`prod_args`, `staging_args`, `dev_args`, `client-sandbox_args`, plus `prod_parameters_overrides`, `client-sandbox_parameters_overrides`, etc.). The diff is only meaningful if you reconcile against the *correct* overlay. Comparing prod-flavored JSON against a staging Prefect connection produces a flood of false-positive "drifts" — every staging org ID and S3 path looks like drift when it isn't. + +The Prefect MCP is connected to exactly one workspace (env) per session. You can't choose; you can only detect. There are two signals — use both: + +**1a. Ask the MCP directly.** This is the authoritative source. + +``` +mcp__prefect__get_identity() +``` + +The response should include account/workspace identifiers. Look for the workspace name or URL — it usually contains the env (e.g. `prod`, `staging`, `dev`, `client-sandbox`). + +**1b. Cross-check with deployment-data fingerprints.** After you fetch the live deployment in step 4, the `parameters` dict (and especially `pipeline_params`) carries clear env tells. Use this table to confirm — and if the fingerprints disagree with the env you announced from `get_identity`, **stop and re-announce** before continuing into the diff: + +| Signal in Prefect | prod | staging | dev | client-sandbox | +|---|---|---|---|---| +| `bwell_organization_id` (default/value) | `078cf112-f802-4e4b-9b62-ed44cdf05d27` | `c9511013-0862-4cdd-a466-c6e028d42bf1` | `c08bf3b8-8949-4f94-a013-cc490873f4ea` | `a7a9c609-6bef-4599-9f83-8aed9390a08d` | +| S3 path prefix | `s3://bwell-helix-data-lake-ue1/...` (no env in name) | `s3://bwell-staging-helix-data-lake-ue1/...` | `s3://bwell-dev-helix-data-lake-ue1/...` | `s3://bwell-helix-data-lake-client-sandbox-ue1/...` | +| `token_service_url` | `aperture-token-service-pipelines.prod.bwell.zone` | `...staging.bwell.zone` | `...dev.bwell.zone` | `...client-sandbox.bwell.zone` | + +If `get_identity` and the fingerprints agree, you're good. If they conflict (rare — usually means someone is testing with mixed configs), trust the fingerprints over `get_identity` and call out the conflict for the user. + +**1c. Announce it.** Before doing any diffing, tell the user, plainly: + +``` +Connected Prefect MCP: (detected via ) +Will reconcile against the JSON's _args / _parameters_overrides overlays. +If this is wrong, stop me now. +``` + +If the user wants a *different* env's view, they need to reconfigure the MCP — this skill cannot switch envs mid-session. Tell them so and stop. + +The user's CLI args (`--env prod`) only become relevant when the MCP is connected to a *multi-env* workspace, which is unusual; if the user passes `--env` and the detected env disagrees, surface the conflict and ask before proceeding. + +### 2. Resolve the deployment name + +The user gives you a deployment name. The `deployment_name` in the JSON file is exactly the `name` field on the Prefect deployment object. They match verbatim, including capitalization and spacing (e.g. `"High Resource Tokens - Provider Existing Patient Access Pipeline"`). + +### 3. Find the JSON file + +This skill ships inside the `helix.orchestration` repo, so the repo root is wherever Claude Code is running. Resolve it once with `git rev-parse --show-toplevel` (call it `$REPO`) and search under its `pipeline_configs/`: + +```bash +REPO=$(git rev-parse --show-toplevel) +grep -rln "\"deployment_name\": \"\"" "$REPO/pipeline_configs/" +``` + +If multiple files match (rare but possible — different versions of a pipeline can share a name across subfolders), list them and ask the user which one. Don't guess. + +If zero files match, try a case-insensitive search on a distinctive substring of the name: + +```bash +grep -rli "" "$REPO/pipeline_configs/" +``` + +If still nothing: tell the user the deployment exists in Prefect but has no corresponding JSON file in the repo. That's a different kind of drift — outside this skill's scope. Stop. + +### 4. Fetch the live Prefect deployment + +``` +mcp__prefect__get_deployments(filter={"name": {"any_": [""]}}) +``` + +The default response is a *summary* and **does not include `parameters`**. To get the full deployment record you need for diffing, follow up with an id-filtered call: + +``` +mcp__prefect__get_deployments(filter={"id": {"any_": [""]}}) +``` + +This returns the full `parameters` dict, `parameter_openapi_schema`, `job_variables`, work-pool details, and recent runs. + +Capture from the result: + +- `parameters` — the full dict. The keys you care about: `pipeline_params` (list of strings), `executors`, `driver_memory`, `spark_memory`. Other keys may exist; preserve them mentally but don't try to sync them unless they overlap with the sync scope above. +- `schedule` (and `schedules` — Prefect 2.x can have multiple). Most deployments have one cron schedule; if there are multiple, show all of them in the diff. +- `work_pool_name`, `tags`, `id` — for context only, not for syncing. + +If `get_deployments` returns more than one match (e.g. similarly named deployments in the same workspace), show the user the candidates with `id` + `work_pool_name` and ask which one. **Do not pick silently.** + +If `get_deployments` returns zero: confirm you used the exact name from the JSON's `deployment_name` field. If still empty, the deployment doesn't exist in the env you're querying — tell the user and stop. + +### 5. Build the diff + +This is the core of the skill. Compare Prefect's view to the JSON's view, but be careful about *where* in the JSON each piece lives. + +#### 5a. `pipeline_params` diff + +The Prefect deployment's `parameters.pipeline_params` is what's *actually being passed at runtime*. In the JSON file, it is constructed at deploy time as roughly: + +``` +parameters.pipeline_params + _args + flatten(_parameters_overrides) +``` + +So before flagging a Prefect param as "drift", check if it appears in any of these JSON sources for the active env: + +- `parameters.pipeline_params` (base list) +- `_args` (e.g. `prod_args`) +- `_parameters_overrides` (e.g. `prod_parameters_overrides`) + +A param is **drifted** only if its `--key=value` form is present in Prefect but NOT present in *any* of those three JSON sources, OR if the value differs. + +For each drifted param, classify it: + +- **Modified value** — same `--key`, different `=value`. Most common. +- **Added in Prefect** — `--key` doesn't exist anywhere in the JSON. +- **Removed in Prefect** — `--key` exists in the JSON but not in Prefect's runtime params. + +**Write-target rule** — where to put the change in the JSON: + +- **Modified value, param exists in `_args` or `_parameters_overrides`:** edit it *in place* in that overlay. You know its home — use it. Writing the new value to `parameters.pipeline_params` instead would create a contradictory pair (e.g. `--run_intelligence_layer=False` in base, `=True` in `staging_args`) which is worse than the original drift. +- **Modified value, param exists only in `parameters.pipeline_params` (the base list):** edit it in place there. +- **Modified value, param exists in *both* base and an overlay:** edit the overlay (the overlay wins at deploy time, so that's the value Prefect is actually seeing). +- **Added in Prefect (param doesn't exist anywhere in the JSON):** append to `parameters.pipeline_params`. In the diff summary, note: "If any of these belong in `_args` or `_parameters_overrides` instead, move them manually before committing." +- **Removed in Prefect:** delete the param from wherever it lives (base or overlay). Confirm with the user before doing this — removed-in-Prefect can occasionally mean ops disabled a flag temporarily, not that the code should drop it. + +Always state the write target for each line in the diff (e.g. `lives in staging_args` / `lives in parameters.pipeline_params`) so the user can sanity-check before approving. + +#### 5b. Infra fields diff + +Compare Prefect's `parameters.executors` / `driver_memory` / `spark_memory` to the JSON's `parameters.executors` / `driver_memory` / `spark_memory` directly. These are not env-overlaid (or if they are via `*_parameters_overrides`, the override takes precedence — surface that in the diff if relevant). + +#### 5c. Schedule diff + +Prefect's schedule shape varies. Two common forms: + +- `{"cron": "45 6 * * *", "timezone": "UTC", ...}` — cron schedule. +- `{"interval": , "anchor_date": "...", ...}` — interval schedule. + +The JSON file represents schedules as: + +```json +"schedule": { "type": "cron", "value": "45 6 * * *" } +``` + +When building the diff: + +- Cron in Prefect, cron in JSON, different value → modified. +- Cron in Prefect, no schedule in JSON → added. +- No schedule in Prefect, cron in JSON → removed (deployment was un-scheduled — confirm with user before writing this; might be intentional pause). +- Different timezone → flag it; the JSON shape doesn't carry timezone, so warn the user. +- Interval-based schedule in Prefect → tell the user the JSON `schedule` shape used here is cron-only. Don't try to convert; ask how they want to handle it. + +### 6. Present the diff and ask for approval + +Always show the diff in chat in this exact shape, before touching the file: + +``` +## Drift summary: (env: — detected via ) + +JSON file: +Prefect deployment id: + +### Pipeline params +- **Modified:** + - `--max_concurrent_tasks`: `50` → `75` + - ... +- **Added in Prefect:** + - `--new_flag=true` + - ... +- **Removed in Prefect:** + - `--deprecated_flag=...` + +### Infra +- `executors`: 18 → 24 +- `driver_memory`: unchanged +- `spark_memory`: unchanged + +### Schedule +- `45 6 * * *` → `30 7 * * *` + +### Out-of-scope differences (informational only — will NOT be synced) +- tags: vs +- ... + +--- +Apply these changes to ? (yes/no) +``` + +If the user says no or asks for changes (e.g. "skip the schedule change, that's intentional"), respect that and re-show the narrowed diff before applying. + +If there is **no drift**, say so plainly: `No drift detected between Prefect and . Nothing to sync.` Stop. + +### 7. Apply the changes + +Once approved: + +- Use the Edit tool. For each change, target the JSON location you announced in the diff (base list, `_args`, `_parameters_overrides`, or top-level fields like `parameters.executors` and `schedule.value`). +- Edits to lines that aren't unique in the file (e.g. `--run_intelligence_layer=True` appears in both `staging_args` and `prod_args`) need an anchor: include the surrounding `"_args": [` line in the `old_string` to disambiguate. +- Preserve the existing JSON file's formatting style (indentation, trailing newline, key ordering). The original is 2-space indented and key-ordered the way `build_deployment_configs.py` writes it. Don't reformat. +- For `pipeline_params`: keep the existing ordering of params that didn't change; insert new params at the end of the list (matching the file's existing convention); remove deleted params in place. +- After the edits, run a quick sanity check: `python -c "import json; json.load(open(''))"` to make sure the file is still valid JSON. If it fails, undo and tell the user — don't try to fix bad JSON by guessing. + +### 8. Tell the user what happened — and what's next + +Show a short summary: which file changed, how many params/fields, and the next steps. Example: + +``` +Updated : + - 7 pipeline_params modified + - 1 added, 0 removed + - executors: 18 → 24 + - schedule unchanged + +Next steps (left for you to do): + cd /helix.orchestration + git checkout -b sync/ + git add + git diff --cached # final review + git commit -m "sync(): pull live config from Prefect " + git push -u origin sync/ + gh pr create --title "Sync config from Prefect " --body "..." +``` + +Keep the suggested branch name short and descriptive. Don't run any of these commands yourself — git/PR is explicitly the user's responsibility. + +## Common footguns + +- **Param ordering in `pipeline_params` is not stable.** Don't treat order changes as drift. Compare as a set keyed by the `--` portion before the `=`. +- **Empty-value params are a thing.** `--exclude_service_slugs=` (trailing equals, no value) is valid and not the same as the param being absent. Match on the full literal `--key=value` string when checking presence. +- **Param values can themselves contain `=`.** e.g. `--token_service_url=https://...?foo=bar`. When parsing into key/value, split on the *first* `=` only. +- **Templated values like `{token_service_client_id}` in the JSON** are placeholders that get substituted at deploy time. If you see a concrete value in Prefect (e.g. `--token_service_client_id=abc-123`) where the JSON has `{token_service_client_id}`, this is **not drift** — that's the templating system working as designed. Skip these. +- **`enable_on_demand` and `enable_driver_on_demand`** live in `*_parameters_overrides` (see `prod_parameters_overrides`, `client-sandbox_parameters_overrides` in the example). These are not in `pipeline_params`. If they differ between Prefect and the JSON, flag them in the diff but be cautious — these are env-specific overrides, not drift in the usual sense. +- **The Prefect MCP is read-only.** You cannot push back to Prefect from this skill. The direction of sync is always Prefect → code, never the reverse. +- **Multiple env overlays compound.** A param that looks "modified" might actually exist in *both* `parameters.pipeline_params` and `_args` — the overlay wins at runtime, so that's the value Prefect sees. Always check all three JSON sources (base, `_args`, `_parameters_overrides`) before declaring drift, and follow the write-target rule in step 5a so the edit lands where it actually matters. +- **Connected env is fixed for the session.** The Prefect MCP can only point at one workspace at a time. If the user says "sync from prod" but the MCP is on staging, you can't switch — tell them and stop. Don't fake the diff against an env you're not actually connected to. +- **Don't auto-pick the file or the deployment when there's ambiguity.** Always ask. The cost of a wrong sync (silently overwriting an unrelated config) is much higher than the cost of one extra clarifying question. +- **Don't reformat the file.** Preserve indentation, key order, trailing newline. The repo's `build_deployment_configs.py` writes JSON with `indent=4`-style for new files, but existing files in the repo may use 2-space; match what's already there. +- **Don't run git commands.** Even if it would be convenient. The user has explicitly asked for git/PR to remain manual. diff --git a/.claude/skills/resolve-route/.gitignore b/.claude/skills/resolve-route/.gitignore new file mode 100644 index 00000000..c2658d7d --- /dev/null +++ b/.claude/skills/resolve-route/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/.claude/skills/resolve-route/SKILL.md b/.claude/skills/resolve-route/SKILL.md new file mode 100644 index 00000000..3952b832 --- /dev/null +++ b/.claude/skills/resolve-route/SKILL.md @@ -0,0 +1,232 @@ +--- +name: resolve-route +description: + Resolves between ui-platform URL paths and React components in both + directions. Use this skill when the user asks "what component renders at this + URL?", "which component is at this path?", "what's rendered here?", provides a + localhost:4200 URL, wants to trace a route to its source code, OR asks "where + is this component rendered?", "what URL uses this component?", "what route + maps to this?", or asks about unreachable/dead components. This skill depends + on having route config from Client Hub — invoke the client-hub skill first if + no cached route config exists. +--- + +# Resolve Route + +Maps between ui-platform URL paths and React components in both directions: + +- **Forward**: URL → component (what renders at this route?) — instant +- **Reverse**: component → URL (where is this component rendered?) — instant +- **Trace**: component → full parent chain via TypeScript references (~8s) + +## STOP — Config Dependency Check + +**Before doing ANYTHING else, check if the route config cache exists AND is +fresh:** + +```bash +find ~/.claude/skills/client-hub/.cache -name "config-*-*-embeddable_configuration.json" -mmin -10 2>/dev/null +``` + +**If NO file is returned** (either missing or older than 10 minutes): + +1. You MUST invoke the `client-hub` skill using the Skill tool NOW. +2. When invoking, specify: config type = `Client Config`, module = + `embeddable configuration`. +3. Do NOT explore the repo. Do NOT search for route files. Do NOT proceed + without the config. +4. Wait for the skill to complete and confirm the cache file exists before + continuing. + +**If a fresh cache file exists** (returned by the find command), proceed to +Step 1. + +--- + +## NEVER write inline scripts to parse route data. ALWAYS use `resolve_route.mjs` below. + +--- + +## Determine Repo Root + +All commands require `--repo-root`. Set it to the ui-platform checkout path: + +```bash +REPO_ROOT="/path/to/ui-platform" # adjust to actual path in the current working directory +``` + +If working from a ticket directory or worktree outside the repo, you MUST pass +`--repo-root` explicitly. + +--- + +## Step 1: Determine Direction + +- If the user provides a **URL or path** → use **Forward Resolution** (Step 2) +- If the user provides a **component name, file path, or registry key** → use + **Reverse Resolution** (Step 3) + +--- + +## Step 2: Forward Resolution (URL → Component) + +```bash +node ~/.claude/skills/resolve-route/scripts/resolve_route.mjs \ + --repo-root "$REPO_ROOT" --path "" +``` + +Or with a full URL: + +```bash +node ~/.claude/skills/resolve-route/scripts/resolve_route.mjs \ + --repo-root "$REPO_ROOT" --url "http://localhost:4200/#/health-circle/share-data" +``` + +Optional flags: + +- `--env dev|staging|client-sandbox|prod` — use a specific env's cached config +- `--client-slug ` — use a specific client's cached config +- `--list-routes` — show all available routes and their registry keys +- `--verbose` — show full matched node JSON + +The script outputs: + +- **Registry Key** — the `elementId` used in `mainRegistry.tsx` +- **Route Trail** — the full path of matched nodes from root to the target +- **Related registry keys** — other components from the same MFE package + +--- + +## Step 3: Reverse Resolution (Component → URL) + +### 3a: Direct lookup + +```bash +node ~/.claude/skills/resolve-route/scripts/resolve_route.mjs \ + --repo-root "$REPO_ROOT" --component +``` + +Input formats: component export name, registry key, or file path. + +**Exit code 0 + URL(s) found** → done. Report the result. + +**Exit code 1 + "NOT IN ROUTE CONFIG"** → go to Step 3b. + +**Exit code 1 + "No registry entry found"** → **IMMEDIATELY** go to Step 3c. Do +not try other approaches. + +### 3b: Registered but not routed + +The component has a registry key in `mainRegistry.tsx` but no corresponding +`elementId` in the client's `routeDefinition`. This means either: + +- The route exists in a different client/env +- The route is genuinely unreachable (dead code) + +List available cached environments and try each: + +```bash +ls ~/.claude/skills/client-hub/.cache/config-*-embeddable_configuration.json 2>/dev/null +``` + +For each config file found (extract env from filename +`config---...`): + +```bash +node ~/.claude/skills/resolve-route/scripts/resolve_route.mjs \ + --repo-root "$REPO_ROOT" --component --env --client-slug +``` + +If still "NOT IN ROUTE CONFIG" across all available cached envs → report as +**UNREACHABLE**. + +### 3c: Not in registry — trace the import chain + +**This is the primary resolution method for nested components.** Use it +immediately when `--component` says "No registry entry found". Do NOT try to +manually grep or explore — the trace does everything. + +```bash +node ~/.claude/skills/resolve-route/scripts/resolve_route.mjs \ + --repo-root "$REPO_ROOT" --trace --mfe +``` + +Or if you have the file path: + +```bash +node ~/.claude/skills/resolve-route/scripts/resolve_route.mjs \ + --repo-root "$REPO_ROOT" --file +``` + +The `--file` flag auto-infers component name and MFE from the path. Use it when +the user references a specific file. + +**Key points:** + +- Takes ~8s (loads TypeScript project). Outputs JSON with the **complete** + reference chain in one call. +- Exit code 0 = resolved to a registry key. Exit code 1 = unreachable. +- **The script automatically checks the route config** and appends `Route:` + lines — no separate `--component` call needed. +- If the component has **multiple parents leading to different registry keys**, + all paths are shown under `alternatePaths` in the JSON output. + +Read the JSON output: + +- `resolved.registryKey` — the registry key(s) at the top of the primary chain +- `resolved.component` — the registered screen component +- `chain` — each hop from the queried component up to the registered screen +- `alternatePaths` — other registered parents found along the way (if any) + +**You do NOT need a separate `--component` call after trace.** The route check +is included in the output. + +### 3d: Report result + +Format the final report to the user: + +``` +Component: +Import chain: +Registry Key: (or NONE if chain never reached registry) +URL: (or UNREACHABLE — not in routeDefinition for /) +``` + +--- + +## Flags Reference + +| Flag | Purpose | +| ---------------------- | --------------------------------------------------------- | +| `--repo-root ` | Explicit ui-platform repo root (bypasses auto-detection) | +| `--path ` | Forward: resolve URL path to component | +| `--url ` | Forward: resolve full URL (extracts hash fragment) | +| `--component ` | Reverse: component name, registry key, or file path → URL | +| `--trace ` | Trace: full ts-morph reference chain (~8s) | +| `--file ` | Trace from file: infers component name + MFE, then traces | +| `--mfe ` | Narrow trace to specific MFE (e.g., `mfe-medicine`) | +| `--depth ` | Max trace depth (default: 5) | +| `--env ` | Use specific environment's cached config | +| `--client-slug ` | Use specific client's cached config | +| `--list-routes` | List all available routes | +| `--verbose` | Show full matched node JSON | + +## Exit Codes + +| Code | Meaning | +| ---- | --------------------------------------------------- | +| 0 | Found / resolved successfully | +| 1 | Not found / unreachable | +| 2 | Usage error (missing args, no config, no repo root) | + +## Setup + +The script requires `ts-morph` for `--trace` mode. One-time install: + +```bash +npm install --prefix ~/.claude/skills/resolve-route +``` + +If not installed, `--trace` will fail with an import error. The `--path`, +`--component`, and `--list-routes` modes work without it (they only need the +cached config JSON). diff --git a/.claude/skills/resolve-route/package-lock.json b/.claude/skills/resolve-route/package-lock.json new file mode 100644 index 00000000..818c617d --- /dev/null +++ b/.claude/skills/resolve-route/package-lock.json @@ -0,0 +1,123 @@ +{ + "name": "resolve-route-skill", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "resolve-route-skill", + "version": "1.0.0", + "dependencies": { + "ts-morph": "24.0.0" + } + }, + "node_modules/@ts-morph/common": { + "version": "0.25.0", + "resolved": "https://registry.npmjs.org/@ts-morph/common/-/common-0.25.0.tgz", + "integrity": "sha512-kMnZz+vGGHi4GoHnLmMhGNjm44kGtKUXGnOvrKmMwAuvNjM/PgKVGfUnL7IDvK7Jb2QQ82jq3Zmp04Gy+r3Dkg==", + "license": "MIT", + "dependencies": { + "minimatch": "^9.0.4", + "path-browserify": "^1.0.1", + "tinyglobby": "^0.2.9" + } + }, + "node_modules/balanced-match": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", + "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", + "license": "MIT" + }, + "node_modules/brace-expansion": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.0.tgz", + "integrity": "sha512-TN1kCZAgdgweJhWWpgKYrQaMNHcDULHkWwQIspdtjV4Y5aurRdZpjAqn6yX3FPqTA9ngHCc4hJxMAMgGfve85w==", + "license": "MIT", + "dependencies": { + "balanced-match": "^1.0.0" + } + }, + "node_modules/code-block-writer": { + "version": "13.0.3", + "resolved": "https://registry.npmjs.org/code-block-writer/-/code-block-writer-13.0.3.tgz", + "integrity": "sha512-Oofo0pq3IKnsFtuHqSF7TqBfr71aeyZDVJ0HpmqB7FBM2qEigL0iPONSCZSO9pE9dZTAxANe5XHG9Uy0YMv8cg==", + "license": "MIT" + }, + "node_modules/fdir": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", + "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", + "license": "MIT", + "engines": { + "node": ">=12.0.0" + }, + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/minimatch": { + "version": "9.0.9", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.9.tgz", + "integrity": "sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg==", + "license": "ISC", + "dependencies": { + "brace-expansion": "^2.0.2" + }, + "engines": { + "node": ">=16 || 14 >=14.17" + }, + "funding": { + "url": "https://github.com/sponsors/isaacs" + } + }, + "node_modules/path-browserify": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/path-browserify/-/path-browserify-1.0.1.tgz", + "integrity": "sha512-b7uo2UCUOYZcnF/3ID0lulOJi/bafxa1xPe7ZPsammBSpjSWQkjNxlt635YGS2MiR9GjvuXCtz2emr3jbsz98g==", + "license": "MIT" + }, + "node_modules/picomatch": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", + "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/tinyglobby": { + "version": "0.2.16", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz", + "integrity": "sha512-pn99VhoACYR8nFHhxqix+uvsbXineAasWm5ojXoN8xEwK5Kd3/TrhNn1wByuD52UxWRLy8pu+kRMniEi6Eq9Zg==", + "license": "MIT", + "dependencies": { + "fdir": "^6.5.0", + "picomatch": "^4.0.4" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/ts-morph": { + "version": "24.0.0", + "resolved": "https://registry.npmjs.org/ts-morph/-/ts-morph-24.0.0.tgz", + "integrity": "sha512-2OAOg/Ob5yx9Et7ZX4CvTCc0UFoZHwLEJ+dpDPSUi5TgwwlTlX47w+iFRrEwzUZwYACjq83cgjS/Da50Ga37uw==", + "license": "MIT", + "dependencies": { + "@ts-morph/common": "~0.25.0", + "code-block-writer": "^13.0.3" + } + } + } +} diff --git a/.claude/skills/resolve-route/package.json b/.claude/skills/resolve-route/package.json new file mode 100644 index 00000000..1757014a --- /dev/null +++ b/.claude/skills/resolve-route/package.json @@ -0,0 +1,8 @@ +{ + "name": "resolve-route-skill", + "version": "1.0.0", + "private": true, + "dependencies": { + "ts-morph": "24.0.0" + } +} diff --git a/.claude/skills/resolve-route/scripts/resolve_route.mjs b/.claude/skills/resolve-route/scripts/resolve_route.mjs new file mode 100644 index 00000000..4b89e2e9 --- /dev/null +++ b/.claude/skills/resolve-route/scripts/resolve_route.mjs @@ -0,0 +1,906 @@ +#!/usr/bin/env node +/** + * Unified route ↔ component resolver for ui-platform. + * + * Modes: + * --path Forward: URL → component (instant) + * --url Forward: full URL → component (instant) + * --component Reverse: component/registry key → URL (instant) + * --trace Trace: component → full parent chain via ts-morph (~8s) + * --list-routes List all routes in config (instant) + * + * Common flags: + * --repo-root (required for --component/--trace) ui-platform root + * --env Environment for config lookup + * --client-slug Client for config lookup + * --mfe Narrow trace to specific MFE (e.g., mfe-medicine) + * --depth Max trace depth (default: 5) + * --verbose Show full matched node JSON + * + * Exit codes: 0 = found, 1 = not found/unreachable, 2 = usage error + */ +import { execSync } from 'child_process'; +import { existsSync, readdirSync, readFileSync, statSync } from 'fs'; +import { homedir } from 'os'; +import { basename, dirname, join, relative, resolve } from 'path'; +import { parseArgs } from 'util'; + +// --- Argument Parsing --- + +const { values: args } = parseArgs({ + options: { + path: { type: 'string' }, + url: { type: 'string' }, + component: { type: 'string' }, + trace: { type: 'string' }, + file: { type: 'string' }, + 'repo-root': { type: 'string' }, + 'list-routes': { type: 'boolean', default: false }, + env: { type: 'string' }, + 'client-slug': { type: 'string' }, + mfe: { type: 'string' }, + depth: { type: 'string', default: '5' }, + verbose: { type: 'boolean', default: false }, + help: { type: 'boolean', default: false }, + }, + strict: false, +}); + +if (args.help) { + console.error(`Usage: + node resolve_route.mjs --path /medicines # URL → component + node resolve_route.mjs --component medicineDetails # registry key → URL + node resolve_route.mjs --trace PricingItem --mfe mfe-medicine # component → chain + node resolve_route.mjs --file libs/mfe-medicine/src/component/PricingItem/PricingItem.tsx # file → trace + node resolve_route.mjs --list-routes # show all routes`); + process.exit(2); +} + +const EXIT_FOUND = 0; +const EXIT_NOT_FOUND = 1; +const EXIT_USAGE_ERROR = 2; + +const CACHE_DIR = join(homedir(), '.claude', 'skills', 'client-hub', '.cache'); + +// --- Path Validation --- + +function validatePath(target, base) { + const resolvedTarget = resolve(target); + const resolvedBase = resolve(base); + const rel = relative(resolvedBase, resolvedTarget); + if (rel.startsWith('..') || resolve(resolvedBase, rel) !== resolvedTarget) { + throw new Error(`Invalid file path: ${target} is outside ${base}`); + } + return resolvedTarget; +} + +// --- Repo Root Detection --- + +function findRepoRoot() { + if (args['repo-root']) { + const p = resolve(args['repo-root']); + if ( + !existsSync( + join( + p, + 'apps', + 'composite', + 'src', + 'app', + 'embeddable', + 'mainRegistry.tsx', + ), + ) + ) { + console.error( + `WARNING: --repo-root '${p}' does not contain mainRegistry.tsx`, + ); + } + return p; + } + + const hasRegistry = (d) => { + const target = join( + resolve(d), + 'apps', + 'composite', + 'src', + 'app', + 'embeddable', + 'mainRegistry.tsx', + ); + const rel = relative(resolve(d), target); + if (rel.startsWith('..')) return false; + return existsSync(target); + }; + + const envRoot = process.env.UI_PLATFORM_ROOT; + if (envRoot && existsSync(envRoot) && hasRegistry(resolve(envRoot))) + return resolve(envRoot); + + try { + const gitRoot = execSync('git rev-parse --show-toplevel', { + encoding: 'utf-8', + timeout: 2000, + }).trim(); + if (hasRegistry(gitRoot)) return gitRoot; + } catch {} + + let dir = process.cwd(); + while (dir !== dirname(dir)) { + if (hasRegistry(dir)) return dir; + dir = dirname(dir); + } + return null; +} + +// --- Config File Discovery --- + +function findConfigFile() { + if (!existsSync(CACHE_DIR)) return null; + + const env = args.env; + const slug = args['client-slug']; + let pattern; + if (env && slug) + pattern = `config-${env}-${slug}-embeddable_configuration.json`; + else if (env) pattern = `config-${env}-`; + else if (slug) pattern = `-${slug}-embeddable_configuration.json`; + else pattern = 'config-'; + + const files = readdirSync(CACHE_DIR) + .filter( + (f) => + f.includes('embeddable_configuration.json') && f.startsWith('config-'), + ) + .filter((f) => { + if (env && slug) return f === pattern; + return f.includes(pattern); + }); + + if (files.length === 0) return null; + if (files.length === 1) return join(CACHE_DIR, files[0]); + // Pick most recent + const sorted = files + .map((f) => { + try { + return { f, mtime: statSync(join(CACHE_DIR, f)).mtimeMs }; + } catch { + return null; + } + }) + .filter(Boolean) + .sort((a, b) => b.mtime - a.mtime); + return sorted.length > 0 ? join(CACHE_DIR, sorted[0].f) : null; +} + +function loadRouteTree(configFile) { + const validPath = validatePath(configFile, CACHE_DIR); + const data = JSON.parse(readFileSync(validPath, 'utf-8')); + const rd = data.routeDefinition; + if (!rd) { + console.error('ERROR: No routeDefinition field in config'); + process.exit(EXIT_USAGE_ERROR); + } + return typeof rd === 'string' ? JSON.parse(rd) : rd; +} + +// --- Registry Parsing --- + +function parseRegistry(repoRoot) { + const registryPath = join( + repoRoot, + 'apps', + 'composite', + 'src', + 'app', + 'embeddable', + 'mainRegistry.tsx', + ); + if (!existsSync(registryPath)) return { entries: {}, content: '' }; + + const validPath = validatePath(registryPath, repoRoot); + const content = readFileSync(validPath, 'utf-8'); + const entries = {}; + const entryRe = /^\s+(\w+):\s*\{/gm; + const importRe = /import\(['"]([^'"]+)['"]\)\)\s*\.(\w+)/s; + + const allMatches = []; + let match; + while ((match = entryRe.exec(content)) !== null) { + allMatches.push({ key: match[1], start: match.index }); + } + + for (let i = 0; i < allMatches.length; i++) { + const { key, start } = allMatches[i]; + const end = + i + 1 < allMatches.length ? allMatches[i + 1].start : content.length; + const block = content.slice(start, end); + const imp = importRe.exec(block); + if (imp) { + const pkg = imp[1]; + const comp = imp[2]; + if (!entries[comp]) entries[comp] = []; + entries[comp].push({ + key, + package: pkg, + line: content.slice(0, start).split('\n').length, + }); + } + } + return { entries, content }; +} + +// --- Route Tree Walking --- + +function pathMatches(pathSegments, urlSegments) { + if (pathSegments.length !== urlSegments.length) return false; + return pathSegments.every( + (ps, i) => ps.startsWith(':') || ps === urlSegments[i], + ); +} + +function findRoute(nodes, segments, trail = []) { + if (!segments.length) { + const indexChild = nodes.find((n) => n.index); + return indexChild + ? { match: indexChild, trail: [...trail, indexChild] } + : null; + } + + for (const node of nodes) { + const path = (node.path || '').replace(/^\/|\/$/g, ''); + + // Layout wrapper + if (!path && !node.index && node.children) { + const result = findRoute(node.children, segments, [...trail, node]); + if (result) return result; + continue; + } + + if (path === '*') return { match: node, trail: [...trail, node] }; + + const pathParts = path.split('/').filter(Boolean); + if (!pathParts.length) continue; + + if (pathMatches(pathParts, segments.slice(0, pathParts.length))) { + const remaining = segments.slice(pathParts.length); + if (!remaining.length) { + const indexChild = (node.children || []).find((c) => c.index); + if (indexChild) + return { match: indexChild, trail: [...trail, node, indexChild] }; + return { match: node, trail: [...trail, node] }; + } + if (node.children) { + const result = findRoute(node.children, remaining, [...trail, node]); + if (result) return result; + } + } + } + return null; +} + +function findUrlsForRegistryKey(routeTree, targetKey) { + const matches = []; + function walk(nodes, currentPath) { + for (const node of nodes) { + const path = (node.path || '').replace(/^\/|\/$/g, ''); + const elementId = node.elementId || node.id || ''; + const isIndex = !!node.index; + + const nodePath = isIndex + ? currentPath || '/' + : path + ? `${currentPath}/${path}` + : currentPath; + + if (elementId === targetKey) { + matches.push(isIndex ? currentPath || '/' : nodePath || '/'); + } + + if (node.children) { + walk(node.children, path ? `${currentPath}/${path}` : currentPath); + } + } + } + walk(routeTree, ''); + return matches; +} + +function listAllRoutes(nodes, prefix = '') { + for (const node of nodes) { + const path = (node.path || '').replace(/^\/|\/$/g, ''); + const key = node.elementId || node.id || ''; + const isIndex = !!node.index; + + const displayPath = isIndex + ? `${prefix} [index]` + : path + ? `${prefix}/${path}` + : prefix || '/'; + + if (key) { + console.log(` ${displayPath.padEnd(60)} → ${key}`); + } + if (node.children) { + listAllRoutes(node.children, path ? `${prefix}/${path}` : prefix); + } + } +} + +// --- Component Lookup in Registry --- + +function findRegistryKeysForComponent(query, registry) { + const normalized = query.replace(/^\/|\/$/g, ''); + const searchTerms = normalized.includes('/') + ? [basename(normalized).replace(/\.tsx?$/, '')] + : [normalized]; + + const results = []; + for (const [comp, entries] of Object.entries(registry)) { + for (const term of searchTerms) { + const tl = term.toLowerCase(); + const matchesKey = entries.some( + (e) => tl === e.key.toLowerCase() || e.key.toLowerCase().includes(tl), + ); + const matchesComp = + tl === comp.toLowerCase() || comp.toLowerCase().includes(tl); + if (matchesKey || matchesComp) { + for (const entry of entries) { + if ( + tl === entry.key.toLowerCase() || + tl === comp.toLowerCase() || + comp.toLowerCase().includes(tl) || + entry.key.toLowerCase().includes(tl) + ) { + results.push({ + key: entry.key, + component: comp, + package: entry.package, + line: entry.line, + }); + } + } + break; + } + } + } + return results; +} + +function findRelatedRegistryKeys(pkg, registry) { + const results = []; + for (const [comp, entries] of Object.entries(registry)) { + for (const entry of entries) { + if (entry.package === pkg) { + results.push({ key: entry.key, component: comp, package: pkg }); + } + } + } + return results; +} + +// --- ts-morph Tracing (lazy loaded) --- + +async function traceComponent(componentName, repoRoot, mfeName, maxDepth) { + let Project; + try { + ({ Project } = await import('ts-morph')); + } catch { + console.error('ERROR: ts-morph is not installed.'); + console.error('Run: npm install --prefix ~/.claude/skills/resolve-route'); + process.exit(EXIT_USAGE_ERROR); + } + const tsConfigPath = validatePath( + join(repoRoot, 'tsconfig.base.json'), + repoRoot, + ); + if (!existsSync(tsConfigPath)) { + console.error(`ERROR: tsconfig.base.json not found at ${tsConfigPath}`); + process.exit(EXIT_USAGE_ERROR); + } + + console.error('Loading TypeScript project...'); + const startTime = Date.now(); + const project = new Project({ + tsConfigFilePath: tsConfigPath, + skipAddingFilesFromTsConfig: false, + }); + console.error( + `Project loaded in ${((Date.now() - startTime) / 1000).toFixed(1)}s (${project.getSourceFiles().length} files)`, + ); + + const registry = parseRegistry(repoRoot).entries; + const chain = []; + let currentName = componentName; + let currentMfe = mfeName; + const visitedFiles = new Set(); + const alternateRegistered = []; + + for (let depth = 0; depth < maxDepth; depth++) { + // Check registry before finding source file + const regEntries = registry[currentName]; + if (regEntries) { + const filtered = currentMfe + ? regEntries.filter((r) => r.package.includes(currentMfe)) + : regEntries; + if (filtered.length > 0) { + chain.push({ + component: currentName, + registered: filtered.map((r) => r.key), + depth, + }); + break; + } + } + + // Find source file (prefer non-barrel) + const sourceFile = findSourceFile(project, currentName, currentMfe); + if (!sourceFile) { + chain.push({ + component: currentName, + error: 'not found in project', + depth, + }); + break; + } + + const absFilePath = sourceFile.getFilePath(); + if (visitedFiles.has(absFilePath)) { + chain.push({ component: currentName, error: 'cycle detected', depth }); + break; + } + visitedFiles.add(absFilePath); + + const filePath = relative(repoRoot, absFilePath); + const referencingFiles = getReferencingFiles(sourceFile, currentName); + + if (!referencingFiles.length) { + chain.push({ component: currentName, file: filePath, usedBy: [], depth }); + break; + } + + // Map referencing files to parent info + const parents = referencingFiles + .map((refPath) => { + const refFile = project.getSourceFile(refPath); + const primaryExport = refFile ? getPrimaryExport(refFile) : null; + return { + file: relative(repoRoot, refPath), + component: primaryExport || basename(dirname(refPath)), + }; + }) + .filter( + (p) => !p.file.endsWith('index.ts') && !p.file.endsWith('index.tsx'), + ); + + const allParents = referencingFiles.map((refPath) => { + const refFile = project.getSourceFile(refPath); + const primaryExport = refFile ? getPrimaryExport(refFile) : null; + return { + file: relative(repoRoot, refPath), + component: primaryExport || basename(dirname(refPath)), + }; + }); + + const usedBy = parents.length > 0 ? parents : allParents; + + chain.push({ + component: currentName, + file: filePath, + usedBy: usedBy.map((p) => ({ + component: p.component, + file: p.file, + registered: registry[p.component] + ? registry[p.component].map((r) => r.key) + : undefined, + })), + depth, + }); + + // Collect ALL registered parents (not just the one we follow) + const candidates = usedBy.filter( + (p) => p.component && p.component !== currentName, + ); + for (const c of candidates) { + if ( + registry[c.component] && + c !== candidates.find((p) => registry[p.component]) + ) { + alternateRegistered.push({ + component: c.component, + registryKeys: registry[c.component].map((r) => r.key), + reachedFrom: currentName, + }); + } + } + + // Pick next parent: prefer registered, then screens, then first + const registeredParent = candidates.find((p) => registry[p.component]); + const screenParent = candidates.find( + (p) => p.file.includes('/screens/') || p.file.includes('/pages/'), + ); + const nextParent = registeredParent || screenParent || candidates[0]; + + if (!nextParent) break; + currentName = nextParent.component; + const mfeMatch = nextParent.file.match(/libs\/(mfe-[^/]+)/); + if (mfeMatch) currentMfe = mfeMatch[1]; + } + + const resolved = + chain.length > 0 && chain[chain.length - 1].registered + ? { + registryKey: chain[chain.length - 1].registered, + component: chain[chain.length - 1].component, + } + : null; + + return { + query: componentName, + chain, + resolved, + alternatePaths: + alternateRegistered.length > 0 ? alternateRegistered : undefined, + }; +} + +function findSourceFile(project, name, mfeName) { + let barrelMatch = null; + for (const sourceFile of project.getSourceFiles()) { + const filePath = sourceFile.getFilePath(); + if ( + /\/(spec|test|mock|__mocks__)\/|\.stories\.|\.cy\.|\.test\.|\.spec\./.test( + filePath, + ) + ) + continue; + if (mfeName && !filePath.includes(`/libs/${mfeName}/`)) continue; + if (!sourceFile.getExportedDeclarations().has(name)) continue; + + if (filePath.endsWith('/index.ts') || filePath.endsWith('/index.tsx')) { + if (!barrelMatch) barrelMatch = sourceFile; + } else { + return sourceFile; + } + } + return barrelMatch; +} + +function getReferencingFiles(sourceFile, name) { + const exported = sourceFile.getExportedDeclarations().get(name); + if (!exported?.length) return []; + + try { + const refs = exported[0].findReferences(); + const files = new Set(); + for (const ref of refs) { + for (const entry of ref.getReferences()) { + const refPath = entry.getSourceFile().getFilePath(); + if (refPath === sourceFile.getFilePath()) continue; + if ( + /\/(spec|test|mock|__mocks__)\/|\.stories\.|\.cy\.|\.test\.|\.spec\./.test( + refPath, + ) + ) + continue; + files.add(refPath); + } + } + return [...files]; + } catch (err) { + console.error(`Warning: findReferences failed for ${name}: ${err.message}`); + return []; + } +} + +function getPrimaryExport(sourceFile) { + const exports = sourceFile.getExportedDeclarations(); + for (const [name, decls] of exports) { + if (name === 'default') continue; + for (const decl of decls) { + const kind = decl.getKindName(); + if ( + kind.includes('Function') || + kind.includes('Variable') || + kind.includes('Class') + ) + return name; + } + } + for (const [name] of exports) { + if (name !== 'default') return name; + } + return null; +} + +// --- Main --- + +async function main() { + // === FILE MODE → delegates to trace === + if (args.file) { + const repoRoot = findRepoRoot(); + if (!repoRoot) { + console.error( + 'ERROR: Cannot determine repo root. Pass --repo-root /path/to/ui-platform', + ); + process.exit(EXIT_USAGE_ERROR); + } + + const filePath = resolve(args.file); + // Infer MFE from path + const mfeMatch = filePath.match(/libs\/(mfe-[^/]+)/); + const mfeName = args.mfe || (mfeMatch ? mfeMatch[1] : null); + + // Extract component name: use file stem (strip .component, .container, etc.) + let componentName = basename(filePath) + .replace(/\.tsx?$/, '') + .split('.')[0]; + if (componentName === 'index') componentName = basename(dirname(filePath)); + // Convert kebab-case to PascalCase (e.g., motion-scan → MotionScan) + if (componentName.includes('-')) { + componentName = componentName + .split('-') + .map((s) => s[0]?.toUpperCase() + s.slice(1)) + .join(''); + } + + console.error( + `Inferred: component=${componentName}, mfe=${mfeName || '(all)'}`, + ); + + // Delegate to trace + args.trace = componentName; + if (mfeName && !args.mfe) args.mfe = mfeName; + // Fall through to trace mode below + } + + // === TRACE MODE (ts-morph, ~8s) === + if (args.trace) { + const repoRoot = findRepoRoot(); + if (!repoRoot) { + console.error( + 'ERROR: Cannot determine repo root. Pass --repo-root /path/to/ui-platform', + ); + process.exit(EXIT_USAGE_ERROR); + } + + const result = await traceComponent( + args.trace, + repoRoot, + args.mfe, + parseInt(args.depth, 10) || 5, + ); + console.log(JSON.stringify(result, null, 2)); + + // Check route config for resolved + alternate paths + const configFile = findConfigFile(); + if (configFile && (result.resolved || result.alternatePaths)) { + const routeTree = loadRouteTree(configFile); + + if (result.resolved) { + for (const key of result.resolved.registryKey) { + const urls = findUrlsForRegistryKey(routeTree, key); + if (urls.length) { + console.log(`\nRoute (${key}): http://localhost:4200/#${urls[0]}`); + } else { + console.log(`\nRoute (${key}): NOT IN ROUTE CONFIG`); + } + } + } + + if (result.alternatePaths) { + console.log('\nAlternate paths:'); + for (const alt of result.alternatePaths) { + for (const key of alt.registryKeys) { + const urls = findUrlsForRegistryKey(routeTree, key); + const route = urls.length + ? `http://localhost:4200/#${urls[0]}` + : 'NOT IN ROUTE CONFIG'; + console.log(` - ${alt.component} (${key}) → ${route}`); + } + } + } + } + process.exit(result.resolved ? EXIT_FOUND : EXIT_NOT_FOUND); + } + + // === COMPONENT MODE (instant) === + if (args.component) { + const repoRoot = findRepoRoot(); + if (!repoRoot) { + console.error( + 'ERROR: Cannot determine repo root. Pass --repo-root /path/to/ui-platform', + ); + process.exit(EXIT_USAGE_ERROR); + } + const { entries } = parseRegistry(repoRoot); + const registryEntries = findRegistryKeysForComponent( + args.component, + entries, + ); + + if (!registryEntries.length) { + console.log(`No registry entry found matching: ${args.component}`); + console.log(`\nThis component is not directly in mainRegistry.tsx.`); + console.log( + `Use --trace ${args.component} to find its parent component chain.`, + ); + process.exit(EXIT_NOT_FOUND); + } + + const configFile = findConfigFile(); + if (!configFile) { + console.error('ERROR: No cached embeddable configuration found.'); + console.error( + 'Run the client-hub skill to fetch: config type = Client Config, module = embeddable configuration', + ); + process.exit(EXIT_USAGE_ERROR); + } + + console.error(`Using: ${basename(configFile)}`); + const routeTree = loadRouteTree(configFile); + + let anyFound = false; + for (const entry of registryEntries) { + console.log(`\nRegistry Key: ${entry.key}`); + console.log(`Component: ${entry.component} from ${entry.package}`); + console.log(`Registry Line: ${entry.line}`); + + const urls = findUrlsForRegistryKey(routeTree, entry.key); + if (urls.length) { + anyFound = true; + console.log('URL(s):'); + for (const url of urls) console.log(` http://localhost:4200/#${url}`); + } else { + console.log('URL(s): NOT IN ROUTE CONFIG'); + console.log( + 'Status: Registered in mainRegistry but no matching elementId in routeDefinition', + ); + const related = findRelatedRegistryKeys(entry.package, entries); + if (related.length) { + console.log(`Related keys from ${entry.package}:`); + for (const r of related) { + const rUrls = findUrlsForRegistryKey(routeTree, r.key); + console.log( + ` - ${r.key} (${r.component}) → ${rUrls[0] || '(also not routed)'}`, + ); + } + } + } + } + process.exit(anyFound ? EXIT_FOUND : EXIT_NOT_FOUND); + } + + // === FORWARD / LIST-ROUTES MODE (instant) === + const configFile = findConfigFile(); + if (!configFile) { + console.error('ERROR: No cached embeddable configuration found.'); + console.error( + 'Run the client-hub skill to fetch: config type = Client Config, module = embeddable configuration', + ); + process.exit(EXIT_USAGE_ERROR); + } + + console.error(`Using: ${basename(configFile)}`); + const routeTree = loadRouteTree(configFile); + + if (args['list-routes']) { + console.log('Available routes:'); + const rootChildren = routeTree[0]?.children || []; + listAllRoutes(rootChildren); + process.exit(EXIT_FOUND); + } + + // Forward resolution + const urlInput = args.url || args.path; + if (!urlInput) { + console.error( + 'ERROR: --path, --url, --component, or --trace required (or use --list-routes)', + ); + process.exit(EXIT_USAGE_ERROR); + } + + let pathStr = urlInput; + if (pathStr.startsWith('http')) { + const url = new URL(pathStr); + pathStr = (url.hash || url.pathname).replace(/^#?\/?/, ''); + } + pathStr = pathStr.replace(/^\/|\/$/g, ''); + const segments = pathStr.split('/').filter(Boolean); + + if (!segments.length) { + console.error('ERROR: Empty path'); + process.exit(EXIT_USAGE_ERROR); + } + + console.error(`Resolving: /${segments.join('/')}`); + const rootChildren = routeTree[0]?.children || []; + const result = findRoute(rootChildren, segments); + + if (!result) { + console.log(`\nNo match found for: /${segments.join('/')}`); + console.log('\nSearching for partial matches...'); + const searchTerm = segments[0]; + const matches = []; + function search(nodes, target, currentPath = '') { + for (const node of nodes) { + const p = (node.path || '').replace(/^\/|\/$/g, ''); + const nodePath = p ? `${currentPath}/${p}` : currentPath; + if (p.includes(target)) + matches.push({ + path: nodePath, + key: node.elementId || node.id || '(no key)', + }); + if (node.children) search(node.children, target, nodePath); + } + } + search(routeTree, searchTerm); + if (matches.length) { + for (const m of matches) console.log(` ${m.path} → ${m.key}`); + } else { + console.log(` No nodes containing '${searchTerm}' found`); + } + process.exit(EXIT_NOT_FOUND); + } + + const matchedNode = result.match; + const registryKey = matchedNode.elementId || matchedNode.id || null; + console.log(`\nRegistry Key: ${registryKey}`); + + // Show component name from registry + const repoRoot = findRepoRoot(); + if (registryKey && repoRoot) { + const { entries } = parseRegistry(repoRoot); + const matched = findRegistryKeysForComponent(registryKey, entries); + const exact = matched.find((e) => e.key === registryKey); + if (exact) { + console.log(`Component: ${exact.component} from ${exact.package}`); + } else { + console.log(`Component: (not in mainRegistry — may use defaultRegistry)`); + } + } + + const trailStr = result.trail + .map( + (n) => + n.elementId || + n.id || + (n.path || '?').replace(/^\/|\/$/g, '') || + '(layout)', + ) + .join(' → '); + console.log(`Route Trail: ${trailStr}`); + + // Show related keys + if (registryKey && repoRoot) { + const { entries } = parseRegistry(repoRoot); + const matched = findRegistryKeysForComponent(registryKey, entries); + if (matched.length) { + const related = findRelatedRegistryKeys(matched[0].package, entries); + if (related.length > 1) { + console.log(`\nRelated registry keys from ${matched[0].package}:`); + for (const r of related) { + if (r.key !== registryKey) { + const rUrls = findUrlsForRegistryKey(routeTree, r.key); + console.log( + ` - ${r.key} (${r.component}) → ${rUrls[0] || '(not in route config)'}`, + ); + } + } + } + } + } + + if (args.verbose) { + console.log('\nFull Node:'); + console.log(JSON.stringify(matchedNode, null, 2)); + } + + process.exit(EXIT_FOUND); +} + +main().catch((err) => { + console.error(`ERROR: ${err.message}`); + process.exit(EXIT_USAGE_ERROR); +}); diff --git a/.claude/skills/sync-to-async/SKILL.md b/.claude/skills/sync-to-async/SKILL.md new file mode 100644 index 00000000..505cd989 --- /dev/null +++ b/.claude/skills/sync-to-async/SKILL.md @@ -0,0 +1,384 @@ +--- +name: sync-to-async +description: Intervenes when synchronous service-to-service calls are added, shows async alternative +triggers: + - Creating synchronous HTTP call to internal service (axios, fetch, HttpClient, RestTemplate) + - URLs containing internal domains (.internal, .svc.cluster.local, internal service names) + - New service-to-service dependencies +--- + +# Sync-to-Async Intervention + +You catch synchronous service-to-service calls and challenge them. b.well's architecture is **event-driven first** (AGENTS.md line 20). Sync calls create temporal coupling and cascade failures. + +## Tone + +Direct but educational. You're preventing a common antipattern. Explain **why** async is better and **show** how to do it. + +## Detection + +**Trigger when user writes:** +- `axios.get('http://patient-service.internal/...')` +- `fetch('http://insurance-service:8080/...')` +- `RestTemplate.getForObject("http://billing-service/...", ...)` +- `WebClient.get().uri("http://pharmacy-service.svc.cluster.local/...")` +- Any HTTP client targeting internal service URLs + +**Internal service indicators:** +- `.internal` domain +- `.svc.cluster.local` (Kubernetes service) +- Port numbers like `:8080`, `:3000` (not external APIs) +- Service names in b.well namespace (patient-service, billing-service, etc.) + +**Don't trigger for:** +- External APIs (Stripe, Twilio, third-party services) +- Health check endpoints +- Service mesh control plane calls +- Explicit sync calls to cache/database (Redis, PostgreSQL) + +## Response Template + +When you detect a sync call: + +``` +Stop. You're adding a synchronous call to [SERVICE_NAME]. + +**Architecture is event-driven first** (AGENTS.md line 20). Here's why this matters: + +**Sync Problems:** +1. **Temporal coupling:** Your service is down when [SERVICE_NAME] is down +2. **Cascading failures:** [SERVICE_NAME] slow → you're slow → your callers are slow +3. **Brittle under load:** [SERVICE_NAME] spikes → timeouts → retries → death spiral +4. **Tight coupling:** Changes to [SERVICE_NAME] API require coordinated deployments + +**Async Benefits:** +1. **Resilience:** You work when [SERVICE_NAME] is down (eventual consistency) +2. **Independent scaling:** Services scale based on their own load +3. **Loose coupling:** Event contracts evolve independently +4. **Natural back-pressure:** Kafka handles load buffering + +--- + +**How to do this async:** + +[Generate saga pattern based on the operation] + +--- + +**When sync IS justified:** +- Real-time user-facing response requiring <200ms latency +- Strong consistency required (rare in healthcare workflows) +- Synchronous by nature (authentication, authorization checks) + +If you genuinely need sync here, document the justification. Want me to scaffold an ADR? +``` + +## Saga Pattern Generation + +Based on the operation being performed, generate appropriate saga: + +### Example 1: Data Query + +**User code:** +```javascript +const patient = await axios.get('http://patient-service.internal/patients/123'); +const insurance = await axios.get('http://insurance-service.internal/verify', { patientId: patient.id }); +return { patient, insurance }; +``` + +**Your response:** +"You're doing data queries. **This should be async with eventual consistency:** + +### Async Pattern: Read Replicas with Event Sync + +**Architecture:** +``` +insurance-service publishes: + → patient.insurance.verified event + +your-service subscribes: + → Maintains local read replica of insurance verification data + → Queries local data (no network call) +``` + +**Implementation:** +```javascript +// 1. Subscribe to insurance verification events +kafkaConsumer.subscribe('patient.insurance.verified', async (event) => { + await localDb.insuranceVerification.upsert({ + patientId: event.data.patientId, + status: event.data.status, + verified: event.data.verified, + updatedAt: event.timestamp + }); +}); + +// 2. Query local replica (no HTTP call) +async function getPatientInsurance(patientId) { + return await localDb.insuranceVerification.findOne({ patientId }); +} +``` + +**Benefits:** +- Zero network calls for reads +- Insurance service down → you still serve data (stale but available) +- Queries are fast (local database) + +**Tradeoffs:** +- Data is eventually consistent (stale by event lag time) +- Storage: you maintain copy of data + +**Is this acceptable for your use case?** If insurance data must be real-time (< 5 second staleness), we can discuss sync with circuit breaker." + +### Example 2: Multi-Step Transaction + +**User code:** +```javascript +// Create order +const order = await axios.post('http://order-service.internal/orders', orderData); + +// Charge payment +const payment = await axios.post('http://payment-service.internal/charge', { + orderId: order.id, + amount: order.total +}); + +// Allocate inventory +await axios.post('http://inventory-service.internal/allocate', { + orderId: order.id, + items: order.items +}); +``` + +**Your response:** +"You're orchestrating a multi-step transaction across 3 services. **This is a saga:** + +### Async Pattern: Choreography Saga with Compensations + +**Architecture:** +``` +1. your-service publishes: order.requested +2. order-service subscribes → creates order → publishes order.created +3. payment-service subscribes → charges payment → publishes payment.completed +4. inventory-service subscribes → allocates inventory → publishes inventory.allocated +5. your-service subscribes to all completion events → marks order as fulfilled +``` + +**Implementation:** + +```javascript +// Saga orchestrator +class OrderFulfillmentSaga { + async startOrder(orderRequest) { + const sagaId = uuid(); + + // Persist saga state + await sagaState.create({ + sagaId, + status: 'started', + steps: ['order_created', 'payment_charged', 'inventory_allocated'], + completed: [], + compensations: [] + }); + + // Publish initial event + await kafka.publish('order.requested', { + sagaId, + ...orderRequest + }); + + return sagaId; + } + + // Event handlers for saga progression + async onOrderCreated(event) { + await sagaState.markStepComplete(event.sagaId, 'order_created'); + // Saga continues automatically as services publish events + } + + async onPaymentCompleted(event) { + await sagaState.markStepComplete(event.sagaId, 'payment_charged'); + } + + async onInventoryAllocated(event) { + await sagaState.markStepComplete(event.sagaId, 'inventory_allocated'); + await sagaState.markSagaComplete(event.sagaId); + } + + // Compensation for failures + async onPaymentFailed(event) { + // Compensate: cancel order + await kafka.publish('order.cancelled', { + sagaId: event.sagaId, + reason: 'payment_failed' + }); + } + + async onInventoryUnavailable(event) { + // Compensate: refund payment, cancel order + await kafka.publish('payment.refund', { sagaId: event.sagaId }); + await kafka.publish('order.cancelled', { + sagaId: event.sagaId, + reason: 'inventory_unavailable' + }); + } +} +``` + +**Failure Matrix:** +| Step | Failure | Compensation | +|------|---------|--------------| +| Order creation fails | No action needed (nothing committed) | - | +| Payment fails | Cancel order | order.cancelled | +| Inventory unavailable | Refund payment, cancel order | payment.refund + order.cancelled | + +**Benefits:** +- Each service fails independently +- Compensations handle partial failures +- No distributed transactions or 2PC +- Services loosely coupled + +**Kafka Topic Naming:** +- `order.requested` - Initial saga trigger +- `order.created` - Order service completed +- `payment.completed` - Payment succeeded +- `payment.failed` - Payment failed (trigger compensation) +- `inventory.allocated` - Inventory reserved +- `inventory.unavailable` - Out of stock (trigger compensation) + +**DLQ Configuration:** +All events have Dead Letter Queue for unprocessable messages: +- `order.requested.dlq` +- `payment.completed.dlq` +etc. + +**Is this more complex than sync?** Yes, initially. But sagas handle real-world failures that sync chains don't. Sync gives you the illusion of simplicity until production breaks." + +### Example 3: Real-Time User Query + +**User code:** +```javascript +// User clicked "Verify Insurance Now" button +const verification = await axios.get('http://insurance-service.internal/verify-realtime', { + patientId: req.params.patientId +}); +return res.json(verification); +``` + +**Your response:** +"You're doing real-time user-facing query with <200ms latency requirement. **This might justify sync**, but let's examine: + +**Option 1: Async with Push Updates (Better)** +```javascript +// Backend: Async verification +async function startInsuranceVerification(patientId) { + await kafka.publish('insurance.verification.requested', { patientId }); + return { status: 'pending', estimatedTime: '3-5 seconds' }; +} + +// Frontend: WebSocket or SSE for real-time updates +websocket.on('insurance.verification.completed', (data) => { + updateUI(data); +}); +``` + +**Option 2: Sync with Circuit Breaker (If required)** +```javascript +import CircuitBreaker from 'opossum'; + +const verifyInsurance = new CircuitBreaker(async (patientId) => { + return await axios.get(`http://insurance-service.internal/verify`, { + params: { patientId }, + timeout: 200 // 200ms max + }); +}, { + timeout: 200, // Timeout threshold + errorThresholdPercentage: 50, // Open circuit at 50% errors + resetTimeout: 30000, // Try again after 30s + fallback: async (patientId) => { + // Fallback: return cached or degraded response + const cached = await cache.get(`insurance:${patientId}`); + return cached || { status: 'unavailable', cached: true }; + } +}); + +const verification = await verifyInsurance.fire(patientId); +``` + +**Sync Justification (Document in ADR if you go this route):** +- User-facing <200ms requirement: YES / NO +- Acceptable to show 'pending' with async updates: YES / NO +- Circuit breaker + fallback implemented: YES / NO +- Cache strategy for insurance service downtime: DESCRIBED +- SLA with insurance-service established: YES / NO + +If you can't answer YES to all, **use async**." + +## Specific Antipattern: Synchronous Callback Chains + +**Detection:** +Saga implemented with synchronous callbacks between services (the worst pattern): + +```javascript +// Service A calls Service B synchronously +await axios.post('http://service-b/callback', { data }); + +// Service B calls Service C synchronously +await axios.post('http://service-c/callback', { data }); + +// Service C calls Service A synchronously (cycle!) +await axios.post('http://service-a/complete', { data }); +``` + +**Your response:** +"HARD STOP. This is 'synchronous callback chains disguised as sagas' - explicitly called out as antipattern in AGENTS.md line 20. + +**This is the worst of both worlds:** +- All the complexity of distributed transactions +- None of the resilience of async +- Circular dependency hell +- Impossible to reason about failures + +**Fix:** Use proper event-driven choreography where services publish events and subscribe to events. No service calls another service's HTTP endpoints for saga orchestration. + +**If you need orchestration:** Use saga orchestrator service that subscribes to all events and publishes commands. But even then, commands are Kafka messages, not HTTP calls." + +## When User Pushes Back + +**User:** "But async is more complex!" + +**Response:** +"Async is more upfront complexity for much better runtime resilience. + +**The question:** Do you want complexity in your code (saga patterns) or complexity in your operations (cascading failures, incident response, postmortems)? + +Sync is simpler **until** it breaks in production. Then you're retrofitting circuit breakers, retries, timeouts, and fallbacks - all the complexity you should have had from the start. + +b.well's architecture default is async for this reason. Sync requires justification, not the other way around." + +**User:** "The latency requirement is <200ms!" + +**Response:** +"Let's verify: +1. Is this user-facing? (Server-to-server batch jobs don't need <200ms) +2. Is 200ms the actual requirement or a guess? +3. Can we do async with push updates instead (WebSocket, SSE)? +4. Is eventual consistency (3-5 seconds stale) actually unacceptable? + +Many 'real-time' requirements aren't actually real-time when you examine the use case. Healthcare workflows are often asynchronous by nature - results take hours, prescriptions take days. + +If it genuinely needs sync, document the justification and implement circuit breaker + fallback." + +## MCP Integration + +**Use FHIR Server MCP (when available) to:** +- Check if data being fetched already exists in FHIR +- Suggest FHIR subscriptions instead of polling + +**Use Atlassian MCP to:** +- Search for existing ADRs justifying sync calls to this service +- Pull async patterns from Confluence architecture docs + +--- + +**Remember:** You're not blocking all sync. You're ensuring sync is **justified**, **safe** (circuit breaker), and **documented**. Most sync calls should be async. Challenge them. diff --git a/.claude/skills/tech-design-review/SKILL.md b/.claude/skills/tech-design-review/SKILL.md new file mode 100644 index 00000000..2b27d5db --- /dev/null +++ b/.claude/skills/tech-design-review/SKILL.md @@ -0,0 +1,56 @@ +--- +name: tech-design-review +description: Use to review or grade an existing Technical Design Document, FDR, RFC, or architecture proposal against b.well's EA rubrics — grades it AND adversarially verifies its load-bearing claims against the real repos, Confluence/Jira, and FHIR IGs, returning a scored verdict with concrete cited gaps the way EA/Bill would. Trigger when asked to "review this design / TDD / FDR", "grade this design doc", "is this ready for architecture review", or via /tech-design-review with a Confluence page, file, or pasted text. +--- + +# /tech-design-review — grade a design and break its claims + +You are the EA reviewer. You do two things a naive reviewer doesn't: you grade against the **rubrics**, and you **adversarially verify the design's load-bearing claims against ground truth** — because the expensive failures come from designs that *read* fine but rest on an unverified assertion ("we already do X", "this is US Core conformant", "throughput is fine", "follow the orchestrator pattern"). You are **advisory and local**: output the review in-session; do not post to Jira/Confluence or modify anything. + +## Model & rigor + +Run this review — and any sub-reviewers you dispatch — on a **high-capability model (Opus)**. Design review is high-stakes and adversarial; do not down-tier it. For a large or multi-domain design, **dispatch adversarial sub-reviewers in parallel (each `model: opus`)**, one per claim cluster (below), each mandated to *falsify*, then synthesize. + +## Inputs + +A Confluence page ID/URL, a local file path, or pasted text. To read Confluence, discover the cloudId at runtime (`mcp__plugin_atlassian_atlassian__getAccessibleAtlassianResources`) — **never hardcode it** — then `getConfluencePage` + `getConfluencePageFooterComments` (prior EA verdicts). Read-only. + +## Rubrics (source of truth — load the relevant ones) + +- `rubrics/tech-design-rubric.md` — always, for a TDD/RFC. +- `rubrics/fhir-feasibility-rubric.md` — if it touches FHIR resources/profiles. +- `rubrics/api-design-rubric.md` — if it changes a public API/SDK/GraphQL/event contract. + +Supporting context to cite: `patterns/`, `decision-guides/datastore-selection.md`, `standards/events.md`, `reference-architectures/`. (Read from `icanbwell/.github` if not present locally.) + +## Phase 1 — grade against the rubric + +Go through **every** applicable criterion; decide **pass / fail / n-a (with reason)**; capture the specific evidence *in the design* (quote the offending line or note the absence). Cite the criterion anchor **and the authoritative source** the design should be appealing to — the substrate anchor (`patterns/…#pat-…`, `standards/…#std-…`), the real doc (Confluence id / IG / RFC), or the code path — and **name any anti-pattern by name** (FHIR-as-FSM, security-label overloading, Kafka-as-work-queue, distributed policy enforcement, hot-partition key, resource explosion, …). Enforce TD17 in both directions: flag where the **design itself** asserts a pattern/standard/"best practice" with **no citation and no mechanism** ("we ensure idempotency / follow Kafka best practices" is a finding, not a pass), and flag a cited source that **doesn't actually fit** the case. Distinguish severity: tenant-isolation/PHI (TD8/TD11) are **blocking**; missing framing/options/placeholders (TD1/TD5/TD13) mean **Needs-revision**. + +## Phase 2 — adversarially verify the load-bearing claims (the part that matters) + +**Mandate: assume every load-bearing claim is overstated until proven against ground truth. A claim you cannot verify is `unproven`, not `pass`. Try to break the design.** Work three clusters (dispatch one Opus verifier each for a big design): + +1. **Factual / prior-art / reuse claims.** For every "we already do X", "follow the Y pattern", "reuse service Z", or performance/scale number: verify it. Use `gh` against the real repos (does the service/pattern/topic actually exist and work as claimed?), Confluence/Jira (was this decided/approved? is there a contradicting decision?), and the substrate `reference-architectures/` + `patterns/`. Reinventing a blessed pattern, or citing prior art that doesn't exist/match, is a finding (TD4/TD15). **Also check the doc matches the as-built reality** (TD16): does the described stack / file tree / metric names actually match the repo? A design doc describing a Node.js service for a Java implementation, or fabricated metric names, is a finding — "the document must match reality." +2. **FHIR compliance + IG conformance.** Don't take "FHIR-conformant" on faith. Verify the resource choice reconciles with the **standard resource** (e.g. EOB→ExplanationOfBenefit, not a bespoke "Composition"); that canonical URLs are real (not `example.org`); that datatypes/value-set bindings are correct; and that it conforms to the **named Implementation Guide** (US Core / CARIN / DaVinci / NDH — check the actual IG profile: must-support elements, cardinality, bindings) and any Helix profile. Then check **feasibility** (`rubrics/fhir-feasibility-rubric.md`): resource-count & growth math, write amplification (every FHIR write = new version + AuditEvent), cardinality — conformant-but-infeasible is a fail. +3. **System-design best practices — scalability & patterns.** Verify the approach *fits the stated workload attributes* (throughput, read/write ratio, latency, consistency, volume & growth, burstiness) and won't hit a known scaling failure: hot partitions / low-cardinality keys, unbounded queries / N+1, resource explosion, FHIR-as-operational-state, sync cascade (missing timeout/retry/circuit-breaker), shared limiter backpressure, non-idempotent consumers. Confirm the named pattern actually matches the design (`patterns/`), the datastore fits (`decision-guides/datastore-selection.md`), and partition/key design holds (`patterns/event-key-and-partition-design.md`). If the design gives no workload numbers, that itself is a TD3 finding — you can't certify scalability against unstated load. + +Don't reward volume: a doc can be deep on mechanism and still fail because the top-level technology/pattern choice is unverified (the EA-2394 outcome). + +## Phase 3 — synthesize + +Merge Phase 1 + Phase 2. **A claim that fails verification is a finding regardless of how polished the doc is.** Output: + +``` +# Design Review: — Verdict: <Approved | Approved-with-changes | Needs-revision> +## Blocking +- <criterion/claim + the named anti-pattern, if any> — <what's wrong> — <evidence: quote / repo path / IG profile / data, plus the authoritative source the design should cite (substrate anchor / doc / code path)> — <fix> +## Required changes +## Suggestions +## Verified claims (what checked out, with evidence) +## Unverifiable — needs author to substantiate +``` + +Be honest: if something can't be assessed from the doc + ground truth, say "cannot verify — author must substantiate," don't assume it passes. Never invent approvals, metrics, IG profiles, or ticket numbers. + +**Tone (mirror how EA actually reviews):** blunt but constructive and evidence-first — state the **verdict**, then the **mechanism/proof**, and cleanly separate hard **blockers** from **follow-ups** (a good reviewer files the follow-up rather than blocking on it). Acknowledge good catches. Not adversarial for its own sake — the goal is a defensible decision, not a body count. diff --git a/.claude/skills/tech-design/SKILL.md b/.claude/skills/tech-design/SKILL.md new file mode 100644 index 00000000..d4d2276c --- /dev/null +++ b/.claude/skills/tech-design/SKILL.md @@ -0,0 +1,45 @@ +--- +name: tech-design +description: Use when authoring, writing, scoping, or revising a Technical Design Document (TDD), FDR, design doc, RFC, or architecture proposal — for a new service, feature, datastore, Kafka event, FHIR resource, or any significant change at b.well. Walks the author through the tech-design rubric and the pattern library so the design passes EA review the first time instead of being sent back. +--- + +# Tech Design authoring + +You help an engineer produce a Technical Design Document that would pass EA review. b.well designs get sent back for the same reasons every time — no problem framing, no quantified NFRs, no alternatives, wrong datastore, reinvented patterns, unbounded FHIR cardinality. Your job is to prevent that by walking the design through the rubric **before** it reaches review. + +The rubric and patterns are the source of truth — read them, don't restate them: +- Rubric: `rubrics/tech-design-rubric.md` (and `rubrics/fhir-feasibility-rubric.md`, `rubrics/api-design-rubric.md` when relevant). +- Patterns: `patterns/` (orchestrated-long-running-work, temporal-coalescing, event-key-and-partition-design). +- Decision guides: `decision-guides/datastore-selection.md`. Standards: `standards/events.md`. + +These live in the org `.github` repo; if they aren't in the current repo, read them from `icanbwell/.github` (raw) or the synced copies. + +## How to run it + +Work through the rubric in order, section by section, filling real content. Don't let the author skip ahead to the data model — the failures are almost always in A–B. + +1. **Frame the problem (TD1) with evidence + the driving requirement.** What's broken, who's affected, and a real current-state number or code observation. **Ask for the link to the PRD / product-requirements doc or ticket** (or, for engineering-driven work, the triggering incident/ticket); if there genuinely isn't one, record that and why. The success/acceptance criteria should trace back to it. If they open with a schema or code, stop and get the problem + requirement first. +2. **Requirements + quantified NFRs (TD2).** Functional list + numeric NFRs (throughput, latency, availability, freshness). Replace every `<TBD>`. If they claim an improvement, ask "measured how?" +3. **Characterize the workload (TD3), then fit.** Make them state: throughput, read/write ratio, latency budget, consistency, data volume **and growth**, access pattern, burstiness. Then use `decision-guides/datastore-selection.md` to pick a store *from those attributes*. Flag wrong-tool-for-workload. +4. **Name the pattern / right abstraction (TD4).** Point them at the closest pattern in `patterns/` and have them justify fit (or justify divergence). Actively catch the §3h wrong-abstractions: run/FSM state on FHIR `Task`, operational state or access policy on `meta.security`, a denormalized table branded a FHIR resource, distributed policy enforcement, a god-orchestrator. +5. **Options at the right altitude (TD5).** Require a real comparison table for the highest-blast-radius decision (usually the core technology/datastore/engine choice) **before** mechanism detail. "Benefits" bullets are not a comparison. +6. **Contracts.** FHIR → run `rubrics/fhir-feasibility-rubric.md` (conformance + named IG/profile + cardinality/resource-explosion math). Events → `standards/events.md` + `patterns/event-key-and-partition-design.md` (tenant key, partition count justified, event-vs-command naming, producer-owned topic, DLQ). API → `rubrics/api-design-rubric.md`. +7. **Scale & multitenancy (TD8).** Cardinality/growth math; tenant isolation on every path + an isolation test. +8. **Failure & observability (TD9–TD10).** Idempotency, retry/backoff, timeout+circuit-breaker, DLQ, outbox for audited writes; and a *meaningful* alert (volume/freshness/success-rate, not just 5xx) with health checks that assert on real output. +9. **Security/PHI (TD11) & reversibility (TD12).** For PHI outputs, decide access/audit/retention now. No secrets/PHI in logs. Infra reversibility + baseline-chart check. +10. **Completeness & governance (TD13–TD15).** No placeholders; invoke the right process (TDR / FDR / contract update); **cite prior art** — search Confluence/Jira for an existing design or blessed service before inventing one. + +## Ground every claim in the code and the org — verify, don't trust + +You have full tool access. A design authored from the author's memory is how wrong numbers, reinvented patterns, and doc-vs-reality drift get in. **Treat what the author tells you as claims to verify against the actual system**, and prefer a measured number over a stated one. This is the authoring-time twin of `/tech-design-review`'s adversarial pass — do it *before* the doc is written, not after it's rejected. + +- **Search the codebase.** `grep`/read the repo for the entities, services, topics, and FHIR resources in scope. Ground current-state and workload claims in what's actually there — count real references, read the real schema/handler — rather than a remembered figure. If the design says "millions of X," find N(X). +- **Verify prior art / reuse.** Discover the Atlassian cloudId at runtime (`getAccessibleAtlassianResources` — never hardcode it); search Confluence (`searchConfluenceUsingCql`) + Jira (`searchJiraIssuesUsingJql`, EA project / "Tech Design Review") and use `gh` against the real repos. Does the service/pattern/topic you're about to cite actually exist and behave as claimed? If a blessed pattern/service exists, use it and justify any divergence; don't reinvent it. +- **Check `approved-tech.yaml` directly.** Read `policies/approved-tech.yaml` — never assume a datastore/library/vendor is approved. Not listed → it needs a TDR; say so in the doc. +- **Run something, safely.** Where a claim is checkable, check it read-only: run the repo's existing tests, a typecheck/lint/schema-validate, a FHIR-validator pass on an example resource, or a quick query/count to pressure-test a cardinality/feasibility/perf claim (e.g. count current `Task` docs to ground a growth estimate). Never guess a number you can measure. Don't run anything that mutates shared state. +- **Anything you can't verify becomes an open item with an owner** — never assert it in the doc. "Unverified: MedStar rate limit, owner: <author>, confirming by <date>" beats a confident fabrication (mark it, exactly as `/tech-design-review` would flag it). +- **Cite the source; name the pattern (TD17).** Every time the doc appeals to a pattern, standard, or "best practice," write in the **link to the authoritative source** — the substrate anchor (`patterns/…#pat-…`, `standards/…#std-…`), the real doc (Confluence id / IG / RFC), or the code path — and **name the pattern/anti-pattern**. Don't write "we ensure idempotency" — write "idempotent under at-least-once (`standards/events.md#std-events-idempotency`) via an upsert on `ce_id`." Don't write "we store run state on a Task" — name it the **FHIR-as-FSM** anti-pattern and cite `decision-guides/datastore-selection.md#dg-fhir-not-fsm`. Cite deliberately: the source must actually fit — a reflexive citation that doesn't apply is worse than none. + +## Output + +Produce the TDD in the repo's/Confluence's expected shape, every rubric section filled, with **every pattern/standard/best-practice claim carrying its authoritative link and every anti-pattern named (TD17)** — no bare assertions. Then self-grade against `rubrics/tech-design-rubric.md` and show the author which criteria pass and which still need work, so they hit review at "Approved," not "Needs-revision." Do not invent metrics, page IDs, or approvals — if a number or prior-art link is unknown, mark it an open item with an owner. diff --git a/.claude/skills/verify-descope-mfa/SKILL.md b/.claude/skills/verify-descope-mfa/SKILL.md new file mode 100644 index 00000000..52dae8ed --- /dev/null +++ b/.claude/skills/verify-descope-mfa/SKILL.md @@ -0,0 +1,130 @@ +--- +name: verify-descope-mfa +description: Drive a real signup/login cycle against the dev Descope project to verify ktc-mfa (or other hosted-flow) changes at runtime, not just via ExportFlow diffing. +--- + +# Verify a `ktc-mfa` / hosted-flow change end-to-end + +Use this when a fix lives in Descope-hosted flow JSON (pushed via `flows_write` → +`ImportFlow`) rather than app code — `ExportFlow` diffing proves the config changed, not that +the runtime behaves as intended. This recipe drives the actual signup/login UI. + +## 1. Bring up the stack against real dev Descope + +```bash +make up-descope # needs .env.descope (TESTMAIL_API_KEY, TESTMAIL_NAMESPACE, DESCOPE_*, + # and — for the SMS path in Section 3b — TWILIO_ACCOUNT_SID/TWILIO_API_KEY_SID/ + # TWILIO_API_KEY_SECRET/TWILIO_TEST_PHONE_NUMBER) +``` +Wait for `Container kill_the_clipboard_scanner is healthy` — app at `http://localhost:5050`. +`make down-descope` when done. Don't run alongside another worktree's `make up`/`make tests` +(shared Docker Compose project name — see root CLAUDE.md's Docker caveat). + +## 2. Drive signup via Playwright, retrieving codes from testmail.app (no human inbox needed) + +`.env.descope`'s `TESTMAIL_API_KEY`/`TESTMAIL_NAMESPACE` give a real receiving mailbox at +`{namespace}.{tag}@inbox.testmail.app` — any tag auto-receives, no pre-registration. Compute the +address and poll for the code with a small script (mirrors `frontend/e2e/support/testmail.ts` +without needing Node/the e2e harness): + +```python +# reads .env.descope directly; never prints TESTMAIL_API_KEY itself +email_for_tag(namespace, tag) -> f"{namespace}.{tag}@inbox.testmail.app" +wait_for_code(api_key, namespace, tag) -> polls https://api.testmail.app/api/json, + matches email.to == expected address, regex \d{6} out of html/text body +``` + +Drive the `descope-wc` custom element (open shadow DOM — Playwright pierces it automatically, +no `>>>` needed) screen by screen: Welcome (email) → Verify OTP (code, **6 separate single-char +boxes** — see gotcha below) → Set Password → User Info → MFA. + +## 3. TOTP enrollment/challenge: read the secret from the network response, don't scan the QR + +Descope's `update-user-totp` flow/next response embeds the raw base32 secret directly in JSON — +no OCR needed: + +``` +POST .../v1/flow/next response body: + screen.state.totp.key <- base32 secret, e.g. "AWOSQTQO..." + screen.state.totp.provisionUrl <- otpauth://totp/... (same secret) +``` + +Capture it via `browser_network_requests` (filter `/v1/flow/next`) → +`browser_network_request(index, part: "response-body")` on the call that fired when you clicked +"Use Authenticator App". Then compute the current code yourself, RFC 6238 (30s step, HMAC-SHA1, +6 digits) — same algorithm as `frontend/e2e/support/totp.ts`'s `generateTotpCode`, portable to +a one-file Python script (`base64.b32decode` + `hmac.new(key, counter_bytes, "sha1")`). + +## 3b. SMS enrollment/challenge: poll the provisioned Twilio number + +`.env.descope`'s `TWILIO_ACCOUNT_SID`/`TWILIO_API_KEY_SID`/`TWILIO_API_KEY_SECRET`/ +`TWILIO_TEST_PHONE_NUMBER` (see `docs/sms-mfa-e2e-testing.md`) give a +real, reusable receiving number — enter `TWILIO_TEST_PHONE_NUMBER` (national digits only; the +country-code combobox already defaults to +1) at the MFA method screen instead of a real phone, +then poll for the OTP with Twilio's REST API, authenticating with the API Key (not the account's +Auth Token) the same way `frontend/e2e/support/smstest.ts` does: + +```bash +curl -s -u "$TWILIO_API_KEY_SID:$TWILIO_API_KEY_SECRET" \ + "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Messages.json?To=$TWILIO_TEST_PHONE_NUMBER&PageSize=5" \ + | jq -r '[.messages[] | select(.direction == "inbound")][0].body' +``` + +Results are newest-first, so `[0]` after the `direction == "inbound"` filter is the most recent +inbound SMS — filtering on `direction` matters because this number is reused across runs and picks +up unrelated real-world texts over time (observed on this exact number: a stray "Reschedule ..." +appointment reminder, apparently carried over from whoever had it before). Trigger the send, wait +a couple seconds, then poll — don't poll before clicking through the phone-number screen, or you'll +retrieve a stale message instead of the one you just requested. + +**Gotcha: the phone-number screen's submit button reads "Verify", not "Continue"/"Submit"/"Next"** +— the only screen in either MFA path with different button text than everywhere else in the flow. +Confirm via a `browser_evaluate` walk of `<descope-button>` elements (same technique Section 6 +already uses for the sign-in button-order gotcha) before clicking if unsure — clicking the wrong +(non-existent) "Continue" button here just does nothing, so the flow silently sits on the same +screen with no SMS ever sent, which then shows up downstream as "no message received" when polling +Twilio, not as an obvious failure at the click itself. + +Unlike testmail's per-run addresses, this is **one shared number** — don't run this alongside an +automated `@mfa-sms` CI run against the same Descope project, or the two polls will race on the +same inbound SMS. + +## 4. Gotcha: the passcode/TOTP code inputs are 6 separate single-character `<input>`s, not one field + +Typing the whole code into the first box via `pressSequentially`/keyboard-type is unreliable — +characters land in the wrong boxes or get dropped (observed: typing "882817" produced "827" split +across boxes 1-3). **Fill each box individually** with its one digit +(`browser_type` targeting each box's own ref, one call per digit) — reliable every time, and each +fill auto-advances focus/submits on the last digit. + +## 5. The actual regression check + +After first enrollment (TOTP or SMS), log out, log back in with the same credentials. The screen +that appears immediately after the password step tells you which bug state you're in: +- **"Enter the 6-digit code from your authenticator app"** (TOTP) or **"Enter the 6-digit code + sent to +1\*\*\*\*\*\*NNNN"** (SMS) — goes straight to the challenge — correct; `preferredMfaMethod` + persisted, `mfa-exists-check` found it non-empty. +- **"MFA is required — choose a method"** reappearing — the KTCS-16 bug (or a regression of its + fix): the custom-attribute write isn't persisting. + +Also cross-check via the Descope MCP directly: `users_read` → `LoadUser` (by loginId/email) → +`customAttributes.preferredMfaMethod` should show the enrolled method. Do the logout/login cycle +**twice** — a single pass can't rule out a one-off race. + +## 6. Sign-in button ambiguity + +`ktc-sign-in`'s password screen renders "Forgot password" **before** "Continue" in DOM order — +clicking the first button by position (rather than by matched text) submits a password-reset +request instead of signing in. If this happens, it's recoverable (just complete the reset with +the same password via the emailed code) but wastes a round trip. Confirm button text via a +quick `browser_evaluate` walk of `<descope-button>` elements before clicking when order is +ambiguous. + +## 7. Clean up afterward + +``` +tenants_write -> DeleteTenant {id: <tenantId from LoadUser>, cascade: true} +``` +Get the tenant ID from `users_read` → `LoadUser`'s `userTenants[].tenantId`, or from the +session-storage JWT's `tenantId` claim captured mid-flow. Needs a fresh `session.elevate` (cite +this exact call) if the prior elevation from pushing the fix has expired (900s TTL). diff --git a/.claude/skills/workflow-knowledge/SKILL.md b/.claude/skills/workflow-knowledge/SKILL.md new file mode 100644 index 00000000..8d4c5628 --- /dev/null +++ b/.claude/skills/workflow-knowledge/SKILL.md @@ -0,0 +1,513 @@ +--- +name: workflow-knowledge +description: Explains GitHub Actions CI/CD workflows, pipelines, and troubleshooting for the health-data-service repository. Use when a user asks about CI/CD, deployments, releases, GitHub Actions, workflow failures, how to deploy, how to promote a tag, ephemeral environments, or why a pipeline didn't trigger. +--- + +# GitHub Actions Workflow Knowledge + +## Contents + +1. PR Merge to Dev Deployment Pipeline — the automated release chain +2. Dev to Prod Promotion Pipeline — manual promotion through environments +3. PR Checks Pipeline — what runs on every PR +4. Ephemeral Environments Pipeline — label-triggered PR environments + +> **Script Jobs:** `script-job.yml` provides a `workflow_dispatch`-triggered mechanism to run ad-hoc shell commands in any environment (`dev-ue1`, `staging-ue1`, `prod-ue1`, `client-sandbox-ue1`). It uses the same `cie.gha-deploy` action with `trigger-script-job: 'true'`. Requires inputs: `script-job-name`, `script-job-command`, `env`, and `tag`. + +--- + +## PR Merge to Dev Deployment Pipeline + +When a PR is merged to `main`, an automated chain of workflows runs to create a release, build a Docker image, and deploy to the dev environment. The full chain is: + +``` +PR merged to main + → release.yml (creates semver tag + GitHub Release) + → docker-publish.yml (builds image, pushes to ECR) + → deploy.dev-ue1.yml (pre-deployment checks + deploy) +``` + +Each step triggers the next via `gh workflow run`. If any step fails, the chain stops — downstream workflows won't fire. + +--- + +### Step 1: Release (`release.yml`) + +**Trigger:** `pull_request` closed on `main` +**Condition:** PR must be merged AND must NOT have the `do-not-release` label + +**What it does:** + +1. Determines version bump from PR labels: + - `Major` label → major bump (e.g., 1.0.0 → 2.0.0) + - `Minor` label → minor bump (e.g., 1.0.0 → 1.1.0) + - `Patch` label (or no label) → patch bump (e.g., 1.0.0 → 1.0.1) +2. Creates a git tag using `anothrNick/github-tag-action` +3. Creates a GitHub Release from that tag +4. Triggers `docker-publish.yml` with the new tag and `deployDev=true` + +**Key details:** + +- Requires exactly one of `Patch`, `Minor`, `Major` labels (enforced by `check_labels.yml` on the PR) +- If no version label is present, defaults to `patch` +- The `do-not-release` label skips the entire pipeline — nothing is tagged, built, or deployed + +**Common failure points:** + +- Release not created → PR was closed without merging, or has `do-not-release` label +- Tag already exists → manual tag was created with the same version; delete it or bump manually + +--- + +### Step 2: Docker Publish (`docker-publish.yml`) + +**Trigger:** `workflow_dispatch` (called by release.yml with `tagOverride` and `deployDev=true`) +**Also triggered by:** GitHub Release `created` event (but in the automated pipeline, it's the `workflow_dispatch` path) + +**What it does:** + +1. Checks out the code at the tag ref (`refs/tags/<tag>`) +2. Authenticates to AWS ECR (region: `us-east-1`) +3. Builds Docker image with: + - Target: `server` stage in Dockerfile + - BuildKit secrets: `JFROG_READ_TOKEN` + - Build args: `SENTRY_AUTH_TOKEN`, FHIR codegen credentials +4. Pushes image to: `856965016623.dkr.ecr.us-east-1.amazonaws.com/health-data-service:<tag>` +5. If `deployDev=true`: triggers `deploy.dev-ue1.yml` with the same tag + +**Key details:** + +- Image tag matches the git tag exactly +- Uses `DOCKER_BUILDKIT=1` +- Can be run manually via workflow_dispatch with a `tagOverride` to republish an existing tag + +**Common failure points:** + +- Docker build fails → usually a dependency issue (JFrog token expired, FHIR endpoint down during codegen) +- ECR push fails → AWS credentials or permissions issue +- Dev deploy not triggered → `deployDev` was not set to `true` + +--- + +### Step 3: Dev Deployment (`deploy.dev-ue1.yml`) + +**Trigger:** `workflow_dispatch` (called by docker-publish.yml) +**Inputs:** `tag` (required), `skip_deployment_branch_checks` (optional), `use_main_branch_for_deployment_plan` (optional) + +**What it does:** + +#### 3a. Pre-Deployment Checks (`pre_deployment_checks.yml`) + +1. **Tag/branch validation** — verifies the workflow ref matches the tag (can be skipped with `skip_deployment_branch_checks=true`) +2. **Schema check** — checks out code at the tag, generates GraphQL SDL, runs `npm run wgc-check` against Cosmo, then publishes the schema with `npm run wgc-publish` + +#### 3b. Deploy (`deploy.common.yml`) + +1. Checks out code at the tag (or `main` if `use_main_branch_for_deployment_plan=true`) +2. Checks out `icanbwell/cie.gha-deploy` (external deploy action) +3. Runs deployment with: + - service-name: `health-data-service` + - env: `dev-ue1` + - image-tag: the release tag + - Updates Jira with deployment info + - Annotates ephemeral environment + +**Concurrency:** Only one deployment per environment can run at a time (concurrency key: `dev-ue1`) + +**Key details:** + +- The deploy action lives in a separate repo (`icanbwell/cie.gha-deploy`), accessed via `BWELL_DEV_PAT` +- Schema is published to Cosmo during pre-deployment (using `COSMO_API_KEY_DEV` / `COSMO_API_URL_DEV`) +- Jira tickets are updated automatically on deploy + +**Common failure points:** + +- Tag validation fails → the `--ref` used in the workflow_dispatch doesn't match the tag input; use `skip_deployment_branch_checks=true` to bypass +- Schema check fails → GraphQL schema is incompatible with the federated graph in Cosmo; fix schema issues and re-release +- Deploy action fails → check `cie.gha-deploy` action logs; often infrastructure or permissions issues + +--- + +### Pipeline Summary Table + +| Stage | Workflow File | Triggered By | Produces | +| ------------ | -------------------- | ---------------------------------------- | ------------------------------------- | +| Release | `release.yml` | PR merge to main | Git tag + GitHub Release | +| Docker Build | `docker-publish.yml` | release.yml via `gh workflow run` | ECR image `health-data-service:<tag>` | +| Dev Deploy | `deploy.dev-ue1.yml` | docker-publish.yml via `gh workflow run` | Running service in dev-ue1 | + +--- + +### How to Re-run Parts of the Pipeline + +| Scenario | Action | +| ------------------------------------------------ | ----------------------------------------------------------------------------------- | +| Release created but image not built | Manually trigger `docker-publish.yml` with `tagOverride=<tag>` and `deployDev=true` | +| Image built but dev not deployed | Manually trigger `deploy.dev-ue1.yml` with `tag=<tag>` | +| Need to skip schema check | Not possible — schema check is mandatory in pre-deployment | +| Need to skip branch validation | Set `skip_deployment_branch_checks=true` when triggering deploy | +| Need to use main's deploy plan with an older tag | Set `use_main_branch_for_deployment_plan=true` | + +--- + +### Deployment Override Options Explained + +#### `skip_deployment_branch_checks` + +**What it does:** Bypasses the pre-deployment validation that the workflow's `--ref` (branch/tag the workflow was dispatched from) matches the `tag` input. + +**Why it exists:** When `docker-publish.yml` triggers `deploy.dev-ue1.yml` automatically, it passes `--ref <tag>` so the ref and tag match naturally. But when you trigger a deploy _manually_ from the GitHub Actions UI, the "Use workflow from" dropdown defaults to `main` — meaning the ref is `main` while the tag input is something like `1.5.2`. This mismatch causes the validation to fail. + +**When to use it:** + +- You're manually triggering a deployment from the Actions UI and the "Use workflow from" field doesn't match the tag +- You need to redeploy an older tag without switching the workflow ref + +**Risk:** Low — this only skips the ref/tag consistency check. The schema check and actual deployment still use the correct tag. + +--- + +#### `use_main_branch_for_deployment_plan` + +**What it does:** Makes `deploy.common.yml` check out `main` instead of `refs/tags/<tag>` when running the deploy action. The _image_ deployed is still the one tagged with the specified version — only the deployment configuration/plan files come from main. + +**Why it exists:** The deploy action (`cie.gha-deploy`) reads deployment plan files (Helm charts, task definitions, config) from the checked-out code. Sometimes you need to deploy an older image but with updated infrastructure configuration that only exists on `main` (e.g., new environment variables, updated resource limits, changed health check paths). + +**When to use it:** + +- Infrastructure config on `main` has been updated and you need to redeploy an existing tag with the new config +- A hotfix to deployment configuration was merged to main but the service code hasn't changed (no new release needed) +- Rolling back the application version while keeping recent deployment plan changes + +**Risk:** Medium — you're deploying image version X with deploy config from a potentially newer state. Ensure the deploy config on `main` is compatible with the image version you're deploying. + +--- + +## Dev to Prod Promotion Pipeline + +Promoting a release from dev through to production is a **fully manual** process. There is no automated chain — each environment is deployed independently via `workflow_dispatch`. The promotion path is: + +``` +dev-ue1 (automatic after PR merge) + → staging-ue1 (manual trigger) + → client-sandbox-ue1 (manual trigger) + → prod-ue1 (manual trigger) +``` + +--- + +### How Promotion Works + +Every environment uses the same pattern: + +1. Human triggers the deploy workflow with a `tag` +2. Pre-deployment checks run (tag validation + schema check/publish to that environment's Cosmo) +3. `deploy.common.yml` deploys the image to the target environment + +The **same Docker image** (built once during the PR-to-Dev pipeline) is reused across all environments. You're promoting an immutable artifact — only the deployment config and target environment change. + +--- + +### Environment Workflows + +| Environment | Workflow File | Cosmo Secrets | Extra Steps | +| -------------- | ------------------------------- | --------------------------------------------------------------- | ----------------------------------------------- | +| Dev | `deploy.dev-ue1.yml` | `COSMO_API_KEY_DEV` / `COSMO_API_URL_DEV` | — | +| Staging | `deploy.staging-ue1.yml` | `COSMO_API_KEY_STAGING` / `COSMO_API_URL_STAGING` | Slack notification to `#bwell-staging-releases` | +| Client Sandbox | `deploy.client-sandbox-ue1.yml` | `COSMO_API_KEY_CLIENT_SANDBOX` / `COSMO_API_URL_CLIENT_SANDBOX` | — | +| Prod | `deploy.prod-ue1.yml` | `COSMO_API_KEY_PROD` / `COSMO_API_URL_PROD` | — | + +All four workflows accept the same inputs: + +- `tag` (required) — the version tag to deploy +- `skip_deployment_branch_checks` (optional, default: false) +- `use_main_branch_for_deployment_plan` (optional, default: false) + +--- + +### Promoting a Tag: Step by Step + +#### Dev → Staging + +1. Confirm the tag is deployed and validated in dev +2. Go to Actions → "Staging Deployment to ue1" → Run workflow +3. Set "Use workflow from" to the tag (e.g., `1.5.2`), or use `main` and set `skip_deployment_branch_checks=true` +4. Enter the tag in the `tag` field +5. Run — pre-deployment checks will publish the schema to Cosmo staging +6. On success, a Slack message is posted to `#bwell-staging-releases` + +#### Staging → Client Sandbox + +1. Confirm the tag is validated in staging +2. Go to Actions → "Client Sandbox Deployment to ue1" → Run workflow +3. Same input pattern (tag + optional overrides) +4. Run — schema is published to Cosmo client-sandbox, then the image is deployed + +#### Client Sandbox → Prod + +1. Confirm the tag is validated in client sandbox +2. Go to Actions → "Prod Deployment to ue1" → Run workflow +3. Same input pattern (tag + optional overrides) +4. Run — schema is published to Cosmo prod, then the image is deployed + +--- + +### Key Differences from the Dev Pipeline + +| Aspect | Dev | Staging / Client Sandbox / Prod | +| --------------------- | --------------------------------------- | -------------------------------------------------- | +| Trigger | Automatic (chained from docker-publish) | Manual (workflow_dispatch) | +| Schema publish target | Cosmo dev | Environment-specific Cosmo instance | +| Notifications | None | Staging posts to Slack (`#bwell-staging-releases`) | +| Concurrency | One deploy at a time per env | Same — `concurrency: ${{ inputs.env }}` | + +--- + +### Common Failure Points + +| Failure | Cause | Fix | +| ------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | +| Schema check fails on staging/prod | Schema incompatibility with that environment's federated graph | Fix schema, re-release, redeploy to dev first, then promote up | +| "Branch must match tag" error | Triggered workflow from `main` instead of the tag ref | Re-run with `skip_deployment_branch_checks=true`, or select the tag in "Use workflow from" | +| Deploy succeeds but service unhealthy | Image works in dev but environment-specific config differs | Check env-specific secrets, environment variables, and downstream service availability | +| Slack notification not posted | Only staging has Slack integration; deploy may have succeeded anyway | Check the deploy job status independently of the Slack step | + +--- + +### Rollback + +To roll back an environment to a previous version: + +1. Identify the previous working tag (check GitHub Releases or deployment history) +2. Trigger the environment's deploy workflow with that older tag +3. Optionally set `use_main_branch_for_deployment_plan=true` if recent deploy config changes on main are needed for the rollback to work correctly + +There is no dedicated rollback workflow — rolling back is just deploying an older tag. + +--- + +## PR Checks Pipeline + +When a pull request is opened (or updated) against `main`, several workflows run in parallel to validate the change before it can be merged. These are all independent — they don't trigger each other. + +``` +PR opened/updated against main + ├── ci.yml (lint, build, unit-test, integration-test) + ├── schema-check.yml (GraphQL schema compatibility) + ├── check_commit_msg.yml (JIRA issue key format) + └── check_labels.yml (version/release label present) +``` + +--- + +### CI/CD (`ci.yml`) + +**Trigger:** `pull_request` to main (opened, synchronize, reopened), also `push` to main and `workflow_dispatch` + +**Jobs and dependencies:** + +``` +lint (independent) +build (independent) + ├── unit-test (needs: build) + └── integration-test (needs: build) +``` + +#### Lint Job + +- Node.js 22.x setup with JFrog registry +- Runs `npm run lint` and `npm run format:check` +- Catches ESLint violations and Prettier formatting issues + +#### Build Job + +- Builds the Docker image (same as production build) to verify it compiles +- Does NOT push to ECR — this is a validation-only build +- Uses the same build args and secrets as the production docker-publish + +#### Unit Test Job (depends on: build) + +- Copies `.env.example` → `.env` +- Runs FHIR code generation (`npm run generate:fhir` + `npm run generate:types`) +- Runs `npm run test:unit -- --runInBand --coverage` +- On non-PR events (push to main): also runs SonarQube scan for code quality metrics +- SonarQube quality gate is NOT enforced (commented out in workflow) + +#### Integration Test Job (depends on: build) + +- Same setup as unit tests +- Runs `npm run test:integration -- --runInBand` +- Uses additional secret: `CLIENT_CONFIGURATION_SECRET_KEY` + +**Why tests depend on build:** The build job validates that the Docker image compiles successfully. Tests run after to avoid wasting CI time on tests if the build itself is broken. + +--- + +### Schema Check (`schema-check.yml`) + +**Trigger:** `pull_request` to main (all types) + +**What it does:** + +1. Installs dependencies +2. Generates GraphQL SDL files (`npm run generate:sdl`) +3. Runs `npm run wgc-check` — validates the schema is compatible with the federated graph in Cosmo + +**Key details:** + +- On PRs, it only **checks** — it does NOT publish the schema +- Uses dev Cosmo credentials as fallback: `secrets.COSMO_API_KEY || secrets.COSMO_API_KEY_DEV` +- Schema is only **published** during deployment (when called via `workflow_call` with `publishSchema: true`) +- Includes Docker Compose setup/cleanup (always runs cleanup even on failure) + +--- + +### Commit Message Check (`check_commit_msg.yml`) + +**Trigger:** `pull_request` to main (opened, synchronize, reopened, edited) + +**What it validates:** + +- PR title must match: `^(Bump|Merge|([?[A-Z]+-[0-9]+]?)):?\s+[\w\s]+` +- In practice: must start with a JIRA issue key (e.g., `PHR-1234 Add feature`) or `Bump`/`Merge` + +**Exceptions:** + +- Skipped entirely for `dependabot[bot]` PRs +- `Bump` prefix is allowed for dependency update PRs +- `Merge` prefix is allowed for merge commits + +--- + +### Label Check (`check_labels.yml`) + +**Trigger:** `pull_request` to main (opened, labeled, unlabeled, ready_for_review, reopened, synchronize) + +**What it validates:** + +- PR must have **exactly one** label from: `do-not-release`, `Patch`, `Minor`, `Major` +- Fails if zero labels or more than one of these labels is applied + +**Why this matters:** The release workflow uses these labels to determine the semver bump. Without exactly one, the release pipeline won't know how to tag. + +--- + +### PR Checks Summary + +| Check | Workflow | What Fails It | How to Fix | +| ----------------- | --------------------------- | ------------------------------------- | ------------------------------------------------------------------------------ | +| Lint | `ci.yml` → lint | ESLint errors or Prettier formatting | Run `npm run lint -- --fix` and `npm run format` locally | +| Build | `ci.yml` → build | Docker image won't compile | Fix build errors; check if FHIR codegen dependencies are available | +| Unit Tests | `ci.yml` → unit-test | Test failures | Run `npm run test:unit` locally to reproduce | +| Integration Tests | `ci.yml` → integration-test | Test failures | Run `npm run test:integration` locally; may need FHIR secrets in `.env` | +| Schema | `schema-check.yml` | SDL incompatible with federated graph | Run `npm run generate:sdl && npm run wgc-check` locally with Cosmo credentials | +| Commit Message | `check_commit_msg.yml` | Missing JIRA key in PR title | Rename PR to `JIRA-123 Description` format | +| Labels | `check_labels.yml` | Missing or multiple version labels | Add exactly one of: `Patch`, `Minor`, `Major`, or `do-not-release` | + +--- + +### Notes + +- All CI jobs run on self-hosted runners (label: `main`) +- Node.js version is 22.x across all jobs +- npm registry is JFrog (configured via `.npmrc.sample` → `.npmrc`) +- FHIR code generation is required before tests can run — tests depend on generated types +- SonarQube only runs on pushes to main (not on PRs) to avoid token exposure in fork PRs + +--- + +## Ephemeral Environments Pipeline + +Ephemeral environments are short-lived deployments tied to a specific PR. They let you test your branch in a real environment before merging. The lifecycle is fully automatic once the label is applied. + +``` +PR labeled with 'ephemeral-environment' + → build-ephemeral-image.yml (build + push Docker image tagged with commit SHA) + → deploy-ephemeral-environment (external action from icanbwell/actions) + +PR closed (merged or not) + → ephemeral-environment-cleanup.yml (destroys the environment) +``` + +--- + +### Activation + +Ephemeral environments are **opt-in per PR**. To activate: + +1. Add the `ephemeral-environment` label to your PR +2. The deploy workflow triggers on the next PR event (push, reopen, or add the label to an already-open PR) + +Without the label, none of the ephemeral environment workflows run. + +--- + +### Deploy (`ephemeral-environment.yml`) + +**Trigger:** `pull_request` to main (opened, reopened, synchronize) +**Condition:** PR must have the `ephemeral-environment` label + +**Jobs:** + +#### 1. Build Ephemeral Image (`build-ephemeral-image.yml`) + +- Checks out the PR's head commit SHA (not the merge commit) +- Builds the Docker image with the same Dockerfile and build args as production +- Tags the image with the full git commit SHA: `856965016623.dkr.ecr.us-east-1.amazonaws.com/health-data-service:<commit-sha>` +- Pushes to ECR +- Outputs: `image-uri` (the full ECR URI with tag) + +**Difference from production build:** The image tag is a commit SHA (not a semver tag), and it uses `BWELL_DEV_PAT` for NPM auth instead of `GITHUB_TOKEN`. + +#### 2. Deploy Ephemeral Environment + +- Calls the shared workflow from `icanbwell/actions` repo +- Passes: organization, repository name, PR number, and the built image URI +- The external action handles the actual infrastructure provisioning (namespace, deployment, routing) + +--- + +### Cleanup (`ephemeral-environment-cleanup.yml`) + +**Trigger:** `pull_request` to main — type `closed` (covers both merged and unmerged) + +**What it does:** + +- Calls `icanbwell/actions/.github/workflows/destroy-ephemeral-environment.yml` +- Passes: organization, repository name, PR number +- Destroys all infrastructure associated with that PR's ephemeral environment + +**Key detail:** This runs on ALL PR closures, regardless of whether the `ephemeral-environment` label is present. The external destroy action handles the case where no environment exists (no-op). + +--- + +### Lifecycle Summary + +| Event | What Happens | +| ----------------------------------- | ---------------------------------------------------------------------------------- | +| Label `ephemeral-environment` added | Next PR event triggers build + deploy | +| New commit pushed to labeled PR | Image rebuilt with new SHA, environment updated | +| PR reopened (with label) | Environment redeployed | +| PR closed or merged | Environment destroyed automatically | +| Label removed (PR still open) | No new deploys on future pushes, but existing environment stays up until PR closes | + +--- + +### Common Failure Points + +| Failure | Cause | Fix | +| ---------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------- | +| Environment not deploying | Missing `ephemeral-environment` label | Add the label; push a commit or reopen PR to trigger | +| Build fails | Same reasons as CI build (dependency issues, FHIR endpoint down) | Check build-ephemeral-image job logs | +| Deploy fails | Infrastructure issue in the external `icanbwell/actions` workflow | Check the deploy-ephemeral-environment job; may need platform team help | +| Environment not cleaned up | Cleanup workflow failed or was skipped | Manually trigger the destroy workflow or notify platform team | +| Stale environment after force-push | Image was rebuilt but deploy may have used cached state | Close and reopen PR to force a full redeploy | + +--- + +### Notes + +- The ephemeral image is tagged with the commit SHA, not a version — it's never promoted to other environments +- Each push to a labeled PR rebuilds and redeploys (no caching between pushes) +- The external deploy/destroy actions live in `icanbwell/actions` — this repo only controls the build step +- The cleanup is intentionally unconditional on the label — ensures no orphaned environments if the label is removed before closing diff --git a/.github/workflows/check-commit-message.yml b/.github/workflows/check-commit-message.yml new file mode 100644 index 00000000..59d2fb2d --- /dev/null +++ b/.github/workflows/check-commit-message.yml @@ -0,0 +1,18 @@ +name: 'Commit Message Check' +on: + pull_request: + types: [opened, edited, reopened, synchronize] + push: + branches: [main, 'releases/*'] + +jobs: + check-commit-message: + name: Check Commit Message + runs-on: ubuntu-latest + steps: + - name: Check For Issue Key + uses: gsactions/commit-message-checker@v2 + with: + pattern: '^(Bump|Merge|Revert|Reapply|build\(deps(-dev)?\): bump|([A-Z]+-[0-9]+))\s+.+' + flags: 'gm' + error: 'Commit message must begin with a JIRA Issue Key: e.g. HP-123 <message>' diff --git a/.github/workflows/pr-architecture-review.yml b/.github/workflows/pr-architecture-review.yml new file mode 100644 index 00000000..09a2e4ec --- /dev/null +++ b/.github/workflows/pr-architecture-review.yml @@ -0,0 +1,22 @@ +name: PR Architecture Review + +on: + pull_request_target: + types: [opened, synchronize, reopened] + +permissions: + contents: read + pull-requests: write + +jobs: + call-review: + uses: icanbwell/.github/.github/workflows/pr-architecture-review-reusable.yml@main + with: + pr_number: ${{ github.event.pull_request.number }} + pr_author: ${{ github.event.pull_request.user.login }} + base_ref: ${{ github.event.pull_request.base.ref }} + head_ref: ${{ github.event.pull_request.head.ref }} + head_sha: ${{ github.event.pull_request.head.sha }} + secrets: + gh_token: ${{ secrets.GITHUB_TOKEN }} + bwell_dev_pat: ${{ secrets.BWELL_DEV_PAT }} diff --git a/AGENTS.md b/AGENTS.md index bd763279..2c953890 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,6 +16,22 @@ Treat these as hard constraints, not suggestions. Code that violates tenant isol --- +## Design-Time Quality Kit & Knowledge Substrate + +The worst failures happen at **design time**, before code review can catch them. b.well maintains a tool-neutral **knowledge substrate** — the single source of truth for rules, patterns, and gradeable review rubrics — and a design-time kit that uses it. Reach for these instead of re-deriving (or reinventing) an approach: + +- **Authoring a design?** Use the **`tech-design`** skill — it walks you through the rubric so the design passes EA review the first time. +- **Reviewing a design?** Use **`/tech-design-review`** — it grades a TDD/FDR against the rubric and returns concrete, cited gaps. +- **Rubrics** (`rubrics/`) — what "good" means, gradeably: `tech-design-rubric.md`, `fhir-feasibility-rubric.md` (conformance + IG conformance + resource-explosion feasibility), `api-design-rubric.md`. +- **Patterns** (`patterns/`) — named, blessed shapes to appeal to by name, not reinvent: `orchestrated-long-running-work`, `temporal-coalescing`, `event-key-and-partition-design`. +- **Decision guides** (`decision-guides/`) — e.g. `datastore-selection.md` (including *do not put run/FSM state on a FHIR `Task`*). +- **Standards** (`standards/`) — canonical rules, e.g. `events.md` (Kafka/event conventions; supersedes the inline `patient.updated`-style examples elsewhere in this file). +- **Reference architectures** (`reference-architectures/`) — annotated real exemplars (the DEQM orchestrator; a good API/SDK). + +Overview + the stable-anchor citation convention: `docs/knowledge-substrate.md`. Cite substrate content by anchor (e.g. `standards/events.md#std-events-partition-key`), never by line number. + +--- + ## Architecture Non-Negotiables ### Event-Driven First @@ -105,24 +121,6 @@ Keep interfaces small and focused. If a consumer only needs read access, do not ### Dependency Inversion Depend on abstractions at module and service boundaries. Inject dependencies through constructors. Never instantiate infrastructure inside business logic. Never use service locators when constructor injection is available. -### IoC Container (`simple_container`) - -This project uses the `simple_container` library for inversion of control. All service registrations live in `languagemodelcommon/container/container_factory.py`. - -**Rules:** -- Register new services as singletons in `LanguageModelCommonContainerFactory.register_services_in_container()` — do not instantiate services directly in calling code. -- When creating a new service class, add its registration to `container_factory.py` and inject its dependencies from the container (not by constructing them inline). -- Tests can override registrations by constructing a test container with mock implementations. - -### Environment Variables (`LanguageModelCommonEnvironmentVariables`) - -Access environment variables through the `LanguageModelCommonEnvironmentVariables` class (`languagemodelcommon/utilities/environment/language_model_common_environment_variables.py`), not via `os.environ` directly. This class is registered in the IoC container. - -**Rules:** -- Add new environment variable access as `@property` methods on `LanguageModelCommonEnvironmentVariables`. -- Inject the environment variables class via the container — do not instantiate it inline or call `os.environ.get()` in business logic. -- Provide sensible defaults in the property when the variable is optional. - --- ## Architectural Boundaries @@ -170,25 +168,6 @@ Use current language idioms for the repo's language version. Do not write legacy **Python:** Use dataclasses or Pydantic models, not manual dict manipulation. Use type hints everywhere. Use structural pattern matching (3.10+) where it improves clarity. Use `Protocol` for structural typing. Use `async`/`await` for IO-bound operations in async services. -**Keyword arguments are mandatory in Python code.** This applies to both definitions and call sites: -- **Definitions:** All public functions, methods, and constructors must use the `*` separator to enforce keyword-only arguments (after `self`/`cls` if present). The only exception is callback signatures required by frameworks (e.g., LangGraph `RunnableLambda` expects `func(state, config)` positionally). -- **Call sites:** Always pass parameters by keyword, not by position — this applies to function calls, constructor invocations, and method calls. Positional arguments are fragile and break when signatures change. -- **Tests:** Test code must also use keyword arguments when calling production code. No exceptions. - -```python -# Good — definition enforces keyword-only -def resolve_mcp_servers(*, configs: list[ChatModelConfig], mcp_config: McpJsonConfig) -> None: ... - -# Good — call site uses keywords -resolve_mcp_servers(configs=[config], mcp_config=mcp) - -# Bad — positional args at call site -resolve_mcp_servers([config], mcp) - -# Bad — definition allows positional -def resolve_mcp_servers(configs: list[ChatModelConfig], mcp_config: McpJsonConfig) -> None: ... -``` - **TypeScript:** Use discriminated unions for variant types, not type casting chains. Use strict mode. Use `readonly` and `as const` where appropriate. Use modern `satisfies` operator for type-safe object literals. Use optional chaining and nullish coalescing instead of manual null checks. --- @@ -305,7 +284,7 @@ Understand the constraints of the system you are working in. If branch protectio When a user tells you an action is blocked or explains a constraint, internalize it. Do not re-propose the same blocked action with different wording. If you are unsure whether a constraint applies, ask once. If the answer is "no, that won't work", do not ask again or try to find a loophole. ### Don't Guess Commands -Find and use the repo's canonical build, test, and lint commands. Check the Makefile, package.json scripts, build.gradle, pyproject.toml, or uv.lock. This repo uses `uv` for Python dependency management (not pip/pipenv). If the commands are unclear, say so and point to where you looked. Do not invent commands. +Find and use the repo's canonical build, test, and lint commands. Check the Makefile, package.json scripts, build.gradle, or Pipfile. If the commands are unclear, say so and point to where you looked. Do not invent commands. ### Don't Introduce Dependencies Casually Check `approved-tech.yaml` before adding any new dependency. If the dependency is not listed, flag it for review. Do not assume a library is approved because it is popular. @@ -323,9 +302,6 @@ If you encounter code that violates these principles - tenant isolation missing Do not add "Co-Authored-By", "Generated by", or any other AI attribution to commits, PRs, or code comments. Engineers own their code regardless of what tool assisted in writing it. The tool is irrelevant. The author on the commit is the owner. -### Branch Protection -Direct commits and merges to `main` are blocked by branch protection rules. All changes must go through a pull request. If you are not already on a feature branch, create one before making any commits. Do not attempt to push directly to `main`, force-push to `main`, or bypass branch protection. If the user asks you to make a change and you are on `main`, create a branch first following the naming convention below. - ### Branch Naming Branch names must follow the pattern: `{initials}-{project}-{ticket-number}` diff --git a/CODEOWNERS b/CODEOWNERS new file mode 100644 index 00000000..be66473b --- /dev/null +++ b/CODEOWNERS @@ -0,0 +1,3 @@ +# Architecture instruction files owned by Enterprise Architecture +/AGENTS.md @icanbwell/enterprise-architecture +/CLAUDE.md @icanbwell/enterprise-architecture diff --git a/policies/approved-tech.yaml b/policies/approved-tech.yaml new file mode 100644 index 00000000..ad74071d --- /dev/null +++ b/policies/approved-tech.yaml @@ -0,0 +1,364 @@ +# Approved Technology Stack - icanbwell +# Owner: Enterprise Architecture +# Last Updated: 2026-03-03 +# Version: 1.0.0 +# +# Purpose: Centralized registry of approved technologies, frameworks, and tools. +# Any addition requires Tech Design Review with EA approval. +# +# Enforcement: +# - severity: "warn" = Flag for review, does not block +# - severity: "error" = Blocks PR merge, requires exception approval +# - Phase 2 enforcement starts at "warn" for all categories + +--- +version: "1.0.0" +updated: "2026-03-03" +enforcement: + defaultSeverity: "warn" + exceptionProcess: "Tech Design Review with EA (JIRA ticket in EA project)" + +categories: + # ============================================================================ + # Languages + # ============================================================================ + languages: + severity: "warn" + approved: + - name: "TypeScript" + versions: ["5.x"] + notes: "Preferred for new Node.js services" + + - name: "JavaScript" + versions: ["ES2022+"] + notes: "Approved for Node.js services alongside TypeScript" + + - name: "Python" + versions: ["3.11", "3.12"] + notes: "Preferred for data pipelines, AI/ML services" + + - name: "Java" + versions: ["17", "21"] + notes: "Spring Boot services, FHIR processing" + + - name: "Kotlin" + versions: ["1.9+"] + notes: "Mobile SDKs, Android native" + + - name: "Swift" + versions: ["5.9+"] + notes: "iOS SDK only" + + - name: "HCL" + versions: ["Terraform 1.5+"] + notes: "Infrastructure as Code only" + + # ============================================================================ + # Frameworks & Libraries + # ============================================================================ + frameworks: + severity: "warn" + approved: + # Backend Frameworks + - name: "NestJS" + category: "backend" + versions: ["10.x"] + notes: "Preferred Node.js backend framework" + + - name: "FastAPI" + category: "backend" + versions: ["0.109+"] + notes: "Preferred Python backend framework" + + - name: "Flask" + category: "backend" + versions: ["3.x"] + notes: "LEGACY ONLY - Do not use for new services. Migrate to FastAPI." + legacy: true + + - name: "Django" + category: "backend" + versions: ["4.x", "5.x"] + notes: "Legacy bwell-platform only. Do not use for new services." + legacy: true + + - name: "Spring Boot" + category: "backend" + versions: ["3.x"] + notes: "Java microservices standard" + + # Frontend Frameworks + - name: "React" + category: "frontend" + versions: ["18.x"] + notes: "Web UI standard" + + - name: "React Native" + category: "mobile" + versions: ["0.73+"] + notes: "Mobile app standard" + + # AI/ML Frameworks + - name: "LangChain" + category: "ai-ml" + versions: ["0.1.x"] + notes: "LLM orchestration, agent frameworks" + + - name: "LangGraph" + category: "ai-ml" + versions: ["0.0.x"] + notes: "Stateful agent workflows" + + # ============================================================================ + # Data Storage + # ============================================================================ + datastores: + severity: "warn" + approved: + - name: "FHIR Server" + type: "document" + implementation: "MongoDB" + notes: "Platform system-of-record; access via FHIR APIs only" + + - name: "MongoDB" + type: "document" + versions: ["7.x"] + notes: "Service-private datastores; tenant isolation mandatory" + + - name: "PostgreSQL" + type: "relational" + versions: ["15.x", "16.x"] + notes: "First-class approved relational store, incl. operational run/FSM state with JPA + Flyway migrations (the clinical-reasoning-orchestrator uses PostgreSQL). Choose relational (PostgreSQL) vs document (MongoDB) by workload, not by default — see decision-guides/datastore-selection.md. A new-service datastore still requires a Tech Design Review; tenant isolation mandatory." + + - name: "Elasticsearch" + type: "search" + versions: ["8.x"] + notes: "Search and analytics; requires Tech Design Review for new use cases" + + - name: "Neo4j" + type: "graph" + versions: ["5.x"] + notes: "Graph database for data operations; requires Tech Design Review for new use cases" + + - name: "ClickHouse" + type: "analytics" + versions: ["23.x", "24.x"] + notes: "OLAP database for analytics and time-series data. Also an active EA-approved track (EA-2126) as a hybrid FHIR storage backend for resources that exceed MongoDB's 16MB BSON limit (e.g. Group at 1M+ members) — see reference-architectures/fhir-server-group-scaling.md. That is a reviewed, in-flight direction for the specific fhir-server use case, not a blanket approval to use ClickHouse as a general operational datastore; a new use case still requires a Tech Design Review." + + - name: "Databricks" + type: "analytics-platform" + versions: ["Runtime 13.x+"] + notes: "Data lakehouse platform for big data analytics and ML pipelines" + + # ============================================================================ + # Messaging & Events + # ============================================================================ + messaging: + severity: "warn" + approved: + - name: "Kafka" + versions: ["3.x"] + notes: "Primary async communication; CloudEvents envelope required" + patterns: + - "CloudEvents format mandatory" + - "Idempotent consumers required" + - "Partition keys must ensure entity ordering" + + # ============================================================================ + # Caching + # ============================================================================ + caching: + severity: "warn" + approved: + - name: "Redis" + versions: ["7.x"] + notes: "Distributed cache standard; tenant keys required; cache introduction requires ADR" + + - name: "In-Process Caching" + examples: ["Caffeine (Java)", "node-cache (Node.js)", "cachetools (Python)"] + notes: "Service-local caching; requires ADR documenting strategy, size limits, TTL, eviction policy" + + # ============================================================================ + # Observability + # ============================================================================ + observability: + severity: "warn" + approved: + - name: "OpenTelemetry" + versions: ["1.x"] + notes: "Tracing, metrics, logs; trace context propagation mandatory" + required: true + + - name: "Groundcover" + versions: ["Current"] + notes: "K8s observability platform" + + # ============================================================================ + # Infrastructure & DevOps + # ============================================================================ + infrastructure: + severity: "warn" + approved: + - name: "Terraform" + versions: ["1.5+"] + notes: "IaC standard; all infra changes via Terraform PRs" + required: true + + - name: "Docker" + versions: ["24+"] + notes: "Containerization standard" + required: true + + - name: "Kubernetes" + versions: ["1.28+"] + provider: "AWS EKS" + notes: "Orchestration standard" + + - name: "Helm" + versions: ["3.x"] + notes: "K8s package management; .helm/ structure standard" + + - name: "GitHub Actions" + versions: ["Current"] + notes: "CI/CD standard" + + # ============================================================================ + # Testing Frameworks + # ============================================================================ + testing: + severity: "warn" + approved: + - name: "Jest" + language: "TypeScript/JavaScript" + versions: ["29.x"] + notes: "TypeScript/Node.js testing standard" + + - name: "pytest" + language: "Python" + versions: ["8.x"] + notes: "Python testing standard; pytest-cov for coverage" + + - name: "JUnit 5" + language: "Java" + versions: ["5.10+"] + notes: "Java testing standard; Jacoco for coverage" + + - name: "Playwright" + category: "e2e" + versions: ["1.x"] + notes: "Browser testing for web UIs" + + - name: "Detox" + category: "mobile-e2e" + versions: ["20.x"] + notes: "React Native E2E testing" + + - name: "Karate" + category: "api" + versions: ["1.4+"] + notes: "API contract testing" + + # ============================================================================ + # Linting & Formatting + # ============================================================================ + linting: + severity: "warn" + approved: + - name: "ESLint" + language: "TypeScript/JavaScript" + versions: ["9.x"] + notes: "Flat config (eslint.config.js) preferred; migrate from .eslintrc" + + - name: "Ruff" + language: "Python" + versions: ["0.3+"] + notes: "Python linter; replaces Flake8; combines linting + formatting" + + - name: "Prettier" + language: "TypeScript/JavaScript" + versions: ["3.x"] + notes: "Code formatting; consistent config org-wide" + + # ============================================================================ + # Security Scanning + # ============================================================================ + security: + severity: "warn" + approved: + - name: "Aikido" + category: "SAST/SCA" + versions: ["Current"] + notes: "SAST and dependency scanning; managed by Security team" + + - name: "CodeQL" + category: "SAST" + versions: ["Current"] + notes: "GitHub native SAST; language-specific queries" + +# ============================================================================ +# Migration Notes +# ============================================================================ +migrations: + - from: "Flake8" + to: "Ruff" + status: "Mandatory" + notes: "All repos must migrate to Ruff. Flake8 is deprecated." + + - from: "Black" + to: "Ruff Formatter" + status: "Mandatory" + notes: "Ruff combines linting + formatting; simplifies toolchain. Black is deprecated." + + - from: "Flask" + to: "FastAPI" + status: "Recommended" + notes: "Migrate Flask services to FastAPI for new features; full migration not required immediately" + + - from: ".eslintrc.json" + to: "eslint.config.js" + status: "Recommended" + notes: "ESLint 9+ flat config; improved composability" + +# ============================================================================ +# Restricted Technologies +# ============================================================================ +restricted: + - name: "Go" + reason: "On hold pending language strategy review. Existing Go services (provider-directory, document-processing-handlers) are grandfathered." + exception: "Requires Tech Design Review with EA approval for new services." + severity: "error" + + - name: "SQS" + reason: "Prefer Kafka for cross-service async communication. SQS creates AWS lock-in and lacks CloudEvents support." + exception: "Requires Tech Design Review with EA approval. Must document why Kafka is insufficient." + severity: "warn" + + - name: "New Cache Introduction" + reason: "All cache introductions (Redis, Caffeine, in-memory) require ADR or Tech Design Review" + requiresADR: true + exception: "Document cache strategy, size limits, TTL, eviction policy, tenant isolation approach" + + - name: "New NoSQL Databases" + reason: "Datastores require Tech Design Review; prefer MongoDB/PostgreSQL" + exception: "EA-approved use cases only" + + - name: "Service Meshes" + reason: "Complexity vs. benefit trade-off under evaluation" + exception: "EA-approved pilots only" + + - name: "Client-Side State Management (Redux, MobX, etc.)" + reason: "React Context + hooks preferred for new code" + exception: "Legacy codebases maintain existing patterns" + +# ============================================================================ +# Review Process +# ============================================================================ +reviewProcess: + trigger: "Introduction of technology not listed in this file" + steps: + - "Create JIRA ticket in EA project (type: Tech Design Review)" + - "Submit Technical Design Document with rationale" + - "EA review and approval/rejection" + - "If approved, update this file via PR to icanbwell/.github" + sla: "5 business days for EA review" + appeals: "Escalate to VP Engineering if rejected"