From 8cea7964bb4c971fb1adf94d8d8c075858ccc6d3 Mon Sep 17 00:00:00 2001 From: Nesmesnhnoy Date: Mon, 3 Aug 2026 13:36:29 +0300 Subject: [PATCH] docs(mpf): fold the MPF pipeline into Deploy, correct drift from the implementation The MPF pipeline had its own page under Run Payerbox, outside the agreed IA (Architecture / Deploy / Maintain), and mixed three altitudes: what the module publishes, how to deploy it, and how to operate it. Provider Directory never mentioned MPF at all, so a reader looking for the Medicare Plan Finder feed found nothing there. Split per the agreed IA, whose Deploy contract already covers "scope configuration, env vars": - Deploy gains a trailing "MPF provider-directory pipeline" section: buckets, sync client, access policy, env vars, export scope, first run, daily schedule. Placed after Upgrade and uninstall so the mandatory runbook stays contiguous and Troubleshooting keeps referring to the install. - Provider Directory gains "MPF feed for Medicare Plan Finder": the published artifacts, the six resource types, scope, crawl cadence, path choice. - run-payerbox/mpf-pipeline.md is gone; the old URL redirects to the anchor. Corrections verified against backend/src/mpf in HealthSamurai/smartbox: - Export scope is admin-editable (Settings -> MPF, GET/PUT /admin/mpf/settings, stored as a DocumentReference). The old text said scope ids were fixed in the image and changing them needed a release. - The feed carries six resource types (pipeline.ts PHASE1_RESOURCE_TYPES); HealthcareService and Endpoint stay REST-only. The composition was documented nowhere. - MPF_STORAGE_ACCOUNT_ID is the id of an Aidbox account resource, required on AWS and Azure (config.ts). The old text called it the Azure storage account name. - MPF_PUBLIC_BASE_URL and MPF_FULL_URL_BASE are not required: both fall back to built-in defaults, which is the more dangerous failure. - Added MPF_EXPORT_STORAGE_PROVIDER/_BUCKET/_ACCOUNT_ID (Aidbox v2605+), MPF_DEFAULT_CONTRACT, MPF_DEFAULT_YEAR. - API reference gains the trigger error table (400/403/404/409/500), the settings endpoint, HEAD, Cache-Control, and path validation rules. The MPF -> compliance/cms-9115 cross-link is dropped: CMS-9115-F covers the Plan-Net REST API, not the Plan Finder feed, and no regulatory anchor for MPF has been verified. The redirect target omits the .md extension on purpose. normalizeRedirectUri in the site repo strips .md with an end-anchored regex, so "page.md#anchor" would resolve to the raw-markdown endpoint. Refs #341 Co-Authored-By: Claude Opus 5 --- SUMMARY.md | 1 - .../operations/mpf-pipeline-api.md | 64 ++++- docs/interop-apis/provider-directory.md | 34 +++ docs/releases.md | 2 +- docs/run-payerbox/deploy.md | 223 ++++++++++++++++++ .../provider-directory-pipeline.md | 200 ---------------- redirects.yaml | 1 + 7 files changed, 317 insertions(+), 208 deletions(-) delete mode 100644 docs/run-payerbox/provider-directory-pipeline.md diff --git a/SUMMARY.md b/SUMMARY.md index 3382c23..97baddd 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -9,7 +9,6 @@ * [Run Payerbox](run-payerbox/README.md) * [Architecture](run-payerbox/architecture.md) * [Deploy](run-payerbox/deploy.md) - * [MPF Pipeline](run-payerbox/provider-directory-pipeline.md) * [Maintain](run-payerbox/maintain/README.md) * [Observability](run-payerbox/maintain/observability.md) * [Upgrade](run-payerbox/maintain/upgrade.md) diff --git a/docs/api-reference/operations/mpf-pipeline-api.md b/docs/api-reference/operations/mpf-pipeline-api.md index d721249..9b94788 100644 --- a/docs/api-reference/operations/mpf-pipeline-api.md +++ b/docs/api-reference/operations/mpf-pipeline-api.md @@ -1,11 +1,20 @@ --- description: >- - MPF pipeline endpoint reference: the sync and refresh triggers and the - public directory endpoint. + MPF pipeline endpoint reference: the sync and refresh triggers, the export + scope settings, and the public directory endpoint. --- # MPF Endpoints +The endpoints of the MPF provider-directory module. They mount only when the portal runs with `MPF_ENABLED=true`; otherwise every path below answers `404`. What the module publishes: [Provider Directory](../../interop-apis/provider-directory.md#mpf-feed-for-medicare-plan-finder). Setup: [Deploy](../../run-payerbox/deploy.md#mpf-provider-directory-pipeline). + +| Endpoint | Purpose | +|---|---| +| `POST /admin/mpf/sync` | Full run: export, filter, bundle, publish | +| `POST /admin/mpf/refresh` | Re-bundle a previous export without re-exporting | +| `GET` / `PUT /admin/mpf/settings` | Read and write the export scope | +| `GET` / `HEAD /mpf-provider-directory/{contract}/{year}/{file}` | Public, the URL CMS crawls | + ## Auth The trigger endpoints take a Bearer token of a `client_credentials` client listed in `MPF_TRIGGER_CLIENT_IDS`. Mint the token from Aidbox: @@ -99,15 +108,58 @@ Content-Type: application/json {% endtab %} {% endtabs %} -## GET /mpf-provider-directory/{contract}/{year}/{file} +## Trigger errors + +Both triggers share these. + +| Status | Meaning | +|---|---| +| `400` | `folder` missing on `/refresh`, a `contract` that is not `H` plus digits, or a `year` outside 2024–2099. | +| `403` | The token's client is not in `MPF_TRIGGER_CLIENT_IDS`. | +| `404` | `/refresh` only: the folder holds no files for the exported resource types. | +| `409` | A run is already in flight. Runs never overlap; retry after the current one finishes. | +| `500` | `MPF_EXPORT_CLIENT_ID` / `MPF_EXPORT_CLIENT_SECRET` unset, or the export scope could not be read. A failing scope read aborts rather than publishing the wrong scope. | + +## GET / PUT /admin/mpf/settings + +The export scope: which `InsurancePlan` ids and which network `Organization` ids the pipeline keeps. Stored on the admin Aidbox as a `DocumentReference`, so it survives restarts and upgrades, and both triggers resolve it at the start of every run. The Admin Portal edits the same values under **Settings → MPF**. + +Same auth as the triggers. + +{% tabs %} +{% tab title="GET response" %} +```json +{ + "planIds": ["snp-plan", "map-plan"], + "networkIds": ["network-a", "network-b"], + "source": "settings" +} +``` +{% endtab %} +{% tab title="PUT request" %} +```json +{ + "planIds": ["snp-plan", "map-plan"], + "networkIds": ["network-a", "network-b"] +} +``` +{% endtab %} +{% endtabs %} + +`source` reports where the current values come from: `settings` once saved, `defaults` while the deployment still runs on the ids compiled into the image. `PUT` takes non-empty arrays of FHIR ids for both fields, answers `400` otherwise, and writes an `AuditEvent` on success. + +## GET / HEAD /mpf-provider-directory/{contract}/{year}/{file} -Public, no auth: the endpoint CMS crawls. Proxies the storage bucket (which can stay private) and supports conditional GET, so repeat crawls only download files that changed. `index.json` lists the bundle URLs. The resources inside the bundles conform to the Plan-Net profiles. +Public, no auth: the endpoint CMS crawls. Proxies the storage bucket (which can stay private) and terminates conditional requests, so repeat crawls only download files that changed. `HEAD` returns the same headers without a body. Responses carry `Cache-Control: public, max-age=0, must-revalidate` plus the store's `ETag` and `Last-Modified`. `index.json` lists the bundle URLs. ```http -GET /mpf-provider-directory/H1234/2026/index.json -GET /mpf-provider-directory/H1234/2026/PractitionerRole-001.json +GET /mpf-provider-directory/H1234/2026/index.json +HEAD /mpf-provider-directory/H1234/2026/index.json +GET /mpf-provider-directory/H1234/2026/PractitionerRole-001.json ``` +Path segments are validated before anything is fetched, and anything else answers `404`: `contract` is `H` followed by digits, `year` is a four-digit year in the 2000s, and `file` is either `index.json` or `-.json`. + | Status | Meaning | |---|---| | `200` | File served. | diff --git a/docs/interop-apis/provider-directory.md b/docs/interop-apis/provider-directory.md index d36ffd1..82500e7 100644 --- a/docs/interop-apis/provider-directory.md +++ b/docs/interop-apis/provider-directory.md @@ -12,6 +12,7 @@ The [Da Vinci PDex Plan-Net IG](https://hl7.org/fhir/us/davinci-pdex-plan-net/) - Public unauthenticated `GET` on the Plan-Net directory resource types (`Practitioner`, `PractitionerRole`, `Organization`, `Location`, `HealthcareService`); everything else stays authenticated. - `Location.near` geographic search. - FHIR Bulk Data `$export` for periodic directory snapshots. +- [MPF feed](#mpf-feed-for-medicare-plan-finder) for the CMS Medicare Plan Finder crawler, published daily as static bundles per Medicare Advantage contract and year (optional module). ## Caller and auth @@ -120,3 +121,36 @@ GET /fhir/PractitionerRole ## Bulk download CMS recommends — but does not mandate — periodic full-directory downloads alongside REST search. Payerbox supports FHIR Bulk Data system-level `$export` with `_type=Practitioner,PractitionerRole,Organization,Location,HealthcareService` returning NDJSON, suitable for nightly snapshots a CDN or third party can mirror. + +## MPF feed for Medicare Plan Finder + +The CMS Medicare Plan Finder does not call the REST API above. It crawls a static feed: a manifest at a fixed URL, plus the FHIR `Bundle` files that manifest points to. Payerbox builds that feed from the same directory data and republishes it on a daily schedule, as an optional module of the [FHIR App Portal](../fhir-app-portal/README.md). Operators set it up in [Deploy](../run-payerbox/deploy.md#mpf-provider-directory-pipeline). + +### What gets published + +| Artifact | Shape | +|---|---| +| Manifest | `index.json`, a single object: `{"provider_urls": ["", …]}`. A file the manifest does not list is invisible to CMS. | +| Bundle files | `-001.json`, `-002.json`, and so on. Each is a `Bundle` with `type: collection`, and every `entry` carries a `fullUrl` plus the resource. | +| Path | `/{contract}/{year}/{file}`. Files roll over at 1000 entries or 250 MiB, whichever comes first. | + +Six resource types reach the feed, and the manifest lists them in dependency order: `InsurancePlan`, `Organization`, `Practitioner`, `PractitionerRole`, `Location`, `OrganizationAffiliation`. `HealthcareService` and `Endpoint` stay REST-only, so a consumer that needs services or electronic endpoints has to query the API rather than read the feed. + +Each `Bundle` carries the run's generation timestamp in `meta.lastUpdated`, while every resource inside keeps the `meta.lastUpdated` it has in the FHIR engine. CMS tracks change per resource, so preserving the resource-level value is what makes an unchanged provider read as unchanged. + +### Scope + +The feed is not the whole directory. It carries the plans and networks the deployment declares in scope, and everything the graph pulls in with them: the affiliations and practitioner roles attached to those networks, the facility organizations those affiliations point at, and the practitioners and locations those roles reach. Everything else stays out. The in-scope plan and network ids are set per deployment in the Admin Portal, not baked into the image. + +### Cadence and the year segment + +CMS registers one URL per contract and calendar year, and crawls it daily with conditional requests, so an unchanged directory costs one round trip per file. Publish one run per day per contract. Around open enrollment, when both the current and next plan year are live, run a second sync with the next year to publish both concurrently. + +### Choosing a path + +| Path | When | +|---|---| +| Prebuilt pipeline | The published feed matches the six resource types and the scope model above. Configuration is buckets, credentials, and the in-scope ids. | +| Custom export flow | Different resource types, a different scope model, or post-processing between the export and the publish. Reuses the same `$export`, client, policy, and storage setup; a runnable example lives in the [Aidbox examples repository](https://github.com/Aidbox/examples). | + +Endpoint contracts, status codes, and the settings API: [MPF Endpoints](../api-reference/operations/mpf-pipeline-api.md). diff --git a/docs/releases.md b/docs/releases.md index 1f0958e..6a361c9 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -19,7 +19,7 @@ A new `payerbox` umbrella Helm chart deploys the full stack (portals, Interop AP **Provider Directory** -- The CMS Medicare Plan Finder (MPF) provider-directory pipeline now runs its scope filters inside the `$export` query and gzip-compresses the export output. A runnable reference implementation is public in the [Aidbox examples](https://github.com/Aidbox/examples/tree/main/aidbox-features/medicare-plan-finder). See the [MPF Pipeline](run-payerbox/provider-directory-pipeline.md). +- The CMS Medicare Plan Finder (MPF) provider-directory pipeline now runs its scope filters inside the `$export` query and gzip-compresses the export output. A runnable reference implementation is public in the [Aidbox examples](https://github.com/Aidbox/examples/tree/main/aidbox-features/medicare-plan-finder). See [Deploy](run-payerbox/deploy.md#mpf-provider-directory-pipeline). ### Prior Auth (ePA) APIs [`2606`](https://hub.docker.com/r/healthsamurai/prior-auth) diff --git a/docs/run-payerbox/deploy.md b/docs/run-payerbox/deploy.md index fc9b3a8..08b955e 100644 --- a/docs/run-payerbox/deploy.md +++ b/docs/run-payerbox/deploy.md @@ -300,6 +300,229 @@ helm upgrade payerbox healthsamurai/payerbox -n payerbox --values values.yaml helm uninstall payerbox -n payerbox ``` +## MPF provider-directory pipeline + +Optional, and only for Medicare Advantage deployments that publish to the CMS Medicare Plan Finder. The pipeline ships inside the FHIR App Portal image and stays off unless `MPF_ENABLED=true`. It builds a scoped provider directory out of the FHIR engine and publishes it as static files the CMS crawler reads. For what it publishes, see [Provider Directory](../interop-apis/provider-directory.md#mpf-feed-for-medicare-plan-finder); for endpoint contracts, [MPF Endpoints](../api-reference/operations/mpf-pipeline-api.md). + +```mermaid +graph LR + T(scheduler):::yellow2 --> S(portal
sync endpoint):::violet2 + S --> A(FHIR engine
/$export):::blue2 + A --> SRC(source bucket):::green2 + SRC --> B(portal
scope filter):::violet2 + B --> C(bundles + index.json):::violet2 + C --> PUB(storage bucket):::green2 + PUB --> E(portal
public endpoint):::violet2 + E --> F(CMS crawler):::yellow2 +``` + +All bucket access goes through Aidbox-signed URLs, so neither bucket needs to be public. The pipeline is triggered over HTTP, typically by a daily Kubernetes CronJob. At production scale a run takes 30 to 40 minutes. + +Beyond the install above, the module needs: + +- Two buckets: a **source bucket** for `$export` output and a **storage bucket** for the published files. +- Aidbox connected to both. All bucket access goes through it: GCP and Azure use workload identity, AWS an `AwsAccount` resource. [File storage](https://www.health-samurai.io/docs/aidbox/file-storage) in the Aidbox docs covers the setups and IAM roles, and [the signing probe](#verify-bucket-signing) below proves them. +- Headroom on the portal pod. The module is a spiky daily job: on a directory of roughly a million source resources a run peaks near 2 GB of memory and stages about 11 GB in `MPF_OUTPUT_DIR` before upload. Short of either, the run is OOM-killed or the pod evicted, and a killed run restarts from scratch rather than resuming. + +{% hint style="warning" %} +Every run adds a new folder to the source bucket. Set a lifecycle rule to expire old ones, keeping a few days for `folder` re-bundling. The storage bucket holds only the latest set. +{% endhint %} + +{% stepper %} +{% step %} + +### Create the sync client + +The pipeline authenticates to Aidbox as its own client. `PUT` this (and the next step's policy) with admin credentials. The secret reappears in [Configure the environment](#configure-the-environment). + +{% code title="PUT /Client/mpf-sync" %} +```json +{ + "resourceType": "Client", + "id": "mpf-sync", + "secret": "", + "grant_types": ["client_credentials"], + "auth": { "client_credentials": { "access_token_expiration": 3600 } } +} +``` +{% endcode %} + +{% endstep %} +{% step %} + +### Create the access policy + +Least privilege: only the calls the portal makes. + +{% code title="PUT /AccessPolicy/mpf-sync-policy" %} +```json +{ + "resourceType": "AccessPolicy", + "id": "mpf-sync-policy", + "engine": "matcho", + "matcho": { + "$one-of": [ + { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/fhir/\\$export(\\?|$)" }, + { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/fhir/\\$export-status/" }, + { "client": { "id": "mpf-sync" }, "request-method": "put", "uri": "#^/Notification/" }, + { "client": { "id": "mpf-sync" }, "request-method": "post", "uri": "#^/Notification/[^/]+/\\$send$" }, + { "client": { "id": "mpf-sync" }, "request-method": "post", "uri": "#^/gcp/workload-identity/storage/" }, + { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/gcp/workload-identity/storage/" }, + { "client": { "id": "mpf-sync" }, "request-method": "delete", "uri": "#^/gcp/workload-identity/storage/" }, + { "client": { "id": "mpf-sync" }, "request-method": "post", "uri": "#^/aws/storage/" }, + { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/aws/storage/" }, + { "client": { "id": "mpf-sync" }, "request-method": "delete", "uri": "#^/aws/storage/" }, + { "client": { "id": "mpf-sync" }, "request-method": "post", "uri": "#^/azure/workload-identity/storage/" }, + { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/azure/workload-identity/storage/" }, + { "client": { "id": "mpf-sync" }, "request-method": "delete", "uri": "#^/azure/workload-identity/storage/" } + ] + } +} +``` +{% endcode %} + +{% hint style="warning" %} +The matcho engine has no `$or` operator. Use `$one-of` at the root with `client` inlined into each alternative, as above. A policy written with `$or` never grants, and every request answers `403`. +{% endhint %} + +{% endstep %} +{% step %} + +### Configure the environment + +On **Aidbox**, point `$export` at the source bucket: + +| Variable | Description | +|---|---| +| `BOX_FHIR_BULK_STORAGE_PROVIDER` (required) | `gcp`, `aws`, or `azure`. Lets `$export` write to object storage. | +| `BOX_FHIR_BULK_STORAGE_GCP_BUCKET` (required) | The source bucket (setting name is provider-specific, GCP shown). | + +On the **portal**: + +| Variable | Description | +|---|---| +| `MPF_ENABLED` (required) | `true` to turn the module on. The endpoints mount only when it is set, and the Admin Portal's MPF tab appears only when they do. | +| `MPF_EXPORT_CLIENT_ID`, `MPF_EXPORT_CLIENT_SECRET` (required) | `mpf-sync` and the secret from [Create the sync client](#create-the-sync-client). Without both, `/sync` answers `500`. | +| `MPF_STORAGE_PROVIDER` (required) | `gcp`, `aws`, or `azure`. Same provider as Aidbox's bulk storage. | +| `MPF_STORAGE_BUCKET` (required) | The bucket the bundles and `index.json` are published to. | +| `MPF_PUBLIC_BASE_URL` (set it) | Prefix for the bundle links in `index.json`: the portal's public endpoint (`https:///mpf-provider-directory`) or a public bucket. It falls back to a built-in default, so leaving it unset publishes links pointing at someone else's base URL. | +| `MPF_FULL_URL_BASE` (set it) | FHIR base URL for bundle entries' `fullUrl`, e.g. `https://fhir./fhir`. Same built-in-default caveat. | +| `MPF_TRIGGER_CLIENT_IDS` | Clients allowed to trigger runs. Defaults to `admin-api`, which lacks `mpf-sync`. Set `admin-api,mpf-sync`. | +| `MPF_STORAGE_ACCOUNT_ID` | Id of the Aidbox account resource that signs the storage URLs (`AwsAccount`, `AzureAccount`). Required on AWS and Azure. On GCP the signing goes through workload identity and this can stay unset. | +| `MPF_DEFAULT_CONTRACT` | Contract used when a request body omits it. | +| `MPF_DEFAULT_YEAR` | Year used when a request body omits it. Defaults to the current UTC year. | +| `MPF_ALERT_EMAIL_TO` | Failure-alert recipients via Aidbox `Notification` (needs its email provider configured). Unset: log-only. | +| `MPF_BUCKET_PREFIX` | Source bucket root URL. Only `folder` refresh uses it, and only when the export-storage override below is unset. | +| `MPF_BUNDLE_SIZE` | Max entries per bundle. Default `1000`. | +| `MPF_MAX_BUNDLE_BYTES` | Max bytes per bundle before rolling to a new file. Default 250 MiB, which keeps files under the 300 MB CMS recommends. | +| `MPF_OUTPUT_DIR` | Local directory where bundles are staged. Default `./mpf-output`. | + +Optionally, send the raw `$export` output to a bucket other than Aidbox's `BOX_FHIR_BULK_STORAGE_*` default. This needs Aidbox v2605 or newer; older versions ignore it. + +| Variable | Description | +|---|---| +| `MPF_EXPORT_STORAGE_PROVIDER` | `gcp`, `aws`, or `azure`. Activates the override together with the bucket below. | +| `MPF_EXPORT_STORAGE_BUCKET` | Bucket the NDJSON is written to. Also takes precedence over `MPF_BUCKET_PREFIX` for `folder` refresh. | +| `MPF_EXPORT_STORAGE_ACCOUNT_ID` | Id of the Aidbox account resource holding the storage credentials. Required on AWS and Azure, omit on GCP. | + +{% endstep %} +{% step %} + +### Verify bucket signing + +Prove the signing chain with one object before running the pipeline: + +{% code title="Signing probe" %} +```bash +# get a token +TOKEN=$(curl -s -X POST https:///auth/token \ + -H 'Content-Type: application/json' \ + -d '{"grant_type":"client_credentials","client_id":"mpf-sync","client_secret":""}' \ + | jq -r .access_token) + +# get a presigned upload URL +URL=$(curl -s -X POST https:///gcp/workload-identity/storage/ \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ + -d '{"filename":"_probe.json"}' | jq -r .url) + +# put data through it +curl -i -X PUT "$URL" \ + -H 'Content-Type: application/json' -d '{"probe":true}' +``` +{% endcode %} + +On AWS or Azure, the endpoint prefix is `/aws/storage//` or `/azure/workload-identity/storage//`. + +{% endstep %} +{% step %} + +### Set the export scope + +The scope decides which plans and networks reach the published feed. Set it before the first production run: the image ships defaults that belong to another deployment, and a run against the wrong ids publishes the wrong directory. + +Edit it in the Admin Portal under **Settings → MPF**, or over the API: + +{% code title="PUT /admin/mpf/settings" %} +```json +{ + "planIds": ["", "..."], + "networkIds": ["", "..."] +} +``` +{% endcode %} + +Values persist as a `DocumentReference` on the admin Aidbox, so they survive portal restarts and upgrades. Both `/sync` and `/refresh` resolve the scope at the start of every run: saved values win, an empty or missing resource falls back to the image defaults, and a failing read aborts the run instead of publishing the wrong scope. + +Resource types and profile filters are not configurable. Changing them is a portal release, so coordinate with Health Samurai, or use the [custom export flow](#custom-export-flow). + +{% endstep %} +{% step %} + +### Run and verify + +Trigger a sync as `mpf-sync` (listed in `MPF_TRIGGER_CLIENT_IDS`): + +{% code title="First run" %} +```bash +TOKEN=$(curl -s -X POST https:///auth/token \ + -H 'Content-Type: application/json' \ + -d '{"grant_type":"client_credentials","client_id":"mpf-sync","client_secret":""}' \ + | jq -r .access_token) + +curl -X POST https:///admin/mpf/sync \ + -H "Authorization: Bearer $TOKEN" \ + -H 'Content-Type: application/json' \ + -d '{"contract":"H1234"}' +``` +{% endcode %} + +The endpoint is asynchronous and the pipeline runs in the background. Verify, in order: + +1. Pod logs: `[mpf:sync] export kicked off`, later `export completed`. +2. Source bucket: a new folder of NDJSON files. +3. Logs: the resolved scope, `[mpf] export scope: source=settings planIds=… networkIds=…`. `source=defaults` means [Set the export scope](#set-the-export-scope) did not take. +4. Logs: `publishing via signed URLs`. A `403` here means the policy is missing the signing branches from [Create the access policy](#create-the-access-policy). +5. Logs: `run completed` with `uploaded=true`. The storage bucket holds bundles and `index.json`. +6. The public endpoint (the URL the crawler will go to) works: `GET https:///mpf-provider-directory/H1234/2026/index.json`. + +A run that fails on an empty scope stopped deliberately: the filter kept zero `PractitionerRole` resources, which means the ids do not match the data. Nothing is published in that case, so the previous generation stays live. + +{% endstep %} +{% step %} + +### Schedule the daily run + +CMS crawls the registered URL daily, so the feed needs one run per day per contract. A thin CronJob that mints a token and posts to `/sync` is enough; the run itself happens inside the portal pod. + +Place the schedule after the upstream data load and before the crawl. Publishing wipes the target prefix and uploads `index.json` last, so a crawler reading mid-publish gets a `404` rather than a mix of two generations. On a schedule that window is harmless, which is also why a manual run during the crawl window is not. + +{% endstep %} +{% endstepper %} + +### Custom export flow + +The prebuilt pipeline covers the whole path out of the box. A custom flow (different scope, resource types, or post-processing) can reuse the same `$export`, client, policy, and storage setup. A runnable example lives in the [Aidbox examples repository](https://github.com/Aidbox/examples). + ## Next steps - [Architecture](architecture.md) — component topology, network, and the authentication chain diff --git a/docs/run-payerbox/provider-directory-pipeline.md b/docs/run-payerbox/provider-directory-pipeline.md deleted file mode 100644 index c398e9d..0000000 --- a/docs/run-payerbox/provider-directory-pipeline.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -description: >- - Set up and run the Payerbox pipeline that builds a scoped CMS Plan-Net - provider directory from the FHIR engine and publishes it to object storage. ---- - -# MPF Pipeline - -An optional module of the [FHIR App Portal](../fhir-app-portal/README.md), built into its [image](https://hub.docker.com/r/healthsamurai/fhir-app-portal) and enabled with `MPF_ENABLED=true`. It builds a CMS Plan-Net provider directory and publishes it as static FHIR `Bundle` files for the CMS [Medicare Plan Finder](https://www.medicare.gov/plan-compare/) (MPF) crawler. - -Data flow: - -```mermaid -graph LR - T(scheduler):::yellow2 --> S(portal
sync endpoint):::violet2 - S --> A(FHIR engine
/$export):::blue2 - A --> SRC(source bucket):::green2 - SRC --> B(portal
scope filter):::violet2 - B --> C(bundles + index.json):::violet2 - C --> PUB(storage bucket):::green2 - PUB --> E(portal
public endpoint):::violet2 - E --> F(CMS crawler):::yellow2 -``` - -All bucket access goes through Aidbox-signed URLs, so neither bucket needs to be public. The pipeline is triggered over HTTP, typically by a daily Kubernetes CronJob. A production-scale run takes upwards of half an hour. Endpoint details live in the [API reference](../api-reference/operations/mpf-pipeline-api.md). - -## Prerequisites - -- Aidbox access (Payerbox's FHIR engine). -- Two buckets: a **source bucket** for `$export` output and a **storage bucket** for the final files. -- Aidbox connected to both buckets. All bucket access goes through it. GCP and Azure use workload identity, AWS an `AwsAccount` resource. [File storage](https://www.health-samurai.io/docs/aidbox/file-storage) in the Aidbox docs covers the setups and IAM roles. [Step 4](#verify-bucket-signing) verifies the setup. - -{% hint style="warning" %} -Every run adds a new folder to the source bucket. Set a lifecycle rule to expire old ones, keeping a few days for `folder` re-bundling. The storage bucket holds only the latest set. -{% endhint %} - -## Set up - -{% stepper %} -{% step %} - -### Create the sync client - -The pipeline authenticates to Aidbox as its own client. `PUT` this (and [step 2](#create-the-access-policy)'s policy) with admin credentials. The secret reappears in [step 3](#configure-the-environment). - -{% code title="PUT /Client/mpf-sync" %} -```json -{ - "resourceType": "Client", - "id": "mpf-sync", - "secret": "", - "grant_types": ["client_credentials"], - "auth": { "client_credentials": { "access_token_expiration": 3600 } } -} -``` -{% endcode %} - -{% endstep %} -{% step %} - -### Create the access policy - -Least privilege: only the calls the portal makes. - -{% code title="PUT /AccessPolicy/mpf-sync-policy" %} -```json -{ - "resourceType": "AccessPolicy", - "id": "mpf-sync-policy", - "engine": "matcho", - "matcho": { - "$one-of": [ - { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/fhir/\\$export(\\?|$)" }, - { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/fhir/\\$export-status/" }, - { "client": { "id": "mpf-sync" }, "request-method": "put", "uri": "#^/Notification/" }, - { "client": { "id": "mpf-sync" }, "request-method": "post", "uri": "#^/Notification/[^/]+/\\$send$" }, - { "client": { "id": "mpf-sync" }, "request-method": "post", "uri": "#^/gcp/workload-identity/storage/" }, - { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/gcp/workload-identity/storage/" }, - { "client": { "id": "mpf-sync" }, "request-method": "delete", "uri": "#^/gcp/workload-identity/storage/" }, - { "client": { "id": "mpf-sync" }, "request-method": "post", "uri": "#^/aws/storage/" }, - { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/aws/storage/" }, - { "client": { "id": "mpf-sync" }, "request-method": "delete", "uri": "#^/aws/storage/" }, - { "client": { "id": "mpf-sync" }, "request-method": "post", "uri": "#^/azure/workload-identity/storage/" }, - { "client": { "id": "mpf-sync" }, "request-method": "get", "uri": "#^/azure/workload-identity/storage/" }, - { "client": { "id": "mpf-sync" }, "request-method": "delete", "uri": "#^/azure/workload-identity/storage/" } - ] - } -} -``` -{% endcode %} - -{% endstep %} -{% step %} - -### Configure the environment - -On **Aidbox**, point `$export` at the source bucket: - -| Variable | Description | -|---|---| -| `BOX_FHIR_BULK_STORAGE_PROVIDER` (required) | `gcp`, `aws`, or `azure`. Lets `$export` write to object storage. | -| `BOX_FHIR_BULK_STORAGE_GCP_BUCKET` (required) | The source bucket (setting name is provider-specific, GCP shown). | - -On the **portal**: - -| Variable | Description | -|---|---| -| `MPF_ENABLED` (required) | `true` to turn the module on. | -| `MPF_EXPORT_CLIENT_ID`, `MPF_EXPORT_CLIENT_SECRET` (required) | `mpf-sync` and the secret from [step 1](#create-the-sync-client). | -| `MPF_STORAGE_PROVIDER` (required) | Same provider as Aidbox's bulk storage. | -| `MPF_STORAGE_BUCKET` (required) | The bucket the bundles and `index.json` are published to. | -| `MPF_PUBLIC_BASE_URL` (required) | Prefix for the bundle links in `index.json`: the portal's public endpoint (`https:///mpf-provider-directory`) or a public bucket. | -| `MPF_FULL_URL_BASE` (required) | FHIR base URL for bundle entries' `fullUrl`, e.g. `https://fhir./fhir`. | -| `MPF_TRIGGER_CLIENT_IDS` (required) | Clients allowed to trigger runs. Set `admin-api,mpf-sync` (the default lacks `mpf-sync`). | -| `MPF_BUCKET_PREFIX` | Source bucket root URL. Only `folder` refresh uses it. | -| `MPF_ALERT_EMAIL_TO` | Failure-alert recipients via Aidbox `Notification` (needs its email provider configured). Unset: log-only. | -| `MPF_STORAGE_ACCOUNT_ID` | On AWS: the `AwsAccount` resource id. On Azure: the storage account name. Not used on GCP. | -| `MPF_BUNDLE_SIZE` | Max entries per bundle. Default `1000`. | -| `MPF_MAX_BUNDLE_BYTES` | Max bytes per bundle before rolling to a new file. Default 250 MB. | -| `MPF_OUTPUT_DIR` | Local directory where bundles are staged. Default `./mpf-output`. | - -Resource types, profile filters, scope IDs, and the default contract are fixed in the portal image. Changing them is a portal release (coordinate with Health Samurai), or use the [Custom export flow](#custom-export-flow). - -{% endstep %} -{% step %} - -### Verify bucket signing - -Prove the signing chain from [Prerequisites](#prerequisites) with one object before running the pipeline: - -{% code title="Signing probe" %} -```bash -# get a token -TOKEN=$(curl -s -X POST https:///auth/token \ - -H 'Content-Type: application/json' \ - -d '{"grant_type":"client_credentials","client_id":"mpf-sync","client_secret":""}' \ - | jq -r .access_token) - -# get a presigned upload URL -URL=$(curl -s -X POST https:///gcp/workload-identity/storage/ \ - -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ - -d '{"filename":"_probe.json"}' | jq -r .url) - -# put data through it -curl -i -X PUT "$URL" \ - -H 'Content-Type: application/json' -d '{"probe":true}' -``` -{% endcode %} - -On AWS or Azure, the endpoint prefix is `/aws/storage//` or `/azure/workload-identity/storage//`. - -{% endstep %} -{% step %} - -### Run and verify - -Trigger a sync as `mpf-sync` (listed in `MPF_TRIGGER_CLIENT_IDS`, [step 3](#configure-the-environment)): - -{% code title="First run" %} -```bash -TOKEN=$(curl -s -X POST https:///auth/token \ - -H 'Content-Type: application/json' \ - -d '{"grant_type":"client_credentials","client_id":"mpf-sync","client_secret":""}' \ - | jq -r .access_token) - -curl -X POST https:///admin/mpf/sync \ - -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' \ - -d '{"contract":"H1234"}' -``` -{% endcode %} - -The endpoint is asynchronous and the pipeline runs in the background. Verify, in order: - -1. Pod logs: `[mpf:sync] export kicked off`, later `export completed`. -2. Source bucket: a new folder of NDJSON files. -3. Logs: `publishing via signed URLs`. A `403` here means the policy is missing the signing branches from [step 2](#create-the-access-policy). -4. Logs: `run completed` with `uploaded=true`. The storage bucket holds bundles and `index.json`. -5. The public endpoint (the URL the crawler will go to) works: `GET https:///mpf-provider-directory/H1234/2026/index.json`. - -{% endstep %} -{% endstepper %} - -## Custom export flow - -The prebuilt pipeline covers the whole path out of the box. A custom flow (different scope, resource types, or post-processing) can reuse the same `$export`, client, policy, and storage setup. A runnable example lives in the [Aidbox examples repository](https://github.com/Aidbox/examples). - -## Related - -{% content-ref url="../api-reference/operations/mpf-pipeline-api.md" %} -[mpf-pipeline-api.md](../api-reference/operations/mpf-pipeline-api.md) -{% endcontent-ref %} - -{% content-ref url="../interop-apis/provider-directory.md" %} -[provider-directory.md](../interop-apis/provider-directory.md) -{% endcontent-ref %} - -{% content-ref url="../compliance/cms-9115.md" %} -[cms-9115.md](../compliance/cms-9115.md) -{% endcontent-ref %} diff --git a/redirects.yaml b/redirects.yaml index b4ea465..a4eed98 100644 --- a/redirects.yaml +++ b/redirects.yaml @@ -1,4 +1,5 @@ redirects: + run-payerbox/provider-directory-pipeline: run-payerbox/deploy#mpf-provider-directory-pipeline run-payerbox/maintain/event-notifications.md: prior-auth/event-notifications.md fhir-app-portal/developer-portal#register-a-smart-app: fhir-app-portal/smart-app.md#register-a-smart-app fhir-app-portal/developer-portal#test-your-app: fhir-app-portal/smart-app.md#test-your-app