Skip to content

Fix MS FHIR setup and stop the benchmark reporting failures as throughput - #31

Open
octoshikari wants to merge 1 commit into
mainfrom
msfhir-parity
Open

Fix MS FHIR setup and stop the benchmark reporting failures as throughput#31
octoshikari wants to merge 1 commit into
mainfrom
msfhir-parity

Conversation

@octoshikari

@octoshikari octoshikari commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

MS FHIR's import numbers were far too fast for a server doing real work. Three
separate problems, all reproduced against a local stack.

MS FHIR answers 500 to every request

SecurityProvider's constructor asserts Security.Authentication.Authority is
non-null before it checks whether security is enabled, and it is an
IProvideCapability, so it gets constructed while building the
CapabilityStatement — which ValidateFormatParametersAttribute triggers on
every request. With auth off and no Authority, everything returns 500.

The assert has been there since 2021 but only started firing recently:

image built /metadata without Authority
4.0.815 2026-07-11 200
5.0.0 2026-07-15 500
latest 2026-07-31 500

Fixed by setting Authority to a non-null value. Auth stays off — BuildAsync
still gates on Enabled, the capability statement has no security block, and
unauthenticated writes return 201.

Separately: the compose file pins :latest with pull_policy: always, so the
MS version changes silently between runs. Those two builds are four days apart
and one of them does not work at all. Worth pinning.

The tests scored failures as throughput

A server that rejects everything rejects it quickly. Same image, same script,
the only difference being whether the server works:

working 500 on everything
http_reqs 120/s 3399/s
iterations 3.35/s 850/s
checks_succeeded 100% 0%

Nothing failed the run — checks went red, throughput went up, and only the
check rate told them apart. Now:

  • thresholds on checks and http_req_failed with abortOnFail, so a broken
    server ends its run instead of filling five minutes with noise
  • import.js aborts if a seed bundle is rejected; its result was never looked
    at before, so a run could import into a broken dataset. teardown counts
    what actually persisted, because HTTP 200 on a transaction is not evidence
    the rows are queryable
  • search.js probes every configured parameter before measuring. Each entry's
    values[0] cannot match anything, so a parameter that still returns the
    whole collection is being dropped under lenient handling — that aborts the
    run. A values[1] miss only warns, since that depends on what was imported.
    Verified: an unknown parameter returns all 5159 Observations and is caught;
    every parameter in searchConfig passes on MS.

The pipeline still finishes

k6 exiting non-zero would have been worse than the problem it fixes: runner.sh
runs set -e, so a failing server would have skipped every server after it in
the loop and every later suite — and search runs after import. Failures are now
collected per server, reported loudly at the end, and the exit code stays 0, so
the remaining servers and suites run and the report still gets published.

run_test_on_server also returns k6's status now. It ended with echo, so the
function always reported success and a failed run was indistinguishable from a
completed one.

MS was not validating writes

ProfileValidationOnCreate and OnUpdate ship as false, so MS only
parse-checked writes while aidbox and octofhir ran full validation. Both are on
now. Confirmed it is really profile validation and not the base attribute
validator — an obs-6 violation (value[x] and dataAbsentReason together) is
a 400 quoting the FHIRPath invariant, and a 201 again when
x-ms-profile-validation: false turns it off.

CRUD validates cleanly: its seeds declare no profiles, checks stay at 100%, and
the cost is about 15% throughput and 50% on p95 at 20 VUs.

Open decision: imports against MS fail

Deliberate, and needs a call. The import suite is the one place where
validation cannot currently work, and the cause is in the dataset.

705 of the ~1150 entries in every Synthea bundle declare a US Core profile:

us-core-procedure          165
us-core-observation-lab    114
us-core-medicationrequest   79
us-core-encounter           64
us-core-documentreference   64
... 21 distinct profiles

Servers disagree on what to do when that canonical cannot be resolved:

Patient with meta.profile: us-core-patient, IG not loaded
aidbox 201 — profile silently skipped, validates base spec only
aidbox, profile http://example.org/total-nonsense 201 — same
aidbox, base invariant pat-1 violated 422 — base validation does work
MS FHIR 400 Unable to resolve reference to profile ..., whole transaction fails

MS has no setting for this. ProfileValidator builds a stock
Hl7.Fhir.Validation.Validator, and the only code that touches missing-profile
issues just logs them:

if (issue.Details?.Text?.Contains("Unable to resolve reference to profile", ...) == true)
{
    _logger.LogDebug("Validation failure due to missing profile. {Details}", issue.Details.Text);
}

Measured with validation on and no profiles loaded: 0 of 20 import bundles
succeeded
, and the run looked six times faster than the working one — 2.07
iterations/s against 0.35 — because rejections return immediately. Exactly the
failure-as-throughput pattern the threshold work above now catches, so the
suite is marked untrustworthy instead of publishing that number, and the other
servers still get measured.

Options

  1. Load US Core into MS only. scripts/load-us-core.py is in the repo and
    verified: a full bundle returns 200 with all 1155 entries at 201.
    Deliberately not wired into bootstrap — it makes MS validate against an IG
    that aidbox and octofhir do not have, which moves the asymmetry rather than
    removing it.
  2. Load US Core into every server. Aidbox takes BOX_FHIR_PACKAGES;
    octofhir needs its own mechanism. Genuinely equal, and the most work. It will
    also surface real US Core violations in the Synthea data, so the import suite
    may need fixing separately.
  3. Turn MS validation off for imports and state in the UI which servers
    validate and which do not. Gives up on comparing writes.

Editing the bundles to drop meta.profile was ruled out — the dataset is
generated and stays as-is.

Also still open

  • MS has no terminology loaded, so binding validation does not run even
    with the flag on. Prefer: handling=strict is not sent either, so anything
    the validator reports as a warning is ignored.
  • _total. MS ships IncludeTotalInBundle: None and skips the COUNT that
    aidbox and HAPI pay on every search.
  • x-bundle-processing-logic: parallel is still sent only to microsoft.

scripts/parity-check.sh runs after a benchmark: persisted counts, reference
resolution, whether a parameter is applied, and whether a server rejects an
invalid resource. Its octofhir port is a guess, octofhir is not in this compose
file.

…hput

MS FHIR's import numbers were far too fast for a server doing real work. Three
separate problems, all reproduced against a local stack.

The server answers 500 to every request. SecurityProvider's constructor asserts
Security.Authentication.Authority is non-null before it checks whether security
is enabled, and it is an IProvideCapability, so it gets constructed while
building the CapabilityStatement — which ValidateFormatParametersAttribute
triggers on every request. The assert dates from 2021 but only started firing
between images 4.0.815 (2026-07-11, /metadata 200) and 5.0.0 (2026-07-15, 500).
Setting Authority to any non-null value fixes it; auth stays off because
BuildAsync still gates on Enabled, and unauthenticated writes return 201.

The tests could not tell that apart from a fast server. A server that rejects
everything rejects it quickly: the same crud.js against the same image measured
3399 req/s and 850 iterations/s while answering 500 to everything, against
120 req/s and 3.35 iterations/s working properly. Only the check rate differed,
and nothing failed the run. So: thresholds on checks and http_req_failed with
abortOnFail; import.js aborts when a seed bundle is rejected (its result was
never inspected) and counts what actually persisted in teardown, since HTTP 200
on a transaction is not evidence the rows are queryable; search.js probes every
configured parameter first, using the values[0] entries that cannot match
anything — a parameter still returning the whole collection is being dropped
under lenient handling, which now aborts the run. An unknown parameter returns
all 5159 Observations and is caught; every parameter in searchConfig passes.

Because k6 can now exit non-zero, runner.sh had to stop letting one server take
the run down with it: `set -e` plus a failing k6 would have skipped every server
after it and every later suite, and search runs after import. Failures are
collected per server, reported at the end, and the exit code stays 0 so the
pipeline still finishes and publishes a report. run_test_on_server also returns
k6's status now — the trailing echoes were masking it, so a failed run looked
like a completed one.

MS was not validating writes. ProfileValidationOnCreate/OnUpdate ship as false,
so it only parse-checked while aidbox and octofhir validated fully. Both are on
now: an obs-6 violation is a 400 quoting the FHIRPath invariant and a 201 again
with x-ms-profile-validation: false, so it is profile validation and not the
base attribute validator. The CRUD seeds declare no profiles, so that suite
validates cleanly at a cost of roughly 15% throughput and 50% on p95.

Imports against MS fail, and that is deliberate. 705 of the ~1150 entries in
every Synthea bundle declare a US Core profile; MS errors on a canonical it
cannot resolve where aidbox accepts the resource and validates the base spec
only — it accepts a made-up profile URL just as happily. There is no setting
for it: ProfileValidator builds a stock Hl7.Fhir.Validation.Validator and the
only code touching missing-profile issues logs them. scripts/load-us-core.py
loads the IG and makes a full bundle return 200 with all 1155 entries at 201,
but it is left out of bootstrap because loading US Core into MS alone makes it
validate against an IG nobody else does. That trade-off needs a decision.

scripts/parity-check.sh checks, after a run, what actually persisted, whether
references resolved, whether a search parameter is applied, and whether a
server rejects an invalid resource.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant