Fix MS FHIR setup and stop the benchmark reporting failures as throughput - #31
Open
octoshikari wants to merge 1 commit into
Open
Fix MS FHIR setup and stop the benchmark reporting failures as throughput#31octoshikari wants to merge 1 commit into
octoshikari wants to merge 1 commit into
Conversation
octoshikari
force-pushed
the
msfhir-parity
branch
from
August 6, 2026 12:16
b67164f to
4f42e60
Compare
…hput MS FHIR's import numbers were far too fast for a server doing real work. Three separate problems, all reproduced against a local stack. The server answers 500 to every request. SecurityProvider's constructor asserts Security.Authentication.Authority is non-null before it checks whether security is enabled, and it is an IProvideCapability, so it gets constructed while building the CapabilityStatement — which ValidateFormatParametersAttribute triggers on every request. The assert dates from 2021 but only started firing between images 4.0.815 (2026-07-11, /metadata 200) and 5.0.0 (2026-07-15, 500). Setting Authority to any non-null value fixes it; auth stays off because BuildAsync still gates on Enabled, and unauthenticated writes return 201. The tests could not tell that apart from a fast server. A server that rejects everything rejects it quickly: the same crud.js against the same image measured 3399 req/s and 850 iterations/s while answering 500 to everything, against 120 req/s and 3.35 iterations/s working properly. Only the check rate differed, and nothing failed the run. So: thresholds on checks and http_req_failed with abortOnFail; import.js aborts when a seed bundle is rejected (its result was never inspected) and counts what actually persisted in teardown, since HTTP 200 on a transaction is not evidence the rows are queryable; search.js probes every configured parameter first, using the values[0] entries that cannot match anything — a parameter still returning the whole collection is being dropped under lenient handling, which now aborts the run. An unknown parameter returns all 5159 Observations and is caught; every parameter in searchConfig passes. Because k6 can now exit non-zero, runner.sh had to stop letting one server take the run down with it: `set -e` plus a failing k6 would have skipped every server after it and every later suite, and search runs after import. Failures are collected per server, reported at the end, and the exit code stays 0 so the pipeline still finishes and publishes a report. run_test_on_server also returns k6's status now — the trailing echoes were masking it, so a failed run looked like a completed one. MS was not validating writes. ProfileValidationOnCreate/OnUpdate ship as false, so it only parse-checked while aidbox and octofhir validated fully. Both are on now: an obs-6 violation is a 400 quoting the FHIRPath invariant and a 201 again with x-ms-profile-validation: false, so it is profile validation and not the base attribute validator. The CRUD seeds declare no profiles, so that suite validates cleanly at a cost of roughly 15% throughput and 50% on p95. Imports against MS fail, and that is deliberate. 705 of the ~1150 entries in every Synthea bundle declare a US Core profile; MS errors on a canonical it cannot resolve where aidbox accepts the resource and validates the base spec only — it accepts a made-up profile URL just as happily. There is no setting for it: ProfileValidator builds a stock Hl7.Fhir.Validation.Validator and the only code touching missing-profile issues logs them. scripts/load-us-core.py loads the IG and makes a full bundle return 200 with all 1155 entries at 201, but it is left out of bootstrap because loading US Core into MS alone makes it validate against an IG nobody else does. That trade-off needs a decision. scripts/parity-check.sh checks, after a run, what actually persisted, whether references resolved, whether a search parameter is applied, and whether a server rejects an invalid resource.
octoshikari
force-pushed
the
msfhir-parity
branch
from
August 6, 2026 12:19
4f42e60 to
c2b918a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MS FHIR's import numbers were far too fast for a server doing real work. Three
separate problems, all reproduced against a local stack.
MS FHIR answers 500 to every request
SecurityProvider's constructor assertsSecurity.Authentication.Authorityisnon-null before it checks whether security is enabled, and it is an
IProvideCapability, so it gets constructed while building theCapabilityStatement — which
ValidateFormatParametersAttributetriggers onevery request. With auth off and no Authority, everything returns 500.
The assert has been there since 2021 but only started firing recently:
/metadatawithout AuthorityFixed by setting Authority to a non-null value. Auth stays off —
BuildAsyncstill gates on
Enabled, the capability statement has no security block, andunauthenticated writes return 201.
Separately: the compose file pins
:latestwithpull_policy: always, so theMS version changes silently between runs. Those two builds are four days apart
and one of them does not work at all. Worth pinning.
The tests scored failures as throughput
A server that rejects everything rejects it quickly. Same image, same script,
the only difference being whether the server works:
http_reqsiterationschecks_succeededNothing failed the run — checks went red, throughput went up, and only the
check rate told them apart. Now:
checksandhttp_req_failedwithabortOnFail, so a brokenserver ends its run instead of filling five minutes with noise
import.jsaborts if a seed bundle is rejected; its result was never lookedat before, so a run could import into a broken dataset.
teardowncountswhat actually persisted, because HTTP 200 on a transaction is not evidence
the rows are queryable
search.jsprobes every configured parameter before measuring. Each entry'svalues[0]cannot match anything, so a parameter that still returns thewhole collection is being dropped under lenient handling — that aborts the
run. A
values[1]miss only warns, since that depends on what was imported.Verified: an unknown parameter returns all 5159 Observations and is caught;
every parameter in
searchConfigpasses on MS.The pipeline still finishes
k6 exiting non-zero would have been worse than the problem it fixes:
runner.shruns
set -e, so a failing server would have skipped every server after it inthe loop and every later suite — and search runs after import. Failures are now
collected per server, reported loudly at the end, and the exit code stays 0, so
the remaining servers and suites run and the report still gets published.
run_test_on_serveralso returns k6's status now. It ended withecho, so thefunction always reported success and a failed run was indistinguishable from a
completed one.
MS was not validating writes
ProfileValidationOnCreateandOnUpdateship as false, so MS onlyparse-checked writes while aidbox and octofhir ran full validation. Both are on
now. Confirmed it is really profile validation and not the base attribute
validator — an obs-6 violation (
value[x]anddataAbsentReasontogether) isa 400 quoting the FHIRPath invariant, and a 201 again when
x-ms-profile-validation: falseturns it off.CRUD validates cleanly: its seeds declare no profiles, checks stay at 100%, and
the cost is about 15% throughput and 50% on p95 at 20 VUs.
Open decision: imports against MS fail
Deliberate, and needs a call. The import suite is the one place where
validation cannot currently work, and the cause is in the dataset.
705 of the ~1150 entries in every Synthea bundle declare a US Core profile:
Servers disagree on what to do when that canonical cannot be resolved:
meta.profile: us-core-patient, IG not loadedhttp://example.org/total-nonsenseUnable to resolve reference to profile ..., whole transaction failsMS has no setting for this.
ProfileValidatorbuilds a stockHl7.Fhir.Validation.Validator, and the only code that touches missing-profileissues just logs them:
Measured with validation on and no profiles loaded: 0 of 20 import bundles
succeeded, and the run looked six times faster than the working one — 2.07
iterations/s against 0.35 — because rejections return immediately. Exactly the
failure-as-throughput pattern the threshold work above now catches, so the
suite is marked untrustworthy instead of publishing that number, and the other
servers still get measured.
Options
scripts/load-us-core.pyis in the repo andverified: a full bundle returns 200 with all 1155 entries at 201.
Deliberately not wired into bootstrap — it makes MS validate against an IG
that aidbox and octofhir do not have, which moves the asymmetry rather than
removing it.
BOX_FHIR_PACKAGES;octofhir needs its own mechanism. Genuinely equal, and the most work. It will
also surface real US Core violations in the Synthea data, so the import suite
may need fixing separately.
validate and which do not. Gives up on comparing writes.
Editing the bundles to drop
meta.profilewas ruled out — the dataset isgenerated and stays as-is.
Also still open
with the flag on.
Prefer: handling=strictis not sent either, so anythingthe validator reports as a warning is ignored.
_total. MS shipsIncludeTotalInBundle: Noneand skips the COUNT thataidbox and HAPI pay on every search.
x-bundle-processing-logic: parallelis still sent only to microsoft.scripts/parity-check.shruns after a benchmark: persisted counts, referenceresolution, whether a parameter is applied, and whether a server rejects an
invalid resource. Its octofhir port is a guess, octofhir is not in this compose
file.