release: promote solvis Strawberry migration to prod#97
Merged
Conversation
… sibling) Captures the repo-specific inventory and runbook deltas before code: - PynamoDB surface is a single wrapped model (BinaryLargeObject) - repo ships its own cross-stage differential validator (cli_ab_test) -> reuse for Phase 5 instead of building drive_live.py - still on poetry; yarn age gate missing; container entry is the serverless-wsgi handler.py workaround (-> Mangum) - both serverless-s3-local + serverless-dynamodb load at boot (§4.7) Epic: GNS-Science/nshm-toshi-api#359
chore: migrate Poetry → uv, flake8/black/isort → ruff
- add tools/dump_legacy_sdl.py (stdout->stderr guard; pyvista warns on import) - commit schema.legacy.graphql (449 lines, 29 types) - log: poetry2uv verified green (ruff + 62 tests); Phase 0 SDL done Refs #86
- 11 real queries lifted from the test suite into fixtures/corpus/*.graphql - test_corpus_replay.py validates each against schema_root (no fixtures needed); re-points to the Strawberry schema later to catch field/type drift - kororaa UI queries to follow (real frontend traffic) Refs #86
- vendor 3 pure-solvis kororaa Relay ops, transformed from gateway shape (SOLVIS_<f> -> <f>, on Query -> on QueryRoot); validated against schema - adds coverage the tests lacked: locations_by_id + radius_geojson(style:), rich rupture node fields (fault_surfaces, magnitude, rate_weighted_mean, ...) - ComboRuptureMap* skipped (multi-API: also fetch KORORAA_textual_content) - log the stale-checkout lesson: kororaa was 9mo behind and showed retired inversion_solution/analyse_solution usage -> false parity alarm Refs #86
- add strawberry/fastapi/mangum/pydantic (legacy graphene/flask kept until cutover)
- app.py: FastAPI + GraphQLRouter + Mangum + CORS
- strawberry_schema.py: minimal QueryRoot.about at SDL parity
(auto_camel_case=False, root type QueryRoot to match legacy + kororaa corpus)
- serverless.yml: image.command -> solvis_graphql_api.app.handler; drop
serverless-wsgi plugin + custom.wsgi
- Dockerfile: CMD -> app.handler; drop bogus /bin/bash -c entrypoint
- verified: amd64 docker build OK + handler imports in-image; uvicorn {about}
byte-matches legacy; full legacy suite 77 passed
Refs #86
…r API) - BinaryLargeObjectModel(pynamodb.Model) -> pydantic BaseModel + thin boto3 DynamoDB CRUD; public BinaryLargeObject wrapper API unchanged so cli + composite_solution.cached are untouched - object_meta stored as a JSON string (JSONAttribute parity; ints not Decimals) - migrate()/drop_tables()/tables preserved (create_table wait -> boto3 waiter) - drop pynamodb dependency; uv re-locked (164 pkgs) - verified: moto contract 4/4, full suite 77 passed, ruff + mypy clean Refs #86
…rity - strawberry_schema.py: full 29-type schema; tools/schema_parity.py gate confirms byte-identical vs schema.legacy.graphql - custom Node interface (id: ID!, not strawberry.relay/GlobalID); QueryRoot root type; auto_camel_case=False; relay PageInfo camelCase via name= - nullability (X|None for nullable-with-default), UNSET vs =null args, and the partial 3-key style-arg default all matched - resolvers reuse the graphene-free compute helpers; light query-root resolvers verified vs legacy output - corpus replay re-pointed at the Strawberry schema (15 pass); add strawberry mypy plugin (new code mypy-clean) - archive-dependent runtime resolvers stubbed -> Phase 3 (need fixtures + tests) Refs #86
- test_strawberry_parity.py: run each query vs Graphene AND Strawberry, assert identical data (in-process analogue of cli_ab_test) - implement CompositeRuptureDetail computed fields (magnitude/area/length/ rake_mean/rate_*) via rupture_detail; fix get_parent_fault_names shape - harness caught 2 bugs the SDL gate can't: relay global-id encoding (id -> to_global_id, Model C1) and float opacity defaults (1.0 not 1); switch the style-arg default to a mapping (instance default breaks graphql-core coercion) - SDL parity still byte-identical; suite 85 passed; ruff + mypy(new) clean Refs #86
- reuse legacy paginated_filtered_ruptures via a dict adapter (_legacy_filter) that exposes the dict+attr access it expects; convert the graphene connection to the Strawberry RuptureDetailConnection - fix: filter_set_options default must be a plain mapping, not an input instance (an instance default trips strawberry argument coercion) - same class of fix as the style-arg default; SDL parity preserved - test_filter_ruptures_parity: pagination + cursors + node global-id + magnitude all byte-match legacy (note: legacy can't serve pageInfo.hasPreviousPage) Refs #86
- CompositeRuptureSections: delegate every resolver to the legacy graphene static resolvers (aggregates/geojson/MFD/colour reused verbatim) via a graphene root built from the input; colour-scale normalisation converted to string value - node(): legacy has no get_node -> null on both schemas (parity-correct stub) - mypy: None-safe _fso_dict, edges/normalisation narrowing, type:ignore on untyped graphene constructors -> new code clean - 11 differential parity checks (full surface) + suite 88 passed; SDL byte-identical Refs #86
- container: amd64 build green with migrated deps; 360MB (< 500MB aim, no multi-stage); Mangum handler + full schema import in-image - pip-audit runtime deps: no known vulnerabilities - plugin chain (s3rver + serverless-dynamodb) loads; no resolutions needed - drop dead serverless-wsgi npm plugin + stale requirements_* scripts; yarn lockfile updated, --immutable clean (§4.6) - memory stays 2096; CI already on -uv test+deploy workflows Refs #86
- docs/PHASE5_CUTOVER.md: in-place replacement, rollback (image-only, no new write shapes), differential validation via the built-in cli_ab_test (prod-vs-test, covers the full migrated surface), promote + 30-min watch, post-healthy legacy-cleanup scope - reuse cli_ab_test (no drive_live.py needed); in-process parity already locked by test_strawberry_parity.py - actual deploy/promote needs AWS creds + prod go-ahead Refs #86
…ationEnum type stub CI gates on mypy. poetry2uv added follow_untyped_imports=true, making mypy type-check into the legacy graphene + solvis code (29 errors). Restore the original lenient posture (ignore_missing_imports only); add a TYPE_CHECKING stub enum for SetOperationEnum so it stays a valid type without follow. Whole package mypy-clean (30 files); parity + 11 differential tests + ruff green.
# Conflicts: # README.md # poetry.lock
Migrate Graphene → Strawberry (Phases 0–5, combined)
Live kororaa-test smoke surfaced a RuntimeError every 5 min: the warmup plugin
pings with a non-HTTP event Mangum can't infer a handler for. The legacy
serverless-wsgi handler swallowed these. app.handler now short-circuits the
warmup event ({statusCode:200}) before delegating to Mangum.
- tests/test_app_handler.py: warmup guard + delegation
- all real POST /graphql traffic was already 200; this only silences the
5-minute warmup error in the logs
- full suite 90 passed; ruff + mypy clean
…armup fix - fold the cutover plan into MIGRATION_LOG.md (## Cutover plan section); delete the standalone docs/PHASE5_CUTOVER.md - add 2026-06-26 entry: warmup-event fix verified live (clean warmup ping, no RuntimeError) + cli_ab_test prod-vs-test 9/9 PASS (cutover gate green) + weka/ kororaa animation clean - refresh front-matter status + Phase 5 checklist (test done + validated)
docs: consolidate cutover plan into MIGRATION_LOG + log A/B 9/9
…§4.11) # Conflicts: # .yarnrc.yml # DEPLOYMENT.md # README.md # package.json # pyproject.toml # serverless.yml # setup.cfg # solvis_graphql_api/data_store/model.py # solvis_graphql_api/solution_schema.py # uv.lock # yarn.lock
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #97 +/- ##
==========================================
+ Coverage 95.73% 96.54% +0.80%
==========================================
Files 19 12 -7
Lines 1126 1099 -27
Branches 53 63 +10
==========================================
- Hits 1078 1061 -17
+ Misses 34 25 -9
+ Partials 14 13 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The Phase 1 removal (for stacked-PR CI, Trap #14) was the wrong call here — the branch rule is a deliberate team choice. The stack is collapsed/merged now, so restore it. Log: notes the §4.11 forward-port + this correction.
First step of the post-cutover legacy cleanup. handler.py (serverless-wsgi container workaround) and solvis_graphql_api.py (Flask app) are dead — app.py's Mangum handler is the entry. Removed them + the create_app test. Graphene schema + the strawberry delegation stay until the resolver de-graphene (checklist in PR). Suite still green (86 passed).
color_scale/compute.py: graphene-free compute (get_normaliser, log_intervals, get_colour_values, compute_colour_scale -> ColourScaleResult dataclass). strawberry QueryRoot.color_scale now uses it instead of the graphene get_colour_scale. Differential parity green (vs live graphene).
Build the LocationDetail connection directly from location_by_id + graphql_relay.offset_to_cursor (matches graphene's array cursors) instead of the graphene get_location_detail_list. Differential parity green (incl. Node id + radius_geojson).
Replace the legacy paginated_filtered_ruptures + _to_strawberry_rupture_connection path with graphene-free helpers built directly in strawberry_schema: - _ATTRIBUTE_COLUMN_MAP (was CompositeRuptureDetail.ATTRIBUTE_COLUMN_MAP) - _auto_sorted (port of composite_solution.schema.auto_sorted_dataframe) - _paginated_ruptures (cached.matched_rupture_sections_gdf + graphql_relay cursors) filter_ruptures resolver now builds RuptureDetailConnection itself. Drops the _LegacyFilter/_legacy_filter shims and the composite_solution.schema import. SDL byte-identical; 11/11 differential parity green.
Port the section compute off the legacy graphene CompositeRuptureSections root. The strawberry type now carries the filter input directly (filter_input Private) and resolves graphene-free via new module helpers: - _fault_section_aggregates / _matched_rupture_sections (cached.* wrappers) - _mfd_histogram_rows (build_mfd port) - _fault_surfaces_geojson / _fault_traces_geojson (get_colour_values from compute) - section color_scale via compute_colour_scale with participation-rate fallback Drops the _GrapheneSections/_GrapheneFilterArgs imports and _graphene_sections_root shim; _to_strawberry_color_scale no longer needs the legacy .color_map branch. Adds test_filter_rupture_sections_colour_parity covering the colour paths the plain sections query skips (section color_scale fallback, fault_surfaces/fault_traces + color_scale). SDL byte-identical; 12/12 differential parity green.
…free The strawberry schema imported two pure (pandas/dict) helpers through graphene- importing modules. Move them to graphene-free homes so the schema names no graphene module: - apply_geojson_style -> new geojson_style_util.py (geojson_style.py re-exports it) - rupture_detail -> cached.py (composite_rupture_detail.py re-exports it) strawberry_schema now imports both from the graphene-free modules. The legacy graphene modules keep working via re-export shims until they are deleted at cutover (the package __init__ still eager-loads them, which is removed then). SDL byte-identical; 12/12 differential parity green.
…om tests
The 14 legacy graphene-Client behavioural tests now drive the Strawberry schema via a
graphene.test.Client drop-in (tests/_strawberry_client.py). This keeps their broad coverage
of the migrated resolvers — sorting, filter-set logic, pagination, geojson, node ids — which
the parity corpus alone does not reach. 83 pass, 10 skipped.
test_strawberry_parity converted from an in-process differential (needed the graphene oracle)
to a Strawberry-only snapshot guard; golden outputs captured under tests/__snapshots__ from
the proven-identical schema (regenerate with SNAPSHOT_UPDATE=1).
Fixes one resolver bug the broader suite caught: filter_ruptures passed strawberry.UNSET
through as the sortby 'ascending' value (pandas rejected it); now UNSET fields are omitted so
the legacy '.get("ascending", True)' default applies — matching graphene.
No test imports graphene now. SDL byte-identical; ruff clean.
…e-free
Removes the graphene schema entirely now that the Strawberry schema, the test suite, and the
SDL gate no longer depend on it:
deleted: schema.py (root), location_schema.py, solution_schema.py (dead), geojson_style.py,
color_scale/color_scale.py, composite_solution/{composite_rupture_detail,
composite_rupture_sections,composite_solution,filter_set_logic_options,filtered_ruptures_args,
schema}.py, tools/dump_legacy_sdl.py (the legacy SDL baseline is frozen in schema.legacy.graphql).
Two graphene-free helpers the deleted modules held were relocated to stable homes first:
- auto_sorted_dataframe -> new composite_solution/ruptures.py (also used by the sort unit test)
- _solvis_join -> cached.py (its only consumer)
color compute already lived in color_scale/compute.py.
Gutted composite_solution/__init__.py and color_scale/__init__.py (they eager-loaded the
deleted graphene types). Repointed the two compute unit tests (sort, colour) at the new homes
and dropped the stale get_composite_solution monkeypatches in conftest (everything resolves
through cached now).
No 'import graphene' remains under solvis_graphql_api/. SDL byte-identical; 83 pass, 10 skipped.
With the legacy graphene root schema.py deleted, the canonical name is free. Rename the Strawberry module to solvis_graphql_api/schema.py and repoint importers (app.py, the SDL parity gate, _strawberry_client shim, the snapshot + corpus-replay tests). Module path only — local 'as strawberry_schema' aliases kept. Refreshed the module docstring to point at the graphene-free compute homes (cached, ruptures, color_scale.compute, geojson_style_util). app + handler import OK; SDL byte-identical; 83 pass, 10 skipped; ruff clean.
Remove graphene, graphql-server, flask, flask-cors and serverless-wsgi from project dependencies now that no source imports them. Add graphql-relay as an explicit dep — the schema's relay global-id/cursor encoding used it, but it was only present transitively via graphene. Relock: graphene 3.4.3, flask, flask-cors, graphql-server, serverless-wsgi (and blinker/itsdangerous) all removed from the resolved env; graphql-relay 3.2.0 + graphql-core 3.2.11 (via strawberry) remain. Also narrows the section color_scale normalization default with explicit is-checks (mypy couldn't narrow 'not in (None, UNSET)'). Verified with graphene physically uninstalled: app imports, 83 pass / 10 skipped, SDL byte-identical, mypy + ruff clean.
Add the 2026-06-26 cleanup entry (chunks 1-9: graphene removed from source, tests and deps; test suites repointed at Strawberry; differential parity → snapshot guard) and mark the §5 post-healthy cleanup checklist item.
… (CI cross-platform)
The snapshot conversion (chunk 6) made three parity tests platform-fragile: byte-exact golden
files captured on macOS didn't match ubuntu CI, because shapely buffer coords and matplotlib
hex colours differ in their low float digits. The original differential test was immune (both
schemas ran in one env, so the difference cancelled) — snapshots lost that property.
Keep byte-exact snapshots for the 9 deterministic queries (rounded scalars), and assert the 3
geometry/geojson queries structurally instead: locations_by_id checks the bug-prone global-id
encoding decodes to ('LocationDetail', id) + a non-empty buffer; sections checks counts/extents/
mfd shape + that styling/colour reached every feature. cli_ab_test remains the authoritative
same-data byte-for-byte geojson parity gate (already 9/9).
83 pass / 10 skipped locally; ruff clean.
…apshot) The locations snapshot re-stored the entire nzshm_common LOCATIONS constant — a ~2MB, 118k-line fixture of library data. Assert structurally instead (count == len(LOCATIONS) + a known entry). 12 parity tests pass.
chore: remove legacy Graphene/Flask (post-cutover cleanup) [WIP]
The cleanup churned a lot of source (new graphene-free compute + ported resolvers), leaving codecov patch/project red on PR #97. Cover the new lines directly: - tests/test_compute.py — color_scale/compute.py: get_normaliser (log/lin/error), log_intervals (narrow/wide/same-decade/slim+ensure_max), compute_colour_scale (log/lin/ unknown-normalisation error), get_colour_values (incl. the None->"x000000" sentinel). - tests/test_schema_helpers.py — _style_dict(None), _fso_dict(falsy), and the null-sortby-item skip in filter_ruptures. compute.py/schema.py/ruptures.py/geojson_style_util.py now 100%. One genuinely unreachable legacy branch in log_intervals.slim (caller passes the min as 'max', so largest < it never holds) is marked '# pragma: no cover' with a note. 96 pass / 10 skipped; ruff + mypy clean.
test: close the de-graphene coverage gap
tools/schema_parity.py (the Graphene→Strawberry SDL byte-parity gate) ran as a standalone script, so it sat at 0% coverage and dragged the migration's codecov patch/project just under the auto threshold on promote PR #97. Wrap it in tests/test_sdl_parity.py — happy path (diff()=='' , main()==0) and a drift case (different baseline → non-empty diff, main()==1). Bonus: the SDL parity gate is now a permanent CI test, not a manual invocation. schema_parity.py 0% → 100%; package coverage 96% → 98%.
test: SDL parity gate as a CI test (close remaining codecov gap)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the completed Graphene → Strawberry / FastAPI / Mangum migration from
deploy-testto prod.✅ Ready to merge — conflict resolved, re-validated.
Validation
{about}✓); real kororaa/weka traffic ✓; Mangum warmup-event fix verified live.cli_ab_test -A prod -B test→ 9/9 PASS, twice (before and after the main reconciliation below) — test Strawberry is byte-for-byte equal to prod Graphene.mainhad diverged (runbook §4.11) — now reconciledmaincarried two un-forward-ported commits:fix deploy workflow(convergent) andChore/upgrade #85(a parallel poetry→uv on prod). Resolved by mergingmainintodeploy-test(commit1caad74):pynamodbstays removed); lockfiles regenerated.scripts/smoke_test.py, CLAUDE.md edits.mainis now an ancestor ofdeploy-test→ this promote is a clean, conflict-free merge.On merge → prod deploy
deploy-to-aws-uv.ymlonmain).revert: <this merge sha>againstmain; trigger: 5xx spike sustained or anycli_ab_testmismatch. Image-only rollback (no new write shapes).aws logs tail … --follow+ CloudWatch error rate / p95).docs/MIGRATION_LOG.md§ Cutover plan §5).