ci: add workflow_dispatch action to deploy Goldsky pipeline#9
Open
willemneal wants to merge 21 commits into
Open
ci: add workflow_dispatch action to deploy Goldsky pipeline#9willemneal wants to merge 21 commits into
willemneal wants to merge 21 commits into
Conversation
Pipeline: drop hardcoded contract_id allowlist in transform_1 and filter by event symbol instead, so new registries are indexed the moment they emit any of deploy/publish/register/rename/update_address/update_owner/sub_reg. Add transform_3_subregistry_events + subregistry_events_pg sink that captures sub_reg events from the root registry CB7WYFH2...SYA76J2W and writes them to public.registries keyed on the subregistry's contract_id. The authority filter on emitter_contract_id prevents rogue contracts from polluting the lookup table. sql/v4_registries.sql: schema for the new registries lookup table. fly-app: LEFT JOIN public.registries in every contract/wasm query and return COALESCE(r.channel, table.channel) so unknown subregistries surface under their raw contract_id until the root announces them, at which point the friendly name resolves. Drop the hardcoded main/unverified allowlist from the four endpoints that had it. Add GET /v1/registries listing the new lookup table. Dedupe v4_published_wasms and v4_deployed_contracts in contract queries via DISTINCT ON subqueries so shared wasm hashes don't multiply registered rows. Agent-Id: bitswell Session-Id: 277b25ec-bebf-4249-9719-104aae47e81a Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
transform_1: filter out op-none event variants. The Stellar dataset emits every contract event twice — once at the transaction level (id suffix op-none-event-N) and once at the operation level (op-0-event-M). Both rows have identical data, only the id differs, so the postgres sink was keeping both and every /v1/contracts row surfaced twice. NOT LIKE '%-op-none-%' keeps only the operation-level copy. transform_3_subregistry_events: swap authority filter from the old root CB7WYFH2...SYA76J2W to the new root CBT6FE6W...OGUJH5JTI. start_at: 2036131 -> 2038000 to skip the historical events from the two defunct roots (CDVDJX2HX, CB7WYFH2) so a fresh restart doesn't drag their orphaned register events back into v4_registered_contracts. Agent-Id: bitswell Session-Id: 277b25ec-bebf-4249-9719-104aae47e81a Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Matches the v4_* prefix convention used by every other sink in registry-turbo-v4.yaml. The SQL file was already named v4_registries.sql. Agent-Id: bitswell Session-Id: 4ff31451-91fe-4237-bd27-7143f59d56b0 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the LEFT JOIN v4_registries + COALESCE(...) pattern duplicated across seven query sites in fly-app/src/main.rs with two views (v4_published_wasms_named, v4_registered_contracts_named) that expose a resolved_channel column. Query plans are unchanged; the join is now defined once instead of being copy-pasted per call site. Agent-Id: bitswell Session-Id: 4ff31451-91fe-4237-bd27-7143f59d56b0 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The root registry now emits a sub_reg event naming itself "root", so v4_registries maps its contract_id to channel="root". The API's default-channel routes were passing the literal "main" and returning nothing. Align the literal with what the pipeline writes, and rename the default-channel handlers from *_main* to *_root* so function names match their behavior. Agent-Id: bitswell Session-Id: 4ff31451-91fe-4237-bd27-7143f59d56b0 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the previous root CBT6FE6W with the new root CBNBQND6EMYTTRTCUWUJ3VIKF7RUUISK5T4GAKTXRVIQRHGP4XQY4ID7 at the emitter_contract_id filter in transform_3_subregistry_events, and fix a stale reference to a pre-CBT6FE6W root in the v4_registries.sql header comment so documentation matches the pipeline. Agent-Id: bitswell Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Goldsky Turbo auto-creates the v4_deployed_contracts, v4_published_wasms, v4_registered_contracts, v4_rename, v4_update_address, v4_update_owner and v4_raw_events_backup tables on first pipeline deploy based on each transform_3_* projection in registry-turbo-v4.yaml. Until now the shapes lived only inside Goldsky's runtime, so a fresh Postgres (test harness, disaster recovery) couldn't stand up the schema without first deploying a pipeline. Commit the DDL so the repo owns the source of truth; Goldsky's auto-create continues to work unchanged, and downstream DDL like v4_named_views.sql can now be applied to any Postgres. All statements use IF NOT EXISTS so applying this to an already-populated database is a no-op. Agent-Id: bitswell Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split Jest into two projects: "unit" under test/unit (no external services) and "integration" under test/integration (requires Postgres). Add docker-compose.test.yml running postgres:17-alpine on port 5433 for the integration project. Scope tsconfig's main build to bin/ and lib/ only so test and scripts directories don't emit .js/.d.ts next to their .ts sources and confuse ts-jest's resolution. New devDeps: - js-yaml for loading the pipeline YAML in integration tests - @stellar/stellar-sdk for the Soroban RPC fixture fetcher - ts-node for running the fetcher script New npm scripts: - test:unit jest --selectProjects unit - test:integration jest --selectProjects integration --runInBand - validate:pipelines bash scripts/validate-pipelines.sh - fixtures:refresh ts-node scripts/fetch-event-fixtures.ts Remove the old commented-out test/goldsky.test.ts stub; real coverage lands in subsequent commits. Agent-Id: bitswell Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cover the periodic lambda's parseDeploy/parsePublish across all map-key
orderings plus the unexpected-symbol fallthrough, the get-contracts
validation branch (limit bounds, cursor format), and the CDK stack
layout (two lambdas, 1-minute EventBridge rule, API Gateway with CORS,
secrets:GetSecretValue permissions, 1 rps throttle).
Two small refactors to enable pure-function tests:
- Export parseDeploy and parsePublish from lib/periodic-lambda.ts so
they can be tested without booting the Lambda entry point.
- Factor validateParams out of lib/get-contracts.ts's handler so
parameter validation is testable without mocking Secrets Manager or
pg.Pool.
21 unit tests, all passing under `npm run test:unit`.
Agent-Id: bitswell
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stand up a Postgres 17 test harness (test/integration/setup.ts) that
applies the repo's SQL verbatim, plus hand-authored synthetic fixtures
in test/fixtures/soroban-events/ that mirror Goldsky's transform_2
output shape. Two integration suites run against it:
- v4-views.test.ts applies v4_sink_tables.sql + v4_registries.sql +
v4_named_views.sql and asserts that resolved_channel on the _named
views joins via v4_registries when the contract_id is known and
falls back to the raw contract_id when it is not.
- pipeline-transforms.test.ts loads registry-turbo-v4.yaml through
js-yaml, extracts each transform_3_* SQL string, seeds a
transform_2_events_with_command_name table from the fixtures, and
executes each transform verbatim against Postgres 17 (which ships
with JSON_VALUE). Coverage includes deploy/publish/register map-key
permutations, rename, update_address, update_owner, and a
load-bearing assertion that transform_3_subregistry_events filters
out events from emitters other than CBNBQND6. A header comment
flags the known Flink/Calcite vs. Postgres dialect drift — this is
a sanity check, not a correctness proof.
10 integration tests, passing under `npm run test:integration` against
the service from docker-compose.test.yml.
Agent-Id: bitswell
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Synthetic fixtures catch SQL-logic regressions but can't catch drift in
how Goldsky's Flink source serialises Soroban events. Add real-data
coverage pulled directly from Soroban RPC:
- scripts/fetch-event-fixtures.ts (`npm run fixtures:refresh`)
Hits soroban-testnet.stellar.org, filters getEvents to the current
root, walks the XDR ScVal topics/value into Goldsky's tagged-JSON
shape (symbol/string/address/bytes/bool/map/…), and writes one file
per event symbol to test/fixtures/soroban-events-real/. Lookback
window is 9k ledgers — larger ranges are silently rejected by the
RPC's ~10k-ledger event retention. Clears stale per-symbol files on
each run so a shrinking event set doesn't leave leftovers.
- test/fixtures/soroban-events-real/ committed alongside the
synthetic set so CI and local runs get the same coverage without
needing network; refresh lives under `npm run fixtures:refresh`.
- pipeline-transforms-real.test.ts runs every transform_3_* SQL over
whatever real events exist, asserting structurally that every
extracted column populates and row counts match input. The suite
skips automatically (describe.skip) if the real fixture directory
is empty — e.g. just after a root rotation with no on-chain
activity yet.
7 drift-detection tests now pass against the current 17 real events.
Agent-Id: bitswell
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds .github/workflows/test.yml running on every PR and push to main:
1. Install Goldsky's turbo CLI on the runner (ubuntu-24.04 ships
glibc 2.39, so no container wrapper needed unlike local dev).
2. Run `turbo validate` against each registry-turbo-*.yaml via
scripts/validate-pipelines.sh. registry-minimal.yaml has an
independent pre-existing validation error and is intentionally
out of scope.
3. Run the jest unit project (no external services).
4. Bring up postgres:17-alpine as a GitHub Actions service container
and run the jest integration project — both synthetic and real
suites — against TEST_PG_URL.
Also exposes TURBO_BIN in scripts/validate-pipelines.sh so developers
on older glibc can point at a docker-wrapper; the script tells them
where to install from when the binary is missing.
Agent-Id: bitswell
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The main https://goldsky.com/install installs the goldsky wrapper CLI, not the standalone turbo binary the pipeline validator needs. Switch to https://install-turbo.goldsky.com which puts a working turbo at ~/.goldsky/bin/turbo. Also surface the correct URL in the validate-pipelines.sh error message so local runs point at the same command. Agent-Id: bitswell Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
turbo validate requires an authenticated goldsky session even for offline YAML schema checking — my earlier assumption that validate was auth-free was wrong. Write the GOLDSKY_API_KEY secret to ~/.goldsky/auth_token before running the validator, and guard both the auth step and the validate step on the secret being present so fork PRs (which cannot access repo secrets) skip gracefully instead of failing the workflow. Agent-Id: bitswell Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves the route-wiring closure out of HttpServer::new so tests can reuse the same routes via App::new().configure(configure_routes) without spinning up a real HTTP listener. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Pure unit tests for parse_cursor (None, valid, malformed, negative ledger) and serialize_raw (None, valid JSON inline, invalid JSON). - DB-backed HTTP integration tests using actix_web::test::init_service against a real Postgres: wasms list grouping and channel resolution, wasm-by-version, 404s, limit/cursor validation, contracts JOIN across registered+deployed+wasms, and registries ordering. - End-to-end test that loads registry-turbo-v4.yaml, replays each transform_3_* SQL over the real Soroban fixtures in test/fixtures/soroban-events-real/, and asserts the resulting rows surface correctly through /v1/registries, /v1/contracts, and /v1/wasms. Flipping the root contract id in the pipeline YAML makes the sub-registry filter drop to zero rows, failing this suite. DB-gated tests skip silently when TEST_DATABASE_URL is unset; e2e additionally requires the real fixtures directory to be populated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a parallel fly-app job to the test workflow with its own postgres:17-alpine service container. Renames the existing Node job to `node` to make the split explicit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `npm run fixtures:pull` to incrementally append new testnet events without wiping existing fixtures. Both pull and refresh now also fetch events from all sub-registry contracts discovered via sub_reg rows. Fixes RPC retention window issue where startLedger fell outside the ~10k-ledger window, causing the RPC to silently return no events. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a manually-triggered workflow that runs `turbo apply/stop/status` against registry-turbo-v4.yaml using the existing GOLDSKY_API_KEY secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
willemneal
force-pushed
the
feat/deploy-pipeline-action
branch
from
26335b9 to
fb79035
Compare
chadoh
force-pushed
the
feat/v4
branch
6 times, most recently
from
3685a7a to
8e9e08a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
workflow_dispatch) that deploysregistry-turbo-v4.yamlto Goldskyapply,stop, andstatusactions via a dropdown inputGOLDSKY_API_KEYsecret for authenticationTest plan
statusto verify CLI auth worksapplyto deploy the pipeline🤖 Generated with Claude Code