feat(vyper): recover historical storage layouts

[banteg]

Summary

• recover Vyper storage layouts from 0.1.0-beta.16 through 0.4.1b3 with the exact compiler package in an isolated uv process
• preserve historical allocator semantics, nested HashMap order, struct members, module namespaces, fixed-array/Bytes spans, and compiler-assigned lock positions
• recover and expose Vyper transient storage layouts from 0.3.8, including splitting the mixed persistent/transient namespace emitted by 0.3.8–0.3.10
• keep 0.4.1b4+ on the existing native Standard JSON path and treat supplemental extraction failure as non-fatal
• compile releases without GitHub binary assets through the same exact Python fallback and make multi-source output selection work across old formatters
• persist and expose recovered layouts through the existing API v2 storageLayout and transientStorageLayout fields, with an identity-checked resumable backfill for existing rows

Extraction matrix

Vyper releases	Source of truth
0.1.0-beta.16–0.2.12	compiler GlobalContext positions (legacy hashed-composite roots)
0.2.13–0.2.15	annotated AST/compiler-assigned positions
0.2.16–0.4.1b3	native layout captured before the historical Standard JSON formatter drops it, enriched from compiler type objects
0.3.8–0.3.10 transient	native slots split into persistent and transient namespaces using the compiler AST's is_transient metadata
0.4.0–0.4.1b3 transient	native transient_storage_layout captured with persistent layout
0.4.1b4+	unchanged native Standard JSON layout

Structured historical types stay backward-compatible with Sourcify's flat Vyper layout: each affected leaf gets an optional type_definitions graph. Definitions are leaf-scoped, so initialized modules and same-name imported structs remain accurate without changing native leaf type/slot/span values.

Persistence and operations

• API v2 returns persistent artifacts via storageLayout and Vyper transient artifacts via the existing transientStorageLayout field.
• The private backfill update is atomic and only succeeds when compiler version, target, settings, additional input, and the full path-to-source-content map exactly match the stored compilation identity.
• A resumable, one-representative-per-compilation backfill config is included under services/database/massive-replace-script/vyper-storage-layout-backfill.
• The server image and affected CircleCI jobs install pinned uv 0.11.21.
• The frontend's hard-coded pre-0.4.1 warning remains a separate follow-up once this backend is deployed.

Validation

• npm test in packages/compilers — 29 passing; historical exact-package cases remain opt-in
• exact Vyper 0.3.8 and 0.4.0 transient-layout probes — 2 passing, covering both the mixed and separate compiler namespaces
• direct exact-package probes passed for the previously broken 0.4.0b1 and RC compiler-data APIs; the opt-in CI matrix also covers 0.4.0b1 and 0.4.0rc5
• npm run build — compilers-types, compilers, and lib-sourcify
• focused VyperCompilation persistent/transient extraction test passes
• focused server database persistence test passes
• Etherscan utility suite — 29 passing
• custom replacement unit suite — 3 passing
• Ruff, Prettier, YAML parsing, backfill config syntax, and git diff --check pass

The full server integration suite requires its Postgres/local-chain services and is left to CI. Local ESLint invocation is currently blocked before file analysis by the checkout's ESLint 8/flat-config dependency mismatch (ERR_PACKAGE_PATH_NOT_EXPORTED); TypeScript builds and focused tests are clean.

[banteg]

not ready yet, will also add transient storage layout extraction

[marcocastignoli]

not ready yet, will also add transient storage layout extraction

is it now ready?

[banteg]

is it now ready?

yes, it's ready

[banteg]

Updated against current staging (859adf2a) in non-force merge commit 167f285d.

The sole conflict was the renamed/consolidated private API integration test. The historical Vyper source, layout assertions, backfill identity checks, and API-v2 storage-layout coverage are preserved; the verification setup now uses the staging queued v2 helper, and the renamed unit-test import is updated.

Local validation on Node 22.22.0:

• affected package builds: compilers-types, bytecode-utils, compilers, lib-sourcify, server
• ESLint/Prettier checks plus server TypeScript lint build
• Vyper layout helpers: 10 passing
• exact historical compiler probes: 5 passing (0.3.8/0.4.0 transient layouts, 0.4.0rc5 span recovery, initialized-module and colliding-import structs)
• lib-sourcify historical/output-selection tests: 14 passing
• server backfill/persistence units: 4 passing
• Etherscan input/version handling: 29 passing
• YAML and backfill config syntax checks

Socket checks are green. CircleCI created pipeline 11547 / job 69135 for 167f285d, but the setup job is still held as not_running with no queue/start timestamp or error, so it appears to need project-side fork approval/capacity before the downstream matrix can attach.