feat(solidity): recover historical storage layouts - #2882
Conversation
|
The key distinction is between “we can calculate the layout” and “we have a safe database row in which to store it.” Consider two solc 0.4.0 contracts whose state variables are never used: contract C {
uint256 a;
uint256 b;
}contract C {
uint128 a;
uint128 b;
uint256 c;
}Their layouts differ:
But because none of these variables are accessed, the executable code can be identical. In early solc versions, there is also no source-sensitive metadata suffix distinguishing the two compilations. Therefore both creation and runtime bytecode can be byte-for-byte identical. Sourcify’s database effectively sees: for both contracts. It consequently reuses one The problem appears when adding a source-specific field to that shared row: Verification B now exposes source A’s layout, even though B has a different layout. Why an exact bytecode match does not help: both sources genuinely compile to that same bytecode. Matching the deployed contract proves that the source is a valid explanation of the bytecode, but it does not prove it is the only valid source—or the source associated with every verification sharing the row. From 0.4.7 onward, solc adds source-sensitive compiler metadata to the bytecode. That metadata commits to information including source hashes and compilation settings. Changing the source therefore changes the metadata suffix and consequently the bytecode hash: Even if the executable instructions are identical, Sourcify no longer deduplicates the two compilations into one row. Each row can safely carry its own reconstructed layout. Standalone pre-0.4.7 extraction is still correct because it operates on a specific source bundle supplied by the caller: What is unsafe is persisting that result into a bytecode-keyed row potentially shared by other source bundles. To persist pre-0.4.7 layouts safely, Sourcify would need a larger data-model change, such as:
That is why the PR can reconstruct from 0.4.0 onward, but only stores results automatically from 0.4.7 onward. |
|
also i think this applies to 0.6.0+ contracts that turn off the metadata (metadata.bytecodeHash: "none"). this theoretically allows submitting a different source that compiles to the same bytecode but shifts storage layout. i think the safest way is probably to show 'verified storage layout' if you could prove the exact source shape with either bzzr0, bzzr1, ipfs metadata. |
|
Updated this branch to current The only merge conflicts were from the API v1 removal/move. I kept the storage-layout endpoint in Provenance check:
Validation completed:
The database-backed integration test could not run locally because Docker/OrbStack is unavailable in this worktree environment; the migrated test compiles and the focused unit/runtime coverage above is green. GitHub reports the PR mergeable, and both checks attached to the new head are green. |
|
CI follow-up: CircleCI received the exact new head as pipeline 11549 / workflow |
|
@marcocastignoli @kuzdogan could you review this when you have a chance? The staging merge, provenance conclusion, validation evidence, and current CircleCI allocation state are documented immediately above. GitHub does not permit the fork author to add formal reviewer requests on this PR. |
2 participants
Problem
Solidity verification already works for historical contracts, but solc did not expose Standard JSON
storageLayoutuntil 0.5.13. As a result, otherwise fully verified Solidity contracts compiled with 0.4.x through 0.5.12 are stored and returned through v2 with a null layout.This creates two separate gaps:
The reconstruction also has to produce the same shape as native solc output: a top-level ordered variable list plus a recursive type table containing array bases, mapping keys and values, and struct members. The existing TypeScript model did not describe that full schema.
Chosen approach
Reconstruct from the original compiler AST
The compiler package now reconstructs the layout from the AST emitted by the exact historical solc version used for verification:
legacyASTaststorageLayoutThe implementation normalizes both AST dialects into one internal model, resolves the exact compilation target, follows Solidity's linearized inheritance order, excludes constants, and applies the historical packing rules. It emits solc-compatible storage entries and recursively interns the required type descriptions for:
Historical compiler differences are versioned explicitly. This includes the 0.4.14 nightly fixed-point width transition and Sourcify's
-ci.to-nightly.compiler-version normalization.Extraction is fail-closed. If a source, target, declaration, type, array length, inheritance relationship, or AST shape cannot be resolved exactly, no reconstructed artifact is attached.
Integrate it into normal Solidity compilation
For the safe persistence range,
SolidityCompilationrequests the appropriate source-level AST in addition to the existing compiler outputs. After a successful compile it invokes the optional historical extractor only when the compiler did not already returnstorageLayout.This keeps native solc output authoritative and lets new historical verifications receive the artifact without introducing a server-only special case. Extraction failures are logged but do not turn an otherwise valid contract verification into a failure.
Use a conservative persistence boundary
The standalone extractor supports 0.4.0 through 0.5.12, but automatic persistence and database backfill are restricted to 0.4.7 through 0.5.12.
Before 0.4.7, distinct source programs can produce identical creation and runtime bytecode without metadata binding the source identity. Sourcify deduplicates
compiled_contractsby compiler and bytecode identity, so two deployments with different storage layouts can share one compiled-contract row. Writing either reconstructed layout to that shared row would make it incorrect for the other deployment.From 0.4.7 onward, compiler metadata binds the compilation/source identity into the bytecode, making a source-specific artifact safe under the current row model. Pre-0.4.7 extraction remains available as a standalone capability, but this PR deliberately does not persist it.
Add an exact-identity backfill
A new private stateless method,
replace-solidity-storage-layout, recompiles and verifies the submitted contract before updating storage.The update is accepted only when all of the following hold:
storageLayoutis missing or JSON nullThe SQL update uses
jsonb_setto modify onlycompilation_artifacts.storageLayout. It never replaces the surrounding compilation artifacts and never overwrites an existing non-null layout. Identity mismatches, unsupported versions, malformed recovery results, ambiguous rows, and repeat attempts return a 400-level error.Complete the TypeScript schema
The Solidity compiler-output types now model recursive layout data including
base,key,value, andmembers, and correctly allow the native null type table used for contracts with no storage variables. Runtime output remains compatible with native solc JSON.Alternatives considered
Infer layout from bytecode or observed storage accesses
Rejected. Bytecode cannot reliably recover unused variables, source labels, AST IDs, struct boundaries, mapping key/value types, or the distinction between several types with identical runtime widths. Storage access traces would be incomplete and deployment-specific rather than compiler-exact.
Parse historical source with a modern compiler or a separate Solidity parser
Rejected. Historical syntax, name resolution, inheritance, type identifiers, data locations, mutability, and packing behavior changed across these releases. Using the exact original compiler's AST avoids creating a second historical Solidity frontend and keeps the reconstruction tied to the verified compilation.
Implement reconstruction only inside the backfill endpoint
Rejected. That would repair old rows but leave every new pre-0.5.13 verification incomplete. Keeping extraction in the compiler package lets the normal verification path and the repair path share one implementation.
Persist layouts for 0.4.0–0.4.6
Rejected under the current database model. Bytecode identity is not enough to select a source-specific layout safely in this range. Supporting it would require moving layout storage to a per-verification/source-identity record or otherwise changing compiled-row deduplication.
Backfill rows by version/address alone
Rejected. A successful bytecode match is not sufficient to prove that the submitted source/settings are the same compilation already represented by the row. The chosen path recompiles, verifies, compares the complete stored compilation identity, and performs an atomic guarded update.
Replace or normalize native layouts
Rejected. solc 0.5.13+ remains the source of truth, including any version-specific schema details or compiler behavior. Reconstruction runs only when native output is absent.
Continue below Solidity 0.4.x
Out of scope. This PR intentionally starts at 0.4.0 rather than expanding into older compiler archaeology.
Impact on the current Sourcify dataset
The Sourcify v2 dataset snapshot generated at
2026-07-07T02:26:39Zcontains:verified_contracts.deployment_idThe dump contains 40,259,667 distinct verified deployments overall, so 26.44% of the current verified-contract population is in the range addressed by this PR.
The population is concentrated in three compiler releases: 0.4.23 accounts for 3,653,020 deployments, 0.4.11 for 3,221,404, and 0.4.16 for 2,367,139. Together they represent 86.82% of the affected range.
These counts describe the existing population eligible to gain a layout, not a guarantee that every row will be updated. Reconstruction deliberately fails closed on unresolved historical AST constructs, and the backfill additionally requires an exact stored-compilation identity match.
Validation