Fix similarity API database load and timeouts - #2918
Open
manuelwedler wants to merge 4 commits into
Open
Conversation
Similarity candidates are compilation-level data, so key them on compiled_contracts.id instead of (chain, address): - The prefix query returns candidate compilation ids only. The verified- contract check is a LATERAL LIMIT 1 on purpose: EXISTS gets de-correlated by the planner into seq scans over verified_contracts/sourcify_matches (measured in production; see the comment in Database.ts). - Candidate payloads (std_json_input/output etc.) are fetched in batches of 5 via one query per batch, stopping at the first batch that verifies, instead of 20 payloads in 20 parallel (chain, address) queries upfront. - Creation tx data is resolved once in the main thread; the verification worker no longer performs RPC calls. Refs #2891
…error Postgres cancels queries exceeding statement_timeout with SQLSTATE 57014, which previously surfaced as an opaque internal_error on the verification job. Both similarity database queries (the candidate id prefix scan and the batched candidate payload fetch) now map 57014 to the new similarity_search_timeout error code, so API consumers can distinguish 'retrying is pointless for this bytecode' from transient server errors, and the remaining timeout rate stays measurable after the prefix-table fix lands. Refs #2891
Similarity search needs compilations whose runtime code shares its first 75 bytes with a target contract. The existing idx_code_code_first_75 on the code table cannot serve this efficiently: for contract classes that bake immutables into runtime code (e.g. Uniswap V3-style pools), hundreds of thousands of onchain variants share a prefix while only a handful of compilations exist, so a prefix scan walks the whole variant set (measured in production: 142,872 rows scanned for 21 candidates, 36s). Indexing the prefix over compilations makes every index entry a candidate, so a LIMIT short-circuits after ~LIMIT entries. New compilations are covered by an insert trigger; existing rows (~5.2M) are backfilled by the new idempotent, resumable script in schema-updates/. Bytecodes shorter than 75 bytes and NULL code are excluded by design. The old index stays until the server query switch is deployed and verified; it is dropped in a follow-up migration. Refs #2891
…x table Switch the similarity candidate query from prefix-matching on code to the compiled_contracts_runtime_code_prefixes side table, where every index entry is a candidate and the LIMIT short-circuits after ~limit entries (the code-table scan walked 142,872 rows for 21 candidates in production). Also: - Reject bytecodes shorter than the 75-byte prefix upfront with the new bytecode_too_short_for_similarity error (400) instead of running a doomed job; such bytecodes are not indexed at all. - Normalize library call protection on the search input: deployed libraries carry their own address after the PUSH20 at the start of the runtime code while compiled bytecode has zeros there, so libraries could never prefix-match their compilations before. - Share the prefix length as SIMILARITY_PREFIX_LENGTH_BYTES between the query and the length check. Only effective once the prefix table backfill has run; until then similarity search sees fewer candidates. Refs #2891
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the similarity API's production DB load and timeouts (#2891) in four commits:
perf(server): similarity candidates are keyed on compilation id; payloads are fetched in batches of 5 with early exit on the first match, instead of 20 full payloads upfront via 20 parallel (chain, address) queries. Creation-tx data is resolved once in the main thread; the verification worker no longer performs RPC calls.feat(server): pg statement timeouts (SQLSTATE 57014) from the similarity queries surface assimilarity_search_timeoutinstead of an opaqueinternal_error, making the residual timeout rate measurable after deployment.feat(database): newcompiled_contracts_runtime_code_prefixesside table (first 75 bytes of each compilation's runtime code) with an insert trigger, plus an idempotent, resumable backfill script for the existing ~5.2M compilations.feat(server): candidate search reads the prefix table — every index entry is a candidate, soLIMIT 20short-circuits after ~20 entries instead of walking 142kcoderows (36s measured in production for a V3-pool bytecode). Bytecodes shorter than 75 bytes are rejected upfront with the newbytecode_too_short_for_similarity(400). Library call protection is normalized on the search input so deployed libraries can match their compilations.Full analysis and production measurements in #2891 (see the plan-summary comment). Related planner-statistics findings: #2917.
Deployment (order matters)
20260803100000_add_compiled_contracts_runtime_code_prefixes.sql(fast, metadata-only; the trigger covers new compilations from that moment).services/database/schema-updates/backfill-compiled-contracts-runtime-code-prefixes.mjs(~5.2M rows, ~10–15 min; idempotent and resumable, ends with ANALYZE). See the table inservices/database/README.md.idx_code_code_first_75is intentionally kept: the old query needs it until the new server is deployed, and it is our rollback insurance afterwards. It is dropped in a follow-up migration once the new query is verified in production.Notes
sourcify-databaseshould carry a WARNING line about running the backfill script (wording in Similarity API: reduce DB load from redundant candidate detail queries #2891, following the 2.13.0 precedent).LATERAL ... LIMIT 1asEXISTS— see the comment inDatabase.tsand the production plan in Similarity API: reduce DB load from redundant candidate detail queries #2891.