fix: prevent and auto-recover stuck v2 verification jobs (compiler OOM/hang) (#2880)

[kuzdogan]

Background

Fixes #2880. Nine v2 verification jobs on Avalanche C-Chain (43114) got stuck at isJobCompleted: false for 7+ hours, each blocking resubmission for its address with 429 duplicate_verification_request.

Root cause: a verification worker's native solc subprocess was OOM-killed mid-compile. The immediate trigger is memory (see below), but the reason the job wedged is a bug in asyncExec (packages/compilers): it had no error handler on the child's stdin, so a compiler killed mid-write of the standard-JSON input emitted EPIPE with no listener → an uncaught exception in the Piscina worker thread → the compile promise never rejected. So completed_at was never set. The v2 duplicate check is DB-based (completed_at IS NULL ⇒ "in progress"), so the address stays locked — and because the lock is a DB row, a server restart doesn't clear it.

Investigation against production found this is not Avalanche-specific: ~1,260 jobs are currently stuck with completed_at IS NULL, the oldest ~2 months old, across many chains and both /v2/verify and /v2/verify/similarity.

Related previous issues

• Automate stale job recovery #2242 — Automate stale job recovery: stale jobs accumulate and need to be marked failed; proposes a periodic reaper.
• Compiler subprocess invocations have no execution timeout #2792 — Compiler subprocess invocations have no execution timeout: a hung compiler occupies a worker forever (the "hang" sibling of this OOM case).
• Investigate not being able to repeat verification submissions #2669 — Investigate not being able to repeat verification submissions: repeated submissions return the same stuck verificationId, with no way to force a fresh job.
• Investigate Large Contract Compilation Timeout #2670 — Investigate Large Contract Compilation Timeout: large standard-JSON inputs time out or get permanently stuck at isJobCompleted: false.
• Temporarily store standard json input for failed verifications #2359 (closed issue; implemented in Store failed verification inputs in a S3 bucket for debugging purposes #2613) — Temporarily store standard json input for failed verifications: added S3 storage of the submitted input, but only on settled failures.

The OOM trigger

We set solc outputSelection to {"*":{"*":[…]}} — i.e. output every contract in every source file, with heavy artifacts (evm.legacyAssembly, evm.bytecode.generatedSources, source maps, storage layout, metadata). For a 119-source contract that's ~40 MB of output and ~800 MB of solc Resident Set Size (RSS) per compile; several running concurrently exhaust the container and it gets OOM-killed.

Reproduced with the native linux-amd64 solc 0.8.29 on a real contract from the issue:

outputSelection	output	solc peak RSS
current "*":"*"	39.9 MB	794 MB
scoped to target contract	2.0 MB	187 MB

I recall the "*":"*" (all contracts) was originally needed for extra-file-input handling, but that shouldn't be necessary for newer compilers — so we can likely turn it off, or only enable the broad selection for that specific case. For the common path we should be able to safely output only the target contract.

Fixes (multi-step)

• Scope outputSelection to the compilation target (lib-sourcify) — prevents the OOM at the source: ~20× smaller output, ~4× less solc RSS. Also shrinks storage/network payloads for every verification.
• Compiler subprocess robustness + execution timeout (Compiler subprocess invocations have no execution timeout #2792, compilers) — the deeper root cause. asyncExec now settles exactly once, rejects on stdin error / write failure (so a compiler killed mid-write no longer hangs), and enforces a wall-clock timeout (SOLC_COMPILE_TIMEOUT_MS, default 45 min, SIGKILL). Failures are attributed with dedicated compiler_timeout / compiler_out_of_memory codes. Covers Solidity and Vyper.
• Stale-job reaper (Automate stale job recovery #2242) — a pg_cron scheduled job (every 15 min) marks completed_at IS NULL jobs older than 3 h as failed with a new job_abandoned code, releasing the address lock. Follows the existing refresh-signature-stats pg_cron pattern, with the same graceful fallback when the extension isn't available (pg_cron is already enabled in production). Backed by a partial index verification_jobs (started_at) WHERE completed_at IS NULL — required because the table is ~77 M rows / ~50 GB with no timestamp index, turning the reaper query from a ~4.5-min full sequential scan into a sub-ms lookup (index built CONCURRENTLY).

Considered but deferred

• Compiler subprocess memory cap — bounding the solc child's address space via ulimit -v (opt-in SOLC_MAX_ADDRESS_SPACE_KB, default-off) so solc dies as a clean compiler_out_of_memory before the container OOM-killer targets the Node server. Deferred: ulimit -v caps virtual (not resident) memory, needs ops tuning against real metrics, and risks false-failing legitimate large compiles. Straightforward to add later as an opt-in knob if the outputSelection reduction + timeout prove insufficient under load.

Migration / deploy notes

• Two migrations: (1) CREATE INDEX CONCURRENTLY for the partial index (non-transactional), and (2) the cron.schedule for the reaper. The reaper relies on pg_cron being enabled (as it already is for refresh-signature-stats); on a stack without it the schedule is skipped gracefully and jobs can be reaped by running the same UPDATE manually.
• Regenerate the checked-in services/database/sourcify-database.sql dump after applying.
• One-time cleanup of the ~1,260 existing stuck jobs is a separate operator-run SQL step (sets completed_at + error_code = 'internal_error' — recognized by the currently-deployed server — on jobs older than the threshold), releasing those locks immediately.

Open question: forcing a fresh job (#2669)

Should we let a client force a fresh job instead of returning the same stuck verificationId? IMO we shouldn't need this in the first place — an OOM-killed or otherwise dead process should never leave a job stuck (the reaper + timeouts above should guarantee that). Linking #2669 for discussion rather than committing to it.

Related: #2670 (large-contract compilation timeout).

🤖 Generated with Claude Code

[manuelwedler]

These cases are duplicated across SourcifyLibError and here. Remove the occurrence here.

[manuelwedler]

@ethereum-sourcify/compilers-types is only a devDependency of this package (and of lib-sourcify, which gets the same value imports in AbstractCompilation.ts). That was fine while every import was import type (erased at build), but these are value imports, so the built JS now contains a real require('@ethereum-sourcify/compilers-types').

Both packages are published to npm, and npm doesn't install devDependencies of a dependency — external consumers will crash at load with Cannot find module '@ethereum-sourcify/compilers-types'. It only works in this repo because of workspace hoisting.

Please move @ethereum-sourcify/compilers-types to dependencies in both packages/compilers/package.json and packages/lib-sourcify/package.json.

[manuelwedler]

This timeout only covers asyncExec, i.e. the native-binary path. Two compile paths still have no wall-clock timeout:

• the soljson fallback (worker thread in solidityCompiler.ts), used for versions below the native-binary thresholds
• the Fe compiler (spawnSync, which also blocks synchronously)

A hung compile there still wedges a Piscina worker until the 3h reaper marks the job abandoned — so #2792 is only partially closed by this PR. Probably acceptable given soljson is only used for old versions, but worth stating explicitly whether that's intentional (or noting it in #2792).

[manuelwedler]

This introduces the first functional env var in the published packages. Since #2058 removed process.env from lib-sourcify, all functional configuration in the packages flows explicitly through the API (solcRepoPath, vyperRepoPath, ...) — an ambient env read buried here in asyncExec breaks that convention, and library consumers won't discover it. (Browser compatibility isn't the concern here — compilers is inherently Node-only — just consistency of the config design.)

Consider making it an explicit timeoutMs parameter on useSolidityCompiler/useVyperCompiler, threaded down to asyncExec, and reading the env var in the server's config layer instead. The server already wraps these functions (SolcLocal/VyperLocal), so the parameter doesn't need to touch lib-sourcify. That would also solve two smaller issues with the current shape:

• the name: SOLC_COMPILE_TIMEOUT_MS also governs Vyper compiles since asyncExec is shared — the server config would own a compiler-agnostic name (e.g. COMPILER_TIMEOUT_MS) and document it next to the other settings (right now the variable isn't documented anywhere)
• the tests: common.spec.ts could pass timeoutMs: 150 directly instead of mutating and restoring process.env

[manuelwedler]

Attributing every stdin 'error' to OOM can mislabel unrelated failures: any child that exits without reading its stdin — e.g. a missing/corrupt compiler binary (exit 127) — triggers EPIPE on a large input before the exec callback fires, so it surfaces as compiler_out_of_memory instead of the real spawn/exit error. That could mask e.g. a bad-binary incident as OOM.

An alternative that keeps the crash fix but improves attribution: don't settle here — just record that an stdin error happened (the listener alone prevents the uncaught exception). The exec callback always fires eventually and has better signals (killed, signal, exit code) to attribute the death; only reject from the recorded stdin error in the (practically impossible) case where the callback would otherwise resolve successfully.

[manuelwedler]

Consider extracting the UPDATE into a database function and scheduling a one-line call to it:

CREATE FUNCTION public.reap_stale_verification_jobs(stale_threshold interval DEFAULT '3 hours')
RETURNS integer
LANGUAGE sql
AS $$
  WITH reaped AS (
    UPDATE public.verification_jobs
    SET completed_at = NOW(),
        error_code = 'job_abandoned',
        error_id = gen_random_uuid(),
        verified_contract_id = NULL,
        compilation_time = NULL
    WHERE completed_at IS NULL
      AND started_at < NOW() - stale_threshold
    RETURNING 1
  )
  SELECT count(*)::integer FROM reaped;
$$;

-- in the DO block:
PERFORM cron.schedule('reap-stale-verification-jobs', '*/15 * * * *',
  'SELECT public.reap_stale_verification_jobs();');

Benefits:

• Makes the reaper testable. The test DB (postgres:15-alpine) has no pg_cron, so the schedule is skipped there and the reaper logic is currently untestable without duplicating the SQL. A plain CREATE FUNCTION lands in the test DB through the normal migration path, so a server integration test can exercise the production reaper SQL verbatim.
• The manual fallback becomes a single call returning the reaped count, instead of copy-pasting the UPDATE out of a migration file.
• Matches the existing pattern: refresh-signature-stats schedules a one-liner; the logic stays out of the cron entry. Changing the threshold later is a CREATE OR REPLACE FUNCTION instead of unschedule + reschedule.

When addressing this, please also add the integration test: create a pending job (createMockJob-style in jobs.spec.ts), run SELECT reap_stale_verification_jobs('0 seconds'), then assert that GET /v2/verify/:id returns isJobCompleted: true with error.customCode: "job_abandoned", and that resubmitting the same chain/address no longer returns 429 duplicate_verification_request. This also locks in the subtlety that error_id must be set for the API to build the error object at all (SourcifyDatabaseService requires both error_code and error_id).

(The down migration would gain a DROP FUNCTION IF EXISTS, and the function needs to be in the regenerated sourcify-database.sql dump.)