feat: cut simulator over to generated bb-avm-sim IPC service by charlielye · Pull Request #23697 · AztecProtocol/aztec-packages

charlielye · 2026-05-29T14:10:04Z

Summary

Cuts the public simulator over from the legacy in-process NAPI AVM path to the generated @aztec/bb-avm-sim IPC service package introduced in #23084.

The simulator now drives a pool of out-of-process AVM instances over IPC and owns a TypeScript CDB IPC server for contract-data callbacks, while world-state access continues through the generated wsdb package/service from the lower stack.

Stack

Merged into next: #23610 cl/ipc-foundation · #23611 cl/ipc-wsdb-migrate · #23036 cl/ipc-3-avm-wsdb-cutover · #23084 cl/ipc-4-avm-binary

This chain (bottom → top):

feat: cut simulator over to generated bb-avm-sim IPC service #23697 cl/ipc-5-avm-cutover — this PR (base next)
refactor: in-memory MemoryMerkleDB reference; decouple vm2 + AVM fuzzer from world_state #24306 cl/ipc-6-memory-merkle-db — in-memory MemoryMerkleDB reference; drops world_state from vm2
refactor: extract wsdb/lmdblib/kvdb into native-packages, decoupled from barretenberg #24392 cl/wsdb-decouple — extract wsdb/lmdblib/kvdb into native-packages/, decoupled from barretenberg headers

Parallel IPC stack (independent, also based on next): #23612 cl/ipc-bb-migrate → #23613 cl/ipc-bb-rs-migrate → #23614 cl/ipc-bb-js-migrate.

What changes

AVM simulator cutover

Adds AvmSimulatorPool, which owns the out-of-process @aztec/bb-avm-sim instances and implements a small AvmSimulator interface (simulate / simulateWithHints). Checkout/return is fully internal: simulate blocks until an instance is free and releases it when done, so callers never see the pool mechanics or the IPC transport.
Bundles the pool and the CDB server into an AvmExecutor facade. AvmExecutor.forFork(forkId, contractsDB, timestamp) hands out a ForkedAvmSimulator that registers the fork's contracts DB with the CDB server for the duration of each simulate call. The executor is spawned once and injected into the node, public processor, checkpoint builder, TXE, and the fuzzer.
Adds AbortSignal-based cancellation end to end: PublicTxSimulator cancels via an AbortController, which signals the external C++ AVM process (SIGUSR1) to stop at the next opcode / before the next world-state write, replacing the old in-process cancellation.
Renames socket-specific parameters to generic IPC paths where the value may be UDS or SHM.
Uses explicit async disposal (AsyncDisposable / await using) for the AVM executor, CDB server, wsdb service, and the tests/fixtures that own those resources.

CDB server generation

Converts the AVM and CDB schemas to the newer friendly schema format.
Removes the unused CDB shutdown command; process backends are controlled by normal process signalling.
Generates TypeScript CDB server command/response types and request dispatch from the CDB schema.
Backs CdbIpcServer with the shared UdsIpcServer transport from @aztec/ipc-runtime (socket, framing, per-connection response ordering) rather than hand-rolled framing; CdbIpcServer only implements the generated handler interface.
Ignores generated CDB TypeScript output and produces it from the simulator generation step during bootstrap.

Legacy NAPI cleanup

Deletes the old AVM NAPI simulation implementation.
Deletes the remaining msgpack client side-channel code from nodejs_module.
Removes AVM simulation exports from @aztec/native, leaving the remaining native module focused on the native pieces that still live there.

Validation

yarn build from yarn-project/simulator
make bb-avm-sim
cmake --build build --target bb-avm-sim nodejs_module from barretenberg/cpp
yarn build from yarn-project
./bootstrap.sh format --check simulator from yarn-project
JEST_MAX_WORKERS=1 yarn test public/cdb_ipc_server.test.ts public/public_processor/apps_tests/token.test.ts public/public_tx_simulator/apps_tests/token.test.ts public/public_processor/apps_tests/deployments.test.ts public/public_processor/public_processor.test.ts from yarn-project/simulator

socket-security · 2026-06-18T15:57:45Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/@aztec/bb-avm-sim@0.0.0-use.local	N/A	N/A	N/A	N/A	N/A

View full report

charlielye · 2026-07-02T19:51:45Z

-              BlockNumber(block.number - 1),
-              blockIndex === 0 ? l1ToL2Messages : undefined,
-            );
-            this.checkState();


What checkState() is

A cooperative-cancellation poll: it throws HaltExecutionError if this.state is terminal (timed-out/stopped/failed/reorg), set by stop() on deadline/reorg (epoch-proving-job.ts:406-410). It's "have I been told to quit? if so, bail here."

What actually changed

On next, the inner block loop had three of them:

after createFork, before building the config

after processTxs, before addTxs

after the block scope closed

In the new version:

The one after processTxs moved into processTxs — it's now at epoch-proving-job.ts:511, right after publicProcessor.process(...) returns. Not deleted, relocated.

The checkpoint-level check survives at line 177, at the top of each processCheckpoints iteration.

The other two were dropped as redundant. One guarded a stretch of pure synchronous work (building a PublicSimulatorConfig literal — instant, nothing to interrupt); the other was immediately after the block scope, already covered by the next iteration's checks.

Why polling matters less now

The real cancellation path changed. processTxs now passes the abort signal into the processor:

await publicProcessor.process(txs, { deadline, signal: this.abortController.signal }); // :507-509

stop() calls this.abortController.abort() (:428). So a timeout/reorg now interrupts the actual long-running work — publicProcessor.process observes the deadline/abort, and on PublicProcessorAbortError/PublicProcessorTimeoutError it drives the AVM cancel path (public_processor.ts:346-357 → publicTxSimulator.cancel() → the SIGUSR1 mechanism we just discussed).

Under the old NAPI model the simulation was an in-process blocking call that didn't itself honor an abort, so the only way to bail was to poll checkState() at the statement boundaries around it. Now that cancellation reaches into the simulation itself, sprinkling polls between cheap synchronous steps buys nothing — the one check that matters (after the awaitable work returns) is kept, inside processTxs.

One tangential thing in the same diff: await using db became an explicit try/finally with unregisterFork(...) + db.close() (:246-257). That's the IPC/CDB requirement — the CDB server routes callbacks by forkId, so forks must be explicitly unregistered, which await using couldn't express. Unrelated to the checkState removal but part of the same rewrite.

charlielye · 2026-07-02T19:59:31Z

@@ -29,6 +28,9 @@ import type { ProverNodeJobMetrics } from '../metrics.js';
 import type { ProverNodePublisher } from '../prover-node-publisher.js';
 import { type EpochProvingJobData, validateEpochProvingJobData } from './epoch-proving-job-data.js';

+/** Default parallelism for processing checkpoints. The AVM pool already limits concurrent processes. */
+const DEFAULT_PARALLEL_CHECKPOINT_LIMIT = parseInt(process.env.AVM_MAX_CONCURRENT_SIMULATIONS ?? '4', 10);


The old code imported the constant AVM_MAX_CONCURRENT_SIMULATIONS from @aztec/native. This PR's legacy-NAPI cleanup deletes AVM simulation from @aztec/native, so that import no longer exists. The prover-node now reads the same env var directly:

const DEFAULT_PARALLEL_CHECKPOINT_LIMIT = parseInt(process.env.AVM_MAX_CONCURRENT_SIMULATIONS ?? '4', 10);

So it's the same underlying value (AVM_MAX_CONCURRENT_SIMULATIONS, default 4), just sourced from the environment instead of a native constant. The new name describes what the number means at this call site — "how many checkpoints to fan out in parallel" — rather than borrowing the native module's simulation-cap name. The comment (:31) spells out the reasoning: "The AVM pool already limits concurrent processes" — i.e., this is now just an upstream fan-out default, not the actual hard cap on AVM concurrency.

charlielye · 2026-07-02T20:24:10Z

-      this.simulator = new MeasuredPublicTxSimulator(merkleTree, contractsDB, globals, this.metrics, config);
+      // No simulator — this tester can only be used for setup (setFeePayerBalance, createTx, etc.)
+      // To simulate, use PublicTxSimulationTester.create() or pass a simulatorFactory.
+      this.simulator = undefined;


Post-IPC-cutover the constructor can no longer build a working simulator: MeasuredPublicTxSimulator now requires an already-spawned AvmSimulator (external bb-avm-sim process + WSDB/CDB IPC), which can only be created asynchronously in create(). So the sync constructor degrades to setup-only mode (createTx / setFeePayerBalance); simulateTx throws a clear error if you try to simulate without one. Not a regression — real simulation goes through create() or an explicit simulatorFactory.

charlielye · 2026-07-02T20:24:16Z

    feePayer?: AztecAddress,
    privateInsertions?: TestPrivateInsertions,
-    gasLimits?: Gas,
  ): Promise<PublicTxResult> {


Dropped the trailing gasLimits? param from this wrapper (and from executeTxWithLabel): no caller of these base-class label wrappers ever passed it. Custom gas limits are still fully supported via createTx / simulateTx, and the AvmProvingTester proving path keeps gas control through its simProveVerify* methods. So this was an unused pass-through, not a loss of functionality.

Replace the in-process NAPI AVM with an out-of-process bb-avm-sim binary reached over IPC. The simulator drives a pool of bb-avm-sim processes (AvmSimulatorPool) behind an AvmExecutor that also runs the CDB server answering the C++ AVM's contract-data callbacks; per-fork work goes through AvmExecutor.forFork. The pool prewarms one process on spawn so the first simulate() runs at steady-state cost. Also adds the bench_compare script + bench-regression skill for PR perf checks.

The public-execution wiring threaded a concrete `AvmExecutor` (pool + CDB server) through the node, prover-node, checkpoint builder, and TXE, and each call site called `forFork(forkId, contractsDB, timestamp)` to get a fork-bound `ForkedAvmSimulator`. That leaked the IPC transport (the CDB server, its fork-id routing key, and the register/unregister dance) into every consumer of public simulation. Thread a single transport-agnostic `AvmSimulator` interface instead, mirroring the pre-IPC shape where the factory threaded `(fork, contractsDB)` and called an in-process simulate. `simulate` now takes an `AvmContractsDBContext` ({contractsDB, forkId, timestamp}); the implementation owns the CDB register -> run -> unregister lifecycle. The pool (`AvmSimulatorPool`) folds in the CDB server and is the sole concrete `AvmSimulator`, spawned only at program entrypoints. `AvmExecutor`, `forFork`, and `ForkedAvmSimulator` are gone. `PublicTxSimulatorBase` gains `contractsDB`; consumers (PublicProcessorFactory, prover-node, checkpoint builder, TXE synchronizer) hold the `AvmSimulator` interface and pass the DBs, deriving forkId from the fork's revision.

fcarreiro

Approved with minor comments.

fcarreiro · 2026-07-06T09:46:38Z

 * This token is used to signal cancellation from TypeScript to C++ when a simulation
 * times out. The token is created in TypeScript, passed through NAPI, and checked
 * periodically during C++ execution.


Not NAPI/TS anymore I suppose? (this whole comment may be stale now)

fcarreiro · 2026-07-06T10:36:21Z

+  /** IPC backends to clean up on stop (CDB, AVM). WSDB is cleaned up by world state. */
+  private ipcBackends: Array<{ destroy?(): Promise<void> }> = [];


Every other subservice is explicitly stored and stopped (see stop()). Maybe we should just do that? Also no need to call it IPC, it's just something with a destroy().

fcarreiro · 2026-07-08T09:01:25Z

+// process and signals only the in-flight one, so this isn't exercised today;
+// hardening it would need a request-scoped cancel channel rather than a signal.
+const avm2::simulation::CancellationTokenPtr g_sim_cancellation_token =
+    std::make_shared<avm2::simulation::CancellationToken>();


Does this need to be a shared_ptr now that it's global?

fcarreiro · 2026-07-08T09:30:04Z

+   * Run a fast simulation and return the msgpack-encoded result. If `signal` aborts, the simulation stops
+   * at the next cancellation checkpoint.
+   */
+  simulate(inputBuffer: Uint8Array, signal?: AbortSignal): Promise<Uint8Array>;


As you mentioned, this whole interface/IPC impl should probably be autogenerated by the codegen, and should take actual types derived from the schema (that then get messagepacked internally). I understand this will (likely) be done in a later PR so it's ok.

fcarreiro · 2026-07-09T15:19:11Z

    const config = { ...inputConfig }; // Copy the config so we dont mutate the input object
    const log = deps.logger ?? createLogger('node');
-    const packageVersion = getPackageVersion();
+    const packageVersion = getPackageVersion() ?? '';


Is this from your PR? Why was it needed?

fcarreiro · 2026-07-09T15:19:41Z

        rollupVersion: BigInt(config.rollupVersion),
        l1GenesisTime,
        slotDuration: Number(slotDuration),
+        rollupManaLimit,


Why did you need this?

fcarreiro · 2026-07-09T15:25:41Z

+        this.return(simulator);
+      }
+    } finally {
+      this.cdbServer.unregisterFork(context.forkId);


fcarreiro · 2026-07-09T15:27:05Z

    protected readonly metrics: ExecutorMetricsInterface,
    config?: Partial<PublicSimulatorConfig>,
    bindings?: LoggerBindings,
+    forkId?: number,


I still find it strange to have this here (and in the base classes)

…rrency Two leaks/bugs surfaced by a full-CI process-tree capture of the AVM cutover: 1. TXE session leak. `TXESession.dispose()` closed the world state and LMDB but never stopped the synchronizer's AVM pool, so every test orphaned its prewarmed bb-avm-sim process — and, because that process kept a live connection to the WSDB server, pinned the WSDB alive too. Over a run this accumulated to hundreds of undisposed process-triples. Dispose the pool (`closeIpc()`) first, so the bb-avm-sim processes exit and release their WSDB connections before the WSDB is closed. 2. Pool overshoot under concurrency. `AvmSimulatorPool` bumped `createdCount` only after the `await` that spawns a process, so concurrent checkouts could all observe `createdCount < maxSize` and each spawn, growing the pool past maxSize (a pool was seen holding maxSize+1 processes in CI). Reserve the count synchronously at the top of `createSlot`, before the spawn, and roll it back if the spawn fails, so the maxSize guard is exclusive.

AztecBot · 2026-07-09T16:39:16Z

Flakey Tests

🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/ac2929b42c58ccc6�ac2929b42c58ccc68;;�): yarn-project/kv-store/scripts/run_test.sh src/sqlite-opfs/internal/ordered-binary-browser.test.ts (1s) (code: 0)

charlielye requested review from IlyasRidhuan, MirandaWood and jeanmon as code owners May 29, 2026 14:10

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 1c8d4f9 to a4e2e70 Compare May 29, 2026 14:19

charlielye force-pushed the cl/ipc-4-avm-binary branch from 6867e96 to 65f9aed Compare May 30, 2026 13:39

charlielye force-pushed the cl/ipc-5-avm-cutover branch from a4e2e70 to 0df283e Compare May 30, 2026 13:39

charlielye force-pushed the cl/ipc-4-avm-binary branch from 65f9aed to 47b1d3b Compare May 30, 2026 16:57

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 0df283e to 42d8d4f Compare May 30, 2026 16:57

charlielye force-pushed the cl/ipc-4-avm-binary branch from 47b1d3b to 43f900d Compare May 30, 2026 18:30

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 42d8d4f to 7f602eb Compare May 30, 2026 18:30

charlielye force-pushed the cl/ipc-4-avm-binary branch from 43f900d to 85b52b4 Compare May 30, 2026 22:27

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 7f602eb to 498b1ca Compare May 30, 2026 22:27

charlielye force-pushed the cl/ipc-4-avm-binary branch from 85b52b4 to fac9421 Compare May 31, 2026 13:23

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 498b1ca to 4597920 Compare May 31, 2026 13:23

charlielye force-pushed the cl/ipc-4-avm-binary branch from fac9421 to 8b51cfa Compare June 1, 2026 09:48

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 4597920 to 4a53618 Compare June 1, 2026 09:48

charlielye force-pushed the cl/ipc-4-avm-binary branch from 8b51cfa to cbffb29 Compare June 1, 2026 11:39

charlielye force-pushed the cl/ipc-5-avm-cutover branch 2 times, most recently from af3dc85 to 8cc9db5 Compare June 1, 2026 11:54

charlielye force-pushed the cl/ipc-4-avm-binary branch 2 times, most recently from a9908cb to 665dc82 Compare June 1, 2026 15:35

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 8cc9db5 to 3fe908c Compare June 1, 2026 15:54

charlielye force-pushed the cl/ipc-4-avm-binary branch from 665dc82 to 77496a4 Compare June 1, 2026 22:39

charlielye force-pushed the cl/ipc-5-avm-cutover branch 2 times, most recently from 1ea909f to e7f5dc1 Compare June 2, 2026 15:45

charlielye force-pushed the cl/ipc-4-avm-binary branch from 77496a4 to 8b2448a Compare June 2, 2026 15:45

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 2358098 to b248948 Compare June 2, 2026 16:28

charlielye force-pushed the cl/ipc-4-avm-binary branch from 8de6b5d to 389063c Compare June 2, 2026 17:19

charlielye force-pushed the cl/ipc-5-avm-cutover branch from b248948 to 9835c8c Compare June 2, 2026 17:26

charlielye added the ci-full Run all master checks. label Jun 4, 2026

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 3735bd8 to aae8fc7 Compare June 11, 2026 18:37

charlielye force-pushed the cl/ipc-4-avm-binary branch from 3ab720f to 7c38799 Compare June 11, 2026 18:52

charlielye force-pushed the cl/ipc-5-avm-cutover branch from aae8fc7 to 476ea6b Compare June 11, 2026 18:52

charlielye force-pushed the cl/ipc-4-avm-binary branch from 7c38799 to 6b222ba Compare June 11, 2026 19:05

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 476ea6b to f14b0ba Compare June 11, 2026 19:05

charlielye force-pushed the cl/ipc-4-avm-binary branch from 6b222ba to a53f9a5 Compare June 11, 2026 19:17

charlielye force-pushed the cl/ipc-5-avm-cutover branch from f14b0ba to 7c17ac3 Compare June 11, 2026 19:17

charlielye force-pushed the cl/ipc-4-avm-binary branch from a53f9a5 to 4d7c55c Compare June 11, 2026 19:57

charlielye force-pushed the cl/ipc-5-avm-cutover branch 2 times, most recently from 328033c to c947bce Compare June 11, 2026 20:25

charlielye force-pushed the cl/ipc-4-avm-binary branch from 4d7c55c to 6793497 Compare June 12, 2026 10:48

charlielye force-pushed the cl/ipc-5-avm-cutover branch from c947bce to 15daef1 Compare June 12, 2026 10:48

charlielye force-pushed the cl/ipc-4-avm-binary branch from 6793497 to bf67554 Compare June 12, 2026 11:04

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 15daef1 to 3484d44 Compare June 12, 2026 11:04

charlielye force-pushed the cl/ipc-4-avm-binary branch from bf67554 to a0707df Compare June 12, 2026 11:51

charlielye force-pushed the cl/ipc-5-avm-cutover branch 2 times, most recently from 456e460 to 6094676 Compare June 12, 2026 12:23

charlielye force-pushed the cl/ipc-4-avm-binary branch 2 times, most recently from 1b4352c to ebe1ab2 Compare June 12, 2026 12:56

charlielye force-pushed the cl/ipc-5-avm-cutover branch from 6094676 to 0562481 Compare June 12, 2026 12:56

charlielye commented Jul 2, 2026

View reviewed changes

charlielye added 2 commits July 7, 2026 15:13

fcarreiro approved these changes Jul 9, 2026

View reviewed changes

		/** IPC backends to clean up on stop (CDB, AVM). WSDB is cleaned up by world state. */
		private ipcBackends: Array<{ destroy?(): Promise<void> }> = [];

Uh oh!

Conversation

charlielye commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

What changes

AVM simulator cutover

CDB server generation

Legacy NAPI cleanup

Validation

Uh oh!

socket-security Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fcarreiro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AztecBot commented Jul 9, 2026

Flakey Tests

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

charlielye commented May 29, 2026 •

edited

Loading

socket-security Bot commented Jun 18, 2026 •

edited

Loading