feat: merge-train/fairies-v5

docs/docs-developers/docs/resources/migration_notes.md

@@ -9,11 +9,26 @@ Aztec is in active development. Each version may introduce breaking changes that
 
 ## TBD
 
-### [PXE] Local PXE database is reset on upgrade
+### [Aztec.nr] `set_as_fee_payer` now asserts it is called during the setup phase
 
-The persisted tagging stores now key every entry by the self-describing `<kind>:<secret>:<app>` form of `AppTaggingSecret`; unconstrained secrets previously used a two-part `<secret>:<app>` key. This bumps the PXE data schema version, and there is no forward migration for the old keys: on first open the PXE clears any database whose stored schema version differs from the current one. The wipe resets the entire PXE store, not just the tagging data, because all of it shares one backing database.
+`PrivateContext::set_as_fee_payer` now asserts that execution is still in the setup (non-revertible) phase, i.e. that `end_setup` has not yet been called by any function in the transaction. Electing a fee payer in the revertible phase was never safe: compensation collected by the fee payer after `end_setup` can be discarded if a public call later reverts, while the protocol still debits the fee payer's fee-juice balance.
 
-**Impact**: On upgrade your local PXE state is reset. You must re-register accounts and re-sync from genesis. Wallets should surface a "your local state was reset, please re-register accounts and re-sync" path.
+**Impact**: A transaction in which `set_as_fee_payer` runs after the setup phase has ended now fails with `fee payer must be elected during the setup phase`. Standard fee payment flows, which call `set_as_fee_payer` before or together with `end_setup`, are unaffected.
+
+### [PXE] Stores are now selected by `(l1ChainId, rollupAddress, schemaVersion)` instead of being wiped on mismatch
+
+Previously, connecting a PXE or embedded wallet to a different or redeployed rollup, or bumping the store schema version, wiped the existing on-disk store in place. That meant master account keys could be destroyed simply by pointing a wallet at a different network. PXE data stores now exist per `(l1ChainId, rollupAddress, schemaVersion)` triple, and switching networks (or upgrading) selects or creates the matching store instead of overwriting previous ones. The embedded wallet's `wallet_data` store is partitioned the same way, so accounts and aliases are per network: switching networks starts with an empty account list until accounts are re-imported, and switching back finds the originals intact.
+
+**Impact**: The first start after upgrading to this version begins with a fresh, empty store; the pre-upgrade data is not deleted. On Node.js environments (lmdb-v2) pre-upgrade data stays at `<dataDirectory>/<name>` while new per-identity `pxe_data` stores live under `<dataDirectory>/<name>-stores/`. The embedded Node.js wallet previously stored data in cwd-relative, rollup-address-suffixed directories instead (`pxe_data_<rollupAddress>/pxe_data` for the PXE store, `wallet_data_<rollupAddress>/wallet_data` for the wallet store): if you used it before this release, that is where the old data lives. The embedded wallet now defaults its data root to `aztec-wallet-data/`, with per-identity wallet stores under `<dataDirectory>/wallet_data-stores/` on Node.js and OPFS store names prefixed `wallet_data_` in the browser. Browser apps can enumerate and clean up `pxe_data` and `wallet_data` stores for networks no longer in use with the new `listStores()` / `deleteStore()` utilities:
+
+```ts
+import { deleteStore, listStores } from '@aztec/kv-store/sqlite-opfs';
+
+const names = await listStores();
+await deleteStore(names[0]);
+```
+
+This change also removes `createStore` from `@aztec/kv-store/sqlite-opfs` and `@aztec/kv-store/deprecated/indexeddb`: stores are now opened by name with `AztecSQLiteOPFSStore.open` / `AztecIndexedDBStore.open`.
 
 ### [Aztec.nr] `TestEnvironmentOptions::with_tagging_secret_strategy` replaced
 
@@ -131,8 +146,6 @@ Registering classes and instances are now separate, unvalidated operations. `reg
   The new class is used automatically once the upgrade takes effect on chain; no further PXE action is needed. Registering it beforehand is harmless: until the update activates, the node still resolves the contract's current class to the previous one, so it keeps running its old code.
 
 - `pxe.getContractInstance(address)` and `wallet.getContractMetadata(address).instance` now return the contract's **address preimage**, which no longer includes `currentContractClassId`.
-
-
 ### [Aztec.js] `AccountWithSecretKey` removed, read account keys from the `AccountManager` or PXE
 
 `AccountWithSecretKey` was a thin wrapper that bundled an account's transaction signer with its master secret key, used mainly to print or export the secret. It has been removed, and `AccountManager.getAccount()` now returns the plain `Account` signer. The wrapper's extra methods are no longer available on that value:

noir-projects/aztec-nr/aztec/src/context/private_context.nr

@@ -537,11 +537,22 @@ impl PrivateContext {
     /// Only one contract per transaction can declare itself as the fee payer, and it must have sufficient fee-juice
     /// balance (>= the gas limits specified in the TxContext) by the time we reach the public setup phase of the tx.
     ///
+    /// The fee payer must be elected during the setup (non-revertible) phase, i.e. before
+    /// [`end_setup`](PrivateContext::end_setup) is called - this function asserts so. This is because any compensation
+    /// collected by the fee payer during the revertible phase can be discarded if a public call later reverts, while
+    /// the protocol still debits the fee payer's fee-juice balance. Note that `end_setup` does not need to be called
+    /// by the electing function itself: it can be called later in the transaction (e.g. by the fee-juice contract when
+    /// claiming fee juice that pays for the very same transaction).
     pub fn set_as_fee_payer(&mut self) {
+        assert(!self.in_revertible_phase(), "fee payer must be elected during the setup phase");
         aztecnr_trace_log_format!("Setting {0} as fee payer")([self.this_address().to_field()]);
         self.is_fee_payer = true;
     }
 
+    /// Returns whether execution is currently in the revertible (app) phase of the transaction.
+    ///
+    /// A transaction is in the revertible phase if [`end_setup`](PrivateContext::end_setup) has already been called -
+    /// potentially by a different function of the same transaction.
     pub fn in_revertible_phase(&mut self) -> bool {
         let current_counter = self.side_effect_counter;
 

noir-projects/aztec-nr/aztec/src/oracle/message_processing.nr

@@ -66,3 +66,11 @@ pub unconstrained fn get_tx_effect(tx_hash: Field) -> Option<TxEffect> {
 
 #[oracle(aztec_utl_getTxEffect)]
 unconstrained fn get_tx_effect_oracle(tx_hash: Field) -> Option<TxEffect> {}
+
+/// Fetches all effects of settled transactions by hash, preserving request order.
+pub unconstrained fn get_tx_effects(tx_hashes: EphemeralArray<Field>) -> EphemeralArray<Option<TxEffect>> {
+    get_tx_effects_oracle(tx_hashes)
+}
+
+#[oracle(aztec_utl_getTxEffects)]
+unconstrained fn get_tx_effects_oracle(tx_hashes: EphemeralArray<Field>) -> EphemeralArray<Option<TxEffect>> {}

noir-projects/aztec-nr/aztec/src/oracle/version.nr

@@ -11,7 +11,7 @@
 /// immediately if AZTEC_NR_MINOR > PXE_MINOR because if a contract is updated to use a newer Aztec.nr dependency
 /// without actually using any of the new oracles then there is no reason to throw.
 pub global ORACLE_VERSION_MAJOR: Field = 30;
-pub global ORACLE_VERSION_MINOR: Field = 7;
+pub global ORACLE_VERSION_MINOR: Field = 8;
 
 /// Asserts that the version of the oracle is compatible with the version expected by the contract.
 pub fn assert_compatible_oracle_version() {

playground/src/components/navbar/components/NetworkSelector.tsx

@@ -7,7 +7,7 @@ import AddIcon from '@mui/icons-material/Add';
 import { AddNetworksDialog } from './AddNetworkDialog';
 import CircularProgress from '@mui/material/CircularProgress';
 import KeyboardArrowDownIcon from '@mui/icons-material/KeyboardArrowDown';
-import { createStore } from '@aztec/kv-store/sqlite-opfs';
+import { AztecSQLiteOPFSStore, storePoolDirectory } from '@aztec/kv-store/sqlite-opfs';
 import { AztecContext } from '../../../aztecContext';
 import { navbarButtonStyle, navbarSelect } from '../../../styles/common';
 import { NETWORKS } from '../../../utils/networks';
@@ -50,10 +50,12 @@ export function NetworkSelector() {
       }
       setIsContextInitialized(true);
       WebLogger.create(setLogs, setTotalLogCount);
-      const store = await createStore('playground_data', {
-        dataDirectory: 'playground',
-        dataStoreMapSizeKb: 1e6,
-      });
+      const store = await AztecSQLiteOPFSStore.open(
+        WebLogger.getInstance().createLogger('playground_data'),
+        'playground_data',
+        false,
+        storePoolDirectory('playground_data'),
+      );
       const playgroundDB = PlaygroundDB.getInstance();
       playgroundDB.init(store, WebLogger.getInstance().createLogger('playground_db').info);
       setPlaygroundDB(PlaygroundDB.getInstance());

yarn-project/end-to-end/src/automine/phase_check.parallel.test.ts

@@ -1,4 +1,5 @@
 import { AztecAddress } from '@aztec/aztec.js/addresses';
+import { BatchCall } from '@aztec/aztec.js/contracts';
 import { SponsoredFeePaymentMethod } from '@aztec/aztec.js/fee';
 import { SPONSORED_FPC_SALT } from '@aztec/constants';
 import { Fr } from '@aztec/foundation/curves/bn254';
@@ -7,11 +8,23 @@ import { TestContract } from '@aztec/noir-test-contracts.js/Test';
 import { computeFeePayerBalanceLeafSlot } from '@aztec/protocol-contracts/fee-juice';
 import { getContractInstanceFromInstantiationParams } from '@aztec/stdlib/contract';
 import { PublicDataTreeLeaf } from '@aztec/stdlib/trees';
+import { ExecutionPayload } from '@aztec/stdlib/tx';
 import { defaultInitialAccountFeeJuice } from '@aztec/world-state/testing';
 
 import type { TestWallet } from '../test-wallet/test_wallet.js';
 import { AutomineTestContext } from './automine_test_context.js';
 
+/**
+ * Declares a contract as the tx fee payer without contributing any call to the fee payload, unlike
+ * SponsoredFeePaymentMethod which prepends the sponsor call (and thus makes the election run before any app call).
+ * This lets a test place the electing call anywhere in the app payload, e.g. after the setup phase has ended.
+ */
+class DeferredSponsoredFeePaymentMethod extends SponsoredFeePaymentMethod {
+  override async getExecutionPayload(): Promise<ExecutionPayload> {
+    return new ExecutionPayload([], [], [], [], await this.getFeePayer());
+  }
+}
+
 // Private functions should receive automatically a phase check that avoids any nested call changing the phase.
 // Functions that opt out of this phase check can be marked with #[allow_phase_change].
 //
@@ -78,4 +91,21 @@ describe('automine/phase_check', () => {
       },
     });
   });
+
+  it('should fail when the fee payer is elected after the setup phase has ended', async () => {
+    // BatchCall.simulate ignores the fee payment method, which would make the wallet fall back to
+    // PREEXISTING_FEE_JUICE and end setup in the account entrypoint before any app call runs. Build the payload via
+    // request() instead, which merges the payment method's payload (and thus its fee payer), and simulate it directly.
+    const lateElection = await new BatchCall(wallet, [
+      contract.methods.call_function_that_ends_setup_without_phase_check(),
+      sponsoredFPC.methods.sponsor_unconditionally(),
+    ]).request({
+      fee: {
+        paymentMethod: new DeferredSponsoredFeePaymentMethod(sponsoredFPC.address),
+      },
+    });
+    await expect(wallet.simulateTx(lateElection, { from: defaultAccountAddress })).rejects.toThrow(
+      'fee payer must be elected during the setup phase',
+    );
+  });
 });

yarn-project/kv-store/scripts/run-browser-tests.sh

@@ -15,7 +15,9 @@ set -euo pipefail
 
 cd "$(dirname "$0")/.."
 
-files=$(find src/deprecated/indexeddb src/sqlite-opfs -name '*.test.ts' 2>/dev/null | sort)
+# -type f: vitest failure screenshots land in __screenshots__/<test-file>/, creating
+# directories whose names match the *.test.ts glob.
+files=$(find src/deprecated/indexeddb src/sqlite-opfs -type f -name '*.test.ts' 2>/dev/null | sort)
 
 if [ -z "$files" ]; then
   echo "No test files found in src/deprecated/indexeddb or src/sqlite-opfs"

yarn-project/kv-store/src/deprecated/indexeddb/index.ts

@@ -1,34 +1,9 @@
-import { type Logger, createLogger } from '@aztec/foundation/log';
-import type { DataStoreConfig } from '@aztec/stdlib/kv-store';
+import { createLogger } from '@aztec/foundation/log';
 
-import { initStoreForRollupAndSchemaVersion } from '../../utils.js';
 import { AztecIndexedDBStore } from './store.js';
 
 export { AztecIndexedDBStore } from './store.js';
 
-/**
- * @deprecated The IndexedDB backend is being retired. Use `@aztec/kv-store/sqlite-opfs` instead.
- */
-export async function createStore(
-  name: string,
-  config: DataStoreConfig,
-  schemaVersion: number | undefined = undefined,
-  log: Logger = createLogger('kv-store'),
-) {
-  let { dataDirectory } = config;
-  if (typeof dataDirectory !== 'undefined') {
-    dataDirectory = `${dataDirectory}/${name}`;
-  }
-
-  log.info(
-    dataDirectory
-      ? `Creating ${name} data store at directory ${dataDirectory} with map size ${config.dataStoreMapSizeKb} KB`
-      : `Creating ${name} ephemeral data store with map size ${config.dataStoreMapSizeKb} KB`,
-  );
-  const store = await AztecIndexedDBStore.open(createLogger('kv-store:indexeddb'), dataDirectory ?? '', false);
-  return initStoreForRollupAndSchemaVersion(store, schemaVersion, config.rollupAddress, log);
-}
-
 /**
  * @deprecated The IndexedDB backend is being retired. Use `@aztec/kv-store/sqlite-opfs` instead.
  */

yarn-project/kv-store/src/sqlite-opfs/index.ts

@@ -1,28 +1,11 @@
-import { type Logger, createLogger } from '@aztec/foundation/log';
-import type { DataStoreConfig } from '@aztec/stdlib/kv-store';
+import { createLogger } from '@aztec/foundation/log';
 
-import { initStoreForRollupAndSchemaVersion } from '../utils.js';
 import { AztecSQLiteOPFSStore } from './store.js';
 
 export { AztecSQLiteOPFSStore } from './store.js';
 export { SqliteEncryptionError } from './errors.js';
 export type { SqliteEncryptionErrorCode } from './errors.js';
-
-export async function createStore(
-  name: string,
-  config: DataStoreConfig,
-  schemaVersion: number | undefined = undefined,
-  log: Logger = createLogger('kv-store'),
-) {
-  const { dataDirectory } = config;
-  log.info(
-    dataDirectory
-      ? `Creating ${name} SQLite-OPFS data store with map size ${config.dataStoreMapSizeKb} KB`
-      : `Creating ${name} ephemeral SQLite-OPFS data store with map size ${config.dataStoreMapSizeKb} KB`,
-  );
-  const store = await AztecSQLiteOPFSStore.open(createLogger('kv-store:sqlite-opfs'), name, false);
-  return initStoreForRollupAndSchemaVersion(store, schemaVersion, config.rollupAddress, log);
-}
+export { OPFS_POOL_DIR_PREFIX, deleteStore, listStores, storePoolDirectory } from './manage.js';
 
 export function openTmpStore(ephemeral: boolean = false): Promise<AztecSQLiteOPFSStore> {
   return AztecSQLiteOPFSStore.open(createLogger('kv-store:sqlite-opfs'), undefined, ephemeral);

yarn-project/kv-store/src/sqlite-opfs/manage.ts

@@ -0,0 +1,36 @@
+/** Prefix for the per-store OPFS SAH pool directories owned by this package. */
+export const OPFS_POOL_DIR_PREFIX = '.aztec-kv-';
+
+/**
+ * OPFS directory holding a store's SAH pool. One directory per store: the SAH-pool VFS allows only one
+ * concurrent instance per directory and its capacity does not grow automatically, so sharing a pool across
+ * stores would make concurrently opened stores contend for locks and orphaned stores exhaust pool slots.
+ */
+export function storePoolDirectory(effectiveName: string): string {
+  return `${OPFS_POOL_DIR_PREFIX}${effectiveName}`;
+}
+
+/**
+ * Lists the names of every persistent sqlite-opfs store in this origin, by enumerating the per-store pool
+ * directories. Includes stores other than the current one, so wallets can surface and clean up data for
+ * networks/versions no longer in use.
+ */
+export async function listStores(): Promise<string[]> {
+  const root = await navigator.storage.getDirectory();
+  const names: string[] = [];
+  for await (const [entryName, handle] of root.entries()) {
+    if (handle.kind === 'directory' && entryName.startsWith(OPFS_POOL_DIR_PREFIX)) {
+      names.push(entryName.slice(OPFS_POOL_DIR_PREFIX.length));
+    }
+  }
+  return names;
+}
+
+/**
+ * Permanently deletes a store by effective name (as returned by {@link listStores}). The store must be closed:
+ * an open store's SAH pool holds locks on the directory and the removal will reject.
+ */
+export async function deleteStore(effectiveName: string): Promise<void> {
+  const root = await navigator.storage.getDirectory();
+  await root.removeEntry(storePoolDirectory(effectiveName), { recursive: true });
+}

yarn-project/kv-store/src/sqlite-opfs/store_management.test.ts

@@ -0,0 +1,68 @@
+import { mockLogger } from '../interfaces/utils.js';
+import { AztecSQLiteOPFSStore } from './index.js';
+import { deleteStore, listStores, storePoolDirectory } from './manage.js';
+
+const openByName = (name: string) => AztecSQLiteOPFSStore.open(mockLogger, name, false, storePoolDirectory(name));
+
+describe('sqlite-opfs store management', () => {
+  it('round-trips data for a store reopened by name', async () => {
+    const store = await openByName('mech_roundtrip');
+    await store.openSingleton<string>('payload').set('data');
+    await store.close();
+
+    const reopened = await openByName('mech_roundtrip');
+    expect(await reopened.openSingleton<string>('payload').getAsync()).toEqual('data');
+    await reopened.close();
+    await deleteStore('mech_roundtrip');
+  });
+
+  it('opens two different stores concurrently in the same tab', async () => {
+    const a = await openByName('mech_concurrent_a');
+    const b = await openByName('mech_concurrent_b');
+
+    await a.openSingleton<string>('k').set('a');
+    await b.openSingleton<string>('k').set('b');
+    expect(await a.openSingleton<string>('k').getAsync()).toEqual('a');
+    expect(await b.openSingleton<string>('k').getAsync()).toEqual('b');
+
+    await a.close();
+    await b.close();
+    await deleteStore('mech_concurrent_a');
+    await deleteStore('mech_concurrent_b');
+  });
+
+  it('lists created stores and deletes them', async () => {
+    const store = await openByName('mech_managed');
+    await store.openSingleton<string>('k').set('v');
+    await store.close();
+
+    expect(await listStores()).toContain('mech_managed');
+    await deleteStore('mech_managed');
+    expect(await listStores()).not.toContain('mech_managed');
+
+    // Recreating after deletion starts empty.
+    const fresh = await openByName('mech_managed');
+    expect(await fresh.openSingleton<string>('k').getAsync()).toBeUndefined();
+    await fresh.close();
+    await deleteStore('mech_managed');
+  });
+
+  // Regression test: close() must not resolve until the worker has released the SAH pool's OPFS
+  // handles, otherwise deleteStore races Chromium's async reclaim of the terminated worker and
+  // intermittently throws NoModificationAllowedError. Looped to amplify the race window.
+  it('deletes a store immediately after close, repeatedly', async () => {
+    for (let i = 0; i < 20; i++) {
+      const store = await openByName('mech_close_release');
+      await store.openSingleton<string>('k').set(`v${i}`);
+      await store.close();
+      await deleteStore('mech_close_release');
+    }
+  });
+
+  it('refuses to delete a store that is currently open', async () => {
+    const store = await openByName('mech_locked');
+    await expect(deleteStore('mech_locked')).rejects.toThrow();
+    await store.close();
+    await deleteStore('mech_locked');
+  });
+});