Reduce CAccountMap NIF surface with map-owned storage and frozen-only lock

.github/workflows/ci.yml

@@ -48,5 +48,5 @@ jobs:
         mix nif.stress.quick
         mix run --no-start scripts/cmerkle_deadlock_watchdog.exs -- \
           --progress-timeout 120 --wall-timeout 900 \
-          mix run --no-start scripts/cmerkle_parallel_stress.exs -- --waves 1 --tasks 24 --scenario P10 --scenario P11
+          mix run --no-start scripts/cmerkle_parallel_stress.exs -- --waves 1 --tasks 24 --scenario P12 --scenario P14
         mix nif.leak.quick

.github/workflows/nif-nightly.yml

@@ -76,4 +76,4 @@ jobs:
           mix run --no-start scripts/cmerkle_deadlock_watchdog.exs -- \
             --progress-timeout 300 --wall-timeout 1200 \
             mix run --no-start scripts/cmerkle_parallel_stress.exs -- \
-              --waves 2 --tasks 24 --scenario P4 --scenario P5 --scenario P10 --scenario P12
+              --waves 2 --tasks 24 --scenario P12 --scenario P13 --scenario P14 --scenario P16

.gitignore

@@ -14,7 +14,6 @@
 /priv/merkletree_nif.so
 /priv/merkletree_nif.asan.so
 /priv/merkletree_nif.so.bak*
-
 # Profiling / measurement outputs (see scripts/profile_*.sh)
 /tmp/
 /leak_state.bin

AGENTS.md

@@ -28,6 +28,10 @@ Ethereum-compatible JSON-RPC endpoint plus the Diode PEER/EDGE protocols.
   `make -C deps/libsecp256k1/` (see `.github/workflows/ci.yml`). Build artifacts
   are gitignored and persist across sessions, so this is only needed after a
   clean checkout of that dep.
+- **CAccountMap / state NIF semantics** (clone, lock, storage APIs, get shape):
+  see [`docs/caccount-map-nif.md`](docs/caccount-map-nif.md). Difference/clone
+  performance (cached compact roots, COW, trie-driven `difference_full`):
+  [`docs/specs/change-state-diff-perf.md`](docs/specs/change-state-diff-perf.md).
 
 ### Lint
 - `mix lint` = `compile` + `mix format --check-formatted` + `mix credo --only warning` + `mix dialyzer`.
@@ -39,12 +43,12 @@ Ethereum-compatible JSON-RPC endpoint plus the Diode PEER/EDGE protocols.
 - `make test` generates test PEM certs, then runs each `test/*_test.exs` file in
   a separate `mix test --max-failures 1` invocation (per-file isolation). Test
   env pins ports `RPC_PORT=18001`, `EDGE2_PORT=18003`, `PEER_PORT=18004`.
-- `Chain.State` is a MUTABLE NIF-backed `CMerkleTree`: `Chain.Transaction.apply/3`
-  mutates the state passed to it in place. Use `Chain.State.clone/1` to get an
-  independent copy before reapplying. `test/evm_test.exs` "create contract"
-  currently fails for this reason (it reuses `state` across `apply` calls
-  without cloning). This is a pre-existing repo issue — CI has `mix test`
-  commented out and does not gate on it.
+- For `Chain.State` / CAccountMap mutability and storage rules, see
+  [`docs/caccount-map-nif.md`](docs/caccount-map-nif.md). Perf contract tests:
+  `test/state_diff_perf_contract_test.exs`. Benches (no app start):
+  `scripts/state_diff_bench.exs` (`State.difference`),
+  `scripts/state_uncompact_bench.exs` (`state(uncompact:…)`),
+  `scripts/state_delta_apply_bench.exs` (`state(delta:…)`).
 
 ### Running the node (dev mode)
 - `./dev` runs `MIX_ENV=dev iex -S mix run` (wipes `data_dev/` first). For a

Makefile

@@ -8,7 +8,8 @@ TESTDATA := test/pems/device1_certificate.pem test/pems/device2_certificate.pem
 .PHONY: all
 all: evm/evm priv/merkletree_nif.so
 
-priv/merkletree_nif.so: $(wildcard c_src/*.cpp c_src/*.hpp)
+.PHONY: priv/merkletree_nif.so
+priv/merkletree_nif.so:
 	$(MAKE) -C c_src nif
 
 evm/evm: $(wildcard evm/*.cpp evm/*.hpp evm/*/*.cpp evm/*/*.hpp)

c_src/LOCK_ORDER.md

@@ -1,111 +1,87 @@
-# CMerkleTree NIF — lock order and deadlock scenario registry
+# CAccountMap NIF — lock order and deadlock scenario registry
 
-This document inventories mutex layers in [`nif.cpp`](nif.cpp), documents acquisition order, and maps every known deadlock / liveness scenario to regression tests.
+This document inventories mutex layers in [`nif.cpp`](nif.cpp), documents acquisition order, and maps known deadlock / liveness scenarios to regression tests.
 
-See also [`SECURITY_REVIEW.md`](SECURITY_REVIEW.md) (F-5 fix) and [`scripts/cmerkle_parallel_stress.exs`](../scripts/cmerkle_parallel_stress.exs).
+See also [`SECURITY_REVIEW.md`](SECURITY_REVIEW.md) and [`scripts/cmerkle_parallel_stress.exs`](../scripts/cmerkle_parallel_stress.exs).
 
 ## Mutex layers
 
 | Mutex | Type | Scope | Used by |
 |-------|------|-------|---------|
-| `LockedStates::mtx` | `ErlNifMutex` | Global | `enter_lock`, `leave_lock`, `destruct_merkletree_type` |
-| `SharedState::mtx` | `ErlNifMutex` | Per trie | `Lock(mt)` — insert, get, difference, clone, etc. |
+| `SharedState::mtx` | `ErlNifMutex` | Per storage / state trie | `Lock(mt)` during COW / storage reads / root hash; destructor `release_merkletree_shared` |
 | `SharedAccountMap::mtx` | `ErlNifMutex` | Per account map | `AccountMapLock` |
 | `GlobalStripePool::s_mtx` | `std::mutex` | Process-global | PreAllocator stripe reuse |
 | `ItemPool` internal | `std::recursive_mutex` | Per SharedState pool | COW / pair allocation |
 | `PreAllocator` internal | `std::recursive_mutex` | Per tree stripe | Pair slab mutation |
-| `stats_mutex` | `ErlNifMutex` | Global | DEBUG stats only |
+| `stats_mutex` | `ErlNifMutex` | Global | DEBUG stats + `nif_stats_raw` live counters |
 
-**Independent domains:** `AccountMapLock` and `LockedStates::mtx` are never held together in current NIF code.
+Bare-tree Elixir NIFs (`new` / `insert` / `difference_raw` / `lock` / …) are gone. Storage tries exist only as map-owned internals. The old global `LockedStates` / orphan queue was removed with them.
 
 ## Lock acquisition rules
 
 | Path | Order | Notes |
 |------|-------|-------|
-| `difference_raw` | `locked_states_mutex` → snapshot pointers + `read_pins` → **release global** → `SharedState*` mutexes (address order) → unpin `read_pins` | `read_pins` prevents COW from consuming `difference` lifetime while global is dropped |
-| `enter_lock` (dedup) | tree (`mt->locked`) → `locked_states_mutex` → pin `read_pins` on canonical → **release global** → bump `has_clone` / canonical switch | `read_pins` prevents reclaim until registration ref is taken |
-| `leave_lock` / GC destructor | if `mt->locked`: `locked_states_mutex` → tree → decrement registration → erase map when `has_clone == 0`; else tree refcount only | Unlocked resources never touch `LockedStates` map |
-| `merkletree_clone` | `Lock(parent)` | Dirty CPU scheduler; O(1) shallow resource alloc (`locked = false`) |
-| `account_map_clone` | `AccountMapLock` → `Lock(parent_storage)` per trie (sequential) | Dirty scheduler; long hold |
-| `account_map_lock` | `AccountMapLock` → `enter_lock` / `apply_canonical_lock` per unique `root_hash`; optional store trie after map lock released | Dirty scheduler; dedupes by root hash to avoid redundant canonical switches |
-| `switch_local_to_canonical` | tree mutexes (address order) | Abandoned `SharedState` queued on `pending_orphans`; reclaimed via `try_reclaim_orphans` after `enter_lock` / `leave_lock` when mutex trylock succeeds and `has_clone == 0` |
-| `account_map_uncompact_state` | `AccountMapLock(input)` → `materialize_storage` (brief tree lock) → `batch_insert` (state_store lock) | Dirty scheduler |
-| `account_map_put/delete` | `AccountMapLock` only | May `release_resource` → async GC `leave_lock` |
-| `account_map_list_difference_raw` | `SharedAccountMap*` mutexes (address order) → brief tree lock per entry for root read → release all before materialize | Dirty CPU; never hold map lock across `materialize_storage` |
-| Insert / COW | Tree lock → ItemPool / PreAllocator / stripe pool | Same-thread nesting |
+| Storage trie GC destructor | `release_merkletree_shared`: tree mutex → drop `has_clone` / delete when 0 | Immediate reclaim; no deferred orphan queue |
+| `account_map_clone` | `AccountMapLock` → `fork_shared_accountmap` (`Lock` per parent storage / state_trie) | Dirty scheduler; writable fork even if parent `frozen` |
+| `account_map_lock` | `AccountMapLock` → set `frozen=true` only (O(1); no per-trie seal) | Dirty scheduler; put/delete/storage_put_map reject via `frozen` |
+| `account_map_storage_put_map` | `AccountMapLock` → reject if frozen → per-addr `write_storage_slot` → `update_state_trie_for_entry` | Dirty CPU; EVM `su` hot path |
+| `account_map_storage` | `AccountMapLock` → read-only storage query (`:get`/`:range`/`:list`/`:size`) | Dirty CPU |
+| `account_map_storage_roots` / `account_map_state_roots` | `AccountMapLock` → tree lock (live) **or** temp tree from compact slots (never materialize solely for roots) | No live trie export; compact may use cached root |
+| `account_map_proof` | `AccountMapLock` → account or storage proof | Dirty CPU; arities 2 and 3 |
+| `account_map_uncompact_state` | `AccountMapLock(input)` → parse/seed compact roots → `batch_insert` (state_store lock) | Dirty scheduler; compact entries stay lazy when `:root_hash` present |
+| `account_map_compact` | `AccountMapLock` (read-only; OK frozen) → per-account storage list via live tree lock or compact_storage slots | Dirty CPU |
+| `account_map_put/delete` | `AccountMapLock` only; reject if `frozen`; storage arg may be `:keep` / list / `nil` | `:keep` on compact updates `state_trie` via cached storage root (no materialize) |
+| `account_map_difference_full` | `DualAccountMapLock` → if `state_trie` `SharedState*` equal return `[]` → else `SharedStateLock` on both tries → leaf symmetric difference → snapshot candidates → release map lock → per-candidate storage diffs + roots (6-tuple) | Dirty CPU; never hold map lock across `build_storage_diff_list` |
+| `account_map_apply_difference` | `AccountMapLock` → reject if `frozen` → storage/field writes (`write_storage_slot` → `make_writeable_locked`) | Dirty CPU |
+| Insert / COW (internal) | Tree lock → ItemPool / PreAllocator / stripe pool | Same-thread nesting |
 
 ## Deadlock scenario registry
 
-Each scenario has an ID, hypothesis, and test coverage target.
-
-### A. Tree mutex + `LockedStates::mtx`
-
-| ID | Scenario | Hypothesis | Covered by |
-|----|----------|------------|------------|
-| D-A1 | `enter_lock` dedup + `difference` on overlapping SharedStates | Global held during second tree wait (pre-fix F-5) | P9, S9, ExUnit lock concurrency |
-| D-A2 | `leave_lock` (GC) + `difference` on same tree | Liveness stall: leave waits tree; diff never waits global | P11, S14, ExUnit D-A2 |
-| D-A3 | `leave_lock` + `enter_lock` phase1 on same tree | Serialized on global | P10 smoke |
-| D-A4 | `leave_lock`(global→T) + `enter_lock` switch(T,U) + `difference`(T,U) | Three-way liveness under heavy GC | P10 |
-| D-A5 | Many concurrent `leave_lock` on distinct trees | Global mutex convoy | P11 |
-| D-A6 | `enter_lock` dedup from 50+ clones | Canonical switch / refcount edge | P6, P14, ExUnit D-F2 |
-| D-A7 | `lock` on already-locked tree | Re-entrant / double enter | S8 |
-
-### B. Dual-tree ordering (`difference_raw`)
-
-| ID | Scenario | Hypothesis | Covered by |
-|----|----------|------------|------------|
-| D-B1 | `difference(A,B)` vs `difference(B,A)` | Same address order | P2, P3 |
-| D-B2 | `difference(A,B)` vs `difference(A,C)` — shared first tree | Second lock order by B vs C address | P10, ExUnit D-B2 |
-| D-B3 | Long `difference` + concurrent insert | Insert waits tree held by difference | P5 |
-| D-B4 | `difference` on same SharedState (early return) | No dual lock | S9 smoke |
-| D-B5 | `difference` while `switch_local_to_canonical` holds both trees | Post-F-5 residual | P10 |
-
 ### C. Account map × tree mutex
 
 | ID | Scenario | Risk | Covered by |
 |----|----------|------|------------|
-| D-C1 | `account_map_clone` + `difference` on shared storage | Tree mutex contention | P13 |
-| D-C2 | `account_map_uncompact_state` + per-account `difference` | Independent domains unless storage shared | P12, S12, ExUnit D-C2 |
-| D-C3 | `account_map_get` / `to_list` (materialize) + `difference` | Brief tree lock vs diff | S13 |
-| D-C4 | `account_map_put` replacing storage + GC `leave_lock` | Async GC vs diff | P13 |
-| D-C5 | `account_map_lock` / `account_map_clone` + concurrent `State.lock/1` | Correctness / writable fork | P14, P15, chain_state_uncompact_test, ExUnit D-C5, fuzz 16–18 |
-| D-C6 | `cow_copy_accountmap` during concurrent `put` | Refcount race (F-4) | TSan on P4, P13 |
-| D-C7 | `account_map_list_difference_raw` + `account_map_to_list` same map | Map mutex convoy / materialize stall | S20, ExUnit D-D7, D-C7 |
-| D-C8 | Dual-map `list_difference` lock order (A,B) vs (B,A) | Ordering regression | S24, ExUnit D-C8 |
+| D-C1 | `account_map_clone` + `difference_full` on shared storage | Tree mutex contention | P13 |
+| D-C2 | `account_map_uncompact_state` + `difference_full` | Independent domains unless storage shared | P12, fuzz S3, ExUnit D-C2 |
+| D-C3 | `account_map_get` / `to_list` (root hash export) + `difference_full` | No live storage export; brief hash compute | fuzz S4 |
+| D-C4 | `account_map_put` replacing storage + GC `release_merkletree_shared` | Async GC vs diff | P13 |
+| D-C5 | `account_map_lock` / `account_map_clone` + concurrent `State.lock/1` | Correctness / writable fork | P14, P15, chain_state_uncompact_test, ExUnit D-C5, fuzz 6–8 |
+| D-C6 | Concurrent `put` / `apply_difference` on frozen map | Rejected via `make_writeable_accountmap` | TSan on P13 |
+| D-C7 | `account_map_difference_full` + `account_map_to_list` same map | Map mutex convoy / materialize stall | fuzz S10, ExUnit D-D7, D-C7 |
+| D-C8 | Dual-map `difference_full` lock order (A,B) vs (B,A) | Ordering regression | fuzz S14, ExUnit D-C8 |
 
 ### D. Production composite (block sync)
 
 | ID | Scenario | Covered by |
 |----|----------|------------|
 | D-D1 | `Chain.State.difference/2` (storage diffs per account) | P14 |
-| D-D2 | `State.lock/1` on all account trees | P14 |
+| D-D2 | `State.lock/1` sets map `frozen` only (no per-trie seal; get exports root hashes not live storage) | P14 |
 | D-D3 | D-D1 + D-D2 + uncompact concurrent | P14, P12, ExUnit D-D3 |
-| D-D4 | compact → uncompact → clone → apply_difference | P14, S10, S11 |
-| D-D5 | Storage get/insert + block import | P1, P5 |
-| D-D6 | Dirty scheduler saturation | P15 |
-| D-D7 | prepare_state composite (native diff + lock + legacy to_list) | S30, P17, ExUnit D-D7 |
-| D-D8 | `list_difference` compact storage root compare (no full map materialize) | account_map_diff_test, S19 |
+| D-D4 | compact → uncompact → clone → apply_difference | P14, fuzz S1–S2 |
+| D-D6 | Dirty scheduler saturation | P15, P20 |
+| D-D7 | prepare_state composite (native diff + lock + to_list) | fuzz S20, P17, ExUnit D-D7 |
+| D-D8 | `difference_full` on compact storage | account_map_diff_test, fuzz S9 |
+| D-L1 | `clone` after lock + storage_put_map + discard | `cmerkle_storage_map_test`, `cmerkle_lock_clone_regression_test`, P18L |
+| D-L2 | `difference_full` + `apply_difference` round-trip | chain_state_merkle_test, account_map_diff_test |
+| D-M1 | frozen map + eager clone writable fork | chain_state_merkle_test lock→clone |
 
 ### E. C++ internal mutexes
 
 | ID | Scenario | Covered by |
 |----|----------|------------|
-| D-E1 | Concurrent clone+insert on COW SharedState | P4 + TSan |
-| D-E2 | `difference` + insert allocating pairs | P5 |
-| D-E3 | Parallel tree discard + stripe pool | P7, P11 |
+| D-E1 | Concurrent clone+storage_put on COW SharedState | P13 + TSan |
+| D-E3 | Parallel map discard + stripe pool | P12, P16 |
 | D-E4 | Nested ItemPool lock in `fork_for_write` | By design (recursive) |
 
 ### F. Refcount / UAF masquerading as deadlock
 
 | ID | Scenario | Covered by |
 |----|----------|------------|
-| D-F1 | `has_clone` mismatch during canonical switch | P9, S9, ExUnit dedup |
-| D-F2 | Canonical ref reserved vs concurrent `leave_lock` | P9, ExUnit D-F2 |
 | D-F3 | `release_storage_from_map` during delete vs diff | P13, caccount_map_lifetime_test |
 
 ## Remaining structural risk
 
-`enter_lock` and `leave_lock` now both release the global mutex before blocking on tree mutexes. Monitor `nif_stats_raw/0` (`shared_states` vs `locked_states`) in production; sustained growth indicates a reclaim regression. `nif_stats_raw` is read-only (reclaim runs from lock/unlock paths only).
+Monitor `nif_stats_raw/0` (`shared_states` / `merkletree_resources`) in production; sustained growth indicates a reclaim regression. Locked/orphan tuple slots stay zero (API shape preserved).
 
 ## CI / harness commands
 

c_src/Makefile

@@ -49,13 +49,15 @@ uncompact_harness.bin: uncompact_harness.cpp merkletree.hpp merkletree.cpp item_
 uncompact_harness: uncompact_harness.bin
 
 # Optional: libFuzzer + ASan on SHA (requires Clang with fuzzer runtime)
-.PHONY: nif nif-asan fuzz_sha
+.PHONY: nif nif-asan fuzz_sha FORCE
 fuzz_sha: fuzz_sha.bin
 	./fuzz_sha.bin -runs=5000
 
 fuzz_sha.bin: fuzz_sha.cpp sha.cpp Makefile
 	clang++ -std=c++17 -I. -g -O1 -fsanitize=fuzzer,address -fno-omit-frame-pointer -o fuzz_sha.bin fuzz_sha.cpp sha.cpp -lstdc++
 
+FORCE:
+
 nif: ../priv/merkletree_nif.so
 
 ../priv/merkletree_nif.so: nif.cpp merkletree.hpp merkletree.cpp item_pool.cpp sha.cpp preallocator.hpp Makefile