Add Echidna fuzz testing harness + runner by 0xCardiE · Pull Request #306 · ethersphere/storage-incentives

0xCardiE · 2026-02-28T15:44:01Z

Introduce Echidna-based, stateful fuzz testing for this repo.

Add a Docker-based runner and documentation, plus a multi-actor harness that fuzzes all smart contracts with role modeling, cross-actor accounting invariants, non-interference checks, and post-condition properties for all contracts.

What is fuzz testing?
Fuzz testing ("fuzzing") is an automated testing technique that repeatedly calls contract functions with many randomized inputs and stateful sequences of calls. Instead of writing a single expected-output test per scenario, you write properties/invariants that must always hold, and the fuzzer tries hard to find a sequence that breaks them.

Why use it for smart contracts?
Finds edge cases humans miss: weird input combinations, unusual call ordering, boundary values, and unexpected state transitions.
Stateful bugs: catches issues that only appear after many steps (e.g. stake -> freeze -> slash -> migrate -> restake).
Property-driven: you encode "this should never happen" rules (access control, accounting conservation, deletion semantics), and the tool searches for counterexamples.

Echidna coverage of the commit -> reveal -> claim lifecycle

What we test

Commit (well covered)

act_happyCommit in the redistribution harnesses constructs valid overlays, sets staking mock data, and calls commit() through actors
Properties verify commits land in storage correctly (echidna_tracked_commit_matches_storage, echidna_commit_overlays_unique)

Reveal (covered, though hit rate is still probabilistic in the general harnesses)

act_happyReveal replays tracked pre-images from the same round
Properties verify reveal storage matches (echidna_tracked_reveal_matches_storage, echidna_revealed_commit_indices_valid, echidna_reveal_entries_imply_matching_commit)

Winner selection (covered via exposed internal)

act_winnerSelection in the base harness calls exposedWinnerSelection() which runs the real winnerSelection() internal function - this is the core claim logic:
- getCurrentTruth() truth determination
- winner random selection weighted by stake density
- freezing non-revealers and disagreeing revealers
- OracleContract.adjustPrice() call
- currentClaimRound bookkeeping
Properties: echidna_winnerSelection_only_once_per_round, echidna_last_winnerSelection_freezes_nonrevealed

Pot withdrawal (covered via claim stub)

claimStub() in the claim-stub harness runs real winnerSelection() and then calls PostageStamp.withdraw(winner) directly
Properties: echidna_claim_withdraws_pot_to_winner_when_successful, echidna_claim_triggers_oracle_adjustPrice, echidna_claim_only_once_per_round

Real claim() proof path (now covered with fixture-based fuzzing)

EchidnaRedistributionRealClaimHarness exercises the real Redistribution.claim() verifier path with fixed valid CAC and SOC proof bundles
The harness prepares a real commit -> reveal -> claim flow, then runs the actual proof-verifying claim() function
Instead of trying to randomly discover valid cryptographic proofs, it starts from known-good fixtures and fuzzes targeted mutations around them
Mutations cover selected inclusion-proof fields, postage index fields, and SOC fields while keeping the rest of the scenario realistic
Properties:
- echidna_unmutated_fixture_claim_succeeds
- echidna_mutated_fixture_claim_does_not_succeed
- echidna_successful_real_claim_effects_hold

Why the real `claim()` path needs a fixture-based approach

Naive random fuzzing is very unlikely to hit a valid proof-verifying claim() because the inputs must satisfy multiple correlated cryptographic relationships at once:

correct BMT Merkle inclusion proofs
valid postage signatures for the batch owner
valid SOC signatures and address derivation
matching proximity relationships against the round anchor
correct ordering and reserve-estimation conditions

These witnesses need to be constructed, not discovered by chance. The fixture-based harness solves this by seeding Echidna with valid proof bundles and then fuzzing mutations around them while still executing the real on-chain verifier logic.

Is the current coverage sufficient?

The split is now more complete and more intentional:

Echidna covers: the state machine (phase transitions, round bookkeeping, once-per-round constraints), economics (pot distribution, freezing, oracle updates), access control, and now the real claim() verifier path via fixed CAC/SOC fixtures plus targeted proof mutations
Chai unit tests still cover best: targeted, readable proof-verification scenarios with hand-crafted valid/invalid proofs and exact expected revert reasons

Supporting runner/config changes

Also added:

echidna/echidna-real-claim.yaml
ECHIDNA_CONFIG support in scripts/echidna.sh

This lets the real-claim harness use its own isolated corpus and run independently from the shared suite when needed.

Validation

Validated both with the dedicated run:

ECHIDNA_CONTRACT=EchidnaRedistributionRealClaimHarness \
ECHIDNA_CONFIG=echidna/echidna-real-claim.yaml \
yarn echidna

Add a minimal Echidna setup (harness + config + runner) to fuzz StakeRegistry/TestToken invariants and provide a yarn script for local execution.

Update the stake coverage property to use nodeEffectiveStake and ignore Echidna's crytic-export output directory.

Add properties for successful manageStake(add>0) post-state (potential/balance deltas and commitment recomputation) and add non-interference checks so unrelated actors' stake entries never change during actions.

Add Echidna properties for freeze/slash/migrate post-state correctness, reset actor tracking on stake deletions, and make the runner compile on-host to avoid container npx/hardhat dependency.

Expose winnerSelection in the Redistribution harness to assert phase partitioning, reveal↔commit linkage, single winner selection per round, and freezing of non-revealers. Strengthen PostageStamp fuzzing with wider non-interference checks and a pot monotonicity invariant, and document the new properties.

Introduce a fuzz-only RedistributionClaimStub that bypasses proof verification and executes claim→withdraw end-to-end. Add pot mock + invariants for single-claim-per-round, winner payout, oracle adjustPrice call, and freezing non-revealers; wire the harness into the runner and docs.

Add system-level properties for stake accounting (stake token balance covers sum of potential stake), pot accounting (pot never exceeds stamp token balance), and price floors (oracle/stamp never below minimum).

Run Prettier (with Solidity plugin) to satisfy CI formatting checks.

nugaon

Really nice initiative with really a lot of changes.
It would have been better in several different PRs.

I checked the basic setup and the PriceOrcale harness mostly, but I peeked into other harness logics as well.
I stumbled upon an issue with running the test, see below.

nugaon · 2026-04-02T08:41:32Z

+
+Echidna repeatedly calls the harness “action” functions (like `act_manageStake`) with random inputs, building **stateful sequences**. After (and during) those sequences it checks `echidna_*` property functions such as:
+
+- **Token properties**: total supply stays constant; decimals stay 16


I cannot imagine the decimals could change anyhow

True, was aimed for robustness and if in some cases code changes that we check every minor detail.
Logic wise how its now this would never happen.

these kind of tests should be chai manual tests. shouldn't echidna check only state-model correctness?
IMO this does not need detailed testing because it doesn't replace targeted tests.

Yes you are right
The right mental model would be

Chai tests = targeted, readable, scenario-driven tests
Echidna = randomized exploration of both global invariants and per-call correctness across arbitrary state histories

nugaon · 2026-04-02T09:03:23Z

+  echo "==> echidna: running contract $c" >&2
+
+  # Avoid stale Crytic compile artifacts causing old properties/tests to run.
+  rm -rf crytic-export


This is wrong, the docker writes the files with root, and my user permission cannot remove it.

Current scripts/echidna.sh no longer removes anything on the host.
It runs rm -rf crytic-export inside the same docker run as Echidna (sh -c 'rm -rf crytic-export && echidna-test …')

I don't completely understand why crytic-export generated in the first place if you automatically remove it in each iteration?

It's generated as a side effect of echidna-test. Every invocation of Echidna triggers CryticCompile, which writes to crytic-export/. We can't prevent that; we can only clean it before the next iteration so it doesn't contaminate the next contract's run.

nugaon · 2026-04-02T13:43:20Z

+        uint64 minUp = oracle.minimumPriceUpscaled();
+        uint64 expected = uint64(p) << 10;
+        if (expected < minUp) expected = minUp;


why do you override expected like this? seemingly, the bit overflow is surpressed like this, but it can hide other problems too.

0xCardiE · 2026-04-07T15:08:07Z

Really nice initiative with really a lot of changes. It would have been better in several different PRs.

I checked the basic setup and the PriceOrcale harness mostly, but I peeked into other harness logics as well. I stumbled upon an issue with running the test, see below.

yes you are right. I remove one part into another PR that should be separate
https://github.com/ethersphere/storage-incentives/pull/308/changes

- Run rm + echidna in one Docker invocation so host users need not delete root-owned crytic-export - Point root README fuzz blurb at echidna/README.md (avoid stale harness list) - PriceOracle harness: arm setPrice postconditions only after ok+returned; read expected upscaled from oracle; expand _clearPending; merge paused/revert state checks with OR - System harness: replace tautological min(pot,bal)<=bal with pot<=token balance check - Redistribution harness: clear winnerSelection pending at start of every action Made-with: Cursor

- Run rm + echidna in one Docker invocation so host users need not delete root-owned crytic-export - Point root README fuzz blurb at echidna/README.md (avoid stale harness list) - PriceOracle harness: arm setPrice postconditions only after ok+returned; read expected upscaled from oracle; expand _clearPending; merge paused/revert state checks with OR - System harness: replace tautological min(pot,bal)<=bal with pot<=token balance check - Redistribution harness: clear winnerSelection pending at start of every action

…ng scratch - Claim harness: same stale-pending pattern as redistribution winnerSelection; non-claim actions now drop pendingClaim - Postage harness: _clearPending zeros all pending snapshot fields (parity with PriceOracle harness _clearPending)

Solidity public array getters revert on out-of-bounds access (mapped to currentCommits declaration in Redistribution.sol). Echidna traces those as failures even when wrapped in staticcall. - Add RedistributionExposed.sol with currentCommitsLength/currentRevealsLength - Bound redistribution + system harness scans using real lengths (cap 25) - Simplify _scanRevealsLen to use length instead of probing past end

- Run rm + echidna in one Docker invocation so host users need not delete root-owned crytic-export - Point root README fuzz blurb at echidna/README.md (avoid stale harness list) - PriceOracle harness: arm setPrice postconditions only after ok+returned; read expected upscaled from oracle; expand _clearPending; merge paused/revert state checks with OR - System harness: replace tautological min(pot,bal)<=bal with pot<=token balance check - Redistribution harness: clear winnerSelection pending at start of every action

- Remove unused _revealFull() helper from EchidnaRedistributionHarness - Remove dead seedBatch/Batch struct/mapping from redistribution stamp mock - Remove unread withdrawCalls from EchidnaPostageStampPotMock - Remove unread lastFreezeTime and lastRedundancy from shared mocks

… harness Add a shouldRevertWithdraw toggle to the pot mock so Echidna can explore the path where claimStub succeeds but PostageStamp.withdraw() fails via the .call() error-swallowing pattern. New property asserts pot and actor balances are preserved when withdraw reverts, while the round is still consumed (documenting the known H-1 behavior).

Add an Echidna harness that exercises the real redistribution claim verifier with fixed CAC/SOC proof fixtures and targeted mutations. Isolate its corpus/config and document the workflow so it can run independently from the shared suite.

Apply formatter output to the new real-claim Echidna harness and its README updates so the fixture-based claim fuzzing changes match the repository style.

0xCardiE · 2026-04-16T12:41:31Z

Fixed previous discrepancies and comments and also added some fuzzing to claim function and separate config and corpus for it.

nugaon · 2026-04-28T14:09:12Z

I checked the new EchidnaRedistributionRealClaimHarness.sol but it has just partial coverage seemingly, e.g. act_prepareFixtureReveal does not go further after the redist.currentPhaseReveal() check.

Use echidna/corpus/by-contract/<Harness>/ so each fuzz target keeps its own sequences and coverage HTML. Bump default testLimit/seqLen and set workers: 4. Document paths and ECHIDNA_TEST_LIMIT/ECHIDNA_SEQ_LEN/ECHIDNA_WORKERS overrides. Build optional CLI flags as a string so set -u does not fail when no overrides are set.

Real-claim harness: progress flags, tx counter, strict property after warmup; README and script document defaults and overrides.

Remove the fixture-based real claim harness so proof verification stays in Hardhat; trim deterministic library-level checks from the base harness. scripts/echidna.sh now passes --hardhat-ignore-compile and clears stale artifacts/build-info so CryticCompile works without Node inside Docker.

nugaon

I still don't get good coverage on claim at EchidnaPostageStampHarness.

the tests should be simplified, there are two different specific redistribution tests but seemingly none of them get to the claim phase (mostly stops at winnerselection).
as we discussed this PR is too heavy, rereading 3.5k lines all the time is not the best method that we could do... maybe that's why I have the feeling about this approach overcomplicates the codebase.

The fixes on the contracts seem good, we could merge those in a different fix PR.

nugaon · 2026-05-19T12:46:17Z

+        a.callReveal(uint8(depth % 32), hash, nonce);
+    }
+
+    function act_claim(uint8 actorId) external {


It runs, I have multiple echidna corpuses where it was run. Sharing just one here

covered.1777581556.html

0xCardiE · 2026-05-20T20:14:36Z

I still don't get good coverage on claim at EchidnaPostageStampHarness.

the tests should be simplified, there are two different specific redistribution tests but seemingly none of them get to the claim phase (mostly stops at winnerselection). as we discussed this PR is too heavy, rereading 3.5k lines all the time is not the best method that we could do... maybe that's why I have the feeling about this approach overcomplicates the codebase.

The fixes on the contracts seem good, we could merge those in a different fix PR.

Coverage is pretty good on PostageStamp, we call all the actions there and its getting to 80% so very small part is not called.

They get to claim phase. Maybe we need to talk about this how it should be read, seems like we are reading info differently

We have some Claim stubs and mocks so we can try and test parts of it. Redistribution is complex contract that is why we have 2 harnesses, in first we test commit/reveal, winner selection in second we have claim/withdraw testing.
Both dont have real inclusion proofs as this is hard to fuzz so we need to simulate parts of it.

Yes contract fixes should be applied with specific PRs when we deploy new contracts and will be pulled out of this PR.

0xCardiE · 2026-05-20T20:28:08Z

I still don't get good coverage on claim at EchidnaPostageStampHarness.

the tests should be simplified, there are two different specific redistribution tests but seemingly none of them get to the claim phase (mostly stops at winnerselection). as we discussed this PR is too heavy, rereading 3.5k lines all the time is not the best method that we could do... maybe that's why I have the feeling about this approach overcomplicates the codebase.

The fixes on the contracts seem good, we could merge those in a different fix PR.

About line numbers, I dont see it that way. There are tests, they should be verbose, not minimized. Dont see it add any complexity as its separate part of codebase not affecting actual contracts. They are made in a way that we cover all functions in contract, with properties that make sense to be checked. At best we can remove some properties that you might see as not needed. Otherwise think this is fine, it aims to run all the functions in contracts and exposes them to fuzzing best it can.

nugaon

As we discussed, the fuzzing logic with the coverage will be revised later, but overall the current state of tests and configuration seems fine. Configurations, fixes, test setups seem to be in place and correct.

Raw LCOV totals include echidna_* property lines that fuzz txs never hit, making harness coverage look worse than exploration really is. Print actions-only metrics after each harness run.

The virtual _nextSeedValue override was only used by the removed fixture-based real-claim harness; inline private updateRandomness is enough.

Property checks run via eth_call and are not reflected in LCOV, so reporting a properties block percentage was misleading.

Move oracle clamp/cast fixes to #311; keep harness expectations aligned with master PriceOracle until that PR lands.

Move minimumInitialBalancePerChunk overflow fix to #313.

Compute expected balance from on-chain outpayment state before copyBatch so an extra CI-mined block cannot fail the test. Initialize copyBatch fixture contracts in its own beforeEach.

0xCardiE added 8 commits February 27, 2026 19:08

feat: add echidna fuzz harness for staking

48f10c7

Add a minimal Echidna setup (harness + config + runner) to fuzz StakeRegistry/TestToken invariants and provide a yarn script for local execution.

fix image

0788f95

fix: correct echidna stake invariant

9911aca

Update the stake coverage property to use nodeEffectiveStake and ignore Echidna's crytic-export output directory.

add explanation

f2a6b88

introduce more complex properties

50375ab

add more complex tests

b14c60a

feat: add manageStake postconditions and non-interference checks

fd524e0

Add properties for successful manageStake(add>0) post-state (potential/balance deltas and commitment recomputation) and add non-interference checks so unrelated actors' stake entries never change during actions.

feat: add freeze/slash/migrate postconditions

92520d8

Add Echidna properties for freeze/slash/migrate post-state correctness, reset actor tracking on stake deletions, and make the runner compile on-host to avoid container npx/hardhat dependency.

0xCardiE self-assigned this Feb 28, 2026

0xCardiE added 11 commits March 2, 2026 21:17

implement oracle contract, make default run all

74e0c33

fix price oracle problesm

c79e7a3

introduce postagestamp fuzz

60b1824

finalize stamp fuzzing

59b1d4c

basic redis fuzzing added

b946e35

add advanced fuzzing for stamps

c69ea42

add cross contract wiring for full fuzzing

241808d

add economic invariants to system fuzz harness

73b21f3

Add system-level properties for stake accounting (stake token balance covers sum of potential stake), pot accounting (pot never exceeds stamp token balance), and price floors (oracle/stamp never below minimum).

chore: format echidna harnesses

ca61760

Run Prettier (with Solidity plugin) to satisfy CI formatting checks.

0xCardiE mentioned this pull request Mar 12, 2026

Introduce Echidna-based, stateful fuzz testing for Storage Incentives #307

Closed

nugaon reviewed Apr 2, 2026

View reviewed changes

0xCardiE force-pushed the feat/echidna_fuzz branch from 9694920 to 351eb8c Compare April 7, 2026 15:05

0xCardiE added 3 commits April 7, 2026 18:10

0xCardiE force-pushed the feat/echidna_fuzz branch from 4dabb17 to e0bbd29 Compare April 7, 2026 16:10

0xCardiE added 4 commits April 16, 2026 09:54

chore(echidna): format real claim harness docs

1270897

Apply formatter output to the new real-claim Echidna harness and its README updates so the fixture-based claim fuzzing changes match the repository style.

0xCardiE added 8 commits April 29, 2026 16:22

remove custom harness use just default one

70345fb

Fix echidna so it reaches reveal and claim

b4294c7

feat(echidna): default 60k/320 and fixture E2E guard

cc88220

Real-claim harness: progress flags, tx counter, strict property after warmup; README and script document defaults and overrides.

Oracle fix for upper bound limit

6a671db

test(echidna): drop duplicate system harness props

8c8813e

fix for preetier

9a6231d

nugaon self-requested a review May 19, 2026 12:47

nugaon reviewed May 19, 2026

View reviewed changes

optimize readme

aa6edad

nugaon approved these changes May 22, 2026

View reviewed changes

0xCardiE added 8 commits May 22, 2026 11:14

add extra info how it works

1bc951e

feat(echidna): add action-only coverage summary

2aba35a

Raw LCOV totals include echidna_* property lines that fuzz txs never hit, making harness coverage look worse than exploration really is. Print actions-only metrics after each harness run.

refactor(redis): drop unused fuzz randomness hook

0285ef9

The virtual _nextSeedValue override was only used by the removed fixture-based real-claim harness; inline private updateRandomness is enough.

refactor(echidna): drop properties line from coverage summary

2e53478

Property checks run via eth_call and are not reflected in LCOV, so reporting a properties block percentage was misleading.

chore(echidna): drop PriceOracle production fixes from fuzz branch

27dbd09

Move oracle clamp/cast fixes to #311; keep harness expectations aligned with master PriceOracle until that PR lands.

chore(echidna): drop PostageStamp production fix from fuzz branch

21ef16d

Move minimumInitialBalancePerChunk overflow fix to #313.

fix(test): stabilize copyBatch normalised balance assertion

bf5c5af

Compute expected balance from on-chain outpayment state before copyBatch so an extra CI-mined block cannot fail the test. Initialize copyBatch fixture contracts in its own beforeEach.

style(test): format PostageStamp.test.ts

37a30c5

0xCardiE merged commit 5375498 into master May 22, 2026
1 check passed

0xCardiE deleted the feat/echidna_fuzz branch May 22, 2026 10:55


		Echidna repeatedly calls the harness “action” functions (like `act_manageStake`) with random inputs, building stateful sequences. After (and during) those sequences it checks `echidna_*` property functions such as:

		- Token properties: total supply stays constant; decimals stay 16

Conversation

0xCardiE commented Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Echidna coverage of the commit -> reveal -> claim lifecycle

What we test

Why the real claim() path needs a fixture-based approach

Is the current coverage sufficient?

Supporting runner/config changes

Validation

Uh oh!

nugaon left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0xCardiE commented Apr 7, 2026

Uh oh!

0xCardiE commented Apr 16, 2026

Uh oh!

nugaon commented Apr 28, 2026

Uh oh!

nugaon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xCardiE commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xCardiE commented May 20, 2026

Uh oh!

nugaon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

0xCardiE commented Feb 28, 2026 •

edited

Loading

Why the real `claim()` path needs a fixture-based approach

nugaon left a comment •

edited

Loading

0xCardiE commented May 20, 2026 •

edited

Loading