Made autobahn producer use TxMempool by pompon0 · Pull Request #3224 · sei-protocol/sei-chain

pompon0 · 2026-04-10T11:14:00Z

Also

disabled bunch of reactors unused in autobahn mode.
fixed race conditions in mempool processPeerUpdates

wen-coding · 2026-04-10T21:34:57Z

sei-tendermint/internal/mempool/mempool.go

+// overflow, gas limit enforcement no longer works correctly. This preserves the
+// historical behavior for backward compatibility.
+func (txmp *TxMempool) ReapMaxTxsBytesMaxGas(maxTxs int, maxBytes, maxGasWanted, maxGasEstimated int64) (types.Txs, int64) {
+	if maxTxs < 0 {


it's okay for maxTxs to be equal to 0?

I suppose so

sei-tendermint/internal/mempool/mempool.go

sei-tendermint/internal/autobahn/producer/state.go

wen-coding · 2026-04-11T03:20:35Z

sei-tendermint/internal/autobahn/producer/state.go

+		int(s.cfg.maxTxsPerBlock()), // nolint:gosec // config values fit into int on supported platforms.
+		utils.Max[int64](),
+		int64(s.cfg.MaxGasPerBlock), // nolint:gosec // config values stay within int64 range.
+		int64(s.cfg.MaxGasPerBlock), // nolint:gosec // config values stay within int64 range.


So we won't have different maxGasWanted and maxGasEstimated in the future?

I don't know, probably we will

wen-coding · 2026-04-11T03:21:14Z

sei-tendermint/internal/autobahn/producer/state.go

+
+	txs, totalGas := s.txMempool.ReapMaxTxsBytesMaxGas(
+		int(s.cfg.maxTxsPerBlock()), // nolint:gosec // config values fit into int on supported platforms.
+		utils.Max[int64](),


We probably should have maxBytes as well?

eventually yes

sei-tendermint/internal/autobahn/producer/state.go

sei-tendermint/internal/mempool/reactor/reactor.go

wen-coding · 2026-04-11T18:55:25Z

sei-tendermint/internal/autobahn/producer/state.go

+		int64(s.cfg.MaxGasPerBlock), // nolint:gosec // config values stay within int64 range.
+		int64(s.cfg.MaxGasPerBlock), // nolint:gosec // config values stay within int64 range.
+	)
+	s.txMempool.RemoveTxs(txs)


Just an observation: I think it's okay here to acquire mutex twice, since if the tx disappeared in between RemoveTxs still work.
But maybe some day we should re-think the mutex use in mempool, it looks correct and efficient to me in the current context, but if we add more and more features then it might be harder to reason.

agreed, Mempool handling is already very fragile imo

sei-tendermint/node/node.go

sei-tendermint/internal/p2p/giga_router.go

wen-coding · 2026-04-13T23:04:57Z

sei-tendermint/internal/p2p/giga_router_test.go

 	panic("unreachable")
 }

+func (a *testApp) CheckTx(context.Context, *abci.RequestCheckTxV2) (*abci.ResponseCheckTxV2, error) {


Do we have a test where a block failed to execute and commit somewhere?

only happy path tests for now until we get this to work somehow.

wen-coding · 2026-04-13T23:26:28Z

sei-tendermint/internal/p2p/giga_router.go

 	return scope.Run(ctx, func(ctx context.Context, s scope.Scope) error {
 		// Spawn outbound connections dialing.
 		for _, addr := range r.cfg.ValidatorAddrs {
 			s.Spawn(func() error {


I was trying this PR on #3234. There was one error and all Giga Router stopped, no retries.

AI explains:
Spawn uses SpawnBg which calls s.Cancel(err) if the task returns an error. This cancels the entire scope — all other goroutines get cancelled.

So if runExecute (spawned with SpawnNamed which uses Spawn) returns an error early, the whole scope is cancelled including the dial retry loops.

Does that make sense?

execution failure is a valid termination reason, because it means that we are not able to execute a finalized block.

wen-coding · 2026-04-13T23:43:11Z

Testing PR #3224 with autobahn integration tests

Tested this PR using the autobahn docker cluster setup (PR #3220 + integration tests from #3234). Found the following issues:

1. Panic in `executeBlock` — staking params not initialized

panic: UnmarshalJSON cannot decode empty bytes

goroutine 329 [running]:
github.com/sei-protocol/sei-chain/sei-cosmos/x/params/types.Subspace.Get(...)
    sei-cosmos/x/params/types/subspace.go:109
github.com/sei-protocol/sei-chain/sei-cosmos/x/staking/keeper.Keeper.MaxValidators(...)
    sei-cosmos/x/staking/keeper/params.go:18
github.com/sei-protocol/sei-chain/sei-cosmos/x/staking/keeper.Keeper.GetBondedValidatorsByPower(...)
    sei-cosmos/x/staking/keeper/validator.go:221
github.com/sei-protocol/sei-chain/app.(*App).GetValidators(...)
    app/app.go:2192
github.com/sei-protocol/sei-chain/sei-tendermint/internal/p2p.(*GigaRouter).executeBlock(...)
    sei-tendermint/internal/p2p/giga_router.go:98

executeBlock calls app.GetValidators() which reads MaxValidators from the staking params store, but the params haven't been populated yet at the time the first block is being finalized after InitChain. This causes a panic on the very first block.

2. RPC `/status` nil pointer panic

panic: runtime error: invalid memory address or nil pointer dereference

types.(*ValidatorSet).Copy(...)
    types/validator_set.go:261
consensus.(*State).GetValidators(...)
    internal/consensus/state.go:310
rpc/core.(*Environment).validatorAtHeight(...)
    internal/rpc/core/status.go:120

When autobahn is enabled, the CometBFT consensus reactor is skipped, so ValidatorSet is nil. The RPC /status endpoint calls GetValidators() which tries to Copy() a nil set.

Reproduction

# On the wen/test-3224-with-autobahn branch (PR #3224 + #3220 cherry-picked)
AUTOBAHN=true make docker-cluster-start
# Check logs:
grep "panic" build/generated/logs/seid-0.log
# Try RPC:
curl http://localhost:26657/status

Note: The autobahn config also needs allow_empty_blocks: true for the cluster to produce blocks without submitted transactions. This is being fixed on our side.

wen-coding · 2026-04-13T23:47:37Z

Follow-up on Bug #1: Missing Commit after InitChain

Dug deeper into the root cause. The panic in executeBlock calling app.GetValidators() is because runExecute calls InitChain but never calls Commit after it:

// giga_router.go runExecute()
if last == 0 {
    if _, err := app.InitChain(ctx, r.cfg.GenDoc.ToRequestInitChain()); err != nil {
        return fmt.Errorf("App.InitChain(): %w", err)
    }
    // No Commit() here!
}

for n := next; ; n += 1 {
    b, err := r.data.GlobalBlock(ctx, n)
    // ...
    if vals := r.cfg.App.GetValidators(); len(vals) > 0 {  // PANIC

InitChain populates the staking params in the deliverState, but GetValidators() (in app.go:2192) creates a new context via NewUncachedContext which reads from the committed store — which is still empty because Commit was never called.

In CometBFT's normal flow, the handshaker (replay.go) calls Commit after InitChain before any blocks are finalized. The fix is likely to add app.Commit() after InitChain in runExecute, similar to what CometBFT does.

wen-coding · 2026-04-14T16:31:53Z

Follow-up: Height mismatch after InitChain + Commit fix

Tested the latest 5ed9b0686 (with the Commit after InitChain fix). New panic:

panic: invalid height: 1; expected: 2

at app/abci.go:33 in BeginBlock.

The issue is in runExecute:

if last == 0 {
    if _, err := app.InitChain(ctx, ...); err != nil { ... }
    if _, err := app.Commit(ctx); err != nil { ... }
    next, ok = utils.SafeCast[atypes.GlobalBlockNumber](r.cfg.GenDoc.InitialHeight)  // next = 1
}

for n := next; ; n += 1 {
    // executeBlock at height 1, but app already committed height 1 via InitChain+Commit
    // app expects height 2

InitChain + Commit advances the app to height 1. Then next is set to GenDoc.InitialHeight (1), so executeBlock tries to finalize height 1 again. The app rejects it because it expects height 2.

Fix: after InitChain + Commit, set next = InitialHeight + 1 (or query app.Info() again to get the actual height).

wen-coding · 2026-04-14T16:32:34Z

sei-tendermint/internal/p2p/giga_router.go

+			return fmt.Errorf("app.InitChain(): %w", err)
+		}
+		if _, err := app.Commit(ctx); err != nil {
+			return fmt.Errorf("app.Commit(): %w", err)


Tried again and another panic:
panic: invalid height: 1; expected: 2

BeginBlock at app/abci.go:33 expects height 2 but got height 1. This is because InitChain + Commit already advanced the app state to height 1, so when executeBlock tries to finalize block 1, the app expects block 2.

pompon0 added 30 commits April 8, 2026 13:34

removed unused function

3b3071d

snapshot

3c866a6

updated test

4764164

more cleanup

3b992a1

snapshot

9fd05fc

wip

1fd3453

WIP

a0ef9c7

wip

63e6bd3

wip

254426b

wip

44b9288

wip

3061e49

wip

47aa688

wip

aaf3ee9

addressed a flake

4e0faa9

Merge branch 'gprusak-mempool2' into gprusak-mempool3

bd8db80

wip

d2b6a21

snapshot

4a3a579

removed notion of precheck

59b9f59

simplified errors

28a2ff1

Merge remote-tracking branch 'origin/main' into gprusak-mempool2

fab6e8c

Merge branch 'gprusak-mempool2' into gprusak-mempool3

ea8519e

compilation fix

15ed859

separated reactor

ba81b8b

wip

4601391

concurrency fix

b30fc91

non-nil args to buildGigaConfig

05e69c6

mempool in GigaRouter

0d3b1d0

-1 in Reap

f13a871

disabled some components for autobahn mode

cdc7047

disabled mempool notifications when WaitForTxs is disabled

a21d315

Base automatically changed from gprusak-mempool3 to main April 10, 2026 18:29

pompon0 added 2 commits April 10, 2026 20:30

Merge remote-tracking branch 'origin/main' into gprusak-mempool3

70c75a1

Merge branch 'gprusak-mempool3' into gprusak-mempool4

f5a18e0

wen-coding reviewed Apr 13, 2026

View reviewed changes

applied comments

26e74ea

pompon0 requested a review from wen-coding April 13, 2026 12:57

keeping a tx mempool lock for Finalize + Commit + Update

be79fac

sei-will approved these changes Apr 13, 2026

View reviewed changes

wen-coding mentioned this pull request Apr 13, 2026

feat: add autobahn integration tests - initial batch (CON-249) #3234

Open

3 tasks

wen-coding reviewed Apr 13, 2026

View reviewed changes

pompon0 added 5 commits April 14, 2026 16:09

applied comments

baac6b0

applied comments

121b2e4

Merge remote-tracking branch 'origin/main' into gprusak-mempool4

4dbc0d7

moved config

d217ee0

extracted config

5ed9b06

sei-will approved these changes Apr 14, 2026

View reviewed changes

wen-coding reviewed Apr 14, 2026

View reviewed changes

wen-coding approved these changes Apr 14, 2026

View reviewed changes

deflake

6a2cd95

pompon0 enabled auto-merge April 14, 2026 16:37

pompon0 force-pushed the gprusak-mempool4 branch from 6453164 to 6a2cd95 Compare April 14, 2026 16:45

pompon0 added this pull request to the merge queue Apr 14, 2026

Merged via the queue into main with commit 9795d18 Apr 14, 2026
71 of 74 checks passed

pompon0 deleted the gprusak-mempool4 branch April 14, 2026 17:12

wen-coding mentioned this pull request Apr 14, 2026

fix: autobahn InitChain height and GetValidators panic (CON-249) #3243

Open

3 tasks

Conversation

pompon0 commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wen-coding commented Apr 13, 2026

Testing PR #3224 with autobahn integration tests

1. Panic in executeBlock — staking params not initialized

2. RPC /status nil pointer panic

Reproduction

Uh oh!

wen-coding commented Apr 13, 2026

Follow-up on Bug #1: Missing Commit after InitChain

Uh oh!

wen-coding commented Apr 14, 2026

Follow-up: Height mismatch after InitChain + Commit fix

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pompon0 commented Apr 10, 2026 •

edited

Loading

1. Panic in `executeBlock` — staking params not initialized

2. RPC `/status` nil pointer panic