[fusaka-holesky] Proposer lookahead processing times increasing epoch transition

[philknows]

Describe the bug

Proposer lookahead processing takes quite a bit of time - upwards of ~300-350ms to the transition processing.

Luca from Serenita reports seeing similar values and block proposing is affected by seeing proposed blocks come later in the slot.

Expected behavior

Proposer lookahead should have lowest overhead possible to minimize epoch transition processing.

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.35.0-rc.1

[twoeths]

we always compute shuffling synchronously and don't use the async build in beforeEpochTransition postfulu

this shuffling calculation time by source confirm that

prefulu, we have an assumption that next epoch shuffling is not needed right away and can be lazily computed
postfulu, that assumption does not work since we add proposerLookahead to BeaconState, hence we need the next shuffling computation right away

I don't see how to save epoch transition time without turning epoch transition functions to async

[twoeths]

attaching the profile persisted in the above image

holesky_fulu_oct_02.cpuprofile.zip

[KoxyG]

Is this still an issue?

[twoeths]

@KoxyG yes this is still an issue but we cannot resolve it
starting from fulu, we have to eagerly compute shuffling of next epoch (vs lazily as pre-fulu)
I suppose we should close this "as designed" @philknows
we have this PR #8688 to remove the lazy shuffling computation to reflect the real situation

[philknows]

Agreed, thanks for the context, closing.