test: interface_zmq_dash.py InstantLock timeout on branches missing wait_for_instantlock sync fixes

[thepastaclaw]

Summary

interface_zmq_dash.py can time out waiting for an InstantLock on older/stale branches that do not contain the later wait_for_instantlock() sync fixes now present on develop.

Failing CI examples

• PR branch failure (dashpay/dash#7237, head c425124c7e57a3f5e851d9f5ceb8934d850e84ff):
  • workflow run: https://github.com/dashpay/dash/actions/runs/24067647260
  • failing job: https://github.com/dashpay/dash/actions/runs/24067647260/job/70204889207
• Earlier occurrence on the same PR branch: https://github.com/dashpay/dash/actions/runs/23548285435/job/68563333662

Failure mode

test/functional/interface_zmq_dash.py times out at:

• DashZMQTest.test_instantsend_publishers()
• line 300: self.wait_for_instantlock(rpc_raw_tx_1_hash, self.nodes[0])

CI traceback excerpt:

AssertionError: Predicate ''''
    def check_instantlock():
        try:
            return node.getrawtransaction(txid, True)["instantlock"]
        except Exception:
            return False
''' not true after 240.0 seconds

Root cause

This is not caused by PR #7237's own changes.

I verified in the PR worktree:

git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.py test/functional/test_framework/test_framework.py

which returns no commits, so the PR branch itself does not modify the failing test/helper path.

develop already contains the fixes for this exact issue:

• c79c2d981d — refactor: improve is-waiting helper by including mempool sync bump of mocktime
• d8ac7815e3 — test: fix additional IS lock sync sites that passed nodes=[self.nodes[0]]

Those commits restore the wait_for_instantlock() behavior to bump mocktime and sync mempools across the relevant nodes, which is required so the MN signer nodes actually see the transaction before InstantLock signing.

The failing branch is behind develop and does not contain those fixes, so it can still hit the old timeout.

Reproduction notes

1. Use a branch that predates c79c2d981d / d8ac7815e3
2. Run python3 test/functional/interface_zmq_dash.py in a CI-capable environment with python3-zmq installed
3. Observe intermittent timeout waiting for getrawtransaction(...)["instantlock"]

Expected behavior

Branches with the current develop test framework fixes should not rely on the old unsynchronized behavior and should avoid this timeout.

Suggested resolution

For stale backport branches, rebase or cherry-pick the helper/test fixes from:

• c79c2d981d
• d8ac7815e3

[thepastaclaw]

Another occurrence from PR #7242 (wallet deprecation + RPC type checking backport), current head 24daadcf6d.

• CI run: https://github.com/dashpay/dash/actions/runs/24580460650
• Job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71880218247
• Build target: linux64_ubsan-test / Test source
• Date: 2026-04-17

The failure is the same InstantLock timeout in interface_zmq_dash.py:

AssertionError: Predicate "" not true after 240.0 seconds
    def check_instantlock():
        try:
            return all(
                node.getrawtransaction(txid, True)["instantlock"]
                for txid in txids
                for node in nodes
            )
        except Exception:
            return False

I rechecked the current PR head and both of these are empty / unchanged relative to the branch diff:

• git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.py
• the wait_for_instantlock() helper hunk in test/functional/test_framework/test_framework.py

The PR does touch test_framework.py, but only for unrelated RPC bool/type-check adaptations (1 -> True, None handling, stringifying operator_reward) and util.py config plumbing. So this still matches the stale-branch / missing-sync-fix failure mode tracked here, not an in-scope regression from #7242.

[thepastaclaw]

Another occurrence on #7242 (head 24daadcf6d, run https://github.com/dashpay/dash/actions/runs/24580460650):

• failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71880218247
• failure: interface_zmq_dash.py --legacy-wallet timed out waiting for getrawtransaction(txid, True)["instantlock"] in DashZMQTest.test_instantsend_publishers()

Fresh scope check on this PR still says the failure is not caused by the branch changes:

• git diff upstream/develop..HEAD --name-only does not touch test/functional/interface_zmq_dash.py
• this PR does touch test/functional/test_framework/test_framework.py, but only in unrelated RPC-bool / masternode helper hunks; the wait_for_instantlock() helper path is unchanged
• the branch is still behind the develop-side fixes already referenced here (c79c2d981d, d8ac7815e3, merged via 3957e09d59)

So this is another stale-branch / pre-existing InstantLock wait timeout occurrence, not a regression introduced by PR #7242.

[thepastaclaw]

Another occurrence on PR #7242 (head 24daadcf6dd495c76b5ec2ab6c1e6c9dfec979b8):

• failing workflow run: https://github.com/dashpay/dash/actions/runs/24580460650
• failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71880218247

Same failure shape as before:

• interface_zmq_dash.py --legacy-wallet
• timeout while waiting for self.wait_for_instantlock(rpc_raw_tx_1_hash)
• traceback still ends in AssertionError: Predicate "" not true after ... seconds

Scope check for PR #7242 still shows the branch does not touch test/functional/interface_zmq_dash.py, and the diff against upstream/develop does not modify the wait_for_instantlock() helper hunk in test/functional/test_framework/test_framework.py either. This is still stale-branch / unrelated test debt rather than a regression from the PR changes.

[thepastaclaw]

Another occurrence on PR #7290 (head 317c3768959a6b606e1583d5a1b3f1b89846f408):

• failing workflow run: https://github.com/dashpay/dash/actions/runs/24940575732
• failing job: https://github.com/dashpay/dash/actions/runs/24940575732/job/73035655815
• failing check: linux64_ubsan-test / Test source

Same top-level symptom as before:

• interface_zmq_dash.py
• timeout in DashZMQTest.test_instantsend_publishers() while waiting in self.wait_for_instantlock(rpc_raw_tx_1_hash)

Fresh scope check for this PR still says the branch did not introduce the failing test path:

• git diff upstream/develop..HEAD -- test/functional/interface_zmq_dash.py is empty
• the branch diff in test/functional/test_framework/test_framework.py does not touch the wait_for_instantlock() helper hunk

New detail from the archived combined log: this run hit repeated lower-level LLMQ/DKG failures before the InstantLock timeout, for example:

• CSigningManager::AsyncSignIfMember -- failed to select quorum
• CQuorumManager::BuildQuorumFromCommitment -- ... ReadContributions and BuildQuorumContributions ... failed
• ActiveDKGSessionHandler::InitNewQuorum -- ... quorum initialization failed for llmq_test_instantsend

Also unlike the earlier stale-branch reports, this PR head already contains the prior wait/sync fixes referenced here (3957e09d59, c79c2d981d, d8ac7815e3 are all ancestors of 317c376895).

So this looks broader than “branch missing wait_for_instantlock sync fixes”: the same interface_zmq_dash timeout is now also showing up when the wait/sync patches are already present, with the proximate failure being intermittent LLMQ quorum/DKG setup under UBSan rather than a change from PR #7290 itself.

[thepastaclaw]

Another scope-check confirmation for PR #7290 / tracker issue #1106.

• failing workflow run: https://github.com/dashpay/dash/actions/runs/24940575732
• failing job: https://github.com/dashpay/dash/actions/runs/24940575732/job/73035655815
• failing test: interface_zmq_dash.py --legacy-wallet in linux64_ubsan-test / Test source

Fresh verification in the PR worktree (backport-0.26-batch-2-staging, head 317c3768959a6b606e1583d5a1b3f1b89846f408):

git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.py test/functional/test_framework/test_framework.py test/functional/feature_multikeysporks.py src/instantsend/instantsend.cpp src/spork.h

returns no commits. The batch-2 backport merges themselves only touch wallet/rpc/compactblocks/config-args paths; they do not introduce changes in the failing InstantSend/ZMQ path.

The relevant differences in test_framework.py / spork.h / instantsend.cpp are from the branch being behind develop, not from PR #7290's own commits. In particular, HEAD..upstream/develop contains cfad4141d6 / afcf3d148e (SPORK_3 removal follow-ups) plus other post-branch fixes.

So this is still not an in-scope regression from PR #7290 itself. No branch change is appropriate here; tracker issue #1106 should be closed as unrelated CI/test debt tracked here.

[thepastaclaw]

Fresh tracker-1107 scope recheck on PR #7242 head 24daadcf6d:\n\n- failing workflow run: https://github.com/dashpay/dash/actions/runs/24580460650\n- failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71880218247\n- failure: interface_zmq_dash.py --legacy-wallet timed out again in self.wait_for_instantlock(rpc_raw_tx_1_hash)\n\nI rechecked the exact branch diff with git diff upstream/develop..HEAD --name-only on the PR head. It still does not touch test/functional/interface_zmq_dash.py, and the PR’s test_framework.py hunks are unrelated to the wait_for_instantlock() helper path. I also verified this PR head already contains c79c2d981d, d8ac7815e3, and 3957e09d59 in its ancestry, so this occurrence is not explained by a branch-local omission of those earlier wait/sync fixes.\n\nSo this is still unrelated CI/test debt rather than an in-scope regression from PR #7242, and no PR-branch mutation is appropriate for this failure alone.

[thepastaclaw]

Another scope check for PR #7290 / tracker #1121:

• failing workflow run: https://github.com/dashpay/dash/actions/runs/24940575732
• failing job: https://github.com/dashpay/dash/actions/runs/24940575732/job/73035655815
• failing test: interface_zmq_dash.py --legacy-wallet in linux64_ubsan-test / Test source

Rechecked in the PR worktree at 317c3768959a6b606e1583d5a1b3f1b89846f408:

git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.py test/functional/test_framework/test_framework.py src/instantsend/instantsend.cpp src/spork.h

returns no PR commits touching the failing InstantSend/ZMQ path, so the failure is still not coming from this backport batch's own changes.

One new data point: the latest completed develop CI run I checked (25059702916, head 7e524b10eb2b2cc7c8289931c70bc99c65622fcf) had linux64_ubsan-test / Test source green. So this does not look like a deterministic branch-local regression that needs a PR-branch fixup; it still looks intermittent, with this occurrence lining up with the same lower-level LLMQ/DKG instability already noted earlier in this issue.

[thepastaclaw]

Fresh current-develop recheck (tracker #1176):

• Worktree: ~/Projects/dash/worktrees/tracker-1176
• Ref: upstream/develop at 4a0fd854750caf2a8945ac567447ddd3a341c246 (confirmed HEAD...upstream/develop = 0 0 after fetch)
• Confirmed ancestors present:
  • c79c2d981d — refactor: improve is-waiting helper by including mempool sync bump of mocktime
  • d8ac7815e3 — test: fix additional IS lock sync sites that passed nodes=[self.nodes[0]]
  • merge 3957e09d59 — Merge #7241: test: fix intermittent IS lock timeouts by syncing mempools before waiting

Local validation on this current develop head:

PYTHONPATH=/Users/claw/Library/Python/3.9/lib/python/site-packages \
  .venv/bin/python test/functional/interface_zmq_dash.py --legacy-wallet --tmpdir=/tmp/dash-tracker-1176-legacy-1
# passed, PRNG seed 385690915384688966

# repeated legacy runs:
# passed, PRNG seed 6775434089131662380
# passed, PRNG seed 1972982303692681424

PYTHONPATH=/Users/claw/Library/Python/3.9/lib/python/site-packages \
  .venv/bin/python test/functional/interface_zmq_dash.py --tmpdir=/tmp/dash-tracker-1176-default-1
# passed, PRNG seed 4132865026894307226

So I still do not have a current-develop reproduction from the local check: interface_zmq_dash.py passed 3/3 in --legacy-wallet mode and 1/1 in default wallet mode at current develop.

Recommended handling remains:

1. For branches that are actually missing the wait/sync fixes, rebase onto current develop or cherry-pick exactly:
   • c79c2d981d
   • d8ac7815e3
     (or take the containing merge 3957e09d59 if the branch policy allows merge commits).
2. For unrelated PR/backport branches that already contain those commits, do not mutate the PR branch just because interface_zmq_dash.py timed out. First scope-check whether the branch touches test/functional/interface_zmq_dash.py, wait_for_instantlock() in test_framework.py, or the LLMQ/InstantSend code path. If not, treat the failure as intermittent test/CI debt and rerun/track it here rather than force-pushing unrelated changes.

No Dash Core PR opened from this check because I could not reproduce the flake on current develop.