Skip to content

fix(watchdog): handle LND restarts without locking outgoing operations#902

Draft
PatMulligan wants to merge 1 commit intoshocknet:masterfrom
PatMulligan:fix/watchdog-lnd-restart
Draft

fix(watchdog): handle LND restarts without locking outgoing operations#902
PatMulligan wants to merge 1 commit intoshocknet:masterfrom
PatMulligan:fix/watchdog-lnd-restart

Conversation

@PatMulligan
Copy link

Summary

  • When LND restarts, the watchdog's cached payment index becomes stale, causing a false "History mismatch" that locks all
    outgoing operations
  • Instead of immediately locking on a payment index advance, update the cached offset so the watchdog catches up to LND's
    actual state
  • Only lock outgoing operations when both a history mismatch AND a balance discrepancy are detected — indicating a real
    security concern rather than a benign LND restart

Problem

The watchdog caches latestPaymentIndexOffset at startup. When LND restarts, the payment streams reconnect via Warmup(), but
this cache is never refreshed. The next watchdog check sees newLatest > knownMaxIndex and locks outgoing operations with
"History mismatch detected in absolute update, locking outgoing operations", even though the node balance is fine.

This means any LND restart (maintenance, crash recovery, upgrade) requires a manual Lightning.Pub restart to clear the lock.

Fix

In StartCheck(), when a payment index advance is detected:

  1. Update latestPaymentIndexOffset to the new value (catch up)
  2. Only lock if the balance check also fails (deny = true)
  3. If balance is fine, log the index advance and continue operating

Test plan

  • Restart LND while LP is running — verify LP logs "Payment index advanced" and continues operating
  • Verify that a genuine balance discrepancy + history mismatch still triggers the lock
  • Verify normal operation (no LND restart) is unaffected

LP logs from test

# LND restart detected — streams close
17:30:50 lndManager >> Error with subscribeHtlcEvents stream
17:30:50 lndManager >> Error with new block stream
17:30:50 lndManager >> invoice stream closed
17:30:50 lndManager >> LND is dead, will try to reconnect in 20 seconds
17:30:50 lndManager >> subscribeChannelEvents stream closed
17:30:50 lndManager >> onchain tx stream closed

# LND comes back — LP reconnects automatically
17:31:10 unlocker >> the wallet is already unlocked with pub: 03f0919c...
17:31:10 lndManager >> LND is back online

# First watchdog check after reconnect — no lock, no bark
17:31:39 updating tracked balance: 7944398

# Second check — still clean, operations remain unlocked
17:32:39 updating tracked balance: 7944398

@shocknet-justin shocknet-justin requested a review from boufni95 March 4, 2026 17:39
@PatMulligan PatMulligan force-pushed the fix/watchdog-lnd-restart branch from e6513b4 to bf4be80 Compare March 4, 2026 17:54
When LND restarts, its payment index resets while the watchdog's cached
latestPaymentIndexOffset remains at the old value. On the next check,
the watchdog detects a "history mismatch" (newLatest > knownMaxIndex)
and immediately locks outgoing operations — even when the balance is
perfectly fine.

This fix:
1. Updates latestPaymentIndexOffset to catch up when a gap is detected
2. Only locks outgoing operations when BOTH a history mismatch AND a
   balance discrepancy are present

Previously, any LND restart required a manual Lightning.Pub restart to
clear the lock. Now the watchdog gracefully handles LND reconnections
while still protecting against actual payment history anomalies.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@PatMulligan PatMulligan force-pushed the fix/watchdog-lnd-restart branch from bf4be80 to 40fb43b Compare March 4, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant