[Bug]: Station G2 TCP connection drops repeatedly when MeshMonitor is connected (2.7.21 alpha)

[rancur]

Category

WiFi

Hardware

Station G2

Is this bug report about any UI component firmware like InkHUD or Meshtatic UI (MUI)?

• Meshtastic UI aka MUI colorTFT
• InkHUD ePaper
• OLED slide UI on any display

Firmware Version

2.7.21.1370b23 (alpha)

Description

The Station G2 repeatedly drops its TCP connection when MeshMonitor maintains a persistent connection on port 4403. The device accepts the TCP connection, begins the config exchange (LoRa config request), then drops the connection — cycling between connected and disconnected every few minutes.

The device is reachable on the network (HTTP on port 80 responds, TCP port 4403 accepts connections), but the Meshtastic protocol-level connection fails during or shortly after the config handshake.

Symptoms:

• MeshMonitor connects successfully, requests LoRa config
• Device drops the connection during config exchange
• MeshMonitor reconnects, same cycle repeats
• Pattern: connected → config request → disconnected → reconnect delay → repeat
• The device web UI (/) responds normally throughout — only the TCP API connection drops

Environment:

• Station G2 connected via WiFi on IoT VLAN (192.168.3.x)
• MeshMonitor v3.12.0 running on Docker (Synology NAS), connecting via MESHTASTIC_NODE_IP
• Active Arizona mesh network (~600+ nodes)
• Virtual Node enabled on MeshMonitor side

Workaround: Rebooting the Station G2 (via POST /restart) temporarily restores the connection, but it drops again after some time. Currently using a watchdog script that monitors MeshMonitor's connection status and auto-reboots the G2 when it disconnects.

Likely related: This appears to be the same heap memory leak reported in #9632 (Heltec V4 / RAK4631), but affecting the Station G2. In that issue, heap usage continuously increases when a TCP client like MeshMonitor maintains a persistent connection, eventually causing the device to crash/disconnect. The symptoms here (periodic disconnects on a ~1-2 hour cycle) are consistent with a slow memory leak exhausting available heap.

MeshMonitor logs showing the cycle:

[INFO] [DataEventEmitter] Connection status: connected
[INFO] Node reconnect notification sent
[INFO] Skipping module config request on reconnect (already fetched this session)
[INFO] Virtual node server already initialized, refreshing connected clients with fresh config
[INFO] [DataEventEmitter] Connection status: disconnected
[INFO] Node disconnect notification sent
[INFO] [DataEventEmitter] Connection status: connected
[INFO] Node reconnect notification sent
[INFO] Skipping module config request on reconnect (already fetched this session)
[INFO] Virtual node server already initialized, refreshing connected clients with fresh config
[INFO] [DataEventEmitter] Connection status: disconnected
[INFO] Node disconnect notification sent

After a fresh reboot of the G2, the connection is stable initially but degrades over time:

[INFO] [DataEventEmitter] Connection status: connected
[INFO] Requesting LoRa config from device...
[INFO] GetConfigResponse structure: ["lora","payloadVariant"]
[INFO] Session passkey received from local node and stored (expires in 290 seconds)
[INFO] Starting remote admin scanner with 60 minute interval
[INFO] Starting time sync scheduler with 15 minute interval

Steps to Reproduce

1. Connect a Station G2 to WiFi
2. Point MeshMonitor (or any persistent TCP client) at the G2's IP on port 4403
3. Wait 1-2 hours
4. Observe repeated connect/disconnect cycling in MeshMonitor logs

Relevant log output

[INFO] [DataEventEmitter] Connection status: disconnected
[INFO] Node disconnect notification sent
[INFO] [DataEventEmitter] Connection status: connected
[INFO] Node reconnect notification sent
[INFO] Requesting LoRa config from device...
[INFO] [DataEventEmitter] Connection status: disconnected
[INFO] Node disconnect notification sent
[INFO] No device config available yet - returning null

[github-actions]

The issue provides a clear description of the bug, but it lacks device logs for further investigation. Please provide the logs from the Web Flasher or Meshtastic CLI to assist in diagnosing the problem.

[rancur]

Thanks for the quick response. Here are the device details:

Device: Station G2 (hwModel 31, ESP32-S3, 16MB partition)
Firmware: 2.7.21.1370b23 (alpha)
Connection: WiFi on IoT VLAN, MeshMonitor v3.12.0 connected via TCP port 4403

I don't have serial logs at the moment since the device isn't near a USB-connected machine, but I can share what MeshMonitor captures from the TCP side. The pattern is very consistent:

1. TCP connection established successfully
2. Config exchange begins (LoRa config request/response, module config, session passkey)
3. Connection drops after ~1-2 hours of uptime
4. Device remains reachable on HTTP (port 80 responds, POST /restart works)
5. TCP port 4403 accepts new connections but drops them during config handshake
6. Only a full device reboot (POST /restart) restores stable connectivity temporarily

This is nearly identical to the behavior described in #9632 (heap memory leak with MeshMonitor connected on Heltec V4). The timing (~1-2 hours before disconnect) is consistent with gradual heap exhaustion.

I'll try to get serial logs next time I have physical access to the device. Would heap debug output (-D DEBUG_HEAP) be most useful, similar to what was done in #9632?

[rancur]

Update: I downgraded from 2.7.21 alpha to 2.7.20 alpha via OTA — the TCP disconnect issue persists on 2.7.20 as well. The watchdog I set up is catching ~2 disconnects per day so far, each requiring a device reboot to restore the connection.

This confirms the issue is not a 2.7.21 regression — it exists in 2.7.20 too. Next step would be testing on 2.7.15 beta to see if it's a 2.7.x-wide issue.

Timeline since downgrade:

[2026-04-08 06:00] Disconnected — reboot sent, recovered
[2026-04-08 06:50] Disconnected — reboot sent, recovered

Pattern is consistent: stable for several hours, then TCP drops and only a full device reboot restores it.

[rancur]

Important correction: After further investigation, we discovered that the majority of the disconnect/reconnect cycling we reported was caused by MeshMonitor's stale connection detector, not a firmware TCP bug.

MeshMonitor's STALE_CONNECTION_TIMEOUT (default 5 minutes) forces a reconnect when no data is received. On our Station G2 with CLIENT_MUTE role, the node legitimately receives very little mesh traffic, causing MeshMonitor to repeatedly declare the connection stale and force reconnects (973 cycles in a few hours). Increasing the timeout to 30 minutes resolved the rapid cycling.

We filed this as a MeshMonitor bug: Yeraze/meshmonitor#2609

That said, there is still a genuine firmware-side issue: even with MeshMonitor's stale detector disabled, we do see occasional TCP ECONNRESET errors where the device drops the TCP connection after ~1-2 hours. This is consistent with the heap leak reported in #9632. Our NAS instance (Station G2, ROUTER_LATE role, firmware 2.7.20) sees ~2 genuine disconnects per day that require a device reboot to recover — these are real firmware-side drops, not MeshMonitor's stale detector.

So both issues are real, but the rapid cycling we initially reported was MeshMonitor-side, while the slower ~1-2 hour drops are firmware-side.

[ccchdro]

Thanks for the quick response. Here are the device details:

Device: Station G2 (hwModel 31, ESP32-S3, 16MB partition) Firmware: 2.7.21.1370b23 (alpha) Connection: WiFi on IoT VLAN, MeshMonitor v3.12.0 connected via TCP port 4403

I don't have serial logs at the moment since the device isn't near a USB-connected machine, but I can share what MeshMonitor captures from the TCP side. The pattern is very consistent:

1. TCP connection established successfully
2. Config exchange begins (LoRa config request/response, module config, session passkey)
3. Connection drops after ~1-2 hours of uptime
4. Device remains reachable on HTTP (port 80 responds, POST /restart works)
5. TCP port 4403 accepts new connections but drops them during config handshake
6. Only a full device reboot (POST /restart) restores stable connectivity temporarily

This is nearly identical to the behavior described in #9632 (heap memory leak with MeshMonitor connected on Heltec V4). The timing (~1-2 hours before disconnect) is consistent with gradual heap exhaustion.

I'll try to get serial logs next time I have physical access to the device. Would heap debug output (-D DEBUG_HEAP) be most useful, similar to what was done in #9632?

I have the same problem but it is from MeshMonitor which has many bugs. The latest version that works fine for me without disconnections is MeshMonitor v3.9.5.

[nightjoker7]

I hit this exact pattern on a Station G2 pair (same hw_model 31) running 2.7.23 + custom patches this week, with meshlogger as the persistent TCP client (similar role to MeshMonitor). Symptoms match yours byte-for-byte: accept → LoRa config request starts → drop mid-exchange → reconnect → repeat.

Root cause is two things compounding. @rancur's investigation identified MeshMonitor's stale-connection detector as one trigger, which is correct but not the whole story.

(1) meshtastic.tcp_interface.TCPInterface(host) in the Python library leaks a socket + reader thread on every handshake timeout — and a Station G2 with a mid-sized NodeDB takes meaningful time to dump config, making handshake timeout the common case. Every retry leaks resources on both sides. Filed as meshtastic/python#918.

(2) Under the compounding heap/socket pressure from the retry loop, getMeshNode() calls in Router::perhapsDecode and MeshService::handleFromRadio return null, and the next deref hard-faults. That's the actual source of the "drop mid-exchange" behavior — the node is crashing, not dropping the TCP. Filed as #10226 and #10229.

Also contributing: #10219 (merged two days ago — SX1262 sensitivity loss after AGC reset) raises the ambient packet-drop rate your G2 sees, which increases the retry pressure.

All four together:

• meshtastic/python#918 — client handshake-leak fix
• SX126x: re-apply 0x8B5 register in resetAGC() to preserve RX sensitivity #10219 — SX1262 0x8B5 sensitivity restore (merged)
• Router: null-check destination node before dereferencing public_key in PKI decode #10226 — Router PKI destination null-check
• MeshService: null-check sender node before dereferencing has_user on hot RX path #10229 — MeshService sender null-check

Post-patch, my Station G2 pair runs stably under continuous meshlogger polling (5s intervals) for 4+ hours straight, NodeDB at MAX_NUM_NODES, the same monitoring pattern that was causing the cycle pre-patch. Happy to share heap trajectories if that'd help the review.