Fix #306: end the stream when the console suspends

[quadseven]

Size: XS

Why

#306 is a hung console after waking from sleep mid stream. Orange screen, unresponsive, hard power off to recover.

The hang lives in the window where the OS tears the graphics and network services down underneath a process that keeps running. This ends the stream at that point instead of trying to carry decoder and GPU state through it, so there is no stale state left to get wrong.

This matches Moonlight on Android, where Game.onStop() calls stopConnection() then finish(): sleeping the phone mid stream ends the session and returns you to the host list. Suggested in review by @nyanpasu64, and after building and testing both approaches on hardware I agree it is the better one to ship.

What changed

On losing applet focus, drop input and call the existing terminate(false).

terminateApp is false, so only the stream ends. Whatever is running on the host keeps running and can be resumed by reconnecting.

Acceptance criteria

• Sleeping mid stream ends the stream and shows the host list, with no hang
• Reconnecting after waking works
• HOME mid stream behaves the same way
• Nothing is left running on the host that should not be

Test plan

Tested on hardware, Switch on Atmosphere, streaming a game from a PC:

• Three sleep and resume cycles: stream ends, host list appears, reconnect works, no hang.
• HOME mid stream: same behaviour.

I also built and tested the alternative, which keeps the session alive across the suspend and repairs the GPU frame mappings on the first resumed frame. That also survived three sleeps with no hang, so both are reliable. It is on my fork as archive/306-renderer-invalidation if it is ever wanted.

Out of scope

• Preserving the stream across a suspend. That is the alternative above, and it is a bigger change.
• Any behaviour outside applet focus loss.

[quadseven]

Hardware result, on a V1 console with Atmosphere, running in title takeover mode (AppletType_Application, so appletSetFocusHandlingMode(NoSuspend) is actually in effect).

The hang is gone. Sleep during an active stream, then wake, previously gave a dead screen needing a 15 second power hold and a fresh payload injection. With this change the console wakes, the display comes back after a brief white flash while the display stack reinitialises, and the app stays responsive and exits cleanly.

The overlay confirmed the path runs, in order: StreamingView: window focus lost -> MoonlightSession: rendering suspended -> AppletFocusState_InFocus -> StreamingView: window focus gained -> MoonlightSession: rendering resumed.

Also checked the case this could plausibly have regressed: pressing HOME mid stream, waiting, and returning. The stream stays up and resumes, so the behaviour NoSuspend exists to provide is preserved. updateFrameMapping() reported five cached nvmap handles during that stream, which is the decoder surface pool this change invalidates.

One thing this does not fix, which was previously invisible because the console bricked first: after a real sleep the stream does not recover. Wifi is off while asleep, so the connection dies, and AVFrameQueue::pop() returns bufferFrame when the queue is empty, so the renderer keeps redrawing the last decoded frame indefinitely while input goes to a dead socket. The app is healthy and quits cleanly, it just never notices the session is dead.

That looks like a separate issue in the connection lifecycle rather than something to fold in here, so I have left this PR scoped to the hang. I am instrumenting a build to find out whether connection_terminated fires at all across a sleep and what the reconnect in MoonlightSession::connection_terminated does, and will open a separate PR if there is a clean fix.

[quadseven]

Correction to my previous comment, plus the direct evidence for the root cause.

I was wrong that the stream does not recover. It does. With a file log attached, a real sleep and wake gives:

23:13:07.165 StreamingView: window focus lost
23:13:07.175 MoonlightSession: rendering suspended (active=true terminated=false)
23:13:27.221 Audio Receive: recvUdpSocket() failed: 115
23:13:27.221 MoonlightSession: Connection terminated with code: 115
23:13:27.221 MoonlightSession: Reconnection attempt
23:13:27.224 StreamingView: window focus gained
23:13:27.224 MoonlightSession: rendering resumed
23:13:27.795 MoonlightSession: Reconnected

The reconnect in connection_terminated() completes in about 570ms. What I described earlier as a frozen stream was me quitting during that window on a build with no logging. There is no second bug here, and nothing further is needed. Apologies for the noise.

The evidence for the stale mapping. The same log confirms the decoder surfaces really are reallocated across a sleep, which is what makes the invalidation in this PR load bearing rather than merely harmless.

nvmap handles cached by updateFrameMapping() before the sleep:

1236, 1244, 1252, 1276, 1292, 1300

On resume:

23:13:27.261 DKVideoRenderer: dropped 6 hardware frame mapping(s)

and the handles the renderer then maps after reconnecting:

1372, 1380, 1388, 1396, 1404, 1420

A completely disjoint set. Every mapping cached before the sleep was pointing at decoder memory that no longer existed in that form. Without the invalidation, updateFrameMapping() would have matched the stale cache entries by handle and kept sampling them, which is the GPU fault this PR is about.

The instrumentation used to produce this is a local diagnostic build, not part of this PR.

[nyanpasu64]

What happens if you remove appletSetFocusHandlingMode(AppletFocusHandlingMode_NoSuspend)? Please have your human test and reply rather than letting Claude write a response.

[quadseven]

I tested it out. I built three versions: master with that line deleted, master untouched as a control, and this PR's branch unmodified. I used the same commit for the first two, so the only difference is just that one line.

With the line deleted, sleep and resume still crashes. I got a solid orange screen, the console hung, and I had to hold down power to get out of it. Pressing Home mid-stream also kills the stream, and doing it a second time panicked Atmosphere with Title ID: 0100000000000034, std::abort (0xFFE). That title ID is Atmosphere's own fatal module (stratosphere/fatal/fatal.json).

With the untouched master on the same commit: sleep and resume drops the stream, but the app stays up and relaunches fine, and pressing Home works 3/3 time. This PR's branch survives sleep and resume 3/3 with the controls still working afterwards.

So it doesn't fix the crash, and it basically turns a recoverable stream drop into a hung console.

One caveat: stock master degraded gracefully for me rather than hard-crashing the way #306 describes. So, this is really just measuring the effect of removing the call, not a reproduction of the original report.

Also, the call isn't even in this repo. It's in the borealis submodule at library/lib/platforms/switch/switch_platform.cpp:112, and xfangfang's borealis (the one wiliwili uses) has it in the exact same place. It seems like a framework-level thing rather than something this PR could just drop.

[nyanpasu64]

Why not pause the stream and return to the games menu upon console sleep? This should avoid all "stream surviving sleep mode" issues, and is consistent with behavior on Android (though a bit less convenient than fully surviving sleep, but even then it's a "good enough" solution even if a better fix gets added in the future).

[quadseven]

Swapped this PR over to your suggestion.

I built both and tested them on hardware first, three sleeps each plus home. Parity version ends the stream, lands on the host list, reconnects fine. The old approach kept the stream alive with controls working. Neither hangs, so reliability was a wash and it came down to which is less to go wrong later.

This one reuses the existing terminate(false) and carries nothing across the suspend, so there's no state left to get stale. I checked moonlight-android first and Game.onStop() does call stopConnection() then finish(), so it really is the same behaviour.

Old approach is on my fork as archive/306-renderer-invalidation if it's ever wanted.

[nyanpasu64]

Note that I am running a beta version of libnx with incompatible changes, and I'm unsure if that's a factor.

Tried copying the file over the installed filename. Now it sometimes exits properly and sometimes crashes upon wake. I've managed to reproduce the crash via home screen, which should be easier to debug than sleep (since the latter disconnects gdb).

[nyanpasu64]

Thread 2 "libnx Thread_0x1466920fb0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 147.1170]
0x0000001464574124 in MoonlightSession::stop (this=0x30, terminate_app=0) at /home/nyanpasu64/code/Moonlight-Switch/app/src/streaming/MoonlightSession.cpp:373
373         if (m_stop_requested)
=> 0x0000001464574124 <_ZN16MoonlightSession4stopEi+4>: 39456800        ldrb    w0, [x0, #346]
   0x0000001464574128 <_ZN16MoonlightSession4stopEi+8>: 370003e0        tbnz    w0, #0, 0x14645741a4 <_ZN16MoonlightSession4stopEi+132>
(gdb) set width 0
(gdb) bt
#0  0x0000001464574124 in MoonlightSession::stop (this=0x30, terminate_app=0) at /home/nyanpasu64/code/Moonlight-Switch/app/src/streaming/MoonlightSession.cpp:373
#1  0x000000146458489c in StreamingView::terminate (this=0x7315321a60, terminateApp=<optimized out>) at /home/nyanpasu64/code/Moonlight-Switch/app/src/streaming_view.cpp:452
#2  0x00000014646c8cf4 in std::function<void(bool)>::operator() (this=0x616add7cb0, __args#0=<optimized out>) at /opt/devkitpro/devkitA64/aarch64-none-elf/include/c++/16.1.0/bits/std_function.h:581
#3  brls::Event<bool>::fire (this=0x1465489750 <brls::Application::windowFocusChangedEvent>, args#0=false) at /home/nyanpasu64/code/Moonlight-Switch/extern/borealis/library/include/borealis/core/event.hpp:76
#4  0x0000001464f77b0c in appletCallHook (hookType=<optimized out>) at /home/nyanpasu64/code/libnx/nx/source/services/applet.c:526
#5  appletProcessMessage (msg=<optimized out>) at /home/nyanpasu64/code/libnx/nx/source/services/applet.c:3080
#6  0x0000001464f77d24 in appletMainLoop () at /home/nyanpasu64/code/libnx/nx/source/services/applet.c:3124
#7  0x000000146465d324 in brls::Application::internalMainLoop () at /home/nyanpasu64/code/Moonlight-Switch/extern/borealis/library/lib/core/application.cpp:206
#8  0x000000146465dc3c in std::__invoke_impl<bool, bool (*&)()> (__f=@0x616add7f20: 0x146465d280 <brls::Application::internalMainLoop()>) at /opt/devkitpro/devkitA64/aarch64-none-elf/include/c++/16.1.0/bits/invoke.h:63
#9  std::__invoke_r<bool, bool (*&)()> (__fn=@0x616add7f20: 0x146465d280 <brls::Application::internalMainLoop()>) at /opt/devkitpro/devkitA64/aarch64-none-elf/include/c++/16.1.0/bits/invoke.h:116
#10 std::_Function_handler<bool(), bool (*)()>::_M_invoke (__functor=...) at /opt/devkitpro/devkitA64/aarch64-none-elf/include/c++/16.1.0/bits/std_function.h:295
#11 std::function<bool()>::operator() (this=0x616add7f20) at /opt/devkitpro/devkitA64/aarch64-none-elf/include/c++/16.1.0/bits/std_function.h:581
#12 brls::Platform::runLoop (this=<optimized out>, runLoopImpl=...) at /home/nyanpasu64/code/Moonlight-Switch/extern/borealis/library/include/borealis/core/platform.hpp:176
#13 brls::Application::mainLoop () at /home/nyanpasu64/code/Moonlight-Switch/extern/borealis/library/lib/core/application.cpp:196
#14 0x00000014644d42c4 in main (argc=1, argv=0x7313104038) at /home/nyanpasu64/code/Moonlight-Switch/app/src/main.cpp:180

The bug is that you call StreamingView::terminate before MoonlightSession* session is initialized. Though this shouldn't be possible (modulo race conditions) since StreamingView initializes session in its constructor.

If I breakpoint MoonlightSession::MoonlightSession, hit the home button then, then continue execution, then StreamingView::terminate is called, and we get a crash in:

Thread 5 "libnx Thread_0x7696377900" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 149.1325]
GameStreamClient::server_data (this=0x34e718bf8 <Singleton<GameStreamClient>::instance()::instance>, address=...) at /home/nyanpasu64/code/Moonlight-Switch/app/src/streaming/GameStreamClient.hpp:57
57              return m_server_data[address];
(gdb) set width 0
(gdb) bt
#0  GameStreamClient::server_data (this=0x34e718bf8 <Singleton<GameStreamClient>::instance()::instance>, address=...) at /home/nyanpasu64/code/Moonlight-Switch/app/src/streaming/GameStreamClient.hpp:57
#1  operator() (__closure=0x7698139620) at /home/nyanpasu64/code/Moonlight-Switch/app/src/streaming/MoonlightSession.cpp:348
#2  0x000000034d90931c in std::function<void()>::operator() (this=0x2fb0f27ef0) at /opt/devkitpro/devkitA64/aarch64-none-elf/include/c++/16.1.0/bits/std_function.h:581
#3  brls::ThreadPool::threadEntry (this=0x76963774a0, i=<optimized out>) at /home/nyanpasu64/code/Moonlight-Switch/extern/borealis/library/lib/core/thread_pool.cpp:175
#4  0x000000034e2735bc in execute_native_thread_routine ()
#5  0x000000034e213584 in __thread_entry (__arg=0x2f67424e80) at /home/nyanpasu64/code/libnx/nx/source/runtime/newlib.c:160
#6  0x000000034e201728 in _EntryWrap (args=0x2fb0f27fd0) at /home/nyanpasu64/code/libnx/nx/source/kernel/thread.c:57

It appears the change made is wildly concurrency-unsafe. Out of curiosity can you push your old code to a different branch for me to test?