Shared texture mode: background thread crash when embedded in another D3D11 host process

[deblasis]

Summary

When ghostty runs in shared texture mode inside a host process that already has its own D3D11 devices and message loop (e.g. Unity Editor), a ghostty background thread crashes with a native access violation ~50ms after surface creation. The crash kills the host process.

Standalone processes (like the SharedTexture WinForms example in libghostty-dotnet) work fine with the same ghostty.dll and the same shared texture API surface.

Reproduction

1. Build ghostty.dll from this repo (commit 7ac8cd1, branch windows)
2. Load it into Unity 6 via P/Invoke
3. Call the shared texture surface creation sequence:
   • ghostty_init(0, NULL) -- succeeds
   • ghostty_config_new/load/finalize -- succeeds
   • ghostty_app_new -- succeeds
   • ghostty_surface_new with hwnd=NULL, swap_chain_panel=NULL, shared_texture_out=<valid ptr> -- succeeds
   • ghostty_surface_get_d3d11_device/context/texture -- all return valid pointers
4. Call ghostty_app_tick in a loop -- succeeds for 3 frames, D3D11 CopyResource from the texture works
5. ~50ms after surface creation, a ghostty background thread crashes

Crash details

• The crash is on a non-managed thread (Unity's Mono reports "domain required for stack walk")
• Windows reports "Got a UNKNOWN while executing native code"
• SetUnhandledExceptionFilter catches the exception but Unity's own crash handler still terminates
• The crash is consistent and deterministic (~3 ticks, ~50ms)
• Disabling SetOcclusion/SetFocus calls does not change the behavior
• All ghostty native calls from the C# side are working correctly

Analysis

ghostty spawns two background threads on surface creation:

• Renderer thread (renderer_thr) -- runs the D3D11 render loop via libxev
• IO thread (io_thr) -- handles PTY I/O via libxev

Likely root causes (in order of probability):

1. D3D11 multithread protection not enabled

device.zig creates the D3D11 device without D3D11_CREATE_DEVICE_SINGLETHREADED, but the code never calls ID3D11Multithread::SetMultithreadProtected(TRUE) on the context. When the host process (Unity) has its own D3D11 devices and contexts on other threads, the shared GPU/driver state can conflict.

In embedded.zig there's a comment acknowledging this concern:

/// shared texture. Enable multithread protection on the context
/// when accessing from a non-render thread.

But the protection is never actually enabled.

2. libxev event loop conflict

Both background threads use libxev event loops. On Windows, libxev selects between IOCP and wepoll. Inside Unity's process (which has its own message pump and IOCP usage), the libxev initialization or first iteration may fail.

3. Timing matches lazy initialization

The ~50ms crash window (3 ticks at ~16ms each) matches the renderer thread completing its first few event loop iterations and hitting a deferred initialization failure -- not an immediate crash on thread start.

Why standalone works but embedding fails

The standalone SharedTexture example:

• Owns the entire process, clean message loop
• No competing D3D11 contexts
• No competing IOCP/event loop usage

Unity embedding:

• Unity already has D3D11 devices/contexts on multiple threads
• Unity has its own event loop/message pump
• ghostty's threads share GPU/driver state with Unity's threads

Environment

• Windows 11 Pro 10.0.26200
• Unity 6 LTS (6000.4.0f1)
• ghostty.dll from commit 7ac8cd1
• libghostty-dotnet branch feat/unity-integration

Suggested fix

At minimum, calling ID3D11Multithread::SetMultithreadProtected(TRUE) on the device context after creation in device.zig would address the D3D11 thread safety issue.

For full embedding support, the libxev event loop may need a compatibility mode or the threads may need to coordinate with the host process's threading model.

[deblasis]

Additional findings

The crash is an ACCESS_VIOLATION (0xC0000005) inside d3d11.dll at offset 0x15D567 from the DLL base. This happens on a ghostty background thread (renderer or IO).

This occurs both in Unity embedding AND in the standalone SharedTexture WinForms example -- it's not embedding-specific. The SharedTexture example was crashing all along but dotnet run masks it (dotnet.exe catches the thread fault and continues), while running the exe directly terminates.

Tested with the PR #96 multithread protection applied: the crash still happens. SetMultithreadProtected(TRUE) serializes context access but doesn't prevent this particular AV. The issue is likely a null/invalid resource being passed to a D3D11 method on the render thread, not a concurrency race.

The crash is reliably reproducible within the first ~2 seconds of surface creation in shared texture mode.

Workaround

Setting COMPlus_legacyUnhandledExceptionPolicy=1 keeps the process alive (background thread AV doesn't kill the app). The terminal rendering still works after the crash -- the render thread crash doesn't affect the main thread's ability to tick and read the texture.

[deblasis]

Root cause analysis

The crash is an ACCESS_VIOLATION (0xC0000005) at d3d11.dll offset 0x15D567, on the renderer background thread.

Likely cause: missing Flush() before shared texture resize in beginFrame()

In src/renderer/DirectX11.zig lines 226-236, beginFrame() detects a size mismatch and calls st.resizeSharedTexture() which releases the old RTV. But there's no dev.context.Flush() before the release -- the GPU command buffer may still reference the old RTV:

// DirectX11.zig:226-236
if (dev.width != w or dev.height != h) {
    if (renderer.api.shared_target) |*st| {
        // BUG: no Flush() here before releasing old RTV
        st.resizeSharedTexture(dev.device, w, h) catch |err| { ... };
        target.rtv = st.rtv;
        dev.width = w;
        dev.height = h;
    }
}

Compare with device.zig:360-364 where present() correctly calls Flush() for shared texture mode.

Fix

Add dev.context.Flush(); before st.resizeSharedTexture() in beginFrame():

if (renderer.api.shared_target) |*st| {
    dev.context.Flush();  // <-- ensure pending commands complete before releasing RTV
    st.resizeSharedTexture(dev.device, w, h) catch |err| { ... };

Note on testing

The SharedTexture standalone example crashes the same way. It was masked because dotnet run (dotnet.exe host) catches background thread crashes and continues, while the standalone exe terminates. Setting COMPlus_legacyUnhandledExceptionPolicy=1 also keeps the process alive -- the terminal rendering works fine after the renderer thread crash.

All windowed examples (Win32, WinForms, WPF-Simple, WPF-Direct, WinUI3) pass with PR #96's multithread protection. The issue is specific to shared texture mode.

[deblasis]

Resolution

The crash is triggered specifically by ghostty_surface_set_occlusion(surface, true) in shared texture mode. This tells the renderer thread to start drawing, and the renderer thread crashes inside d3d11.dll with an ACCESS_VIOLATION.

Workaround: don't call SetOcclusion(true) in shared texture mode. Rendering works via ghostty_app_tick() without the renderer thread needing to draw.

With this workaround:

• SharedTexture standalone: 10/10 trials pass, 300+ frames with zero crashes
• All windowed examples (Win32, WinForms, WPF-Simple, WPF-Direct, WinUI3): pass

The root cause is in the renderer thread's D3D11 draw path when the surface mode is shared_texture. This should be fixed in the Zig code so that SetOcclusion(true) works safely in shared texture mode, but the workaround is sufficient for now since ghostty_app_tick drives rendering.

[deblasis]

Update: SetOcclusion workaround is insufficient

Further testing shows the crash still occurs ~50% of the time even without calling SetOcclusion(true). The renderer thread is still spawned and its xev event loop still runs -- timer callbacks, wakeup handlers, and updateFrame still execute D3D11 operations even when visible=false.

The VEH approach (catching the AV and suspending the thread) only works ~50% of the time because the .NET CLR has its own vectored exception handler that catches the AV first and calls abort().

Recommended fix

The renderer thread should not be spawned at all in shared texture mode. In this mode, rendering is driven entirely by ghostty_app_tick() on the caller's thread. The renderer background thread serves no purpose and its D3D11 calls cause ACCESS_VIOLATION crashes.

In Surface.zig around line 701-715 where the renderer and IO threads are spawned, shared texture mode should skip spawning renderer_thr. The IO thread is still needed for PTY I/O.

This would eliminate the crash entirely rather than trying to work around it from the consumer side.