Skip to content

fix(docker): set musl pthread stack via PT_GNU_STACK (real fix for #60)#61

Merged
ccomb merged 2 commits into
mainfrom
fix/openblas-musl-stack-pt-gnu-stack
May 16, 2026
Merged

fix(docker): set musl pthread stack via PT_GNU_STACK (real fix for #60)#61
ccomb merged 2 commits into
mainfrom
fix/openblas-musl-stack-pt-gnu-stack

Conversation

@ccomb
Copy link
Copy Markdown
Owner

@ccomb ccomb commented May 16, 2026

Summary

  • PR fix(docker): unblock 8 GB VM target — OpenBLAS musl pthread + RTS memory discipline #60 was a no-op. It diagnosed the cause correctly (musl's 128 KB default pthread stack vs. OpenBLAS DYNAMIC_ARCH Fortran kernels with large auto-arrays → SIGSEGV on the first BLAS3 call from MUMPS) but patched a block that the compiler dropped: the injected pthread_attr_setstacksize(&attr, 8 << 20) sits inside #ifdef NEED_STACKATTR, and blas_server.c itself does #undef NEED_STACKATTR unconditionally on Linux (line 103). Verified post-merge: strings /usr/local/bin/volca | grep pthread_attr_setstacksize returns 0 matches in the volca-with-frontend:latest built from fix(docker): unblock 8 GB VM target — OpenBLAS musl pthread + RTS memory discipline #60.
  • Bug reproduces unchanged. User report: same exit 139 at the same point — [MATRIX] Pre-computing factorization for database 'agribalyse-3-2' — on a 16 GB host running the freshly-pruned-and-rebuilt image. OPENBLAS_NUM_THREADS=1 works around it (single-threaded path skips pthread_create), confirming the root cause is still musl's default thread stack.
  • Real fix: set the default at link time via -Wl,-z,stack-size=8388608. musl reads PT_GNU_STACK->p_memsz at process start and uses it as __default_stacksize. Every pthread the binary creates with NULL attr — OpenBLAS workers, GHC RTS capabilities, anything else — starts with 8 MB. No OpenBLAS source patching, no entrypoint env var, no thread-count downgrade.
  • Removes the no-op sed and grep guards from docker/Dockerfile.

Why not patch OpenBLAS properly instead

Two reasons to prefer the linker flag:

  1. NEED_STACKATTR can't simply be #define'd: the second pthread_create(&blas_threads[i], &attr, …) site in goto_set_num_threads references an attr that isn't declared in that function's scope — defining the macro turns latent dead code into a compile error.
  2. The linker flag covers every pthread in the binary, not just OpenBLAS's. RTS capabilities (+RTS -N) inherit musl's default too, so a Haskell-side BLAS call from a non-OpenBLAS worker would still have been at risk.

Test plan

  • ./volca/docker-build.sh --with-frontend from volca-deploy/ produces a volca-with-frontend image.
  • readelf -l dist/volca | grep -A1 GNU_STACK (or on the image's /usr/local/bin/volca) shows MemSiz = 0x800000 (8 MiB).
  • docker run --rm -p 8080:8080 -v /home/ecobalyse/volca_data:/data volca-with-frontend --config /data/volca.toml server --password … no longer crashes at MUMPS factorization for agribalyse-3-2 without OPENBLAS_NUM_THREADS=1.
  • Run a full impact computation on the loaded DB to confirm BLAS multi-threading is actually happening (no silent fallback).

ccomb added 2 commits May 16, 2026 19:00
…x for #60)

#60 diagnosed the SIGSEGV correctly — musl's 128 KB default pthread
stack overflowed on the first BLAS3 call from MUMPS factorization —
but patched the wrong place: the sed inserted pthread_attr_setstacksize
into a block guarded by `#ifdef NEED_STACKATTR`, which blas_server.c
itself `#undef`s unconditionally on Linux (line 103). The injected
code compiled out. `strings volca | grep pthread_attr_setstacksize`
on the merged image returns zero matches.

User-confirmed: `OPENBLAS_NUM_THREADS=1` avoids the crash (no worker
threads → no 128 KB stacks).

Real fix: bake the desired default into the ELF's PT_GNU_STACK header
via `-Wl,-z,stack-size=8388608`. musl reads p_memsz at process start
and uses it as __default_stacksize, so every pthread created with
`NULL` attr — OpenBLAS workers, GHC RTS capabilities, anything else —
starts with 8 MB. No source patching of OpenBLAS needed.
PR #60 shipped broken because nothing in the build pipeline verified
the injected pthread_attr_setstacksize call survived compilation; the
silent regression was only caught by a production SIGSEGV. The fix in
this branch (-Wl,-z,stack-size=8388608) replaces that with a different
invariant — "PT_GNU_STACK->p_memsz == 0x800000 in the shipped ELF" —
which is again only checked manually.

Add a readelf-based assertion right after UPX so a future linker-flag
refactor (or a UPX-side header rewrite) fails the image build with a
diagnostic naming the actual value found, instead of recurring as a
runtime crash on a customer VM.

The check runs on /build/output/volca (post-strip, post-UPX) because
that's the binary that actually runs. binutils — which provides
readelf — is already in the build-stage apk add.
@ccomb ccomb merged commit f46b6f7 into main May 16, 2026
5 checks passed
@ccomb ccomb deleted the fix/openblas-musl-stack-pt-gnu-stack branch May 16, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant