fix(build): make tap unit_tests wait for the tap library directory (intermittent 'libcurl.so: file too short') - #6024
Conversation
CI-unit-tests-asan-coverage failed its BUILD step with:
/usr/bin/ld: error: /opt/proxysql/test/tap/tap/libcurl.so: file too short
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:794: genai_fts_string_unit-t] Error 1
Not a corrupt dependency -- a build race. In test/tap/Makefile the `all` and
`debug` targets fan out to `tests tests_with_deps unit_tests`, and
`unit_tests` was the only one of the four sibling targets without the
`tap test_deps` prerequisites (tests, tests_no_infra and tests_with_deps all
declare them). Under parallel make it therefore ran CONCURRENTLY with `tap`.
The unit tests link -lcurl with -L$(TAP_LDIR), i.e. test/tap/tap, and the
`tap` recipe populates that directory by copying the vendored library in:
libcurl$(SHLIB_EXT): $(DEPS_PATH)/curl/curl/lib/.libs/libcurl$(SHLIB_EXT)
cp -a $(DEPS_PATH)/curl/curl/lib/.libs/libcurl$(SHLIB_EXT)* .
libcurl.so and libcurl.so.4 are symlinks onto libcurl.so.4.8.0, ~700 KB of
real payload. A unit test's linker following that chain while cp is still
writing the payload reads a truncated ELF, which is what "file too short"
reports. The CI log catches the overlap directly -- the cp line, unrelated
cmake output from another job, and the ld error are consecutive, followed by
`make[2]: *** Waiting for unfinished jobs....`.
Adding the prerequisites serialises the two, matching the siblings.
Intermittent by nature: it needs a copy and a link to overlap, so it hits
some runs and not others. The same workflow passed on the immediately
preceding commit of the same branch, which is why this reads as a random
infrastructure failure rather than a build-order bug. It is not specific to
ASAN or to genai -- that job just happens to build the unit tests under a
slower toolchain, widening the window.
Verified: `make -C test/tap -n unit_tests` now descends into ../deps and tap
before tests/unit (previously it went straight to tests/unit), and a parallel
`make -C test/tap unit_tests -j$(nproc)` with test/tap/tap/libcurl.so*
deleted beforehand completes cleanly and restores the symlink chain.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout. (1)
🔇 Additional comments (1)
📝 WalkthroughWalkthroughThe ChangesTAP build ordering
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Code Review ✅ ApprovedAdds missing OptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## v3.0 #6024 +/- ##
===========================================
+ Coverage 38.72% 53.14% +14.42%
===========================================
Files 257 483 +226
Lines 98801 144166 +45365
Branches 26183 36446 +10263
===========================================
+ Hits 38261 76622 +38361
- Misses 50151 50531 +380
- Partials 10389 17013 +6624
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 participant
A one-line build-ordering fix for an intermittent CI failure that presents as a corrupt dependency.
Symptom
CI-unit-tests-asan-coveragefails in its build step (the tests never run, and the follow-onRequire non-empty coverage/lcov.infofailure is just the same root cause):Observed on #6021, whose diff touches only
PgSQLFFTOand one PG TAP test — nothing that could reach libcurl or a genai unit test. The same workflow passed on the immediately preceding commit of that same branch.Cause
test/tap/Makefilefansall/debugout totests tests_with_deps unit_tests. Of those four sibling targets,unit_testswas the only one lacking thetap test_depsprerequisites:So under parallel make it ran concurrently with
tap.The unit tests link
-lcurlwith-L$(TAP_LDIR)— that is,test/tap/tap(tests/unit/Makefile:83,218) — and thetaprecipe populates that directory by copying the vendored library into it:libcurl.soandlibcurl.so.4are 16-byte symlinks ontolibcurl.so.4.8.0, ~700 KB of actual payload. A linker following that chain whilecpis still writing the payload reads a truncated ELF — which is precisely whatfile too shortmeans. It is a partial read, not a bad library.The CI log catches the overlap in the act; these lines are consecutive:
Fix
Give
unit_teststhe prerequisites its three siblings already have, which serialises it aftertap.Why it looks random
It needs a copy and a link to overlap, so it hits some runs and not others. Nothing about it is specific to ASAN or to genai — that job simply builds the unit tests under a slower toolchain, which widens the window. Any PR can draw the short straw.
Verification
make -C test/tap -n unit_testsnow descends into../depsandtapbeforetests/unit. Before the change it went straight totests/unit:Deleted
test/tap/tap/libcurl.so*and ranmake -C test/tap unit_tests -j$(nproc): completes cleanly and restores the full symlink chain (libcurl.so→libcurl.so.4→libcurl.so.4.8.0).Related
Surfaced while investigating the ASAN failure on #6021. That PR needs no change for this — a re-run of the job would likely have gone green on its own.
Summary by CodeRabbit