FR: SYSROOT generated by rules_distroless for hermetic CC complation

[thesayyn]

While working on a GLIBC related problem, i realized that rules_distroless could have made hermetic cc compilation so much easier by generating a sysroot out of the debian packages that are being fetched.

This is great for one single reason, what's actually going into the container is also what you are linking against!

I can imagine a bzlmod only API, as such

# Generate a sysroot using the apt extensions
apt = use_extension("@rules_distroless//apt:extensions.bzl", "apt")

apt.install("libc6-dev")

apt.sysroot(
    name = "sysroot_amd64"
    arch = "amd64"
)
 
apt.sysroot(
    name = "sysroot_arm64"
    arch = "arm64"
)
 
use_repo(apt, "sysroot_amd64", "sysroot_arm64")
 
# Use with toolchains_llvm
llvm = use_extension("@toolchains_llvm//toolchain/extensions:llvm.bzl", "llvm")
llvm.sysroot(
    name = "llvm_toolchain_with_sysroot",
    targets = ["linux-x86_64"],
    label = "@sysroot_amd64//:sysroot",
)
llvm.sysroot(
    name = "llvm_toolchain_with_sysroot",
    targets = ["linux-arm64"],
    label = "@sysroot_arm64//:sysroot",
)

[JonathanPerry651]

This is a very cool idea

[jjmaestro]

This is awesome!! I was wondering about something like this as well! TIL about bazel-contrib/toolchains_llvm's sysroot!

Having something like this API would be amazing, I had to hack something together for a project I'm working on. Basically a custom Docker image from where I then create toolchains using bazelbuild/bazel-toolchains and I was going to eventually try something like this but much less nice, just create the image with rules_distroless + rules_oci and still have to call rbe_configs_gen.

This is neat and I'd love to have it! Hopefully other Cxx toolchains and other rules support something like this sysroot... and maybe Bazel itself would support it natively, it would be great to be able to create a sysroot(s) for the full linux hermetic sandbox. Currently it's very hard to setup, requires manual fiddling, etc. Being able to produce a sysroot and tell Bazel to use it for sandboxing would be 🔥

[JonathanPerry651]

So just thinking about this one a bit more, and with the caveat that I’m way out of my depth - could we figure out a way to make bazel invoke actions under a root thats completely controlled by rules_distroless? This would be (very close to) the holy grail for making sure you fully understand your environment, and for making sure that your builds tests and production runtime are deeply aligned…

[JonathanPerry651]

Context here is that this repo has been exceptionally helpful for us uplifting our containers across the board (thank you!), but we still have the horrible situation where test environments (ie remote build base containers) contain, say, grep, but our production runtime doesn’t - and grep is required by bazel’s java_binary wrapper script. Sadness ensues.

[lukasoyen]

I was inspired by the idea of this FR and started hacking in https://github.com/lukasoyen/bazel_debian_packages/tree/apt-sysroot.

But then I realized this might contradict the stated goal of the repository, so I decided to fork rules_distroless and strip it down to only the functionality to create sysroots.

If you see it within scope of rules_distroless I am happy to collaborate on integrating the idea.

[thesayyn]

Sure, please make a PR so we can discuss there.

[apsaltis-ddog]

👋 I'd love to see this too -- I independently came to a similar conclusion as y'all last week, and would be happy to collaborate on this.

What I have so far is a bunch of targets for assembling the sysroots externally of where they would actually be used; more befitting of an standalone example, or an example of how code might be generated more than anything else.

[thesayyn]

could we figure out a way to make bazel invoke actions under a root thats completely controlled by rules_distroless

Bazel is cross platform, which means whatever we do here, won't work on macos and windows, as i heavily use macos, i'd like whatever we built works on at least macos and linux.

[alexeagle]

https://github.com/malt3/sysroots looks like a start on this already.

[LittleHuba]

I tried to apply the approach in the linked PR to make toolchains_llvm use a sysroot supplied by rules_distroless.
Unfortunately, this fails with a cyclic dependency:

ERROR: /home/user/.cache/bazel/_bazel_user/e5ca210826d4a557c2389a5d3cc48128/external/gawk+/BUILD.bazel:160:10: in cc_binary rule @@gawk+//:gawk: cycle in dependency graph:
    //:my_target (db67a81c70d6e653f966dd66f09f3820c5d8af0ecf3661af71421f949b807468)
    @@bazel_tools//tools/cpp:current_cc_toolchain (7764605b3dc7bb8643b974d1684cabae78b0fbf40973b66f23136df391ad8f2e)
    @@rules_cc+//cc:current_cc_toolchain (7764605b3dc7bb8643b974d1684cabae78b0fbf40973b66f23136df391ad8f2e)
    @@toolchains_llvm++llvm+llvm_toolchain//:cc-clang-x86_64-linux (7764605b3dc7bb8643b974d1684cabae78b0fbf40973b66f23136df391ad8f2e)
    @@toolchains_llvm++llvm+llvm_toolchain//:module-x86_64-linux (7764605b3dc7bb8643b974d1684cabae78b0fbf40973b66f23136df391ad8f2e)
    @@toolchains_llvm++llvm+llvm_toolchain//:include-components-x86_64-linux (7764605b3dc7bb8643b974d1684cabae78b0fbf40973b66f23136df391ad8f2e)
    @@toolchains_llvm++llvm+llvm_toolchain//:sysroot-components-x86_64-linux (7764605b3dc7bb8643b974d1684cabae78b0fbf40973b66f23136df391ad8f2e)
    @@rules_distroless++apt+ubuntu24_04//:flat (7764605b3dc7bb8643b974d1684cabae78b0fbf40973b66f23136df391ad8f2e)
.-> @@gawk+//:gawk (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
|   @@bazel_tools//tools/cpp:current_cc_toolchain (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
|   @@rules_cc+//cc:current_cc_toolchain (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
|   @@toolchains_llvm++llvm+llvm_toolchain//:cc-clang-x86_64-linux (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
|   @@toolchains_llvm++llvm+llvm_toolchain//:module-x86_64-linux (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
|   @@toolchains_llvm++llvm+llvm_toolchain//:include-components-x86_64-linux (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
|   @@toolchains_llvm++llvm+llvm_toolchain//:sysroot-components-x86_64-linux (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
|   @@rules_distroless++apt+ubuntu24_04//:flat (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
`-- @@gawk+//:gawk (e7344cab726cd15ae7e30f46f86041553a4869d15c7fdf664ed0c023215ceda8)
Target //:my_target was skipped
ERROR: Analysis of target '//:my_target' failed; build aborted

So I think rules_distroless must not rely on any tool that itself requires a toolchain to be compiled.
If the usage of gawk cannot be dropped, it might be a solution to add gawk to https://cosmo.zip/ and pull the prebuilt binary from there.

[thesayyn]

Hmm, that one is a weird case, only place we rely on gawk is flatten. so probably #213 already fixed this and we need to pick that up.