add bats based tests

[adrian-gierakowski]

to make sure sandboxing works as expected

these run as part of top level om ci, or can be ran individually with:

nix develop ./tests -c ./tests/test.bats

the reason I've added this is that:

1. I want to make sure it works as expected before I start using it
2. I would like to add support for macos with sandbox-exec

[srid]

Can we make this a nix flake check to obviate omnix's steps.custom?

This may be useful

https://github.com/juspay/vira/blob/7142be73fd97586ee485e5e2fec20c06de1af951/nix/modules/flake-parts/tests.nix#L6

[adrian-gierakowski]

Can we make this a nix flake check to obviate omnix's steps.custom?

This may be useful

https://github.com/juspay/vira/blob/7142be73fd97586ee485e5e2fec20c06de1af951/nix/modules/flake-parts/tests.nix#L6

I'll try. I assumed this sandboxing functionality would not work inside nix sandbox :)

[srid]

You may need sandbox = relaxed in nix.conf (which is what my CI uses)

[adrian-gierakowski]

You may need sandbox = relaxed in nix.conf (which is what my CI uses)

yeah, although I think it's better to use impure-derivations

[adrian-gierakowski]

btw. what do you use to format nix code?

[srid]

yeah, although I think it's better to use impure-derivations

Sure

btw. what do you use to format nix code?

nixpkgs-fmt

[adrian-gierakowski]

yeah, although I think it's better to use impure-derivations

Sure

got it working with nix build and nix flake check directly in the subflake, but running via omnix I get:

error: pure derivation '/nix/store/dpyp820z4ly5mcwz641qnjfm9if3j56j-devour-output.json.drv' depends on impure derivation '/nix/store/a1iqfbk7f0h0khaj94lbp751lv021c7s-tests.drv'
Error: `nix build` failed; exit code: Some(1)

We'd need to set __impure = true on devour-output as well. Any good way to do this?

btw. what do you use to format nix code?

nixpkgs-fmt

ok, I'm getting diff on file I didn't change, should I format then in a separate commit?

[adrian-gierakowski]

yeah, although I think it's better to use impure-derivations

Sure

got it working with nix build and nix flake check directly in the subflake, but running via omnix I get:

error: pure derivation '/nix/store/dpyp820z4ly5mcwz641qnjfm9if3j56j-devour-output.json.drv' depends on impure derivation '/nix/store/a1iqfbk7f0h0khaj94lbp751lv021c7s-tests.drv'
Error: `nix build` failed; exit code: Some(1)

We'd need to set __impure = true on devour-output as well. Any good way to do this?

ok, so we could check if any of the checks or packages injested by devour-flake has __impure == true and if so, make devour-output.json impure as well. I can open a PR

[srid]

cool

[adrian-gierakowski]

cool

https://github.com/srid/devour-flake/pull/41/files

[srid]

Could you merge from master? That should use the new devour-flake in CI.

Note: no need for rebase in any PRs on my repos; I generally do a squash merge anyway.

[adrian-gierakowski]

@srid all done, vira ci -b passes locally

[adrian-gierakowski]

Issues with CI same as on master. Tests did succeed locally

[adrian-gierakowski]

something is broken with CI on your repo, but I managed to run on my fork. Unfortunately there are multiple layers of failures:

1. the new tests fail in the sandbox but there are no useful logs

2. ever when running the old tests only, it fails on gh-signoff. I guess this could be cause it's running on a fork?

Running outside the sandbox (with nix develop ./tests -c ./tests/test.bats) works, so I think I'll do that for now. Particularly that I don't know how to conditionally build aarch64-darwin depending on the runner when using vira

[adrian-gierakowski]

signoff failed in this repo as well. Disabled it for now.

[adrian-gierakowski]

@srid CI is finally green.

[srid]

Ah, right. I've removed it 6c97965

[adrian-gierakowski]

Ah, right. I've removed it 6c97965

ok, so I will try to add it to vira.hs and see if your runner will manage to build it. You'll need to add:

extra-experimental-features = impure-derivations ca-derivations

[adrian-gierakowski]

Ah, right. I've removed it 6c97965

ok, so I will try to add it to vira.hs and see if your runner will manage to build it. You'll need to add:

extra-experimental-features = impure-derivations ca-derivations

done

[adrian-gierakowski]

not seeing any CI checks, but maybe you can see then on yours side?

[srid]

Does vira ci -b run successfully on your local Linux box? Because ths is what I see:

[..]
copying path '/nix/store/122dqbvxbvj44gqp7g5w09fk56f43njq-bats-1.12.0' from 'https://cache.nixos.org'...
building '/nix/store/awvggiqf2a91mrzhbx19pnxmkm01qhid-nix-shell.drv'...
building '/nix/store/2s0pjl7fbdygkwj6qkcslnsxjs7jgil6-tests.drv'...
nix-shell> Running phase: buildPhase
error: Cannot build '/nix/store/2s0pjl7fbdygkwj6qkcslnsxjs7jgil6-tests.drv'.
       Reason: builder failed with exit code 1.
error: Cannot build '/nix/store/w6dvakaf050jfl4bwgw8qrngqkwbrp77-devour-output.json.drv'.
       Reason: 1 dependency failed.
❌  Task failed with exit code 1 {cmd=/nix/store/x11zi16bhr30l7sb89jbjr0v1b5hyv87-nix-2.32.4/bin/nix build /nix/store/0ihykf5ilwihmcgbilaxgiic6fph2hli-devour-flake#default -L --print-out-paths --no-write-lock-file --override-input flake ./tests --override-input flake/landrun-nix . --out-link ./tests/result}
10:25 ❌ ERROR [🛞 ;19] Process failed: ExitFailure 1
10:25 ❌ ERROR [🛞 ;19] CI pipeline failed

[srid]

not seeing any CI checks, but maybe you can see then on yours side?

For PRs, I need to run manually until we have juspay/vira#292 in place

[adrian-gierakowski]

Does vira ci -b run successfully on your local Linux box?
yes

[adrian-gierakowski]

not seeing any CI checks, but maybe you can see then on yours side?

For PRs, I need to run manually until we have juspay/vira#292 in place

Cool, btw. would love to learn more about vira. Could it be used instead of Hercules CI? Does it support "effects", or only does nix build?

[adrian-gierakowski]

@srid just pushed some more tests. Any luck getting them to run on your side?

[srid]

Asked AI to look into this. It fixed most things except for this one test:

[srid]

Cool, btw. would love to learn more about vira. Could it be used instead of Hercules CI? Does it support "effects", or only does nix build?

For now, just pure Nix build, to keep things simple. I want it to be more like garnix.io but self-hosted.

[srid]

Should I skip this test and push?

[adrian-gierakowski]

Should I skip this test and push?

Could you share link to your changes? I just realised that my last commit broke tests in the sanbox as I was only testing outside of it. But I think it's all fixable. Maybe some limitation of your CI VM related to nested sandboxing ?

[adrian-gierakowski]

just pushed a fix, all tests pass for me in the sandbox:

> nix build ./tests#checks.x86_64-linux.tests -L
tests> 1..21
tests> ok 1 test-true runs successfully
tests> ok 2 test-no-nix-fail: program cannot exec if it cannot access libs from nix store
tests> ok 3 test-no-nix-ldd-ok: program can exec if libs are made accessible with --ldd flag
tests> ok 4 test-add-exec-disabled-fail: program cannot exec if not explicitly allowed
tests> ok 5 test-add-exec-disabled-ldd-ok: script can exec if not explicitly allowed but interpreter and libs are
tests> ok 6 test-extra-args: passes extra arguments to landrun
tests> ok 7 test-ls can list /tmp
tests> ok 8 test-ls can list /nix/store
tests> ok 9 test-ls cannot list / (restricted by default)
tests> ok 10 test-curl-deny fails to connect to example.com
tests> ok 11 test-curl-allow can connect to example.com
tests> ok 12 test-env-var cannot access arbitrary env vars
tests> ok 13 test-env-var inherits configured env var
tests> ok 14 test-read-access: can read allowed file
tests> ok 15 test-write-access: can write allowed file
tests> ok 16 test-no-access: cannot read file not allowed
tests> ok 17 test-multi-paths: respects multiple paths
tests> ok 18 test-nested-paths: rw inside ro works
tests> ok 19 test-multi-env: passes multiple variables
tests> ok 20 test-special-env: passes special characters and multiline
tests> ok 21 test-unrestricted-fs: can access /

if this doesn't work in CI due to some limitation you cannot work around then let's skip the tests (but only when ran in the sanbox) for now so we can get this merged

I've got a macos implementation which passes all tests so going to submit that once this is merged.

[srid]

Okay, with latest commits I see one test failing:

❯ nix log /nix/store/ni2bd4c78dj78jm0yszfhr4236g1i1qm-tests.drv
1..21
ok 1 test-true runs successfully
ok 2 test-no-nix-fail: program cannot exec if it cannot access libs from nix store
ok 3 test-no-nix-ldd-ok: program can exec if libs are made accessible with --ldd flag
ok 4 test-add-exec-disabled-fail: program cannot exec if not explicitly allowed
ok 5 test-add-exec-disabled-ldd-ok: script can exec if not explicitly allowed but interpreter and libs are
ok 6 test-extra-args: passes extra arguments to landrun
ok 7 test-ls can list /tmp
ok 8 test-ls can list /nix/store
ok 9 test-ls cannot list / (restricted by default)
ok 10 test-curl-deny fails to connect to google.com
ok 11 test-curl-allow can connect to google.com
ok 12 test-env-var cannot access arbitrary env vars
ok 13 test-env-var inherits configured env var
ok 14 test-read-access: can read allowed file
ok 15 test-write-access: can write allowed file
not ok 16 test-no-access: cannot read file not allowed
# (in test file /nix/store/prygwpiczyjlzaavs5ppxx0a7fgksj7r-test.bats, line 126)
#   `[ "$status" -ne 0 ]' failed
# Status: 0
# Output: This is a secret
ok 17 test-multi-paths: respects multiple paths
ok 18 test-nested-paths: rw inside ro works
ok 19 test-multi-env: passes multiple variables
ok 20 test-special-env: passes special characters and multiline
ok 21 test-unrestricted-fs: can access /

[adrian-gierakowski]

pushed a commit skipping test-no-access in nix sandbox