Make ephemeral exec, venv repl and scie compatible

[tobni]

Fix so that scie pex (zipapp, venv) can 1. drop into repl and 2. run in ephemeral run mode via -- -c.

#!/usr/bin/env bash
set -euo pipefail

TMPDIR=$(mktemp -d)
PEX_ROOT="$TMPDIR/pex_root"
uv run python -m pex . -c pex -o "$TMPDIR/pex.pex" --scie eager --venv --pex-root "$PEX_ROOT" --runtime-pex-root "$PEX_ROOT" 2>&1 | tail -1

echo "=== Venv scie: -- -c ==="
"$TMPDIR/pex" --pex-root "$PEX_ROOT" --runtime-pex-root "$PEX_ROOT" --interpreter-constraint 'CPython>=3.12' -- -c 'import sys; print(sys.executable)' 2>&1
echo "EXIT: $?"

echo ""
echo "=== Venv scie: REPL ==="
echo 'import sys; print("REPL:", sys.executable); quit()' | "$TMPDIR/pex" --pex-root "$PEX_ROOT" --runtime-pex-root "$PEX_ROOT" --interpreter-constraint 'CPython>=3.12' -- 2>&1
echo "EXIT: $?"

echo ""
echo "=== Non-venv scie: -- -c ==="
TMPDIR2=$(mktemp -d)
uv run python -m pex . -c pex -o "$TMPDIR2/pex.pex" --scie eager 2>&1 | tail -1
"$TMPDIR2/pex" --interpreter-constraint 'CPython>=3.12' -- -c 'import sys; print(sys.executable)' 2>&1
echo "EXIT: $?"

This gives me on branch:

(pex-dev) ➜  pex git:(fix/scie-ephemeral-run-mode) ✗ bash test_scie_ephemeral.sh
/tmp/tmp.Uj3pbC1P18/pex
=== Venv scie: -- -c ===
/tmp/tmp.Uj3pbC1P18/pex_root/scies/0/base/86232383051a0950a74a9486949c87f88da45978bafa8d826c2f268629a3f508/cpython-3.12.13+20260325-x86_64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.12
EXIT: 0

=== Venv scie: REPL ===
Pex 2.91.4 ephemeral hermetic environment with no dependencies.
Python 3.12.13 (main, Mar 24 2026, 22:49:22) [Clang 22.1.1 ] on linux
Type "help", "pex", "copyright", "credits" or "license" for more information.
>>> REPL: /tmp/tmp.Uj3pbC1P18/pex_root/scies/0/base/86232383051a0950a74a9486949c87f88da45978bafa8d826c2f268629a3f508/cpython-3.12.13+20260325-x86_64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.12
EXIT: 0

=== Non-venv scie: -- -c ===
/tmp/tmp.3ibEcHDvMR/pex
/home/tobias/.cache/nce/86232383051a0950a74a9486949c87f88da45978bafa8d826c2f268629a3f508/cpython-3.12.13+20260325-x86_64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.12
EXIT: 0

On main ephemeral exec and venv+repl fail in different ways:

(pex-dev) ➜  pex git:(main) ✗ bash test_scie_ephemeral.sh
/tmp/tmp.tPRXEL8YLa/pex
=== Venv scie: -- -c ===
Ignoring the following environment variables in Pex venv mode:
_PEX_CLI_RUN=/tmp/tmp.tPRXEL8YLa/pex
__PEX_ENTRY_POINT__=/tmp/tmp.tPRXEL8YLa/pex_root/venvs/3/d806b12a09b8ad5281bf3d990cb91bfb694c1be1/414e2ae92b4573e3049bcfe135ed1296986a612d
Could not find script 'import sys; print(sys.executable)' in any distribution  within PEX!
EXIT: 1

=== Venv scie: REPL ===
Ignoring the following environment variables in Pex venv mode:
_PEX_CLI_RUN=/tmp/tmp.tPRXEL8YLa/pex
__PEX_ENTRY_POINT__=/tmp/tmp.tPRXEL8YLa/pex_root/venvs/3/d806b12a09b8ad5281bf3d990cb91bfb694c1be1/414e2ae92b4573e3049bcfe135ed1296986a612d
Ignoring the following environment variables in Pex venv mode:
_PEX_CLI_RUN=/tmp/tmp.tPRXEL8YLa/pex
_PEX_CLI_RUN_NO_ARGS=/tmp/tmp.tPRXEL8YLa/pex
__PEX_ENTRY_POINT__=/tmp/tmp.tPRXEL8YLa/pex_root/venvs/3/d806b12a09b8ad5281bf3d990cb91bfb694c1be1/414e2ae92b4573e3049bcfe135ed1296986a612d
... # this loops forever

=== Non-venv scie: -- -c ===
/tmp/tmp.seG0un8tmn/pex
Could not find script 'import sys; print(sys.executable)' in any distribution  within PEX!
EXIT: 1

[jsirois]

@tobni can you explain why you want to do this? Is this just for conceptual completeness or is this part of a real workflow you use? I partially ask because you're using your own way of building the Pex PEX scie that is not how the official Pex PEX scies are built for release. Those use uv run dev-cmd package -- --scie (or --scies if you want to build them all) and that uses some flags you do not:

pex/scripts/create-packages.py

Lines 71 to 121 in 013a078

	def build_pex_scie(
	platform_info: tuple[PlatformConfig, Path],
	*,
	lock: Path,
	pex_requirement: str,
	verbosity: int = 0,
	env: Optional[Dict[str, str]] = None,
	) -> SpawnedJob[tuple[str, PlatformConfig]]:
	platform_config, complete_platform = platform_info
	command = [
	sys.executable,
	"-m",
	"pex",
	*["-v" for _ in range(verbosity)],
	"--disable-cache",
	"--no-build",
	"--no-compile",
	"--no-use-system-time",
	"--record-git-state",
	"--venv",
	"--no-strip-pex-env",
	"--complete-platform",
	str(complete_platform),
	"--lock",
	str(lock),
	"--scie",
	"eager",
	"--scie-only",
	"--scie-name-style",
	"platform-file-suffix",
	"--scie-platform",
	platform_config.name,
	"--scie-pbs-release",
	platform_config.pbs_release,
	"--scie-python-version",
	platform_config.python_version,
	"--scie-pbs-stripped",
	"--scie-hash-alg",
	"sha256",
	"--scie-busybox",
	"@pex",
	"--scie-busybox-pex-entrypoint-env-passthrough",
	"--pip-version",
	PipVersion.LATEST_COMPATIBLE.value,
	"-o",
	os.path.join(safe_mkdtemp(), "pex"),
	"-c",
	"pex",
	"--project",
	pex_requirement,
	]

[tobni]

My use case is indeed to be able to probe/access the released scies chosen interpreter given some constraint.

I have had trouble with pyenv shims (used for work) confusing both me and the uv dev-cmd so I tried to replicate outputs without having to disable them when working in pex. I will get it to work so the flags match.

The script does reproduce release behaviour for the venv and zipapp scies though.

[jsirois]

My use case is indeed to be able to probe/access the released scies chosen interpreter given some constraint.

Sorry for all the questions, but I want to fully understand this.

So you want this for the released Pex PEX scies in particular, your own builds of these (for whatever reason) or for other PEX scies (not the Pex PEX)?

Also, what do you mean by "given some constraint"? A scie, after all, seals in exactly 1 interpreter with purpose. This is masked for a Pex PEX since Pex can run under ~any Python, but the science scie, for example, very specifically must run under Python 3.14 and so seals that detail into its scie.

Basically, the more verbose you can be describing exactly what you're doing will help me gauge how to look into this.

[jsirois]

I have had trouble with pyenv shims (used for work) confusing both me and the uv dev-cmd

It would be above and beyond, but if you have a chance to bug report backtraces, etc to dev-cmd, I maintain that and would like to have it be robust to pyenv shims. I use these extensively locally and works for me, but: https://github.com/jsirois/dev-cmd

[jsirois]

Alright @tobni I think I've got a more coherent fix that I'll push here shortly (I want to see if your change up to this point goes green first). It passed your new integration test as well as the test script in the OP above and makes more sense to me anyway. Let me know what you think.

My questions above still stand though - I am curious to know what you're doing.

[tobni]

Nice! Yea LGTM, the env propagation was a bit of a maze to grok so I fell back on trial and error in my attempt.

Yes, I believe the test captures my usecase exactly. The interactive repl is not the primary target, the -- -c arg-sequence is.

And I realize I was imprecise sorry; I did not mean to say I wanted to probe the scie bundled interpreter. I wanted to probe the chosen interpreter of the pex cli after it has done its interpreter selection. I will be using the "vendored" scies.

I'd use it for fingerprinting and logging in CI.

[jsirois]

Yes, I believe the test captures my usecase exactly.

Ok - this is what I found very confusing since the test script did not download the released Pex PEX scie, it built its own facsimile.

Can I ask in what context you use a Pex PEX scie? Pants does not use these yet IIUC; so you're doing this in your own tooling?

[tobni]

Can I ask in what context you use a Pex PEX scie? Pants does not use these yet IIUC; so you're doing this in your own tooling?

Yes, for packaging prototype python programs on developer machines, who then send them to internal users. No pants involved. I did stumble upon the tool contributing to pants, though, and intially mirrored its use.

The pex PEX is installed for the devs. Would like for it to be a scie.

[jsirois]

Yes, for packaging prototype python programs on developer machines, who then send them to internal users. No pants involved. I did stumble upon the tool contributing to pants, though and intially mirrored its use.

The pex PEX is installed for the devs. Would like for it to be a scie.

Thank you very much for that background! I find one of the most difficult things developing Pex is lack of a good sense of how Pex is used. I know the Pants use case well. I know Dagster's use case (they blogged about it), but it pretty much ends there.

[jsirois]

@tobni you may be interested in using pexrc to get even faster PEX scies on cold boot above and beyond your Pex 2.91.4 fixes. Here's an example, and it uses pex3 scie create to create scies from PEXes as a second step instead of inline:

 :; pex --python python3.14 ansible -c ansible --venv prepend -o ansible.pex
ansible.pex
 
:; curl -sSL -f -O https://github.com/pex-tool/pexrc/releases/download/v0.1.0/pexrc-x86_64-linux-gnu 
:; chmod +x pexrc-x86_64-linux-gnu

# N.B.: `pexrc` currently injects runtimes for all platforms it supports even if the PEX only supports one platform.
# This will be fixed, but currently just means the injected PEX is bigger than it otherwise needs to be.
:; ./pexrc-x86_64-linux-gnu inject ansible.pex 
Writing aarch64-linux-gnu.libpexrc.so 1042032 bytes to __pex__/.clib/aarch64-linux-gnu.libpexrc.so...done.
Writing aarch64-linux-musl.libpexrc.so 1040867 bytes to __pex__/.clib/aarch64-linux-musl.libpexrc.so...done.
Writing aarch64-macos.libpexrc.dylib 933079 bytes to __pex__/.clib/aarch64-macos.libpexrc.dylib...done.
Writing aarch64-windows.pexrc.dll 918855 bytes to __pex__/.clib/aarch64-windows.pexrc.dll...done.
Writing armv7-linux-gnueabihf.libpexrc.so 992221 bytes to __pex__/.clib/armv7-linux-gnueabihf.libpexrc.so...done.
Writing powerpc64le-linux-gnu.libpexrc.so 1077145 bytes to __pex__/.clib/powerpc64le-linux-gnu.libpexrc.so...done.
Writing riscv64gc-linux-gnu.libpexrc.so 1088571 bytes to __pex__/.clib/riscv64gc-linux-gnu.libpexrc.so...done.
Writing s390x-linux-gnu.libpexrc.so 1009289 bytes to __pex__/.clib/s390x-linux-gnu.libpexrc.so...done.
Writing x86_64-linux-gnu.libpexrc.so 1094304 bytes to __pex__/.clib/x86_64-linux-gnu.libpexrc.so...done.
Writing x86_64-linux-musl.libpexrc.so 1090478 bytes to __pex__/.clib/x86_64-linux-musl.libpexrc.so...done.
Writing x86_64-macos.libpexrc.dylib 973805 bytes to __pex__/.clib/x86_64-macos.libpexrc.dylib...done.
Writing x86_64-windows.pexrc.dll 999934 bytes to __pex__/.clib/x86_64-windows.pexrc.dll...done.
 
:; ls -1sh ansible*
65M ansible.pex
82M ansible.pexrc

:; pex3 scie create ansible.pex --style eager --scie-only -d dist
Saved PEX scie for CPython 3.14 on linux-x86_64 to dist/ansible
 
:; pex3 scie create ansible.pexrc --style eager --scie-only -d dist
Saved PEX scie for CPython 3.14 on linux-x86_64 to dist/ansible.pexrc
 
:; ls -1sh dist/
total 401M
192M ansible
209M ansible.pexrc

:; time dist/ansible --version
ansible [core 2.20.4]
  config file = None
  configured module search path = ['/home/jsirois/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jsirois/.cache/pex/venvs/3/e70210d2785321ce96d88b907aa0a6021e6e9298/0c5c84b639468bee157ad23eec75dbbb3681a927/lib/python3.14/site-packages/ansible
  ansible collection location = /home/jsirois/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jsirois/Downloads/dist/ansible
  python version = 3.14.3 (main, Mar 20 2026, 00:34:32) [Clang 22.1.1 ] (/home/jsirois/.cache/pex/venvs/3/e70210d2785321ce96d88b907aa0a6021e6e9298/0c5c84b639468bee157ad23eec75dbbb3681a927/bin/python)
  jinja version = 3.1.6
  pyyaml version = 6.0.3 (with libyaml v0.2.5)

real	0m3.070s
user	0m2.780s
sys	0m0.216s
 
:; time dist/ansible.pexrc --version
ansible [core 2.20.4]
  config file = None
  configured module search path = ['/home/jsirois/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jsirois/.cache/pexrc/venvs/0/Z5mo26dDxOA4lt5wu-vSZfer0LX6exPBBzg-G2ArfCI/lib/python3.14/site-packages/ansible
  ansible collection location = /home/jsirois/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jsirois/Downloads/dist/ansible.pexrc
  python version = 3.14.3 (main, Mar 20 2026, 00:34:32) [Clang 22.1.1 ] (/home/jsirois/.cache/pexrc/venvs/0/Z5mo26dDxOA4lt5wu-vSZfer0LX6exPBBzg-G2ArfCI/bin/python)
  jinja version = 3.1.6
  pyyaml version = 6.0.3 (with libyaml v0.2.5)

real	0m1.062s
user	0m1.322s
sys	0m0.927s

[jsirois]

Thanks for your patience on this one @tobni. Took me a while to catch up. The fix is now released in https://github.com/pex-tool/pex/releases/tag/v2.91.5.