Support baking PythonCall into a juliacall system image (opt-in)

[ncudlenco]

Summary

Lets PythonCall itself be compiled into a juliacall system image (opt-in). With it baked in, the using PythonCall that import juliacall performs becomes a memory-map instead of a multi-second load+compile. In a containerised juliacall workload we measured, fresh-process startup drops from ~18 s to ~1.9 s. Off by default - no behaviour change unless enabled. It's meant as a low-risk interim until the automatic fix discussed in #436 (or related work) lands; it doesn't conflict with or preclude that.

Motivation

In short-lived Python processes that embed Julia via juliacall - serverless or autoscaled containers (AWS Lambda, queue workers, CI jobs) that start, handle one request, and exit - there's no long-lived process to amortise Julia start-up. Every cold start pays it again, and the dominant part is the using PythonCall that import juliacall runs to bring the bridge up.

Measured, fresh container to first call:

	container init
stock juliacall (only app packages baked)	~18 s
PythonCall also baked into the sysimage	~1.9 s

PYTHON_JULIACALL_SYSIMAGE already exists, but a sysimage with only the application packages still runs using PythonCall every cold start. Baking PythonCall in too is the fix, but it currently fails: when PythonCall is in the sysimage its __init__ runs during jl_init_with_image, before juliacall's bootstrap has set Main.__PythonCall_libptr. init_context() decides "embedded" only from that global, so it sees "not embedded"; since Python is mid-import juliacall, init_juliacall() then errors with 'juliacall' module already exists.

Change

Add one module-level flag, PythonCall._is_embedded, that the user flips inside a PackageCompiler script= to mark the resulting sysimage as embedded. At runtime, PythonCall.__init__ reads the flag and takes the embedded path; libpython is opened from the existing lib preference / JULIA_PYTHONCALL_LIB. If the sysimage is loaded by a julia process with no running Python (e.g. a Base.compilecache child), the embedded path rolls back and PythonCall stays inactive instead of erroring. Docs include a worked PackageCompiler example.

Backward compatibility

Backwards compatible since the flag is off by default. The classic juliacall path and the non-embedded path are unchanged; lib and exe preferences keep their existing semantics. No new preferences or environment variables are introduced.

Testing

The new code path only runs when explicitly enabled, so default behaviour is unchanged and the existing test suites are unaffected.

Verified manually on Linux (AWS Lambda base image public.ecr.aws/lambda/python:3.12) and Windows (desktop, Python 3.13). A plain julia.exe --sysimage= load of a baked sysimage leaves PythonCall inactive without erroring. Not tested on macOS. No automated test is included because exercising it requires building a system image in CI, which is slow - happy to add one, or document the steps, if you'd prefer.

Related issues

Relates to #436 and #600 - this unblocks their use case (baking PythonCall into a custom system image) as an interim. The root-cause fix described in #436 - resetting PythonCall's sysimage-persisted state in __init__ so it works with no opt-in - is the longer-term path; this PR doesn't attempt or preclude it.

Also relevant to #762 ("Improve juliacall startup time?", which explicitly asks about compiling a system image) and a building block for #76 ("Compile and use custom sysimages automatically"). Does not address #129 (a different failure: no environment in the LOAD_PATH depends on CondaPkg).

[cjdoris]

Looks good. My main feedback is about the docs - could you document the basic steps to actually create a sysimg?

I don't think you need to go much into the why (the current second paragraph) nor mention Main.__PythonCall_libptr (which is internal). Instead, you can pretty much just say that you need to set the prefs embedded=true and lib=.... But what's not totally obvious is how you set up a project and these prefs and use PackageCompiler to actually make the sysimg.

[cjdoris]

Another thought - if you know that you are embedded, then you can find the libptr (by calling into the C-API functions, which will be globally available in this case). We only don't do this because it's easier to directly communicate the ptr itself by setting Main.__PythonCall_libptr from Python.

Therefore if we can "bake in" the fact that PythonCall is embedded into the sysimg, then we won't need any of these preferences. I wonder if we could simply do @eval PythonCall _is_embedded=true or something when we make the sysimg, so it's baked into PythonCall, then test for this variable in PythonCall.__init__?

[ncudlenco]

Looks good. My main feedback is about the docs - could you document the basic steps to actually create a sysimg?

I don't think you need to go much into the why (the current second paragraph) nor mention Main.__PythonCall_libptr (which is internal). Instead, you can pretty much just say that you need to set the prefs embedded=true and lib=.... But what's not totally obvious is how you set up a project and these prefs and use PackageCompiler to actually make the sysimg.

---

Another thought - if you know that you are embedded, then you can find the libptr (by calling into the C-API functions, which will be globally available in this case). We only don't do this because it's easier to directly communicate the ptr itself by setting Main.__PythonCall_libptr from Python.

Therefore if we can "bake in" the fact that PythonCall is embedded into the sysimg, then we won't need any of these preferences. I wonder if we could simply do @eval PythonCall _is_embedded=true or something when we make the sysimg, so it's baked into PythonCall, then test for this variable in PythonCall.__init__?

---

Thanks - both points shaped the rewrite. The flag is now baked into the sysimage as you suggested, the docs section has a worked PackageCompiler example, and Main.__PythonCall_libptr is no longer mentioned. libpython is opened via the existing lib preference (JULIA_PYTHONCALL_LIB) - no new preferences or env vars.

I also explored the C-API angle. It works in standalone tests but I couldn't get it to resolve libpython reliably during the sysimage init window across all targets, so the PR sticks with the explicit lib preference. Happy to dig deeper if you'd like more detail.

One implementation note: I used const _is_embedded = Ref(false) rather than a rebound non-const global, so the C submodule can import the name once and read it without parentmodule indirection. User API is the same one line in script=. Happy to switch to the literal @eval form if you'd prefer.

P.S. Also added you as a Write collaborator on buchi-labortechnik-ag/PythonCall.jl so you can push directly to this branch if you'd like to make changes yourself.