Skip to content

Load the libpython of a Python outside the system library directories - #78

Open
morisil wants to merge 4 commits into
mainfrom
issue-54-libpython-path
Open

morisil wants to merge 4 commits into
mainfrom
issue-54-libpython-path

Conversation

@morisil

@morisil morisil commented Nov 11, 2026

Copy link
Copy Markdown
Member

Fixes #54

On Linux, a Python whose libpython lies outside the system library directories (uv, pyenv --enable-shared, a custom prefix) now starts without LD_LIBRARY_PATH, where libjep.so failed with UnsatisfiedLinkError: libpython3.X.so.1.0: cannot open shared object file.

  • The environment probe also reports sysconfig's LIBDIR and INSTSONAME.
  • Only when the dynamic loader would not find that libpython by its soname (loaderFinds imitates LD_LIBRARY_PATH, /etc/ld.so.cache and the default directories), it is loaded before Jep, with its symbols made global (C extensions not linked to libpython, like numpy, need them):
    • Java 22+: dlopen(RTLD_GLOBAL) through the foreign linker (by reflection, the library targets Java 17) — C extensions imported at startup (.pth, sitecustomize) work too;
    • Java 17–21: System.load, then the symbols are made global in the first interpreter (best-effort, synchronized) — a C extension imported at startup still needs LD_LIBRARY_PATH there, documented.
  • A system Python is untouched: no new JVM warning on any JDK. For a uv-like Python, JDK 22/23 show one native-access warning (documented); on 24+ it replaces Jep's existing one.
  • When jep still can't load libpython, the startup error names the directory to put on LD_LIBRARY_PATH (also through Jep's "previously failed to initialize" retry); a failure of the preload itself is reported as such.
  • macOS and Windows: only the error advice; loading is Linux-only.

Review

3 rounds:

  1. Startup imports (.pth/sitecustomize) still failed → foreign-linker dlopen on Java 22+.
  2. Test failing on JDK 17–21, new restricted-method warnings on JDK 21–23 also for system Python → the foreign linker only when needed, and only where available.
  3. No blocking findings — verified on JDK 17/21/22/25 × uv Python 3.10/3.12/3.13/3.14 and system 3.12; polish applied (denied native access keeps main's error shape, preload-failure message, boolean flag, lazy ld.so.cache).

Not tested here: pyenv --enable-shared, actions/setup-python without LD_LIBRARY_PATH (CI's Python matrix now removes it in PythonLibraryTest), conda, macOS.

Follow-ups

#66 will be stacked on this branch — merge this PR first.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

morisil and others added 4 commits November 11, 2026 20:31
…ore jep (#54)

On Linux, libjep.so failed to load with "libpython3.X.so.1.0: cannot open
shared object file" for a Python installed by uv, pyenv, conda or in a custom
prefix, unless its lib directory was on LD_LIBRARY_PATH at JVM start.

The environment probe now reports sysconfig LIBDIR and INSTSONAME; before Jep
loads its native library, PythonEnvironment System.loads LIBDIR/INSTSONAME by
its path (Linux only, shared library only), so the loader resolves libjep's
dependency by soname. As the JVM loads it RTLD_LOCAL, and Jep's own
RTLD_GLOBAL promotion looks it up by file name, which the loader cannot find,
the first interpreter makes its symbols global by loading it as an extension
module with RTLD_GLOBAL dlopen flags, so that C extensions not linked to
libpython (numpy, _ctypes of most builds) still import. If loading still
fails, the error names the directory to put on LD_LIBRARY_PATH.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…54)

Making libpython global only in the first interpreter came after the startup
of CPython, so a C extension imported by a .pth file or sitecustomize, e.g.
numpy or coverage, still failed with an undefined symbol.
On Java 22 or newer, libpython is now loaded with dlopen(RTLD_GLOBAL) through
the foreign linker (via reflection, the library targets Java 17) before Jep
starts CPython. On Java 17 to 21 the System.load + make-global-in-the-first-
interpreter fallback stays, now best-effort (logged, never failing startup),
run by EmbeddedPythonStarter; README tells its .pth limitation.

The LD_LIBRARY_PATH advice now walks the cause chain, also of Jep's retry
Error, and is given only for the soname of the shared libpython found.
PythonLibraryTest compiles a C extension not linked to libpython, imported by
the code, by sitecustomize at startup, and without the foreign linker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
A system Python, whose libpython the loader finds by its soname, is left
alone again, so it gets no new restricted method warning on Java 22+:
loaderFinds checks LD_LIBRARY_PATH, /etc/ld.so.cache and the system
directories for the same file, like the loader does. On Java 21, the
foreign linker is no longer touched (allocateFrom is looked up before any
restricted call), so it prints no warning either.

The startup-import test of PythonLibraryTest is assumed on Java 22+ only,
as on Java 17-21 libpython becomes global after the startup, as documented.
Making libpython global on Java 17-21 is now synchronized, so that a second
interpreter waits for it; the LD_LIBRARY_PATH advice is given only for
"cannot open shared object file" / "Library not loaded" of the soname.
README no longer lists conda as handled (its libjep has a RUNPATH),
mentions RPATH, and DEVELOPMENT.md the pyenv LD_LIBRARY_PATH need on 17-21.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…54)

- A RuntimeException of System.load of libpython, e.g. IllegalCallerException
  with --illegal-native-access=deny, is kept as the preload failure instead
  of escaping initialize() unwrapped.
- A failed preload is named in the startup failure message, instead of the
  LD_LIBRARY_PATH advice, which would not help then.
- The noForeignLinker test seam is parsed as a boolean.
- /etc/ld.so.cache is read only when no LD_LIBRARY_PATH entry decides.
- The CLAUDE.md entry is trimmed to the non-obvious lesson.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

jep cannot load libpython of a Python outside the system library directories on Linux

1 participant