Skip to content

Preload conda-forge's libpython, honour libjep and JVM search paths, accurate advice (#96) - #98

Open
morisil wants to merge 3 commits into
issue-93-loader-findsfrom
issue-96-conda-runpath-advice
Open

morisil wants to merge 3 commits into
issue-93-loader-findsfrom
issue-96-conda-runpath-advice

Conversation

@morisil

@morisil morisil commented Nov 13, 2026

Copy link
Copy Markdown
Member

Fixes #96

Stacked on #97 (fix of #93) → #94 (#77) → #78 (#54) — merge #78, #94, #97, then this one (bottom-up, see CLAUDE.md).

Follow-ups of #93, with the same rule: a prediction may err towards "not found" (a harmless preload), never towards a wrong "found".

  • conda-forge: its Python reports a static INSTSONAME (libpython3.X.a), so libpython was never preloaded nor advised. Now the shared library libjep actually needs is taken from libjep's DT_NEEDED — only when exactly one same-stem .so is needed and that exact file is in LIBDIR (a conda-forge 3.12 package ships lib/libpython3.12.so.1.0, not the .a; not run end to end, no conda here). A shared soname libjep does not need gives "not found".
  • ELF dynamic section (readElfDynamic): read through the program headers like the loader (32/64-bit, both byte orders, stripped section headers), with caps on entries and string lengths; matches readelf -d on every libjep build here (cp310–cp314), and survived every truncation and 30k random corruptions without an exception.
  • libjep with RPATH/RUNPATH, or an unreadable one: "not found".
  • The JVM's own search path: the RPATH of /proc/self/exe ($ORIGIN expanded as a whole token) and java.home/lib can only veto — a different file of the name there, also in glibc-hwcaps subdirectories, gives "not found"; they never make it "found".
  • LD_PRELOAD: an inherited libpythonX.Y….so entry is no longer passed to the processes this library starts (the probe, pip); others are kept.
  • Advice: depends on the Java version and the foreign linker (Java 22+ with --enable-native-access=ALL-UNNAMED first); conditional when Jep's libpython name is only uncertain; LD_PRELOAD only for a Jep built while libpython3.X.a existed.
  • README: the libpython section describes the conservative preload, conda-forge (LD_LIBRARY_PATH with LIBDIR on Java 17–21) and the static case.
  • libjep is read lazily, only when the cheap checks do not decide.

A system Python with its development symlink still gets no preload (checked on Java 17, 21, 22, 25); the real JDKs here get no spurious veto.

Review

3 rounds:

  1. README wrongly sent conda-forge to LD_PRELOAD (Jep compiles in the .so when the .a is absent) → fixed, plus advice and parser hardening.
  2. A JVM directory could make the answer "found" (an executable's RUNPATH, java.home/lib, glibc-hwcaps) → it may only veto now.
  3. No blocking findings.

Left as polish: the KDoc calls the JVM directories a "superset" (a native launcher library with its own RPATH, e.g. jpackage's, is not covered); the RPATH is decoded as ISO-8859-1, not the file system charset; a few older tests use the host JVM's real /proc/self/exe (would fail on a JDK with a relative or $LIB RPATH — pass jvmDirectories = { emptyList() }); ${ORIGIN}_x gives "unknown" (preload); LD_PRELOAD still reaches subprocesses started from Python code (documented).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

morisil and others added 3 commits November 13, 2026 15:34
… as not found (#96)

A Python with a static INSTSONAME, e.g. conda-forge's libpython3.12.a, which it
does not ship, now has its shared libpython taken from LIBDIR when the native
library of Jep needs exactly one soname of the same version and ABI
(DT_NEEDED, e.g. libpython3.12.so.1.0), and that file exists, so it is
preloaded like any other. readElfDynamic parses the ELF dynamic section from
the program headers, reading only the parts it needs.

jepLoadsGlobally tells "not found" when libjep has a DT_RPATH or DT_RUNPATH,
or its dynamic section cannot be read, and reads the dynamic section and the
libpython name of Jep lazily, only after the loader finds the soname.

The LD_PRELOAD advice now offers Java 22 with native access first, is
conditional when Jep's libpython name is uncertain, and scopes LD_PRELOAD to
the launch of the JVM, as its child processes inherit it. The README
describes the conservative preload, the static libpython case and the
default directories. The test of a library only in a default directory now
reads a cache, and checks that listing the library would find it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…hild processes (#96)

The README sent conda-forge users to LD_PRELOAD, while Jep compiles in the
shared libpython3.12.so there, as the static library is missing, so the remedy
is LIBDIR on LD_LIBRARY_PATH; LD_PRELOAD is left for a Jep built while
libpython3.X.a existed.

runProcess removes the libpython entries of LD_PRELOAD from the environment
of the processes it starts, e.g. pip. The advice no longer tells to run on
Java 22 when the JVM already is, but to enable native access, or not to
disable the foreign linker.

pythonLibrary does not take a shared soname which libjep does not need, when
libjep needs another libpython. readElfDynamic caps the DT_NEEDED entries and
the length of a string. loaderFinds also searches the directories of the RPATH
of the java launcher, and the lib directory of java.home, which glibc
searches before LD_LIBRARY_PATH for the libraries of libjep. The ELF library
is read only on Linux.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
loaderFinds returned true when the same file was in a directory of the JVM,
though glibc might not search it for libjep, e.g. java.home/lib, or the
RUNPATH of the executable, and ignored glibc-hwcaps there. Such a directory
now only tells "not found", with another file of the name, also in its
glibc-hwcaps, and the search goes on with LD_LIBRARY_PATH and the cache.

$ORIGIN is expanded only as a whole token, any other $ cannot be told.
runProcess filters only the inherited LD_PRELOAD, not one passed in its
environment, and only of the names of a libpython of a Python version. The
advice with the foreign linker enabled on Java 22 tells that it failed, and
mentions --enable-native-access only if not already set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant