Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ see [DEVELOPMENT.md](DEVELOPMENT.md#markdown-soft-wrapping-in-the-ide) for how t
- The embedded CPython does not detect a virtual environment by itself —
without the venv's `python` as its program name (`PyConfig.setProgramName`), it ignores `pyvenv.cfg` and also imports from the global and user `site-packages` of the base Python.
When a module is unexpectedly importable, check `sys.path` of the embedded interpreter, not of `.venv/bin/python`.
- A running Gradle daemon does not pass a changed `LD_LIBRARY_PATH` on to the test JVM,
so use `--no-daemon` when trying a `-PbasePython` which needs it (see [DEVELOPMENT.md](DEVELOPMENT.md#the-python-of-the-tests)).
- A running Gradle daemon does not pass a changed `LD_LIBRARY_PATH` on to the test JVM, so a stale one can hide a failure to load `libpython` —
run a `-PbasePython` of uv with `--no-daemon`, without `LD_LIBRARY_PATH`.
- Jep passes the `PyConfig` strings to CPython as JNI *modified* UTF-8 (`GetStringUTFChars`), not plain UTF-8 —
a character outside the BMP becomes a CESU-8 surrogate pair, which CPython does not decode as that character.
For a path under a non-UTF-8 locale, see `pythonPath`.
Expand Down
13 changes: 8 additions & 5 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,15 @@ or with the `ORG_GRADLE_PROJECT_basePython` environment variable, e.g. on CI:
```

It has to be Python 3.10 or newer, otherwise the build fails before creating the environment.
On Linux, the tests fail to load `libjep.so` when it cannot find `libpython3.*.so` ([#54](https://github.com/xemantic/xemantic-kotlin-script-python/issues/54)).
A Python outside the system library directories, e.g. installed by `uv`,
needs its `lib` directory in `LD_LIBRARY_PATH`,
which `actions/setup-python` sets on CI.
On Linux, a Python outside the system library directories, e.g. installed by `uv`,
does not need its `lib` directory in `LD_LIBRARY_PATH`,
as `PythonEnvironment` loads its `libpython` before `libjep.so` ([#54](https://github.com/xemantic/xemantic-kotlin-script-python/issues/54)).
`actions/setup-python` sets `LD_LIBRARY_PATH` on CI anyway,
so `PythonLibraryTest` removes it in the JVM it starts, to test the loading with such a base Python,
importing a C extension it compiles with `cc`, which tests nothing with a Python of the system library directories.
`pyenv` builds Python without a shared `libpython` by default,
build it with `PYTHON_CONFIGURE_OPTS="--enable-shared"` (and then set `LD_LIBRARY_PATH` as well).
build it with `PYTHON_CONFIGURE_OPTS="--enable-shared"`.
On Java 17 to 21, also set `LD_LIBRARY_PATH` then, for a C extension imported at the startup of Python, see the README.

An existing `.venv` is judged by its own Python:
it is kept when this Python runs, is 3.10 or newer, and has `pip`,
Expand Down
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,23 @@ The Python executable is, in this order:
The Python code imports the modules from the same paths as this executable,
e.g. the `site-packages` of a virtual environment, but not the ones of its base Python.

The native library of `jep` links to the shared `libpython` of the Python, e.g. `libpython3.13.so.1.0`,
which on Linux the dynamic loader finds by its soname in the directories of `LD_LIBRARY_PATH`, of `ld.so.cache`, and the system ones,
unless the native library of `jep` has a `RUNPATH` of it, e.g. in conda.
For a Python installed elsewhere, e.g. by `uv python install`, pyenv, or in a custom prefix,
the library loads `libpython` from the `LIBDIR` of the Python's `sysconfig` itself, with its symbols global,
as C extension modules, e.g. of numpy, need them,
so that `LD_LIBRARY_PATH` is not needed.
On Java 22 or newer, it does so with the foreign linker, a restricted method,
which, like loading the native library of `jep` on Java 24 or newer, prints a warning without `--enable-native-access=ALL-UNNAMED`.
On Java 17 to 21, the symbols become global only after the startup of Python,
so a C extension imported at the startup, by a `.pth` file of the `site-packages`, e.g. of coverage, or by `sitecustomize`,
fails with an `undefined symbol` and still needs `LD_LIBRARY_PATH`.
If `libpython` is not found, e.g. on macOS, where it is not loaded so,
put its directory, the `LIBDIR` of `sysconfig`, on `LD_LIBRARY_PATH` (`DYLD_LIBRARY_PATH` on macOS),
set before the JVM starts, as the loader reads it only then.
pyenv builds Python without a shared `libpython` by default, build it with `PYTHON_CONFIGURE_OPTS="--enable-shared"`.

The embedded Python, like every Python process started by this library, e.g. `pip`,
runs in the [UTF-8 mode](https://peps.python.org/pep-0540/),
whatever the locale, e.g. cp1252 on Windows, or `C` on Linux:
Expand Down
122 changes: 113 additions & 9 deletions src/main/kotlin/PythonEnvironment.kt
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,17 @@ public object PythonEnvironment {
val library = jepLibrary(Path(jepLocation), python, probe.externallyManaged)
MainInterpreter.setJepLibraryPath(library.toString())
MainInterpreter.setInitParams(embeddedPythonConfig(executable))
val pythonLibrary = pythonLibrary(probe.libraryDirectory, probe.librarySoname)
// only when needed, as the foreign linker prints a warning without --enable-native-access
val loading = if (
pythonLibrary != null && System.getProperty("os.name").startsWith("Linux") && !loaderFinds(pythonLibrary)
) {
loadPythonLibrary(pythonLibrary)
} else PythonLibraryLoading()
this.executable = executable
starter = EmbeddedPythonStarter(executable, probe.error)
starter = EmbeddedPythonStarter(
executable, probe.error, pythonLibrary, loading.failure, loading.localSymbols
)
if (dependencyDirectory == null) {
dependencyDirectory = System.getProperty(DEPENDENCY_DIRECTORY_PROPERTY)?.let { Path(it) }
}
Expand Down Expand Up @@ -266,10 +275,14 @@ public object PythonEnvironment {
val begin = lines.lastIndexOf("\n$PROBE_BEGIN\n").takeIf { it != -1 }?.let { it + PROBE_BEGIN.length + 2 }
val end = begin?.let { lines.indexOf("\n$PROBE_END", it) }?.takeIf { it != -1 }
val values = if (begin == null || end == null) null else lines.substring(begin, end).split(' ')
if (values == null || values.size != 5) return null
val (executable, cacheTag, prefix, jepLocation, externallyManaged) =
values.map { it.hexToPath() ?: return null }
return PythonProbe(executable, cacheTag, prefix, jepLocation, externallyManaged.isNotEmpty())
if (values == null || values.size != 7) return null
val paths = values.map { it.hexToPath() ?: return null }
val (executable, cacheTag, prefix, jepLocation, externallyManaged) = paths
return PythonProbe(
executable, cacheTag, prefix, jepLocation, externallyManaged.isNotEmpty(),
libraryDirectory = paths[5],
librarySoname = paths[6]
)
}

/**
Expand All @@ -281,9 +294,11 @@ public object PythonEnvironment {
* as text, a path which is not valid UTF-8, e.g. `/home/josé` in ISO-8859-1,
* would be surrogate-escaped by Python in UTF-8 mode, and could not be encoded to UTF-8,
* while the JVM would decode a path of UTF-8 bytes to another one, in another encoding of its locale.
* The last value tells whether the Python is externally managed (PEP 668), `1` if it is, empty if not:
* The fifth value tells whether the Python is externally managed (PEP 668), `1` if it is, empty if not:
* a base Python, not a virtual environment, with an `EXTERNALLY-MANAGED` file in its standard library,
* into which pip refuses to install.
* The values after it are the directory and the soname of the `libpython` of the base Python,
* `LIBDIR` and `INSTSONAME` of `sysconfig`, empty if unknown, e.g. on Windows, see [pythonLibrary].
* Without the `jep` package, it exits with the [PROBE_JEP_MISSING_EXIT_CODE],
* or the [PROBE_JEP_MISSING_EXTERNALLY_MANAGED_EXIT_CODE] if the Python is externally managed,
* writing to the standard error the file of a `jep` module, if any, e.g. a `jep.py` shadowing the package.
Expand Down Expand Up @@ -312,6 +327,8 @@ public object PythonEnvironment {
| sys.prefix,
| jep_spec.submodule_search_locations[0],
| '1' if externally_managed else '',
| sysconfig.get_config_var('LIBDIR'),
| sysconfig.get_config_var('INSTSONAME'),
|]
|payload = ' '.join('' if value is None else os.fsencode(str(value)).hex() for value in values)
|sys.stdout.flush()
Expand Down Expand Up @@ -490,8 +507,22 @@ private fun String.hexToPath(): String? {
* telling a failure of its startup with what the probe of the [executable] wrote to the standard error, if anything,
* as a warning of a successful probe, e.g. `Could not find platform libraries` of a wrong `PYTHONHOME`,
* might tell why the embedded CPython, configured the same way, fails.
* When the native library of Jep cannot load the shared [pythonLibrary] of the Python, see [pythonLibrary],
* it tells to put its directory on `LD_LIBRARY_PATH`,
* or the [libraryPreloadFailure] of [loadPythonLibrary], if any, also kept as suppressed.
* The symbols of the [pythonLibrary] loaded by [loadPythonLibrary], if [libraryLocalSymbols],
* are made global in the first interpreter, see [makePythonLibraryGlobal].
*/
internal class EmbeddedPythonStarter(private val executable: String, private val probeError: String) {
internal class EmbeddedPythonStarter(
private val executable: String,
private val probeError: String,
private val pythonLibrary: Path? = null,
private val libraryPreloadFailure: Throwable? = null,
libraryLocalSymbols: Boolean = false
) {

/** Whether the symbols of the [pythonLibrary] are still to be made global, see [makePythonLibraryGlobal]. */
private var libraryLocalSymbols = libraryLocalSymbols

/** Whether an interpreter has been created, so that the embedded CPython has started. */
@Volatile
Expand Down Expand Up @@ -520,6 +551,7 @@ internal class EmbeddedPythonStarter(private val executable: String, private val
}
started = true
try {
if (interpreter is Interpreter) makePythonLibraryGlobal(interpreter)
check(interpreter)
} catch (e: Throwable) {
try {
Expand All @@ -532,15 +564,83 @@ internal class EmbeddedPythonStarter(private val executable: String, private val
return interpreter
}

/**
* Makes the symbols of the [pythonLibrary] loaded by [loadPythonLibrary] global, if [libraryLocalSymbols],
* in the first [interpreter], before any code of the user, but after the startup of Python,
* see [loadPythonLibrary].
* The interpreters created meanwhile on other threads wait for it, so that no code runs before.
*
* CPython can load a library with `RTLD_GLOBAL`,
* loading an extension module with the flags of `sys.setdlopenflags`:
* loading the already loaded `libpython` as one makes it global,
* before it fails as an extension module without its initialization function.
* It is best-effort: a failure is only logged, as the code of the user might not need any C extension.
*/
@Synchronized
private fun makePythonLibraryGlobal(interpreter: Interpreter) {
if (!libraryLocalSymbols) return
val library = pythonLibrary ?: return
// also on a failure, which would only repeat
libraryLocalSymbols = false
try {
interpreter.exec(
"""
|def _xemantic_make_python_library_global():
| import _imp, os, sys
| from importlib.machinery import ModuleSpec
| flags = sys.getdlopenflags()
| sys.setdlopenflags(flags | os.RTLD_GLOBAL)
| try:
| _imp.create_dynamic(ModuleSpec('_xemantic_libpython', None, origin=${pythonPath(library.toString())}))
| except ImportError:
| pass
| finally:
| sys.setdlopenflags(flags)
|try:
| _xemantic_make_python_library_global()
|finally:
| del _xemantic_make_python_library_global
|""".trimMargin()
)
} catch (e: Exception) {
System.getLogger(EmbeddedPythonStarter::class.java.name).log(
System.Logger.Level.WARNING,
"Cannot make the symbols of $library global, " +
"a C extension module might fail to import with an undefined symbol, " +
"put ${library.parent} on LD_LIBRARY_PATH before the JVM starts",
e
)
}
}

private fun startupFailure(cause: Throwable): IllegalStateException {
// Jep wraps the failure of initializing CPython in an Error, whose message is its toString()
val reason = cause.cause?.takeIf { cause.message == it.toString() } ?: cause
val message = "Cannot initialize the embedded Python $executable: ${reason.message ?: reason}"
val headline = "Cannot initialize the embedded Python $executable: ${reason.message ?: reason}"
// e.g. libjep.so: libpython3.13.so.1.0: cannot open shared object file, or Library not loaded on macOS,
// also as the cause of the Error of Jep retrying to initialize CPython
val library = pythonLibrary
val libraryAdvice = if (libraryPreloadFailure != null) {
// LD_LIBRARY_PATH would not help, as the loader would fail to load it the same way, e.g. of another architecture
"\nCannot load the libpython of this Python: ${libraryPreloadFailure.message ?: libraryPreloadFailure}"
} else if (
library != null && generateSequence(cause) { it.cause }.any {
val message = it.message ?: ""
it is UnsatisfiedLinkError && library.fileName.toString() in message &&
("cannot open shared object file" in message || "Library not loaded" in message)
}
) {
"\nThe dynamic loader cannot load ${library.fileName} of this Python, " +
"put its directory ${library.parent} on LD_LIBRARY_PATH (DYLD_LIBRARY_PATH on macOS) before the JVM starts."
} else ""
val message = headline + libraryAdvice
return IllegalStateException(
if (probeError.isEmpty()) message
else "$message\nThe probe of this Python wrote to the standard error, which might tell why:\n$probeError",
cause
)
).apply {
libraryPreloadFailure?.let { addSuppressed(it) }
}
}

}
Expand All @@ -553,6 +653,8 @@ internal class EmbeddedPythonStarter(private val executable: String, private val
* @property prefix the `sys.prefix`, the virtual environment, if any.
* @property jepLocation the directory of the `jep` package.
* @property externallyManaged whether pip refuses to install into it (PEP 668), see [PythonEnvironment.probeCommand].
* @property libraryDirectory the directory of the `libpython` of the base Python, `LIBDIR`, empty if unknown.
* @property librarySoname the soname of the `libpython`, `INSTSONAME`, empty if unknown, see [pythonLibrary].
* @property error the standard error of the probe, e.g. a warning of Python, empty if none.
*/
internal data class PythonProbe(
Expand All @@ -561,6 +663,8 @@ internal data class PythonProbe(
val prefix: String,
val jepLocation: String,
val externallyManaged: Boolean = false,
val libraryDirectory: String = "",
val librarySoname: String = "",
val error: String = ""
)

Expand Down
Loading
Loading