Skip to content

Advise a pinned jep install that works, also for externally managed Pythons - #67

Merged
morisil merged 23 commits into
mainfrom
issue-37-jep-install-advice
Nov 10, 2026
Merged

morisil merged 23 commits into
mainfrom
issue-37-jep-install-advice

Conversation

@morisil

@morisil morisil commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Fixes #37

Stacked on #68 (recover-environment-fixes, which brings the fixes of #24, #14, #15, #21 and #19 onto main) — merge #68 first; this PR's own change is recover-environment-fixes..issue-37-jep-install-advice (README, PythonEnvironment.kt, PythonEnvironmentTest.kt).

Fix

The advice for a missing or broken jep (python3 -m pip install jep) failed on externally managed Pythons (PEP 668), installed an unpinned version, and didn't tell what building jep needs. Now:

  • Pinned version: read at runtime from the jep jar's META-INF/maven/org.ninia/jep/pom.properties (its manifest has no version) — so the advice matches the org.ninia:jep library actually on the classpath; without it, the advice falls back to an unpinned jep.
  • Advice by the kind of Python: the probe reports whether the Python is externally managed (a base Python with an EXTERNALLY-MANAGED marker in its stdlib, as pip checks) — by exit code when jep is missing, or a probe value otherwise.
    • a venv or another Python: <python> -m pip install jep==<version>, or uv pip install --python <python> jep==<version>;
    • an externally managed Python: create a venv at an absolute path, install the pinned jep into it, then point VIRTUAL_ENV or PythonEnvironment.pythonExecutable to it;
    • a broken jep (no native library): reinstall with --force-reinstall --no-cache-dir.
  • Build requirements: PyPI has only an sdist of jep 4.3.2, so it is always built from source — a C compiler, the Python headers and a JDK (apt install python3-dev, plus python3-venv for the venv recipe, on Debian/Ubuntu); on Windows the Microsoft C++ Build Tools and JAVA_HOME.
  • Paths are absolute (no ~ / %USERPROFILE%, which only a shell expands), quoted when they contain spaces, with a PowerShell & note on Windows.
  • README: the installation section explains the same.

Tests (PythonEnvironmentTest)

The advice for a venv, an externally managed Python, Windows paths with spaces, an unknown version, a broken jep; the probe reporting externally managed Pythons (simulated with a sitecustomize); reading the version from malformed properties; the version read at runtime matches the version catalog.

Review rounds

  1. Round 1 — the advice worked verbatim, but blocking: it sent a user whose Python is already a venv away from it (to a new venv), where the old pip install worked. Fixed in 0575a58 (externally-managed detection), with absolute paths, the always-from-source build requirements, a lazy version read, --force-reinstall.
  2. Round 2 — no blocking findings; the reviewer followed the advice verbatim for a venv without jep, the externally managed /usr/bin/python3, and a broken jep — all worked. Minor findings applied in 14b6efa (python3-venv, README examples, PowerShell, uv pip for a venv without pip).

Follow-ups filed

🤖 Generated with Claude Code

https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

morisil and others added 20 commits November 9, 2026 21:04
runProcess wrote the whole input to the process before reading its
output, so a process writing more than the pipe buffer (about 64 KB)
to stdout before consuming all of its stdin blocked both sides forever.
PythonDependencies pipes the script metadata and the imported module
names, tens of KB, into the PLAN process this way.

The input is now written, and the output read, on their own threads,
while the calling thread waits for the process, so it can still be
interrupted, destroying the process. A broken pipe of the input is no
longer an error: when the process ends without reading its input, e.g.
PLAN failing on an import, the caller gets its exit code and output,
and so a PythonDependencyException with Python's traceback, instead of
a bare IOException: Broken pipe.

The test of destroying the process on a broken pipe is replaced by one
destroying it when interrupted, the remaining way to leave runProcess
while the process is alive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
A failure of the output reader other than an IOException, e.g. an
OutOfMemoryError on a huge output, is now rethrown on the calling
thread, instead of surfacing as a NullPointerException of the missing
output. Without input, the standard input is just closed, with no
thread to write it.

The interrupt test stops waiting for the pid file when the runner has
already ended, so that it fails instead of hanging the suite, and the
echo test asserts the length and content of the 10 MB output as
counts, so that power-assert does not render it whole on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
The environment probe, the PLAN and the pip fallback started Python with
-c and -m pip, which put the JVM's working directory first on sys.path.
A json.py, tomllib.py, jep/ or pip/ there - likely in an agent's working
directory full of generated files - replaced the module the installer
imports, running unintended code, or made the probe report the wrong jep.

-P would fix it, but it is unknown before Python 3.11, and 3.10 is
supported. -I would ignore every PYTHON* variable, also the PYTHONUTF8
the process streams depend on, PYTHONPATH and the user site-packages,
which the embedded interpreter still sees. So the code started with -c
first removes the '' entry of the working directory from sys.path, before
any import, and pip is run with runpy instead of -m. PIP_* variables and
pip config files keep working, as nothing about the environment changes.

uv is not changed: it queries the target interpreter itself, in isolated
mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
The KDoc of pythonCommand now says precisely that only the implicit
working directory entry of -c is removed, while the entries the user
configured, e.g. with PYTHONPATH, are kept, as the embedded interpreter
sees them too. The tests of PLAN without tomllib, the only ones of its
Python < 3.11 behavior, now run it with the same prologue as production.
The probe shadowing test runs its reference probe before creating the
temporary directory, so that a failure of it does not leak the directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
PLAN ran with its standard error merged into its standard output,
which was then parsed line by line as the install plan.
Python block-buffers stdout in a pipe, so anything written to stderr
during the run, e.g. a warning without a trailing line break,
lands before or inside a plan line in the merged stream,
corrupting or dropping a requirement, the same class of bug as #4
fixed for the environment probe.

PLAN now runs with mergeErrorStream = false, only stdout is parsed,
and stderr, where sys.exit writes PLAN's error messages, is included
in the PythonDependencyException on failure.
The uv / pip install runs keep the merged output,
as it is only shown in the error message, never parsed.

plan() takes the command as a defaulted parameter, a test seam
for a PLAN run which writes to stderr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…ms (#14)

The PLAN failure and the environment probe failure both combined
stderr and stdout into the error message, in opposite orders;
ProcessResult.failureOutput now does it once, stderr first,
as it tells why the process failed, e.g. the message of sys.exit.

The tests of PLAN without tomllib, or without any TOML parser,
ran it with stderr merged into stdout, so their assertions on
the sys.exit message passed only because of the merging,
while plan() reads the streams separately.
They now run through plan(), with the command as the test seam,
so they check the message of the exception the user gets.
The tests in a working directory with shadowing modules
read stdout separately as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
When the stdout reader thread failed, it stored the failure and ended,
while the calling thread kept waiting for the process. Closing the
output pipe usually ends a writing process with a broken pipe, but a
process which survives it, or never writes again, made runProcess
wait forever and never rethrow the failure, where reading the output
on the calling thread, before #13, would have thrown at once.

The reader now destroys the process on a failure, so that the wait
ends and the failure is rethrown. The output reading is an internal
parameter of runProcess, a seam for the test to make it fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
The probe printed its values one per line and Kotlin took them by line
position, so anything else writing to stdout - a sitecustomize module or
a .pth file - shifted every value, and an empty first value, e.g. an
empty sys.executable, was trimmed away, leaving too few lines and an
IndexOutOfBoundsException instead of the documented IllegalStateException.

The probe now writes its values separated by NUL, which no path can
contain, between unique begin and end sentinel lines, preceded by an
empty line in case the noise before does not end with a newline, and as
bytes, so without newline translation. Kotlin takes the values only from
between the last begin and the following end sentinel, and fails with an
IllegalStateException showing the output when there are not exactly four.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
)

An empty sys.executable was parsed correctly but then accepted silently:
the embedded CPython got an empty program name, so it ignored the
pyvenv.cfg of a virtual environment, and installing dependencies failed
later with a bare IOException of running "". The probe now fails right
away with an IllegalStateException telling that sys.executable is empty.

The probe wrote to sys.stdout.buffer, which a sitecustomize replacing
sys.stdout with a wrapper does not have, so it now writes to the
original sys.__stdout__, after flushing both, which also bypasses any
redirection of sys.stdout.

When the output has no probe values, the message now names the Python
executable and shows its stderr as well, as it tells why; the parsing
returns null for the caller to report it, and is unit-tested on its own
for output around the values, repeated markers, an empty value, a
missing end marker, and a wrong number of values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…bytes (#21)

The embedded interpreter used the encoding of the JVM's locale, so under
LC_ALL=C with PYTHONUTF8=0, or a cp1252 Windows locale, printing non-ASCII
text or reading a UTF-8 file failed. Jep's PyConfig has no UTF-8 mode setter,
but CPython parses `-X utf8` from the argv set with PyConfig.setArgv before
decoding the other values, which enables it, also over PYTHONUTF8=0.

Under a non-UTF-8 POSIX locale, e.g. ISO-8859-1 with a home like /home/josé,
a path is the locale's bytes, surrogate-escaped by Python in UTF-8 mode:
- the environment probe failed encoding such a path to UTF-8; it now writes
  each value as the hex of its os.fsencode bytes, which Kotlin decodes with
  sun.jnu.encoding (UTF-8 on Windows), so it names the same file;
- the dependency directory, a Java String passed by Jep as UTF-8, was another
  path in the embedded sys.path; it is now appended as os.fsdecode of its
  bytes. In argv, to PLAN and pip, it already round-trips unchanged.
A non-ASCII path of the Python executable cannot be fixed here, as Jep passes
the program name as UTF-8 too; the README documents it.

PythonTest compared sys.path of a raw ProcessBuilder decoded as UTF-8; it now
uses runProcess and compares ascii() of the paths on both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
Review findings of the UTF-8 mode fix:
- The os.fsdecode conversion of the dependency directory had no regression
  test; PythonLocaleTest now forks a JVM in an ISO-8859-1 locale, created
  with localedef and LOCPATH (skipped without localedef), importing a module
  from a non-ASCII dependency directory. The forked JVM is awaited with a
  timeout, its output going to a file, so a stuck JVM cannot hang the build.
- A probe value which is not valid in the file system encoding of the JVM
  was replaced with U+FFFD, naming another path, and a sign was accepted in
  its hex digits; it is now decoded strictly, and the probe fails clearly.
- The UTF-8 mode depends on Jep setting argv before the other PyConfig
  values, so a new interpreter now checks sys.flags.utf8_mode, failing
  clearly if a future Jep breaks it.
- The README documents that an inherited PYTHONIOENCODING still wins for the
  embedded streams, and that the ASCII limitation of the executable's path
  applies only to non-UTF-8 POSIX locales; CLAUDE.md keeps only the gotchas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…err (#19)

Any non-zero exit of the environment probe was reported as a missing jep
package with a pip install remedy, also when Python itself could not run,
e.g. a pyenv shim of a version which is not installed. The probe now exits
with a dedicated code (83) when it cannot find jep, and only this one gets
the jep headline; any other exit says Python cannot be run, with its exit
code and output.

The stderr of a successful probe was dropped, so a warning like "Could not
find platform libraries" of a wrong PYTHONHOME left no trace when the
embedded interpreter, configured the same way, failed later. It is kept in
PythonProbe.error, and added to the failure of creating the first
interpreter. The main code has no logging, so it is not logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…on (#19)

Jep 4.3 rethrows the failure of initializing CPython as a plain
java.lang.Error, and every later attempt as another Error ("previously
failed to initialize"), which the catch of JepException and LinkageError
missed, so the probe's stderr was never attached where it matters, e.g. a
broken encodings module. EmbeddedPythonStarter now creates the interpreters
of Python: any failure before the first interpreter exists, and any Error of
Jep at any time, is a startup failure, always wrapped in an
IllegalStateException with the original as cause, so its type does not
depend on whether the probe wrote anything; the probe's stderr is offered as
a possible hint. A later, unrelated failure of an interpreter is thrown as
is, instead of blaming an old warning.

The UTF-8 mode check now runs before the interpreter is kept, closing it on
failure, as a later call skipped the check of the kept interpreter.

The probe names a jep module shadowing the package, as "pip install jep"
would only answer "Requirement already satisfied"; the failure messages no
longer end with an empty line without output; the kept stderr is limited to
its first 8 KiB, where a startup warning of Python is, as it lives as long
as the JVM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
The advice of a missing jep package, and of a missing native Jep library,
was `python -m pip install jep`, which fails on an externally managed Python
(PEP 668, e.g. Homebrew or most Linux distributions), installs the latest jep
instead of the one matching the org.ninia:jep library, and does not tell
that building jep from its source needs a C compiler and a JDK.

Both messages now advise creating a virtual environment, installing
jep==<version> into it, and pointing to it with VIRTUAL_ENV or
PythonEnvironment.pythonExecutable, with the platform's venv layout and the
value of JAVA_HOME. The version is read at runtime from the pom.properties
of the org.ninia:jep jar on the classpath, so that it matches the library
actually in use, also when an application resolves another version than
this build; without it, the advice falls back to an unpinned jep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…19)

Nothing tested the wiring in Python.call, so a forked JVM now starts the
embedded CPython with a fake encodings package, which warns and fails only
when not run with -c, as the probe is, and the test checks that the
IllegalStateException carries the probe's warning; it fails when Python
creates the SharedInterpreter directly.

Jep's MainInterpreter is a singleton, so once an interpreter is created any
failure, also an Error like NoClassDefFoundError, is not a startup failure,
and is thrown as is. Jep wraps the failure of initializing CPython in an
Error whose message is the toString() of its cause, so the headline takes
the message of the cause instead of repeating its class.

The kept stderr of the probe is limited including the note of truncation,
without splitting a surrogate pair. The message of a probe without values
no longer ends with an empty line without output, and a jep package
directory which cannot be listed is reported naming it, instead of a raw
NIO exception. The Python KDoc no longer promises a cause for the failures
of resolving the environment, which have none.

The probe's warning is attached only to a failure of the embedded CPython
to start, and it is not logged, as the main code has no logging, so a later
failure caused by the same misconfiguration carries no hint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
Keeps the one jepLibrary function of #19, with its wrapping of a directory
which cannot be listed, giving the pinned virtual environment advice of #37
in both of its messages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…naged Python (#37)

The advice of a missing jep sent every user to a new ~/.venvs/jep, also when
the Python already was a virtual environment, e.g. of VIRTUAL_ENV, holding
the other packages of the user, where a pip install of jep just works.
The probe now tells whether the Python is externally managed (PEP 668):
a base Python with an EXTERNALLY-MANAGED file in its standard library;
also when jep is missing, with its own exit code 84, as it exits before
writing its values. Only then is a virtual environment advised, otherwise
the pinned pip install into the Python itself.

The virtual environment is an absolute path of user.home, quoted when it
has a space, as ~ and %USERPROFILE% are expanded only by a shell, not by an
IDE run configuration or by PowerShell. As PyPI has no wheels of jep, it is
always built from its source, so the advice tells about the Python headers,
and that JAVA_HOME is optional where javac on the PATH is found. A broken
jep is reinstalled with --force-reinstall --no-cache-dir, as a plain
install would only tell that the requirement is already satisfied.

The jep version is read lazily, a malformed pom.properties giving no
version instead of an ExceptionInInitializerError, and the tests read the
version catalog themselves, so that they also run outside Gradle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…vice (#37)

Following the advice still failed in a few common setups:
- on stock Debian/Ubuntu, creating the virtual environment fails without
  the python3-venv package, so it is named next to python3-dev,
- a virtual environment of uv has no pip, so uv pip install is advised too,
- PowerShell takes a quoted command for a string, so on Windows the advice
  tells to prefix it with &, keeping the quoting which cmd needs.
The README no longer gives the virtual environment as an example of
PythonEnvironment.pythonExecutable, which must be its Python, and tells
that the version of the jep package must match the library.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
Base automatically changed from issue-19-probe-failure-message to issue-21-non-utf8-locale November 10, 2026 18:04
Base automatically changed from issue-21-non-utf8-locale to issue-15-probe-parsing November 10, 2026 18:05
Base automatically changed from issue-15-probe-parsing to issue-14-plan-stderr November 10, 2026 21:18
Base automatically changed from issue-14-plan-stderr to issue-24-installer-sys-path November 10, 2026 21:19
Base automatically changed from issue-24-installer-sys-path to issue-13-runprocess-deadlock November 10, 2026 21:20
* Do not import from the working directory in installer Pythons (#24)

The environment probe, the PLAN and the pip fallback started Python with
-c and -m pip, which put the JVM's working directory first on sys.path.
A json.py, tomllib.py, jep/ or pip/ there - likely in an agent's working
directory full of generated files - replaced the module the installer
imports, running unintended code, or made the probe report the wrong jep.

-P would fix it, but it is unknown before Python 3.11, and 3.10 is
supported. -I would ignore every PYTHON* variable, also the PYTHONUTF8
the process streams depend on, PYTHONPATH and the user site-packages,
which the embedded interpreter still sees. So the code started with -c
first removes the '' entry of the working directory from sys.path, before
any import, and pip is run with runpy instead of -m. PIP_* variables and
pip config files keep working, as nothing about the environment changes.

uv is not changed: it queries the target interpreter itself, in isolated
mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Apply the review of the installer sys.path fix (#24)

The KDoc of pythonCommand now says precisely that only the implicit
working directory entry of -c is removed, while the entries the user
configured, e.g. with PYTHONPATH, are kept, as the embedded interpreter
sees them too. The tests of PLAN without tomllib, the only ones of its
Python < 3.11 behavior, now run it with the same prologue as production.
The probe shadowing test runs its reference probe before creating the
temporary directory, so that a failure of it does not leak the directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Parse only the standard output of PLAN (#50)

* Parse only the standard output of PLAN, not merged with stderr (#14)

PLAN ran with its standard error merged into its standard output,
which was then parsed line by line as the install plan.
Python block-buffers stdout in a pipe, so anything written to stderr
during the run, e.g. a warning without a trailing line break,
lands before or inside a plan line in the merged stream,
corrupting or dropping a requirement, the same class of bug as #4
fixed for the environment probe.

PLAN now runs with mergeErrorStream = false, only stdout is parsed,
and stderr, where sys.exit writes PLAN's error messages, is included
in the PythonDependencyException on failure.
The uv / pip install runs keep the merged output,
as it is only shown in the error message, never parsed.

plan() takes the command as a defaulted parameter, a test seam
for a PLAN run which writes to stderr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Share the failure output of a process, test PLAN without merged streams (#14)

The PLAN failure and the environment probe failure both combined
stderr and stdout into the error message, in opposite orders;
ProcessResult.failureOutput now does it once, stderr first,
as it tells why the process failed, e.g. the message of sys.exit.

The tests of PLAN without tomllib, or without any TOML parser,
ran it with stderr merged into stdout, so their assertions on
the sys.exit message passed only because of the merging,
while plan() reads the streams separately.
They now run through plan(), with the command as the test seam,
so they check the message of the exception the user gets.
The tests in a working directory with shadowing modules
read stdout separately as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Frame the environment probe's values instead of parsing stdout by line position (#61)

* Delimit the values of the Python environment probe (#15)

The probe printed its values one per line and Kotlin took them by line
position, so anything else writing to stdout - a sitecustomize module or
a .pth file - shifted every value, and an empty first value, e.g. an
empty sys.executable, was trimmed away, leaving too few lines and an
IndexOutOfBoundsException instead of the documented IllegalStateException.

The probe now writes its values separated by NUL, which no path can
contain, between unique begin and end sentinel lines, preceded by an
empty line in case the noise before does not end with a newline, and as
bytes, so without newline translation. Kotlin takes the values only from
between the last begin and the following end sentinel, and fails with an
IllegalStateException showing the output when there are not exactly four.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Reject an empty executable, read the probe from the original stdout (#15)

An empty sys.executable was parsed correctly but then accepted silently:
the embedded CPython got an empty program name, so it ignored the
pyvenv.cfg of a virtual environment, and installing dependencies failed
later with a bare IOException of running "". The probe now fails right
away with an IllegalStateException telling that sys.executable is empty.

The probe wrote to sys.stdout.buffer, which a sitecustomize replacing
sys.stdout with a wrapper does not have, so it now writes to the
original sys.__stdout__, after flushing both, which also bypasses any
redirection of sys.stdout.

When the output has no probe values, the message now names the Python
executable and shows its stderr as well, as it tells why; the parsing
returns null for the caller to report it, and is unit-tested on its own
for output around the values, repeated markers, an empty value, a
missing end marker, and a wrong number of values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Run the embedded Python in UTF-8 mode and pass paths losslessly (#64)

* Run the embedded Python in UTF-8 mode, transfer paths as file system bytes (#21)

The embedded interpreter used the encoding of the JVM's locale, so under
LC_ALL=C with PYTHONUTF8=0, or a cp1252 Windows locale, printing non-ASCII
text or reading a UTF-8 file failed. Jep's PyConfig has no UTF-8 mode setter,
but CPython parses `-X utf8` from the argv set with PyConfig.setArgv before
decoding the other values, which enables it, also over PYTHONUTF8=0.

Under a non-UTF-8 POSIX locale, e.g. ISO-8859-1 with a home like /home/josé,
a path is the locale's bytes, surrogate-escaped by Python in UTF-8 mode:
- the environment probe failed encoding such a path to UTF-8; it now writes
  each value as the hex of its os.fsencode bytes, which Kotlin decodes with
  sun.jnu.encoding (UTF-8 on Windows), so it names the same file;
- the dependency directory, a Java String passed by Jep as UTF-8, was another
  path in the embedded sys.path; it is now appended as os.fsdecode of its
  bytes. In argv, to PLAN and pip, it already round-trips unchanged.
A non-ASCII path of the Python executable cannot be fixed here, as Jep passes
the program name as UTF-8 too; the README documents it.

PythonTest compared sys.path of a raw ProcessBuilder decoded as UTF-8; it now
uses runProcess and compares ascii() of the paths on both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Test the Latin-1 dependency directory, decode probe paths strictly (#21)

Review findings of the UTF-8 mode fix:
- The os.fsdecode conversion of the dependency directory had no regression
  test; PythonLocaleTest now forks a JVM in an ISO-8859-1 locale, created
  with localedef and LOCPATH (skipped without localedef), importing a module
  from a non-ASCII dependency directory. The forked JVM is awaited with a
  timeout, its output going to a file, so a stuck JVM cannot hang the build.
- A probe value which is not valid in the file system encoding of the JVM
  was replaced with U+FFFD, naming another path, and a sign was accepted in
  its hex digits; it is now decoded strictly, and the probe fails clearly.
- The UTF-8 mode depends on Jep setting argv before the other PyConfig
  values, so a new interpreter now checks sys.flags.utf8_mode, failing
  clearly if a future Jep breaks it.
- The README documents that an inherited PYTHONIOENCODING still wins for the
  embedded streams, and that the ASCII limitation of the executable's path
  applies only to non-UTF-8 POSIX locales; CLAUDE.md keeps only the gotchas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Tell a missing jep from a Python failing to run, keep the probe's stderr for startup failures (#65)

* Tell a missing jep from a Python failing to run, keep the probe's stderr (#19)

Any non-zero exit of the environment probe was reported as a missing jep
package with a pip install remedy, also when Python itself could not run,
e.g. a pyenv shim of a version which is not installed. The probe now exits
with a dedicated code (83) when it cannot find jep, and only this one gets
the jep headline; any other exit says Python cannot be run, with its exit
code and output.

The stderr of a successful probe was dropped, so a warning like "Could not
find platform libraries" of a wrong PYTHONHOME left no trace when the
embedded interpreter, configured the same way, failed later. It is kept in
PythonProbe.error, and added to the failure of creating the first
interpreter. The main code has no logging, so it is not logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Report the probe's stderr on any startup failure of the embedded Python (#19)

Jep 4.3 rethrows the failure of initializing CPython as a plain
java.lang.Error, and every later attempt as another Error ("previously
failed to initialize"), which the catch of JepException and LinkageError
missed, so the probe's stderr was never attached where it matters, e.g. a
broken encodings module. EmbeddedPythonStarter now creates the interpreters
of Python: any failure before the first interpreter exists, and any Error of
Jep at any time, is a startup failure, always wrapped in an
IllegalStateException with the original as cause, so its type does not
depend on whether the probe wrote anything; the probe's stderr is offered as
a possible hint. A later, unrelated failure of an interpreter is thrown as
is, instead of blaming an old warning.

The UTF-8 mode check now runs before the interpreter is kept, closing it on
failure, as a later call skipped the check of the kept interpreter.

The probe names a jep module shadowing the package, as "pip install jep"
would only answer "Requirement already satisfied"; the failure messages no
longer end with an empty line without output; the kept stderr is limited to
its first 8 KiB, where a startup warning of Python is, as it lives as long
as the JVM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Test the startup failure end to end, wrap only failures before start (#19)

Nothing tested the wiring in Python.call, so a forked JVM now starts the
embedded CPython with a fake encodings package, which warns and fails only
when not run with -c, as the probe is, and the test checks that the
IllegalStateException carries the probe's warning; it fails when Python
creates the SharedInterpreter directly.

Jep's MainInterpreter is a singleton, so once an interpreter is created any
failure, also an Error like NoClassDefFoundError, is not a startup failure,
and is thrown as is. Jep wraps the failure of initializing CPython in an
Error whose message is the toString() of its cause, so the headline takes
the message of the cause instead of repeating its class.

The kept stderr of the probe is limited including the note of truncation,
without splitting a surrogate pair. The message of a probe without values
no longer ends with an empty line without output, and a jep package
directory which cannot be listed is reported naming it, instead of a raw
NIO exception. The Python KDoc no longer promises a cause for the failures
of resolving the environment, which have none.

The probe's warning is attached only to a failure of the embedded CPython
to start, and it is not logged, as the main code has no logging, so a later
failure caused by the same misconfiguration carries no hint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Recovered onto main after the stacked PR was merged into its parent branch (#46).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
Takes the recovery branch for everything which is not the change of #37,
whose original commits of #13 to #19 it carries squashed, along with the
newer work of main, and reapplies the own change of #37 on top of it:
the jep install advice, the externally managed detection, the version of
jep from pom.properties, the README install section, and their tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
@morisil
morisil changed the base branch from issue-13-runprocess-deadlock to recover-environment-fixes November 10, 2026 21:29
Base automatically changed from recover-environment-fixes to main November 10, 2026 22:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

# Conflicts:
#	src/main/kotlin/PythonEnvironment.kt
#	src/test/kotlin/PythonEnvironmentTest.kt
@morisil
morisil merged commit 7e3f623 into main Nov 10, 2026
3 checks passed
@morisil
morisil deleted the issue-37-jep-install-advice branch November 10, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The suggested 'pip install jep' fails on externally managed Pythons, and does not pin the version of the library

1 participant