Skip to content

Recover the reviewed fixes of #24, #14, #15, #21 and #19 onto main - #68

Merged
morisil merged 1 commit into
mainfrom
recover-environment-fixes
Nov 10, 2026
Merged

morisil merged 1 commit into
mainfrom
recover-environment-fixes

Conversation

@morisil

@morisil morisil commented Nov 10, 2026

Copy link
Copy Markdown
Member

Fixes #24
Fixes #14
Fixes #15
Fixes #21
Fixes #19

Why this PR

These fixes were reviewed and approved as the stacked PRs #46 (#24), #50 (#14), #61 (#15), #64 (#21) and #65 (#19), but those were merged into their parent branches after #41 had already been merged to main, so they ended up only on issue-13-runprocess-deadlock (as the squash commit 11aa935), never on main, and their issues stayed open.

This PR re-applies that commit onto the current main, unchanged in substance — see each original PR for the fix, the tests and the review rounds.

Conflicts resolved

main meanwhile got #56 (#23), #51 (#11, #35), #49 (#2, #32, #16), #44 (#12) and #42 (#17). Resolved keeping both sides:

build.gradle.kts, DEVELOPMENT.md, .github, PythonCloseTest.kt and the API dump are unchanged against main; #23's lock, #32's running flag, #16's proxy conversion, #11's basePython and #35's workflow are intact.

Verification

./gradlew build passes, including checkKotlinAbi — 149 tests, among them the forked-JVM PythonLocaleTest (ISO-8859-1 locale) and PythonStartupFailureTest.

The #13 reader-hang fix (4c32923), also pushed to #41's branch, was already on main (included in the squash of #41), so nothing else is missing. #37 (PR #67) follows on top of this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Do not import from the working directory in installer Pythons (#24)

The environment probe, the PLAN and the pip fallback started Python with
-c and -m pip, which put the JVM's working directory first on sys.path.
A json.py, tomllib.py, jep/ or pip/ there - likely in an agent's working
directory full of generated files - replaced the module the installer
imports, running unintended code, or made the probe report the wrong jep.

-P would fix it, but it is unknown before Python 3.11, and 3.10 is
supported. -I would ignore every PYTHON* variable, also the PYTHONUTF8
the process streams depend on, PYTHONPATH and the user site-packages,
which the embedded interpreter still sees. So the code started with -c
first removes the '' entry of the working directory from sys.path, before
any import, and pip is run with runpy instead of -m. PIP_* variables and
pip config files keep working, as nothing about the environment changes.

uv is not changed: it queries the target interpreter itself, in isolated
mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Apply the review of the installer sys.path fix (#24)

The KDoc of pythonCommand now says precisely that only the implicit
working directory entry of -c is removed, while the entries the user
configured, e.g. with PYTHONPATH, are kept, as the embedded interpreter
sees them too. The tests of PLAN without tomllib, the only ones of its
Python < 3.11 behavior, now run it with the same prologue as production.
The probe shadowing test runs its reference probe before creating the
temporary directory, so that a failure of it does not leak the directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Parse only the standard output of PLAN (#50)

* Parse only the standard output of PLAN, not merged with stderr (#14)

PLAN ran with its standard error merged into its standard output,
which was then parsed line by line as the install plan.
Python block-buffers stdout in a pipe, so anything written to stderr
during the run, e.g. a warning without a trailing line break,
lands before or inside a plan line in the merged stream,
corrupting or dropping a requirement, the same class of bug as #4
fixed for the environment probe.

PLAN now runs with mergeErrorStream = false, only stdout is parsed,
and stderr, where sys.exit writes PLAN's error messages, is included
in the PythonDependencyException on failure.
The uv / pip install runs keep the merged output,
as it is only shown in the error message, never parsed.

plan() takes the command as a defaulted parameter, a test seam
for a PLAN run which writes to stderr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Share the failure output of a process, test PLAN without merged streams (#14)

The PLAN failure and the environment probe failure both combined
stderr and stdout into the error message, in opposite orders;
ProcessResult.failureOutput now does it once, stderr first,
as it tells why the process failed, e.g. the message of sys.exit.

The tests of PLAN without tomllib, or without any TOML parser,
ran it with stderr merged into stdout, so their assertions on
the sys.exit message passed only because of the merging,
while plan() reads the streams separately.
They now run through plan(), with the command as the test seam,
so they check the message of the exception the user gets.
The tests in a working directory with shadowing modules
read stdout separately as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Frame the environment probe's values instead of parsing stdout by line position (#61)

* Delimit the values of the Python environment probe (#15)

The probe printed its values one per line and Kotlin took them by line
position, so anything else writing to stdout - a sitecustomize module or
a .pth file - shifted every value, and an empty first value, e.g. an
empty sys.executable, was trimmed away, leaving too few lines and an
IndexOutOfBoundsException instead of the documented IllegalStateException.

The probe now writes its values separated by NUL, which no path can
contain, between unique begin and end sentinel lines, preceded by an
empty line in case the noise before does not end with a newline, and as
bytes, so without newline translation. Kotlin takes the values only from
between the last begin and the following end sentinel, and fails with an
IllegalStateException showing the output when there are not exactly four.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Reject an empty executable, read the probe from the original stdout (#15)

An empty sys.executable was parsed correctly but then accepted silently:
the embedded CPython got an empty program name, so it ignored the
pyvenv.cfg of a virtual environment, and installing dependencies failed
later with a bare IOException of running "". The probe now fails right
away with an IllegalStateException telling that sys.executable is empty.

The probe wrote to sys.stdout.buffer, which a sitecustomize replacing
sys.stdout with a wrapper does not have, so it now writes to the
original sys.__stdout__, after flushing both, which also bypasses any
redirection of sys.stdout.

When the output has no probe values, the message now names the Python
executable and shows its stderr as well, as it tells why; the parsing
returns null for the caller to report it, and is unit-tested on its own
for output around the values, repeated markers, an empty value, a
missing end marker, and a wrong number of values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Run the embedded Python in UTF-8 mode and pass paths losslessly (#64)

* Run the embedded Python in UTF-8 mode, transfer paths as file system bytes (#21)

The embedded interpreter used the encoding of the JVM's locale, so under
LC_ALL=C with PYTHONUTF8=0, or a cp1252 Windows locale, printing non-ASCII
text or reading a UTF-8 file failed. Jep's PyConfig has no UTF-8 mode setter,
but CPython parses `-X utf8` from the argv set with PyConfig.setArgv before
decoding the other values, which enables it, also over PYTHONUTF8=0.

Under a non-UTF-8 POSIX locale, e.g. ISO-8859-1 with a home like /home/josé,
a path is the locale's bytes, surrogate-escaped by Python in UTF-8 mode:
- the environment probe failed encoding such a path to UTF-8; it now writes
  each value as the hex of its os.fsencode bytes, which Kotlin decodes with
  sun.jnu.encoding (UTF-8 on Windows), so it names the same file;
- the dependency directory, a Java String passed by Jep as UTF-8, was another
  path in the embedded sys.path; it is now appended as os.fsdecode of its
  bytes. In argv, to PLAN and pip, it already round-trips unchanged.
A non-ASCII path of the Python executable cannot be fixed here, as Jep passes
the program name as UTF-8 too; the README documents it.

PythonTest compared sys.path of a raw ProcessBuilder decoded as UTF-8; it now
uses runProcess and compares ascii() of the paths on both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Test the Latin-1 dependency directory, decode probe paths strictly (#21)

Review findings of the UTF-8 mode fix:
- The os.fsdecode conversion of the dependency directory had no regression
  test; PythonLocaleTest now forks a JVM in an ISO-8859-1 locale, created
  with localedef and LOCPATH (skipped without localedef), importing a module
  from a non-ASCII dependency directory. The forked JVM is awaited with a
  timeout, its output going to a file, so a stuck JVM cannot hang the build.
- A probe value which is not valid in the file system encoding of the JVM
  was replaced with U+FFFD, naming another path, and a sign was accepted in
  its hex digits; it is now decoded strictly, and the probe fails clearly.
- The UTF-8 mode depends on Jep setting argv before the other PyConfig
  values, so a new interpreter now checks sys.flags.utf8_mode, failing
  clearly if a future Jep breaks it.
- The README documents that an inherited PYTHONIOENCODING still wins for the
  embedded streams, and that the ASCII limitation of the executable's path
  applies only to non-UTF-8 POSIX locales; CLAUDE.md keeps only the gotchas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Tell a missing jep from a Python failing to run, keep the probe's stderr for startup failures (#65)

* Tell a missing jep from a Python failing to run, keep the probe's stderr (#19)

Any non-zero exit of the environment probe was reported as a missing jep
package with a pip install remedy, also when Python itself could not run,
e.g. a pyenv shim of a version which is not installed. The probe now exits
with a dedicated code (83) when it cannot find jep, and only this one gets
the jep headline; any other exit says Python cannot be run, with its exit
code and output.

The stderr of a successful probe was dropped, so a warning like "Could not
find platform libraries" of a wrong PYTHONHOME left no trace when the
embedded interpreter, configured the same way, failed later. It is kept in
PythonProbe.error, and added to the failure of creating the first
interpreter. The main code has no logging, so it is not logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Report the probe's stderr on any startup failure of the embedded Python (#19)

Jep 4.3 rethrows the failure of initializing CPython as a plain
java.lang.Error, and every later attempt as another Error ("previously
failed to initialize"), which the catch of JepException and LinkageError
missed, so the probe's stderr was never attached where it matters, e.g. a
broken encodings module. EmbeddedPythonStarter now creates the interpreters
of Python: any failure before the first interpreter exists, and any Error of
Jep at any time, is a startup failure, always wrapped in an
IllegalStateException with the original as cause, so its type does not
depend on whether the probe wrote anything; the probe's stderr is offered as
a possible hint. A later, unrelated failure of an interpreter is thrown as
is, instead of blaming an old warning.

The UTF-8 mode check now runs before the interpreter is kept, closing it on
failure, as a later call skipped the check of the kept interpreter.

The probe names a jep module shadowing the package, as "pip install jep"
would only answer "Requirement already satisfied"; the failure messages no
longer end with an empty line without output; the kept stderr is limited to
its first 8 KiB, where a startup warning of Python is, as it lives as long
as the JVM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

* Test the startup failure end to end, wrap only failures before start (#19)

Nothing tested the wiring in Python.call, so a forked JVM now starts the
embedded CPython with a fake encodings package, which warns and fails only
when not run with -c, as the probe is, and the test checks that the
IllegalStateException carries the probe's warning; it fails when Python
creates the SharedInterpreter directly.

Jep's MainInterpreter is a singleton, so once an interpreter is created any
failure, also an Error like NoClassDefFoundError, is not a startup failure,
and is thrown as is. Jep wraps the failure of initializing CPython in an
Error whose message is the toString() of its cause, so the headline takes
the message of the cause instead of repeating its class.

The kept stderr of the probe is limited including the note of truncation,
without splitting a surrogate pair. The message of a probe without values
no longer ends with an empty line without output, and a jep package
directory which cannot be listed is reported naming it, instead of a raw
NIO exception. The Python KDoc no longer promises a cause for the failures
of resolving the environment, which have none.

The probe's warning is attached only to a failure of the embedded CPython
to start, and it is not logged, as the main code has no logging, so a later
failure caused by the same misconfiguration carries no hint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Recovered onto main after the stacked PR was merged into its parent branch (#46).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
@morisil
morisil merged commit c1d7243 into main Nov 10, 2026
3 checks passed
@morisil
morisil deleted the recover-environment-fixes branch November 10, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment