Repository navigation
Recover the reviewed fixes of #24, #14, #15, #21 and #19 onto main - #68
Merged
Merged
Conversation
* Do not import from the working directory in installer Pythons (#24) The environment probe, the PLAN and the pip fallback started Python with -c and -m pip, which put the JVM's working directory first on sys.path. A json.py, tomllib.py, jep/ or pip/ there - likely in an agent's working directory full of generated files - replaced the module the installer imports, running unintended code, or made the probe report the wrong jep. -P would fix it, but it is unknown before Python 3.11, and 3.10 is supported. -I would ignore every PYTHON* variable, also the PYTHONUTF8 the process streams depend on, PYTHONPATH and the user site-packages, which the embedded interpreter still sees. So the code started with -c first removes the '' entry of the working directory from sys.path, before any import, and pip is run with runpy instead of -m. PIP_* variables and pip config files keep working, as nothing about the environment changes. uv is not changed: it queries the target interpreter itself, in isolated mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Apply the review of the installer sys.path fix (#24) The KDoc of pythonCommand now says precisely that only the implicit working directory entry of -c is removed, while the entries the user configured, e.g. with PYTHONPATH, are kept, as the embedded interpreter sees them too. The tests of PLAN without tomllib, the only ones of its Python < 3.11 behavior, now run it with the same prologue as production. The probe shadowing test runs its reference probe before creating the temporary directory, so that a failure of it does not leak the directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Parse only the standard output of PLAN (#50) * Parse only the standard output of PLAN, not merged with stderr (#14) PLAN ran with its standard error merged into its standard output, which was then parsed line by line as the install plan. Python block-buffers stdout in a pipe, so anything written to stderr during the run, e.g. a warning without a trailing line break, lands before or inside a plan line in the merged stream, corrupting or dropping a requirement, the same class of bug as #4 fixed for the environment probe. PLAN now runs with mergeErrorStream = false, only stdout is parsed, and stderr, where sys.exit writes PLAN's error messages, is included in the PythonDependencyException on failure. The uv / pip install runs keep the merged output, as it is only shown in the error message, never parsed. plan() takes the command as a defaulted parameter, a test seam for a PLAN run which writes to stderr. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Share the failure output of a process, test PLAN without merged streams (#14) The PLAN failure and the environment probe failure both combined stderr and stdout into the error message, in opposite orders; ProcessResult.failureOutput now does it once, stderr first, as it tells why the process failed, e.g. the message of sys.exit. The tests of PLAN without tomllib, or without any TOML parser, ran it with stderr merged into stdout, so their assertions on the sys.exit message passed only because of the merging, while plan() reads the streams separately. They now run through plan(), with the command as the test seam, so they check the message of the exception the user gets. The tests in a working directory with shadowing modules read stdout separately as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Frame the environment probe's values instead of parsing stdout by line position (#61) * Delimit the values of the Python environment probe (#15) The probe printed its values one per line and Kotlin took them by line position, so anything else writing to stdout - a sitecustomize module or a .pth file - shifted every value, and an empty first value, e.g. an empty sys.executable, was trimmed away, leaving too few lines and an IndexOutOfBoundsException instead of the documented IllegalStateException. The probe now writes its values separated by NUL, which no path can contain, between unique begin and end sentinel lines, preceded by an empty line in case the noise before does not end with a newline, and as bytes, so without newline translation. Kotlin takes the values only from between the last begin and the following end sentinel, and fails with an IllegalStateException showing the output when there are not exactly four. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Reject an empty executable, read the probe from the original stdout (#15) An empty sys.executable was parsed correctly but then accepted silently: the embedded CPython got an empty program name, so it ignored the pyvenv.cfg of a virtual environment, and installing dependencies failed later with a bare IOException of running "". The probe now fails right away with an IllegalStateException telling that sys.executable is empty. The probe wrote to sys.stdout.buffer, which a sitecustomize replacing sys.stdout with a wrapper does not have, so it now writes to the original sys.__stdout__, after flushing both, which also bypasses any redirection of sys.stdout. When the output has no probe values, the message now names the Python executable and shows its stderr as well, as it tells why; the parsing returns null for the caller to report it, and is unit-tested on its own for output around the values, repeated markers, an empty value, a missing end marker, and a wrong number of values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Run the embedded Python in UTF-8 mode and pass paths losslessly (#64) * Run the embedded Python in UTF-8 mode, transfer paths as file system bytes (#21) The embedded interpreter used the encoding of the JVM's locale, so under LC_ALL=C with PYTHONUTF8=0, or a cp1252 Windows locale, printing non-ASCII text or reading a UTF-8 file failed. Jep's PyConfig has no UTF-8 mode setter, but CPython parses `-X utf8` from the argv set with PyConfig.setArgv before decoding the other values, which enables it, also over PYTHONUTF8=0. Under a non-UTF-8 POSIX locale, e.g. ISO-8859-1 with a home like /home/josé, a path is the locale's bytes, surrogate-escaped by Python in UTF-8 mode: - the environment probe failed encoding such a path to UTF-8; it now writes each value as the hex of its os.fsencode bytes, which Kotlin decodes with sun.jnu.encoding (UTF-8 on Windows), so it names the same file; - the dependency directory, a Java String passed by Jep as UTF-8, was another path in the embedded sys.path; it is now appended as os.fsdecode of its bytes. In argv, to PLAN and pip, it already round-trips unchanged. A non-ASCII path of the Python executable cannot be fixed here, as Jep passes the program name as UTF-8 too; the README documents it. PythonTest compared sys.path of a raw ProcessBuilder decoded as UTF-8; it now uses runProcess and compares ascii() of the paths on both sides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Test the Latin-1 dependency directory, decode probe paths strictly (#21) Review findings of the UTF-8 mode fix: - The os.fsdecode conversion of the dependency directory had no regression test; PythonLocaleTest now forks a JVM in an ISO-8859-1 locale, created with localedef and LOCPATH (skipped without localedef), importing a module from a non-ASCII dependency directory. The forked JVM is awaited with a timeout, its output going to a file, so a stuck JVM cannot hang the build. - A probe value which is not valid in the file system encoding of the JVM was replaced with U+FFFD, naming another path, and a sign was accepted in its hex digits; it is now decoded strictly, and the probe fails clearly. - The UTF-8 mode depends on Jep setting argv before the other PyConfig values, so a new interpreter now checks sys.flags.utf8_mode, failing clearly if a future Jep breaks it. - The README documents that an inherited PYTHONIOENCODING still wins for the embedded streams, and that the ASCII limitation of the executable's path applies only to non-UTF-8 POSIX locales; CLAUDE.md keeps only the gotchas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Tell a missing jep from a Python failing to run, keep the probe's stderr for startup failures (#65) * Tell a missing jep from a Python failing to run, keep the probe's stderr (#19) Any non-zero exit of the environment probe was reported as a missing jep package with a pip install remedy, also when Python itself could not run, e.g. a pyenv shim of a version which is not installed. The probe now exits with a dedicated code (83) when it cannot find jep, and only this one gets the jep headline; any other exit says Python cannot be run, with its exit code and output. The stderr of a successful probe was dropped, so a warning like "Could not find platform libraries" of a wrong PYTHONHOME left no trace when the embedded interpreter, configured the same way, failed later. It is kept in PythonProbe.error, and added to the failure of creating the first interpreter. The main code has no logging, so it is not logged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Report the probe's stderr on any startup failure of the embedded Python (#19) Jep 4.3 rethrows the failure of initializing CPython as a plain java.lang.Error, and every later attempt as another Error ("previously failed to initialize"), which the catch of JepException and LinkageError missed, so the probe's stderr was never attached where it matters, e.g. a broken encodings module. EmbeddedPythonStarter now creates the interpreters of Python: any failure before the first interpreter exists, and any Error of Jep at any time, is a startup failure, always wrapped in an IllegalStateException with the original as cause, so its type does not depend on whether the probe wrote anything; the probe's stderr is offered as a possible hint. A later, unrelated failure of an interpreter is thrown as is, instead of blaming an old warning. The UTF-8 mode check now runs before the interpreter is kept, closing it on failure, as a later call skipped the check of the kept interpreter. The probe names a jep module shadowing the package, as "pip install jep" would only answer "Requirement already satisfied"; the failure messages no longer end with an empty line without output; the kept stderr is limited to its first 8 KiB, where a startup warning of Python is, as it lives as long as the JVM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Test the startup failure end to end, wrap only failures before start (#19) Nothing tested the wiring in Python.call, so a forked JVM now starts the embedded CPython with a fake encodings package, which warns and fails only when not run with -c, as the probe is, and the test checks that the IllegalStateException carries the probe's warning; it fails when Python creates the SharedInterpreter directly. Jep's MainInterpreter is a singleton, so once an interpreter is created any failure, also an Error like NoClassDefFoundError, is not a startup failure, and is thrown as is. Jep wraps the failure of initializing CPython in an Error whose message is the toString() of its cause, so the headline takes the message of the cause instead of repeating its class. The kept stderr of the probe is limited including the note of truncation, without splitting a surrogate pair. The message of a probe without values no longer ends with an empty line without output, and a jep package directory which cannot be listed is reported naming it, instead of a raw NIO exception. The Python KDoc no longer promises a cause for the failures of resolving the environment, which have none. The probe's warning is attached only to a failure of the embedded CPython to start, and it is not logged, as the main code has no logging, so a later failure caused by the same misconfiguration carries no hint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Recovered onto main after the stacked PR was merged into its parent branch (#46). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #24
Fixes #14
Fixes #15
Fixes #21
Fixes #19
Why this PR
These fixes were reviewed and approved as the stacked PRs #46 (#24), #50 (#14), #61 (#15), #64 (#21) and #65 (#19), but those were merged into their parent branches after #41 had already been merged to
main, so they ended up only onissue-13-runprocess-deadlock(as the squash commit 11aa935), never onmain, and their issues stayed open.This PR re-applies that commit onto the current
main, unchanged in substance — see each original PR for the fix, the tests and the review rounds.Conflicts resolved
mainmeanwhile got #56 (#23), #51 (#11, #35), #49 (#2, #32, #16), #44 (#12) and #42 (#17). Resolved keeping both sides:Python.ktcall: A callback waiting on another thread that calls Python or close() deadlocks #32'srunning/checkNotClosedordering kept; only the interpreter creation now goes through Environment probe failure message points to jep for any failure, and discards stderr on success #19'sEmbeddedPythonStarter.createInterpreterwith Non-UTF-8 locales still break the embedded interpreter and non-ASCII paths #21's UTF-8 mode check.PythonDependencies.kt: Concurrent installs from different JVMs can corrupt the shared dependency directory #23's unlocked plan → in-JVM lock → file lock → re-plan → install kept; with Installer Pythons import from the JVM working directory, which can shadow stdlib and pip modules #24'splanCommand/pipCommand, PythonDependencies parses PLAN output merged with stderr #14's separate stderr, and Non-UTF-8 locales still break the embedded interpreter and non-ASCII paths #21'spythonPath— whose append to the sharedsys.pathstays inside Concurrent installs from different JVMs can corrupt the shared dependency directory #23'ssynchronizedblock.CLAUDE.md, test imports: both sides kept.build.gradle.kts,DEVELOPMENT.md,.github,PythonCloseTest.ktand the API dump are unchanged againstmain; #23's lock, #32's running flag, #16's proxy conversion, #11'sbasePythonand #35's workflow are intact.Verification
./gradlew buildpasses, includingcheckKotlinAbi— 149 tests, among them the forked-JVMPythonLocaleTest(ISO-8859-1 locale) andPythonStartupFailureTest.The #13 reader-hang fix (4c32923), also pushed to #41's branch, was already on
main(included in the squash of #41), so nothing else is missing. #37 (PR #67) follows on top of this PR.🤖 Generated with Claude Code
https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG