Repository navigation
Convert every PyObject returned by Python to its repr, also nested - #49
Merged
Merged
Conversation
get, eval and invoke now apply one conversion on the Python thread, replacing each PyObject, also nested in a List, a Map or an array, with its repr, and closing it, so that no PyObject escapes the thread of the interpreter. eval with a PyObject type is rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failure (#2) The conversion of returned values is done by PyObjectConverter: - dict keys are converted without touching a closed PyObject, - different keys of the same repr fail with IllegalStateException, instead of dropping an entry silently, - a self-referencing collection set from Java is returned as it is, - when repr fails, the PyObjects not converted yet are closed too, - repr is looked up once per conversion, - a collection without any PyObject is returned as it is, in one pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Java collection returned by Jep can be the same instance which Kotlin passed to Python, still holding the PyObjects Python put there, so a read must not close them. PyObjectConverter now: - returns a value without any PyObject as it is, after one pass, - otherwise copies each reachable List, Map and array, cycles included, never changing or returning the original, - caches the repr of each PyObject by identity, - closes only a top-level PyObject, created by Jep for this value, leaving the nested ones to the MemoryManager of Jep. The converter is skipped for a value which is not a PyObject or a collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of eval (#2) A Set or another Collection than a List, e.g. a Kotlin set filled by Python, is copied as well, a Set to a LinkedHashSet, rejecting elements of the same repr like the keys of a dict. eval rejects an array of PyObjects as the type, and only the containers are tracked while looking for a PyObject, not each element of a large list. The README describes the conversion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…first (#2) A copied array keeps its component type, so that eval can cast it to the requested type, like Array<Array<Any>> or List[], widened to Object only where a repr cannot be held, like in a PyObject[]. eval checks the PyObject type in the call, so that a closed Python fails with "Python is closed" first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…on types (#2) An array typed with a subtype of PyObject, like PyCallable[], is copied to Object[], as a repr cannot be stored in it. eval of a collection type other than a List, a Set, a Map or an array, like ArrayDeque, holding Python objects, fails with an IllegalArgumentException explaining the copy, instead of a ClassCastException. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
This was referenced Nov 9, 2026
* Convert Jep proxies of Python objects to their repr, like PyObjects (#16) Jep converts a Python object to a java.lang.reflect.Proxy of a Java interface, e.g. for eval("lambda: 1", Runnable::class.java), eval("len", AutoCloseable::class.java), the elements of eval("[len]", Comparable[]::class.java), or a Python function passed to a Kotlin method taking a Runnable. The proxy's jep.python.InvocationHandler holds a PyObject bound to the Python thread, so calling the proxy - even toString() or equals() - from the caller's thread throws "Invalid thread access". A proxy is now treated as the PyObject behind it: it becomes its repr, also nested in collections, maps and arrays, and a top-level one is closed. eval to an interface which the repr does not implement, like Runnable, fails with an IllegalArgumentException; interfaces implemented by a converted value (List, Map, CharSequence, Comparable) keep working. A wrapper dispatching each call of the proxy to the Python thread was rejected: it would outlive close(), and need the same re-entrance and deadlock handling as the calls of Python, for little use from a script. The class of a copied array is now decided by the copies of its elements, not by its static component type: Comparable[] can hold a repr and is kept, Runnable[] (or PyObject[]) cannot and is widened to Object[], instead of failing with an ArrayStoreException, while List[] of copied lists is still kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Widen arrays of PyObject types by each dimension again (#16) Deciding the class of a copied array only from the copies of its elements regressed #2: the copy of PyObject[][] became Object[] instead of Object[][], as an inner Object[] does not fit a PyObject[] slot, so eval(..., Object[][]) failed, and an empty PyObject[] kept its class. The declared type is widened first, by each dimension, as on #2, so that no array of a PyObject type is left, even an empty one. Only then the elements are checked, for a proxy whose repr does not fit, like in Runnable[], and such an array is widened to an Object array of the same dimensions. An Object[], the usual array from Jep, skips that check. The KDoc of eval now states that a Python object requested as an interface which String implements, like Comparable, becomes its repr. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Do not close the PyObject of a proxy held by Python (#16) A Python variable can hold a Java proxy created with jep.jproxy, which Jep returns as the same instance, not unwrapped. Closing the PyObject of its handler, as if Jep had created it for the returned value, broke the variable for Python: x.run() then failed with "PyObject has been closed". Only a top-level PyObject is closed now; the PyObject of a proxy, also a top-level one created by Jep for eval, is left to Jep to release once it is garbage collected. Also pin that arrays of a collection type copied to another one, like TreeSet[] or ArrayDeque[], are widened to Object[], and that Comparable[] keeps its class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Decide the copy class of arrays in a cycle independently of elements (#16) An array reached again while deciding the class of its copy was assumed to keep its class, which the arrays deciding in the meantime relied on. If it was widened afterwards, e.g. Cloneable[] holding a jproxy and an array holding it back, copying stored the widened copy into an array which could not hold it, failing with an ArrayStoreException. Such an array now gets a class not depending on its elements: its own, if it can hold a copy of any element, like Object[] or Serializable[], or Object[] otherwise. Also document that eval rejects a proxy also when called back from Python, and that the holder of proxies returned as it is remains #38. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…r it (#52) * Return from Python.close() without waiting while a call runs (#32) close() from another thread waited for the running call to complete, so a Kotlin callback waiting for a thread which calls close() deadlocked: `thread { python.close() }.join()`, `runBlocking(Dispatchers.IO) { python.close() }`, or Python code joining a `threading.Thread` whose target calls close(). #3 avoided this only for close() called on the Python thread itself. Whether the running call waits for the thread calling close() cannot be known, and only the running call is in the way: a call queued before close() sees it closed before running any code. Restricting the deferral to "a Kotlin callback is running" would not help either — in the threading.Thread case the Python thread is in Python code (join), not in a callback, and detecting callbacks would need wrapping every Kotlin value set into Python. So close() now returns without waiting whenever a call is running, generalizing #3: the interpreter is closed on the Python thread right after the running call. Without a running call, close() still waits until the interpreter is closed, so `use { }` from the owning thread keeps releasing it deterministically. The `running` flag is set by the task before it checks `closed`, and read by close() after setting it, so a call which close() does not see running can only fail as closed, and close() never waits behind user code. The same deadlock for other calls (eval, exec, get, set) made from a thread the running call waits for is not fixed, only documented in the KDoc of Python: such a call cannot run before the running call, and cannot be told apart from a legitimate concurrent call which simply waits for its turn, so failing it fast would break those. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG * Pin close() not waiting for a running call, and correct its KDoc (#32) The renamed test released the blocked callback before joining the closer, so it passed whether close() waited or not; a new test asserts that close() from an unrelated thread returns while the call is still blocked in a callback, and that the interpreter is closed only after that call ends. The callback waits longer than the closer's bound, so the old behavior fails it on the closer not returning, instead of both timeouts expiring together. The KDoc of close() now says that it also skips waiting for a call which is just starting, and about to fail as closed, or just ending, as the running flag must be set before the call checks if closed; and that a deferred close is never reported, nor done at all if the JVM exits first, as the thread of Python is a daemon. The class KDoc said a Kotlin function always runs on the thread of Python, which is wrong for one called from a threading.Thread, where a direct call back waits for its turn like any other thread's call, and deadlocks when joined. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2
Fix
get,evalandinvoke(python(code)) returnedPyObjects bound to the interpreter's thread — top-level (python["math"],eval<Any>("len")) and nested in collections (python("[Point()]")) —which failed with
Invalid thread accesson the caller's thread and leaked their Python reference.Every returned value now goes through one conversion on the Python thread (
PyObjectConverterinPython.kt):PyObjectbecomes itsrepr, at the top level or nested at any depth in aList,Set(or any otherCollection),Map(keys too) or array;PyObjectis returned as is; otherwise collections are copied — never mutated, so a Kotlin collection still held by Python keeps working — preservingSetsemantics, self-references, and the component type of arrays (widened toObject[]only for arrays typed withPyObjector a subtype);dictkeys orSetelements with the samereprfail with anIllegalStateExceptioninstead of silently losing an entry;evalto aPyObjecttype, or an array of them, fails withIllegalArgumentException;evalto a concrete collection type (e.g.ArrayDeque) holding Python objects fails with anIllegalArgumentExceptionexplaining the copy;PyObjectis closed; nested ones are left to Jep's GC-based release (documented trade-off, see Nested PyObjects of returned values are released only by the JVM garbage collector #39).KDoc and README describe the conversion. The public API is unchanged (
checkKotlinAbipasses).Review rounds
Setand other non-Listcollections were not converted. Fixed in 02c3194 together with: the array guard ofeval, a full identity-set walk of every large result (now only containers are tracked), the stale README.PyObjectwere copied asObject[], soevaltoArray<Array<Any>>/List[]failed withClassCastException. Fixed in ecc0e7b, also the closed check now precedes theeval<PyObject>rejection.PyObjectsubtype, a clear exception for concrete collection types.The rebase onto
main(close races #1, close from a callback #3) needed one conflict resolution inPython.kt; main's lifecycle logic is untouched.Follow-ups filed
PyObjectarguments of Kotlin callbacks escape inside arbitrary returned Java objects (Pair,Optional) — pre-existing.PyObjects of returned values are released only by the JVM GC.evalwith a primitive class likeInt::class.javafails with Jep'sTypeError— pre-existing.eval<Runnable>) remain Python callables converted to Java interface proxies escape the Python thread #16.🤖 Generated with Claude Code
https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG