Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,8 @@ otherwise the code can declare it as a dependency, see below.
Calling `close()` again returns at once, also while the interpreter is still being closed.
An interrupted `close()` throws an `InterruptedException` at once,
while the interpreter is still closed later, in the background.
A failure of closing the interpreter is thrown by the `close()` waiting for it,
or, when none waits, logged as a `WARNING` of the `System.Logger` named `com.xemantic.kotlin.script.python.Python`.
* The code passed to `exec`, or `python(code)`, can declare its dependencies,
like a `@file:DependsOn` of a Kotlin snippet,
with [inline script metadata](https://packaging.python.org/en/latest/specifications/inline-script-metadata/),
Expand Down
64 changes: 59 additions & 5 deletions src/main/kotlin/Python.kt
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import java.util.concurrent.Executors
import java.util.concurrent.RejectedExecutionException
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicInteger
import java.util.concurrent.atomic.AtomicReference

/**
* A stateful CPython interpreter, embedded with [Jep](https://github.com/ninia/jep).
Expand Down Expand Up @@ -112,6 +113,12 @@ public class Python private constructor(

private val closed = AtomicBoolean(false)

/**
* The failure of closing the interpreter, or [ClosingState.Abandoned] once [close] no longer waits for it,
* whichever is set first, see [abandonClosing].
*/
private val closingState = AtomicReference<ClosingState?>(null)

/** Whether a call is running on the thread of this `Python`, written only by this thread, see [close]. */
@Volatile
private var running = false
Expand Down Expand Up @@ -266,9 +273,9 @@ public class Python private constructor(
* Each call which is not running yet, or which is made afterward, fails with an [IllegalStateException].
* If closing the interpreter fails, its exception is thrown by this `close()`, which waits for it.
* If the waiting is interrupted, it throws an [InterruptedException] at once, clearing the interrupt flag of the thread,
* and the interpreter is still closed afterward, in the background, but a failure of closing it is not reported.
*
* @throws InterruptedException if the waiting is interrupted.
* and the interpreter is still closed afterward, in the background.
* Whenever no `close()` waits for the closing of the interpreter, an exception thrown by it is logged instead,
* as a `WARNING` of the [System.Logger] named after this class, so that it is never lost.
*
* Only the first `close()` can wait.
* Calling it again has no effect, and returns immediately,
Expand All @@ -283,12 +290,14 @@ public class Python private constructor(
* With a call running, or starting or ending, it does not wait for it:
* it returns immediately, the running call continues and completes normally,
* still able to call back this `Python`, and the interpreter is closed after it returns.
* Then nobody waits for the closing, so an exception thrown by closing the interpreter is not reported,
* Then nobody waits for the closing, so an exception thrown by closing the interpreter is only logged,
* and if the JVM exits before the running call returns, e.g. at the end of `main`,
* the interpreter is never closed, as its thread is a daemon one.
* Waiting would deadlock, when the running call waits for `close()`:
* when it is called by a Kotlin function called from Python, or by another thread, which this function,
* or the Python code, waits for, and it cannot be told apart from a `close()` which the running call does not wait for.
*
* @throws InterruptedException if the waiting is interrupted.
*/
@Throws(InterruptedException::class)
override fun close() {
Expand All @@ -300,6 +309,10 @@ public class Python private constructor(
executor.submit(Runnable {
try {
interpreter?.close()
} catch (e: Throwable) {
// logged here, if close() no longer waits for it, see abandonClosing
if (!closingState.compareAndSet(null, ClosingState.Failed(e))) logClosingFailure(e)
throw e
} finally {
// defensive only: nothing can use the interpreter afterward, as the executor is shut down
interpreter = null
Expand All @@ -312,17 +325,41 @@ public class Python private constructor(
// called by a Kotlin function, called from Python, or by a thread it waits for, so waiting here could deadlock;
// read after setting closed, so that a call, which is not running yet, sees it closed before running any code,
// and then the task of close() runs right after it
if (running) return
if (running) {
abandonClosing()
return
}
try {
closing.get()
} catch (e: ExecutionException) {
// thrown here, so not to be kept
closingState.set(ClosingState.Abandoned)
throw e.cause ?: e
} catch (e: InterruptedException) {
// the interpreter is still closed by its task, which is not cancelled
abandonClosing()
throw e
}
}

/**
* Tells the task of [close] that nobody waits for it, so that a failure of closing the interpreter is logged,
* here, if the task has failed already, or by the task otherwise, exactly once.
*/
private fun abandonClosing() {
val state = closingState.getAndSet(ClosingState.Abandoned)
if (state is ClosingState.Failed) logClosingFailure(state.failure)
}

private sealed interface ClosingState {

/** [close] no longer waits for closing the interpreter, or has thrown its failure already. */
object Abandoned : ClosingState

class Failed(val failure: Throwable) : ClosingState

}

private fun <T> call(block: SharedInterpreter.() -> T): T {
// a Kotlin function called from Python, calling back this Python, runs already on its thread,
// where waiting for the executor would deadlock; it belongs to the running call, which completes
Expand Down Expand Up @@ -575,6 +612,23 @@ public class Python private constructor(

private val threadCounter = AtomicInteger()

// only needed on the rare failure path
private val logger by lazy { System.getLogger(Python::class.java.name) }

/**
* Logs the [failure], never throwing, so that a failing logging backend
* neither replaces the exception thrown afterward, nor stops it from being thrown.
*/
private fun logClosingFailure(failure: Throwable) {
runCatching {
logger.log(
System.Logger.Level.WARNING,
"Closing the interpreter failed, with no close() waiting to throw it",
failure
)
}
}

/**
* Returns this value as a [PyObject], also when it is a proxy of a Java interface,
* which Jep creates for a Python object converted to an interface,
Expand Down
139 changes: 134 additions & 5 deletions src/test/kotlin/PythonCloseTest.kt
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,13 @@ import java.util.concurrent.CopyOnWriteArrayList
import java.util.concurrent.CountDownLatch
import java.util.concurrent.ExecutorService
import java.util.concurrent.Future
import java.util.concurrent.LinkedBlockingQueue
import java.util.concurrent.TimeUnit.MILLISECONDS
import java.util.concurrent.TimeUnit.SECONDS
import java.util.logging.Handler
import java.util.logging.Level
import java.util.logging.LogRecord
import java.util.logging.Logger
import kotlin.concurrent.thread
import kotlin.test.Test
import kotlinx.coroutines.Dispatchers
Expand Down Expand Up @@ -578,24 +583,115 @@ class PythonCloseTest {
@Test
fun `should throw failure of closing interpreter only from close waiting for it`() {
// given
val closeFailure = JepException("close failed")
val python = Python.withInterpreterFactory {
object : SharedInterpreter() {
override fun close() {
super.close()
throw JepException("close failed")
throw closeFailure
}
}
}
python.exec("x = 1")
val log = LogCapture(closeFailure)

// when
val firstFailure = runCatching { python.close() }.exceptionOrNull()
val secondFailure = runCatching { python.close() }.exceptionOrNull()
val (firstFailure, secondFailure) = log.use {
runCatching { python.close() }.exceptionOrNull() to runCatching { python.close() }.exceptionOrNull()
}

// then
assert(firstFailure is JepException)
assert(firstFailure.message == "close failed")
assert(firstFailure === closeFailure)
assert(secondFailure == null)
// reported by close(), so not logged as well
assert(log.records.isEmpty())
}

@Test
fun `should log failure of closing interpreter after close interrupted while waiting for it`() {
// given
val closingEntered = CountDownLatch(1)
val releaseClosing = CountDownLatch(1)
val closeFailure = JepException("close failed")
val python = Python.withInterpreterFactory {
object : SharedInterpreter() {
override fun close() {
closingEntered.countDown()
assert(releaseClosing.await(DEADLOCK_TIMEOUT, MILLISECONDS))
super.close()
throw closeFailure
}
}
}
python.exec("x = 1")
val log = LogCapture(closeFailure)
log.use {
var closerFailure: Throwable? = null
val closer = thread(isDaemon = true, name = "closer") {
closerFailure = runCatching { python.close() }.exceptionOrNull()
}
assert(closingEntered.await(DEADLOCK_TIMEOUT, MILLISECONDS))

// when
closer.interrupt()
closer.join(DEADLOCK_TIMEOUT)
releaseClosing.countDown()
val record = log.records.poll(DEADLOCK_TIMEOUT, MILLISECONDS)
val laterFailure = runCatching { python.close() }.exceptionOrNull()

// then
assert(!closer.isAlive)
assert(closerFailure is InterruptedException)
assert(record?.level == Level.WARNING)
assert(record?.thrown === closeFailure)
// already reported by the log, see close()
assert(laterFailure == null)
assert(log.records.isEmpty())
}
}

@Test
fun `should log failure of closing interpreter after close returned while call is running`() {
// given
val closeFailure = JepException("close failed")
val python = Python.withInterpreterFactory {
object : SharedInterpreter() {
override fun close() {
super.close()
throw closeFailure
}
}
}
val callbackEntered = CountDownLatch(1)
val releaseCallback = CountDownLatch(1)
python["block"] = { n: Long ->
callbackEntered.countDown()
assert(releaseCallback.await(DEADLOCK_TIMEOUT, MILLISECONDS))
n
}
val log = LogCapture(closeFailure)
log.use {
var runningResult: Long? = null
val running = thread(isDaemon = true, name = "running") {
runningResult = python.eval<Long>("block(42)")
}
assert(callbackEntered.await(DEADLOCK_TIMEOUT, MILLISECONDS))

// when
// returns at once, as the call is running
val closerFailure = runCatching { python.close() }.exceptionOrNull()
releaseCallback.countDown()
running.join(DEADLOCK_TIMEOUT)
val record = log.records.poll(DEADLOCK_TIMEOUT, MILLISECONDS)

// then
assert(closerFailure == null)
assert(!running.isAlive)
assert(runningResult == 42L)
assert(record?.level == Level.WARNING)
assert(record?.thrown === closeFailure)
assert(log.records.isEmpty())
}
}

@Test
Expand Down Expand Up @@ -673,6 +769,39 @@ class PythonCloseTest {
assert(stopCalled)
}

/**
* Captures the records logged by [Python] with the [failure], through `java.util.logging`,
* the default backend of `System.Logger`, while it is open,
* ignoring the ones of other `Python` instances, and not passing them on to the console.
*/
private class LogCapture(private val failure: Throwable) : AutoCloseable {

val records = LinkedBlockingQueue<LogRecord>()

private val logger = Logger.getLogger(Python::class.java.name)

private val useParentHandlers = logger.useParentHandlers

private val handler = object : Handler() {
override fun publish(record: LogRecord) {
if (record.thrown === failure) records.add(record)
}
override fun flush() {}
override fun close() {}
}

init {
logger.addHandler(handler)
logger.useParentHandlers = false
}

override fun close() {
logger.useParentHandlers = useParentHandlers
logger.removeHandler(handler)
}

}

/**
* Decorates the executor of a [Python], to call on the calling thread:
* [beforeSubmit] before, and [afterSubmit] after submitting each task, telling if it was accepted,
Expand Down
Loading