Stateful Python, embedded with Jep, in Kotlin scripts
xemantic-kotlin-script runs Kotlin snippets in stateful sessions, with diagnostics an LLM can act on. Some tasks are better served by the Python ecosystem (numpy, pandas, scikit-learn, …), so this project embeds a CPython interpreter into a Kotlin script session, keeping the Python state between the snippets as well.
CPython 3.10 or newer, the oldest version supported by the jep package.
Before Python 3.11, which brings tomllib,
reading the inline script metadata of the code requires either the tomli or the pip package to be importable.
An environment without pip, like the one of uv venv, can get tomli with pip install tomli or uv pip install tomli.
Jep is a native CPython extension:
the Java library comes from Maven Central,
but its native part comes with the jep Python package, which has to be installed in the Python environment in use.
The version of the jep package must match the version of the org.ninia:jep Java library, 4.3.2 in these examples:
python -m pip install jep==4.3.2A virtual environment without pip, like the one of uv venv, can get it with uv pip install --python <python> jep==4.3.2.
An externally managed Python (PEP 668),
like the one of Homebrew or of most Linux distributions,
refuses pip install, so install it into a virtual environment instead:
python3 -m venv ~/.venvs/jep
~/.venvs/jep/bin/python -m pip install jep==4.3.2and point to it with an absolute path, as ~ is expanded only by a shell, not by an IDE:
either with the VIRTUAL_ENV environment variable set to the virtual environment, e.g. /home/me/.venvs/jep,
or with PythonEnvironment.pythonExecutable set to its Python, e.g. /home/me/.venvs/jep/bin/python,
which is Scripts\python.exe of the virtual environment on Windows.
PyPI has no wheels of jep, so pip always builds it from its source,
which requires a C compiler, the Python headers,
and a JDK, found through JAVA_HOME, which is optional on Linux and macOS, where javac on the PATH is found as well.
On Debian/Ubuntu, the Python headers, and the venv module, come with apt install python3-dev python3-venv.
The Python executable is, in this order:
PythonEnvironment.pythonExecutable, if set before the first use ofPython,- the one of the virtual environment of the
VIRTUAL_ENVenvironment variable, python3on thePATH.
The Python code imports the modules from the same paths as this executable,
e.g. the site-packages of a virtual environment, but not the ones of its base Python.
The embedded Python, like every Python process started by this library, e.g. pip,
runs in the UTF-8 mode,
whatever the locale, e.g. cp1252 on Windows, or C on Linux:
the standard streams, and open without an encoding, use UTF-8.
Only a PYTHONIOENCODING environment variable of the JVM, if set, still takes precedence over it,
for the standard streams of the embedded Python.
On Linux, or another POSIX system, with a locale whose encoding is not UTF-8, e.g. ISO-8859-1,
the path of the Python executable, e.g. of the virtual environment, must be ASCII:
Jep passes it to the embedded Python as JNI modified UTF-8, which is then another path,
so that the embedded Python would not find the virtual environment,
and fail with ModuleNotFoundError: No module named 'jep'.
It does not apply to macOS and Windows, where Python decodes a path as UTF-8, whatever the locale.
Other paths, like the one of the dependency directory in a home directory like /home/josé, can be non-ASCII.
In build.gradle.kts add:
dependencies {
implementation("com.xemantic.kotlin:xemantic-kotlin-script-python:0.1.0")
// only for running Python in a ScriptSession, like below, as Python does not depend on it
implementation("com.xemantic.kotlin:xemantic-kotlin-script:0.2.1")
}Then:
import com.xemantic.kotlin.script.ScriptResult
import com.xemantic.kotlin.script.ScriptSession
import com.xemantic.kotlin.script.python.Python
suspend fun main() {
Python().use { python ->
ScriptSession {
"python" provides python
}.use { session ->
session.execute("""python.exec("import numpy as np; a = np.arange(10)")""")
session.execute("""python["factor"] = 3""")
val result = session.execute("""python.eval<List<Long>>("(a * factor).tolist()")""")
println((result as ScriptResult.Value).value) // [0, 3, 6, 9, 12, 15, 18, 21, 24, 27]
}
}
}ScriptSession.execute is a suspending function,
called here from a suspend fun main(),
and otherwise from any coroutine, e.g. within runBlocking.
The examples assume numpy installed in the Python environment,
otherwise the code can declare it as a dependency, see below.
- A
Pythonis stateful: the global variables, functions, classes and imports of the executed code stay visible to the code executed after it, also across the snippets of aScriptSession. exec(code)executes Python code,eval<T>(expression)evaluates an expression converted toT, andpython["name"]reads or writes a global variable. The values are converted by Jep, e.g. a Pythonintbecomes aLong, astraString, and alistaList.python(code)executes Python code like a cell of a Jupyter notebook, returning the value of its last statement, if it is an expression, ornullotherwise, so that the code written by an LLM can either provide a result, or be executed only for its effects:The value is converted by Jep to its default Java type.python("import numpy as np; a = np.arange(10)") // null python("a.sum()") // 45L
- A value returned by
python(code),evalorpython["name"], which Jep cannot convert, like a module, a function or an instance of a Python class, is returned as its Pythonrepr, also when nested in aList, aSet, aMapor an array, as a Python object can be used only on the thread of itsPython:If different keys of apython("[len, 1]") // listOf("<built-in function len>", 1L)
dict, or elements of aSet, have the samerepr, anIllegalStateExceptionis thrown, not to drop an entry silently, andeval<PyObject>(or an array of them) fails with anIllegalArgumentException. - For an AI agent, the
Pythoncan be started lazily, only when a script uses it, as the first one initializes the Python environment:Withclass Agency : AutoCloseable { private val lazyPython = lazy { Python() } val python: Python by lazyPython override fun close() { if (lazyPython.isInitialized()) python.close() } }
"agency" provides agency, a script can then callagency.python("..."), or setagency.python["name"] = value. - Each
Pythonhas its own global variables, while the imported modules are shared by all of them, so that native extension modules, likenumpy, can be used. - A Jep interpreter can be used only by the thread which created it,
so each
Pythonruns its code on a thread of its own, and can be called from any thread. - A
ScriptSessiondoes not close its dependencies, close thePythonwhen no longer needed. The firstclose()waits until the interpreter is closed, unless a call is running: then it returns at once, and the interpreter is closed after the call returns, as the call might itself wait for thisclose(), e.g. one made by a Kotlin function called from Python. Callingclose()again returns at once, also while the interpreter is still being closed. An interruptedclose()throws anInterruptedExceptionat once, while the interpreter is still closed later, in the background. - The code passed to
exec, orpython(code), can declare its dependencies, like a@file:DependsOnof a Kotlin snippet, with inline script metadata, as withuv run:The missing dependencies are installed before executing the code, with uv if it is on the# /// script # dependencies = ["rdkit"] # /// from rdkit import Chem
PATH, or withpipof the Python executable otherwise, intoPythonEnvironment.dependencyDirectory, by default the directory of thexemantic.kotlin.script.python.dependencyDirectorysystem property, if set, or otherwise~/.cache/xemantic-kotlin-script-python/<python>-<hash of the environment>, which becomes importable once any code declares its dependencies. The installed dependencies are shared by all thePythoninterpreters of the JVM, and they cannot change the versions of the distributions of the Python environment itself, nor the ones already imported from the dependency directory, as a module cannot be replaced once imported. The installations into the dependency directory are serialized, also across processes, e.g. JVMs sharing the default directory in the user cache, with a lock of its.xemantic-kotlin-script-python.lockfile, so a call with missing dependencies can wait for an installation of another process, and the directory must be writable when something has to be installed. A dependency which cannot be installed fails the call withPythonDependencyException, with the output of the installer. - A Python exception fails the call with
jep.JepException, reported by theScriptSessionas anExecutionPhase.EVALUATIONerror, like:jep.JepException: <class 'ValueError'>: boom at <string>.<module>(<string>:1)
The tests run in the .venv Python virtual environment of the project,
created by Gradle with python3, or another base Python set with the basePython Gradle property
(see DEVELOPMENT.md),
with only the jep package of the version of the library installed,
like a clean environment of an agent.
The tests declare the Python packages they use, like numpy, as dependencies,
installed from PyPI into build/python-dependencies, which Gradle cleans before every test run,
so the tests need network access.
See DEVELOPMENT.md for the documentation conventions, and for maintenance notes on updating the gradle wrapper and the project dependencies.