Kotlin script execution with LLM-friendly diagnostics
LLM agents can act by writing code, and Kotlin scripts are a good medium for it: statically typed, concise, and able to call any JVM API. But the agent needs to understand what went wrong when its script fails, and the raw output of the Kotlin scripting host is not made for that.
This library executes Kotlin snippets in stateful sessions, like a Kotlin notebook,
so an agent can build on what it declared before,
and reports compilation and runtime errors in a form an LLM can act on:
each error points to the offending line of the original snippet,
with the problematic fragment marked with <error> tags.
Extracted from Golem XIV.
In build.gradle.kts add:
dependencies {
implementation("com.xemantic.kotlin:xemantic-kotlin-script:0.1.0")
}Then:
ScriptSession {
"greeting" provides "Hello"
}.use { session ->
session.execute("""
val names = mutableListOf("Alice")
fun greet() = "${'$'}greeting ${'$'}{names.joinToString()}"
""".trimIndent())
session.execute("names += \"Bob\"")
when (val result = session.execute("greet().uppercase()")) {
is ScriptResult.Value -> println(result.value) // HELLO ALICE, BOB
is ScriptResult.Error -> println(result.message)
}
}- A
ScriptSessionis stateful, like a Kotlin notebook: the declarations of an executed snippet, itsvals,vars, functions, classes, objects and imports, stay visible to the snippets executed after it, and can be redeclared by them. A redeclaration is seen by the snippets following it, while the functions of the earlier snippets keep calling the declarations they were compiled against, so after redeclaringgreet, ashoutcalling it has to be redeclared as well to call the new one. A snippet which fails to compile leaves no declarations behind, while a snippet throwing an exception keeps the ones it declared, also the properties it did not initialize before throwing, which holdnullthen, despite their type, or zero, orfalse, if it is a primitive one, likeInt, orBoolean. Redeclaring such a property is not enough, if a function, or a class, compiled against it reads it, like afun greet() = name.uppercase()declared together withval name: String = error("boom"), as it keeps reading the uninitialized one — it has to be redeclared as well, with the declarations calling it, while assigning avaris enough, as all of them read the same one. The error message of the failing snippet names the properties it leftnulldespite their type, the first fivevals and the first fivevars, counting the rest, with a hint to declare thevals again, together with the declarations reading them, and to assign thevars. ANullPointerExceptionof the JVM naming such a property later, like ofs.lengthafterval s: String = error("boom"), gets the hint as well, naming the declarations reading it, and the ones calling them, found in the stack trace, likegreet. Neither hint covers a property of a primitive type, which reads as zero, orfalse, without failing, nor a property of a nullable type, or of a platform type, like ofSystem.getProperty, which isnullby its type. The later hint also needs the JVM to name the property in the message of theNullPointerException. The JVM does not name it for a value copied from the property before the failing read, likeval t = sfollowed byt.length, nor for the property passed as an argument, rejected by the check of a non-null parameter (Parameter specified as non-null is null), nor when it runs without the helpfulNullPointerExceptionmessages (-XX:-ShowCodeDetailsInExceptionMessages). A delegated property is the exception, as it is found by its getter in the stack trace instead. - Each snippet is a regular Kotlin script, compiled by the K2 REPL compiler of the Kotlin scripting API, against the whole classpath of the host application, so any class available to the host can be imported by a snippet.
- The value of the last expression of a snippet becomes
ScriptResult.Value.value, orUnitif there is none. - All the snippets of a session are compiled and evaluated on its own thread, one after another,
so a snippet can use an API bound to the thread which created it.
Snippets executed concurrently wait for their turn.
The thread is given a 16 MB stack, larger than the default one,
so the compiler takes a deeply nested expression, of thousands of nesting levels,
and a snippet can recurse deeper.
The caller of
executeresumes where its dispatcher resumes it, like the main thread forDispatchers.Main.immediate, or fromDispatchers.Defaultif it would resume in place, on the thread of the session: if it has no dispatcher, likesuspend fun main, or one which does not dispatch, likeDispatchers.Unconfined, it runs onDispatchers.Defaultthen, and if it has a continuation interceptor which is not a dispatcher, the interceptor takes it over there, like one posting it to its own thread. Such a caller needs a free worker ofDispatchers.Defaultto resume, so a snippet blocking all of them, likerepeat(n) { launch { latch.await() } }, keeps it waiting, and it should not block its thread for long, while the next snippets do not wait for it, as the turn of the caller ends with its snippet, so the snippet freeing the workers can still run. Only a dispatcher running its tasks on the dispatching thread, likeExecutor(Runnable::run).asCoroutineDispatcher(), resumes its caller on the thread of the session. A dispatcher failing to tell whether it dispatches, or to dispatch, on the thread of the session, is retried fromDispatchers.Default, and one failing there as well, like one always throwing, or the one of an executor shut down, is bypassed: the caller resumes without it, onDispatchers.Default, failing with its exception, so it never stays suspended forever. A fatal error of the dispatcher, aVirtualMachineErrorother thanStackOverflowError, like anOutOfMemoryError, is not retried, but bypasses the dispatcher at once, failing the caller with it, like a fatal error of the snippet. A caller resumes only once, also when its dispatcher, orDispatchers.Default, takes the resumption and throws after it. Separate sessions run in parallel, each with its own state, and each holding its own compiler, which loads the whole classpath, so close the sessions no longer needed. - A caller waiting for the snippets of other callers has its snippet queued on the thread of the session at once,
so it needs no free worker of
Dispatchers.Defaultbefore its snippet, which might be the one freeing them, and it is never resumed by another caller, or on its thread, only after its own snippet, as above. If the session is closed meanwhile, its snippet is rejected on the thread of the session, with anIllegalStateException, which the caller resumes with as above, so a caller of its own dispatcher does not wait forDispatchers.Default, even when queued behind a caller without a dispatcher. A caller cancelled while it waits, or whose job completes then, resumes at once, with itsCancellationException, without executing its snippet, on the cancelling thread, if it would resume in place, unless that is the thread of the session, like a snippet cancelling it, which resumes it as above. So does a caller cancelled already when it callsexecute, sowithTimeoutbounds the wait, unless no other snippet is executed, or waiting, then: such a caller executes its snippet, cancelled. - The snippets are received by the
ScriptScope, aCoroutineScopeliving as long as the session, withkotlinx.coroutines.*imported by default:await { ... }runs suspending code, like the requests of a ktor client, blocking the snippet until it completes, on the thread of the session, e.g.val pages = await { urls.map { async { client.get(it).bodyAsText() } }.awaitAll() };launch { ... }starts a coroutine which keeps running after the snippet, onDispatchers.Default, or thedispatchergiven toScriptSession, until it completes, or the session is closed.
- Cancelling the caller of
executecancels the coroutines ofawait, and interrupts the thread of the session, so a blocking call likeThread.sleepfails, while a snippet which neither blocks interruptibly, nor awaits, runs to its end. A snippet being compiled when the caller is cancelled is compiled and evaluated nevertheless, as the compiler cannot be interrupted, and would evaluate it before the next snippet anyway. - Closing the session cancels the coroutines launched by its snippets,
and the snippet being evaluated, like cancelling its caller does, also a caller which cannot be cancelled, like
suspend fun main, so itsexecutereturns an evaluation error, unless the snippet runs to its end, as above, or catches theInterruptedException, or theCancellationExceptionofawait, and ends without blocking, or awaiting, again, as its evaluation stays cancelled. Then it releases its compiler, thread, and the values of its properties, once that snippet ends, while the snippets and coroutines still running keep reading the properties from the fields of their scripts, unaffected by the release. - Properties of the given name and type are provided to the snippets of the session, like
"greeting" provides "Hello". The type is inferred from the value, including its type arguments, or can be given explicitly, e.g.provide<String?>("name", null)orprovide<Clock>("clock", fixedClock), to make the property nullable, or to expose it only as its supertype. A mutable collection keeps its mutable type, so"names" provides mutableListOf("Alice")lets the snippets modify the list shared with the host, whileprovide<List<String>>("names", names)makes the snippets see it as read-only. A read-only collection of the Kotlin standard library, likeemptyList(), is rejected as a value of a mutable type, while a frozen collection implementing the mutable interface, like the result ofbuildList, or a Java collection, likeCollections.unmodifiableList, is not detected, and fails when mutated. Only the class of the value is checked against the type, as the type arguments are erased at runtime, so a nested collection of another mutability than declared, like anemptySet()value in aMutableMap<String, MutableSet<Int>>, is not detected either, and fails when mutated. OnlytypeOfand the types of the members keep the mutability, whileMutableList::class.createType(...)yields a read-onlyListtype, asMutableList::classis the same class asList::class. A property list built dynamically can be passed instead, asScriptSession(listOf(property("greeting", "Hello"))). Invalid properties are rejected withIllegalArgumentException: a name which is not a Kotlin identifier, a duplicate name, a value which is not of the type, a name of a member of theScriptScope, likeawaitorcoroutineContext, or a type which the snippets cannot refer to, like a private or a local class, or aKTypeimplemented by hand whose classifier is not a class, or whose type arguments do not match the type parameters of its class. A property which passes this validation might still fail to be declared, like one of a type the compiler rejects, e.g.Enum<String>created withcreateType, or one of a class loaded by another class loader than the one of the session, which its value cannot be cast to. Such a failure is reported by everyexecute, as aScriptResult.Errorstarting withThe properties cannot be declared:, in theExecutionPhasein which the declaration failed, without executing the snippet. - A snippet can add Maven dependencies to the session, like a
.main.ktsscript:The dependencies, with their transitive ones, are resolved from Maven Central, or the repositories declared with@file:DependsOn("org.jsoup:jsoup:1.16.1") import org.jsoup.Jsoup Jsoup.parse("<p>Hello</p>").text()
@file:Repository("https://...")instead, by this or any earlier snippet, into the local Maven repository,~/.m2/repositoryby default, and stay available to the snippets executed after it. A class available to the host takes precedence over the one of the same name in a dependency, so a dependency cannot replace a library of the host with another version of it. A dependency which cannot be resolved fails the snippet with anExecutionPhase.COMPILATIONerror pointing to its@file:DependsOnline. - A dependency can be declared without version, like
@file:DependsOn("org.jsoup:jsoup"), as an LLM might know only the versions released before its knowledge cutoff. It is resolved to its latest version, likepip installdoes: the highest stable one in the Maven ordering, among the versions listed by themaven-metadata.xmlof the repositories, neither aSNAPSHOT, nor a pre-release, like2.0-RC1or1.9.20-Beta, or the highest pre-release one, if there is no stable one. A date-style version, like the20040616ofcommons-collections:commons-collections, which Maven orders after any release, gives way to any other version of the same stability, like3.2.2, if there is one. A version with a qualifier, which Maven orders after its release, like0.8.0-0.6.x-compatafter0.8.0, gives way to its release, if there is one. The LLM should be told the resolved version, as it might not know its API: a successful snippet returns it asScriptResult.Value.resolvedVersions, likemapOf("org.jsoup:jsoup" to "1.24.1"), which, unlike the compiler warnings, is worth showing to the LLM always, while a failed snippet reports it in the error message, as a warning pointing to the@file:DependsOnline, likeResolved 'org.jsoup:jsoup' to its latest stable version '1.24.1'. The resolved version is kept for the session, so a later snippet declaring the dependency again does not load another version. The versions are looked up by Maven Resolver like the dependency is resolved, in the repositories declared with their options, like@file:Repository("https://...", options = ["username=...", "password=..."]), with the mirrors, proxies and server credentials of the Maven settings,~/.m2/settings.xmlby default, among themaven-metadata.xmlfiles cached in the local Maven repository, refreshed once a day, or installed into it. A repository blocked by a mirror of the settings, like themaven-default-http-blockerof Maven blocks the plainhttp:ones, is reported as blocked, naming the mirror, when no other repository lists a version. The dependencies themselves, also in the versions looked up, are resolved by Kotlin's own resolver, which applies the settings with two limitations: it resolves a mirrored repository from the mirror directly, bypassing the proxy, and it ignores the block of a mirror, so a dependency held only by the blocked repositories fails to connect to the URL of the blocking mirror, likehttp://0.0.0.0/, instead of being reported as blocked. In the offline mode of the Maven settings, only thefile:repositories are read, along with themaven-metadata.xmlfiles already cached from the other repositories, and a dependency without anymaven-metadata.xmlthere is resolved to the latest of its versions present in the local Maven repository. A version range, like[1.0,), and theRELEASEandLATESTversions of Maven are resolved by Maven itself, which might select a pre-release version. - A repository can be declared again with other options, like corrected credentials,
and is then read with each of its declarations.
Once the lookup of a version is refused (
401) with the credentials of one declaration, while another declaration of the repository is served the metadata of the dependency, the refused declaration is superseded: its credentials are no longer sent, neither to look up a version, nor to resolve a dependency. The supersession is reported once, as a warning pointing to the@file:DependsOnline whose lookup superseded it, naming the line of the superseded@file:Repository, possibly of an earlier snippet, likeSuperseded the declaration of the repository 'https://...' at snippet1.kts:1, with the credentials refused (401), by its other declaration, ..., as a later dependency served only to these credentials would fail without them. Declaring the repository again with them restores the declaration. The warning is part ofScriptResult.Value.warningsof a successful snippet, and of the message of a failed one. It is also logged, at theINFOlevel, as the caller gets no result of a snippet throwing a fatal error, like anOutOfMemoryError, or of a cancelledexecute, while the supersession is not reported again — on every path, also when the result reports it. The user info of the repository URL, likealice:tokenofhttps://alice:token@host/, is masked in the warning and in the log.
The K2 REPL compiler is still experimental, and these of its bugs are reported to the LLM with a hint how to avoid them:
- A class declared in an earlier snippet is unresolved within a declaration of an inferred type calling it in the same snippet,
like
fun isPlain(x: Any) = x is Plainfollowed byisPlain(1), while it resolves with the type declared,fun isPlain(x: Any): Boolean = x is Plain. - A property, or a function, of a value class type, like
DurationorUInt, declared in an earlier snippet, fails withNoSuchMethodErrorwhen called from another snippet, as the compiler calls it by its mangled name, likegetTimeout-UwyO8pc, while it declares it without mangling. The provided properties of value class types are not affected, as their getters are declared with the mangled names. - An
internalproperty, or function, or aninternal setof a property, declared in an earlier snippet, also a member of a class declared there, fails withNoSuchMethodErrorwhen called from another snippet, as the compiler declares it under a name mangled with the module of its snippet, likegetSecret$snippet1_kts, while it calls it by another name, likegetSecret$main, orsecretof a function. It works once declaredpublic, together with the code calling it, directly or not, declared again, as it keeps calling theinternalone, also the subclasses of an open class declaring it, or calling it, as they keep extending the old class, and with the values depending on it created again: the ones of a class declaring it, or calling it, or of its subclasses, as they keep being of the old class, and the ones holding a lambda calling it, likeval f = { secret() }. The hint names the declarations and the classes on the stack trace as examples only, as it cannot tell all of them. The code referring to anobjectdeclaring it, or calling it, and the values holding theobject, keep the old one, so they are declared, and created, again as well. - A
const valdeclared in an earlier snippet cannot be read from another snippet: it is unresolved within a declaration of an inferred type, and crashes the compiler wherever else it is read, also in a top-level expression, or in a property initializer. It works once declared again public, withoutconst, as aninternalone fails like anyinternalproperty. Aconst valof anobjectdeclared in an earlier snippet crashes the compiler as well, without a hint. - A
@JvmFieldproperty declared in an earlier snippet cannot be accessed from another snippet: it is unresolved within a declaration of an inferred type, and fails withIncompatibleClassChangeErrorwherever else it is accessed, as the compiler accesses it as an instance field, while it is static. It works once declared again public, without@JvmField, together with the code accessing it, directly or not, declared again, and with the values depending on it created again, like for aninternalmember. - An
inlinefunction, or property, declared in an earlier snippet cannot be called from another snippet, and the one referring to theScriptScope, likecoroutineContext, cannot be compiled at all, reported as an internal failure of the compiler.
A snippet like:
asdf
println(foo)executed as the third snippet of the session,
produces a ScriptResult.Error with the ExecutionPhase.COMPILATION phase, and the message:
snippet3.kts:1:1: error: Unresolved reference 'asdf'.
| <error>asdf</error>
snippet3.kts:2:9: error: Unresolved reference 'foo'.
| println(<error>foo</error>)
Each diagnostic follows the file:line:column: severity: message format of kotlinc and other compilers,
which LLMs are familiar with,
where the snippets are named after their order in the session, snippet1.kts, snippet2.kts, and so on.
The offending fragment of the source line is marked with <error> tags instead of a caret line below,
so that the LLM does not have to count columns.
Runtime exceptions are reported in the ExecutionPhase.EVALUATION phase,
with the stack frames of the snippets only, also of the earlier ones declaring the functions called,
together with the source line of each frame,
followed by compiler warnings, if any, as they might explain the failure:
java.lang.IllegalArgumentException: value must be positive
at check(snippet1.kts:2)
| require(value > 0) { "value must be positive" }
at <snippet body>(snippet2.kts:2)
| values.forEach { check(it) }
The frames are named after the functions of the snippets, e.g. at Checker.check(snippet1.kts:5),
and repeated frames of a recursion are collapsed.
The later lines of a multi-line exception message are indented like the frames,
also within a Caused by: or a Suppressed: exception,
so that they do not break the structure of the report.
The classes declared by the snippets are named as declared, like Color.BLUE,
not qualified by the generated snippet class, like Snippet1.Color.BLUE.
The warnings of a successful snippet are available as ScriptResult.Value.warnings:
the warnings of its dependencies other than the resolved versions,
like a superseded declaration of a repository, in the order of their lines,
followed by the compiler warnings.
They might reveal a bug in a snippet which nevertheless succeeded,
so it is up to the caller whether to show them to the LLM.
A failed snippet reports these warnings in the error message instead,
also the resolved versions of its dependencies,
except for a snippet with a dependency failing to resolve,
which reports only the warnings and the errors of its dependencies,
as the diagnostics of the compiler follow from the missing dependency then.
A snippet throwing a fatal error, like an OutOfMemoryError, or executed by a cancelled caller, returns no result,
so its warnings are lost then, except for the logged supersession of a repository declaration.
It is up to the caller how to present the error to the LLM, e.g.:
"<impediment phase=\"${result.phase}\">\n${result.message}\n</impediment>"The snippets run with the full privileges of the host application, there is no sandbox.
It is compiled against the whole classpath, and executed in the JVM of the host,
so it can do whatever the host can:
read and write files, open network connections, read the environment variables and system properties,
access the classes and singletons of the host through reflection,
start processes, or terminate the JVM with System.exit.
The provided properties limit what the script is given, not what it can reach.
A script written by an LLM should be treated as untrusted code, as it might be the result of a prompt injection, or just a hallucination. If the LLM can be influenced by content which is not trusted, like web pages, documents, emails or tool results, execute the scripts in an isolated environment, e.g. a separate JVM process running in a container or a virtual machine, with no secrets, limited file system and network access, and bounded CPU, memory and execution time.
The library logs through kotlin-logging over SLF4J.
Add an SLF4J provider, e.g. logback, to see the DEBUG logs of the compilation and evaluation phases.
GNU Affero General Public License v3.0
See DEVELOPMENT.md for the documentation conventions, and for maintenance notes on updating the gradle wrapper and the project dependencies.