Repository navigation
Cross-language extension SDK #949
Replies: 22 comments
Checkpoint 2. First pieceThe first piece is the host API. The Rust CLI calls it in the first slice. The TypeScript and Python host ports are part of the same design and are built later. Today the CLI has two handshake paths. flowchart TB
subgraph CLI["crates/morphir (today)"]
E["extensions.rs<br/>compile, generate, discovery"]
P["commands/extension/probe.rs<br/>install probe"]
end
E -->|"initialize<br/>peer: morphir-cli<br/>versions: MEP_VERSION"| Guest1["guest process"]
P -->|"describe, then session fallback<br/>peer: host_peer()<br/>versions: SUPPORTED_MEP_VERSIONS"| Guest2["guest process"]
P --> CS["check_statement<br/>(morphir-extension-sdk)"]
// crates/morphir/src/extensions.rs
fn host_initialize_params() -> InitializeParams {
InitializeParams {
protocol_versions: vec![MEP_VERSION.into()],
host: PeerInfo {
name: "morphir-cli".into(),
version: env!("CARGO_PKG_VERSION").into(),
},
}
}
// crates/morphir/src/commands/extension/probe.rs
let description = transport
.describe(InitializeParams {
protocol_versions: SUPPORTED_MEP_VERSIONS
.iter()
.map(|version| (*version).into())
.collect(),
host: super::host_peer(),
})
.await?;The two paths send different peer names and different version lists to the same guest. The host API gives both paths one handshake. The next checkpoint decides how much of this the CLI still writes. |
Checkpoint 3. The host is an in-process libraryThe host is a library linked into the flowchart LR
subgraph bin["morphir binary (one process)"]
CLI["CLI client<br/>args, files, cache, output"]
HL["Host library<br/>discovery, request, MEP"]
CLI -->|function call| HL
end
HL <-->|MEP| Guest["guest<br/>(Elm, Gleam, ...)"]
For sequenceDiagram
actor User
participant CLI as CLI client
participant HL as Host library
participant G as Elm guest
User->>CLI: morphir compile src/Foo.elm --package-name My.Package
CLI->>CLI: check compile cache
CLI->>HL: compile(files, language = elm, package = My.Package, peer)
HL->>HL: discover project, normalize name to my/package
HL->>G: initialize, then compile request
G-->>HL: IR and diagnostics
HL-->>CLI: result
CLI->>CLI: write IR to output path, update cache
CLI-->>User: print diagnostics
The CLI tells the library its peer name and version. The library does not hardcode // sketch
let peer = PeerInfo { name: "morphir-cli".into(), version: env!("CARGO_PKG_VERSION").into() };
Words for this thread:
The next checkpoint names which operations the library exposes beside compile. |
Checkpoint 4. Compile, generate, and probeThe host library exposes three operations: compile, generate, and probe. flowchart TB
subgraph CLI["CLI client owns"]
c1["compile: output paths, compile cache, config warnings"]
c2["generate: find IR on disk, choose output dir, write files"]
c3["install: copy files, update catalog"]
end
subgraph HL["Host library owns"]
h1["compile: discover project, build request, call frontend guest"]
h2["generate: select backend, call backend guest"]
h3["probe: handshake, describe, check statement"]
end
c1 --> h1
c2 --> h2
c3 --> h3
HL --> V["one list of supported MEP versions<br/>peer name and version from the client"]
For compile, the CLI passes the files, the language, and the package name. The library discovers the project, builds the protocol request, and talks to the guest. The CLI writes the result, prints the diagnostics, and keeps the output paths and the compile cache. For generate, the CLI finds the IR on disk and chooses the output directory. The library selects the backend and talks to the guest. The CLI writes the generated files. For probe, install still copies files and updates the catalog. It asks the library to probe the artifact. Compile, generate, and probe advertise the same supported protocol versions. The peer name and version still come from the CLI. // sketch
host.compile(CompileInput { files, language, package_name }, &peer).await?;
host.generate(GenerateInput { ir, target }, &peer).await?;
host.probe(&artifact, &declared_statement, &peer).await?;The next checkpoint decides whether a caller can keep a guest warm across many calls. The web workbench does that today, so a playground click does not spawn a new process each time. |
Checkpoint 5. Sessions are the core of the host libraryThe host library gives clients a session. A session is one live guest after the handshake. Opening a session starts the guest, negotiates the protocol version, and stores the guest's capabilities. The session exposes compile, generate, and describe. The client owns the session's lifetime and closes it. A client can hold many sessions at the same time, one for each guest. A server can keep a few sessions for the same guest and use them like workers. The host library opens every guest. Guests do not connect back to the host. The host library has three layers. flowchart TB
subgraph clients["Clients"]
CLI["morphir compile / generate / install"]
WB["web workbench"]
SRV["server"]
BR["browser (WASM + JS)"]
end
subgraph HL["Host library"]
OS["One-shot helpers<br/>compile, generate, probe"]
POOL["Pool (optional)<br/>reuse per key, evict dead guest, retry once"]
S["Session<br/>handshake, capabilities, compile, generate, describe"]
OS --> S
POOL --> S
end
CLI --> OS
WB --> POOL
SRV --> POOL
SRV --> S
BR --> S
S --> T["transport<br/>(later decision)"]
// sketch
pub struct Session { /* negotiated version, capabilities, transport */ }
impl Session {
pub async fn open(guest: GuestSpec, peer: PeerInfo) -> Result<Session>;
pub fn capabilities(&self) -> &Capabilities;
pub async fn compile(&mut self, request: CompileRequest) -> Result<CompileResult>;
pub async fn generate(&mut self, request: GenerateRequest) -> Result<GenerateResult>;
pub async fn describe(&mut self) -> Result<Description>;
pub async fn close(self) -> Result<()>;
}
// One-shot helper: open, call, close.
pub async fn compile_once(guest: GuestSpec, peer: PeerInfo, request: CompileRequest)
-> Result<CompileResult>
{
let mut session = Session::open(guest, peer).await?;
let result = session.compile(request).await;
session.close().await?;
result
}
// Optional pool. The client picks the key and the limits.
pub struct Pool<K> { /* K -> Session */ }
impl<K: Eq + Hash> Pool<K> {
pub async fn compile(&self, key: K, request: CompileRequest) -> Result<CompileResult>;
pub async fn generate(&self, key: K, request: GenerateRequest) -> Result<GenerateResult>;
pub async fn abandon(&self, key: &K);
}Two playground clicks through the pool: sequenceDiagram
actor User
participant WB as Web workbench
participant P as Pool
participant S as Session
participant G as Elm guest
User->>WB: click Compile
WB->>P: compile(key = elm, request)
P->>S: open (no session yet for key)
S->>G: start, initialize
P->>S: compile
S->>G: compile
G-->>WB: IR
User->>WB: click Generate
WB->>P: generate(key = elm, request)
P->>S: generate (same session, no start)
S->>G: generate
G-->>WB: files
Note over P,G: if the guest stopped: evict, open a new session, retry once
The host library keeps no hidden global state. The client decides pool size, idle time, and tenant limits. Reason: the host library must serve the CLI, desktop and server clients, and browser clients through WASM with JS glue. A hidden pool needs global state and timers, and // sketch: browser client through a wasm-bindgen adapter
using session = await Session.open(guest, { name: "morphir-web", version });
const result = await session.compile(request);
// `using` calls session[Symbol.dispose]() here, which frees the guest.Consequence: the session must not depend on one transport. Process stdio, Extism, HTTP, and a browser worker or in-page WASM guest must all fit behind the same session. The transport design is a later decision. |
Checkpoint 6.
|
Checkpoint 7. One MEP contract, two bindingsMEP is one contract. It defines the methods, the capability statement, and the version handshake one time. The contract has two bindings:
flowchart TB
M["MEP contract<br/>methods, capabilities, handshake"]
M --> J["JSON-RPC binding<br/>stdio or HTTP"]
M --> W["WIT binding<br/>morphir:mep world"]
S["Session (morphir-host)"] --> TJ["JSON-RPC transport"]
S --> TW["component transport"]
TJ --- J
TW --- W
TJ <--> G1["process guest"]
TW <--> G2["component guest"]
The session from checkpoint 5 does not change. A component guest is one more transport. A guest author chooses a binding, and the host treats both kinds of guest the same. // sketch
package morphir:mep@0.1.0;
interface frontend {
use types.{compile-request, compile-result, diagnostic};
compile: func(request: compile-request) -> result<compile-result, list<diagnostic>>;
}
interface host {
log: func(level: log-level, message: string);
read-file: func(path: string) -> result<string, string>;
}
world frontend-guest {
import host;
export lifecycle; // initialize, describe
export frontend;
}// sketch
let mut session = Session::open(GuestSpec::component(wasm_bytes), peer).await?; // or GuestSpec::process(command)
let result = session.compile(request).await?; // same call for both kinds of guestThe envelope style of Still open: the versioning rule that the two bindings share, and which file is the source of truth for the contract (WIT, JSON Schema, or Rust types). |
Checkpoint 8. MEP versioning before 1.0.0MEP uses the minor-bump rule until it reaches 1.0.0. Published contracts carry no prerelease tags.
Reason: WIT accepts full SemVer 2.0, prerelease included. The component model Explainer turns each version into a canonical version, and a host matches imports on that string. A version with a prerelease tag is never split, so each draft is a separate contract that needs an exact match. flowchart LR
subgraph A["minor-bump rule"]
g1["guest imports morphir:mep@0.2.6"] -->|"canonical 0.2"| h1["host provides morphir:mep@0.2.1"]
end
subgraph B["prerelease tags"]
g2["guest imports 0.2.0-draft.2"] -.-x|"no match"| h2["host provides 0.2.0-draft.1"]
end
With the minor-bump rule, the JSON-RPC and WIT bindings negotiate and link the same way. After 1.0.0, MEP uses standard SemVer, and the canonical version is the major version. |
Checkpoint 9. The MEP contract is a Gleam model compiled by MorphirThe source of truth for the MEP contract is a Gleam module, flowchart LR
G["mep.gleam<br/>(source of truth)"] -->|"morphir-gleam frontend<br/>works now"| IR["Morphir IR"]
IR -->|"new backend, tracked"| WIT["WIT world<br/>morphir:mep@0.x.y"]
IR -->|"morphir-openapi json-schema<br/>needs record-shape fix"| JS["JSON Schema<br/>(JSON-RPC binding)"]
IR -->|"morphir-rust, works now"| RS["Rust types"]
IR -->|"backend to choose later"| TS["TypeScript types"]
IR -->|"morphir-python, needs List, Maybe, generics"| PY["Python types"]
// sketch: mep.gleam
import gleam/option.{type Option}
pub type Diagnostic {
Diagnostic(severity: Severity, message: String, path: Option(String))
}
pub type CompileRequest {
CompileRequest(language_id: String, sources: List(Source), package: CompilePackage)
}
pub type Frontend {
Frontend(compile: fn(CompileRequest) -> Result(CompileResult, List(Diagnostic)))
}The generators come in steps:
The Gleam over Python: the current |
Checkpoint 10. Transport: a typed connection over a message channel
flowchart TB
S["Session and Pool<br/>handshake, version, capabilities, retry"] --> C{"GuestConnection<br/>typed calls"}
C --> JR["JsonRpcConnection<br/>MEP codec, written once"]
C --> CM["ComponentConnection<br/>WIT exports"]
JR --> CH{"Channel<br/>messages"}
CH --> ST["process stdio"]
CH --> HT["HTTP"]
CH --> WW["Web Worker postMessage"]
CM --> WT["wasmtime (native)"]
CM --> JCO["jco (browser)"]
H["HostServices<br/>log, progress, read-file"] -.->|"guest calls back"| C
// sketch
#[cfg_attr(not(target_arch = "wasm32"), async_trait)]
#[cfg_attr(target_arch = "wasm32", async_trait(?Send))]
pub trait GuestConnection {
async fn initialize(&mut self, params: InitializeParams) -> Result<InitializeResult>;
async fn describe(&mut self) -> Result<Description>;
async fn compile(&mut self, request: CompileRequest) -> Result<CompileResult>;
async fn generate(&mut self, request: GenerateRequest) -> Result<GenerateResult>;
async fn close(&mut self) -> Result<()>;
}
#[cfg_attr(not(target_arch = "wasm32"), async_trait)]
#[cfg_attr(target_arch = "wasm32", async_trait(?Send))]
pub trait Channel {
async fn send(&mut self, message: JsonRpcMessage) -> Result<()>;
async fn receive(&mut self) -> Result<JsonRpcMessage>;
}
pub trait HostServices {
fn log(&self, level: LogLevel, message: &str);
fn read_file(&self, path: &str) -> Result<String>;
}What each piece owns:
Tests follow the layers. Session and pool tests use a fake Cost: a new MEP method touches |
Checkpoint 11. The MEP contract lives in
|
Checkpoint 12. The MEP contract is a published Morphir modelThe MEP contract compiles to a Morphir IR Library, so it is a Morphir model. It uses the Morphir model package mechanism for publishing and downloading. It is the first real model published through that mechanism. flowchart LR
subgraph U["finos/morphir"]
C["spec/mep/contract/mep.gleam"] -->|"compile"| IR["Library IR"]
IR -->|"pack + publish (to build)"| REG[("model registry")]
K["spec/mep/mck"] -->|"mck kit, suite mep"| KIT["conformance kit"]
end
REG -->|"morphir package resolve / restore"| LR["language repo<br/>restored IR"]
LR -->|"morphir generate --target rust, wit, json-schema, ..."| T["types and bindings<br/>committed"]
KIT -->|"mck kit vendor"| LR2["language repo<br/>MEP conformance tests"]
What each piece uses:
Today the consumer side works against one local directory registry: Until publishing exists, each language repo commits a copy of the IR from an exact umbrella commit, with its source and digest: When Requirements on the model package formatMEP adds these requirements to the package system. The current
The digest rules keep working for each file inside the archive. The content path rules must allow the file names that document-tree IR and Ion need. |
Checkpoint 13. IR in MEP messages is a typed payload with a media typeMEP carries IR as a typed payload. The payload names its media type and its layout. Each guest states which IR formats it accepts and which it produces. // sketch: mep.gleam
pub type IrLayout {
Classic
DocumentTree
}
pub type IrPayload {
IrPayload(media_type: String, layout: IrLayout, data: BitArray)
}
// media_type: "application/ion", "application/ion+text", "application/json"flowchart LR
D["IR on disk<br/>Ion binary"] --> H["host library"]
H -->|"guest accepts Ion: send as is"| G1["backend guest A"]
H -->|"guest accepts JSON only: convert once"| G2["backend guest B"]
Rules:
Open gap: the Gleam frontend does not support |
Checkpoint 14. Each client registers its own guests
flowchart TB
subgraph H["morphir-host (no language bindings)"]
R["Registry"]
GS["GuestSource: Native | Process | Component"]
end
CLI["CLI"] -->|"Gleam, Elm when requested (native)<br/>+ installed catalog"| R
BR["browser client"] -->|"component guests"| R
SRV["server"] -->|"its own set"| R
// sketch
let mut registry = Registry::new();
registry.register(GuestSource::native(gleam_guest()))?;
if elm_requested {
registry.register(GuestSource::native(elm_guest()))?;
}
for snapshot in catalog.installed() {
registry.register(GuestSource::from_installed(snapshot))?;
}
|
Checkpoint 15. The root
|
Checkpoint 16. TypeScript and Python host ports wrap one core compiled to a componentThere is one host implementation, flowchart TB
C1["morphir-host<br/>compiled to a component<br/>world morphir:host"] --> TS1["TypeScript wrapper<br/>(jco)"]
C1 --> PY1["Python wrapper<br/>(wasmtime-py)"]
TS1 -.->|"imports: Channel, HostServices"| C1
PY1 -.->|"imports: Channel, HostServices"| C1
K["spec/mep/mck<br/>host suite"] -->|"checks"| C1
K -->|"checks"| TS1
K -->|"checks"| PY1
// sketch: TypeScript wrapper
import { Session } from "@finos/morphir-host";
using session = await Session.open(processGuest("morphir-elm"), peer);
const result = await session.compile(request);Costs: the core must not block on I/O across the component boundary. It needs component model async, or a core that takes messages in and returns actions (sans-IO). Python users need the |
Checkpoint 17. Component model async first, with a sans-IO fallbackThe target is component model async (WASI 0.3). The Support as of 2026-09-23:
WASI 0.3.0 was ratified on 2026-06-11, and 0.3.1 followed on 2026-08-11. flowchart TB
CORE["SessionCore (sans-IO)<br/>handle(event) -> actions"]
CORE --> ASYNC["async shell<br/>world morphir:host<br/>async func exports, async imports"]
CORE --> SYNC["sync world<br/>morphir:host-sync<br/>handle(event) exported as is"]
ASYNC --> RS["native Rust (Wasmtime)"]
ASYNC --> JS["Node and browsers (jco, JSPI)"]
SYNC --> PY["Python (wasmtime-py)<br/>asyncio driver loop"]
// sketch: the sans-IO core under both worlds
pub enum Event { Opened, Received(JsonRpcMessage), Call(Request), Closed }
pub enum Action { Send(JsonRpcMessage), Complete(RequestId, Response), Fail(RequestId, Error), Close }
impl SessionCore {
pub fn handle(&mut self, event: Event) -> Vec<Action>;
}Rules:
|
TrackingThe design in checkpoints 1 to 17 is approved. The work is tracked in #958.
The work issues are sub-issues of the phase issues. They live in finos/morphir, finos/morphir-rust, finos/morphir-typescript, and finos/morphir-python, and they carry "blocked by" links. |
|
The implementation plan for Phase A, part 1 (portable |
Checkpoint 18. Native transports live in
|
|
The implementation plan for Phase A, part 2 (native channels in |
Checkpoint 19.
|
|
The implementation plan for Phase A, part 3 (portable |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Cross-language extension SDK
Working thread for beads
morphir-opua. Each checkpoint is a decision. Later checkpoints are added as comments. Code blocks marked "sketch" show the shape of an idea. They are not a final API.Checkpoint 1. Scope
The design is one extension SDK for the guest side and the host side, with early targets Rust, TypeScript, and Python. The Rust CLI is the first consumer.
flowchart LR subgraph SDK["Extension SDK (morphir-opua)"] direction TB H["Host side<br/>Rust first, then TypeScript, Python"] G["Guest side<br/>Rust, TypeScript, Python"] end CLI["morphir CLI<br/>(first consumer)"] --> H H <-->|MEP| G G --- Elm["Elm guest"] G --- Gleam["Gleam guest"]The session-host extraction and the public compile entry points in #897 stay outside this thread, except where the SDK gives those callers a type they can use.
The first rule already accepted for this SDK is the Elm explicit package-name normal form, in kb
morphir-extensionsdecision 0005.My.Package,My/Package, andmy/packageall reportmy/package. The Rust copy isnormalize_package_nameon the ElmSourceIdentityinmorphir-elm-binding. The shared checks around it live inmorphir-workspace(discover_with_identity). The CLI already calls that crate. Decision 0005 also records a TypeScript copy in finos/morphir-elm and a Scala copy in morphir-scala-elm. The decision says the rule and its cases move into the SDK, and the providers drop their own copies. Scala is not an early target ofmorphir-opua. The case table still has to be something Scala can adopt.The rule today, from
morphir-elm-binding/src/workspace/mod.rs:The case table that every copy must pass:
My.Packagemy/packageMy/Packagemy/packagemy/packagemy/packageThe issue also names protocol types, discovery validation, and the compile envelope, on both the guest side and the host side. Those are separate pieces. The next checkpoint names which piece is designed first.
All reactions