LLM2Jev v0.1.0

Give your LLM / MLLM a decision engine. Zero training required.

LLM2Jev turns supported LLMs / MLLMs into Jev-style decision models using existing model weights. Classify, score, and select actions from text and images with prefill alone, across SGLang, Transformers, and MLX backends. The Python client extends the OpenAI SDK with structured decision requests.

Highlights

• Training-free inference: no additional training or fine-tuning to use existing models for structured decisions.
• Prefill-only decisions: read next-token yes/no logits and assemble results without generating an answer or parsing generated JSON.
• Text and images: combine evidence in state or question instructions; accept HTTP(S) images, local paths, file URIs, and Base64 data URLs.
• Choice, Score, and Noul: mix question types in one request and receive option distributions, rubric scores, and yes/no probabilities.
• Chat and decisions share a service and model: with the SGLang backend, /v1/systemone runs alongside native OpenAI-compatible chat endpoints through the same inference engine and model.
• Chat and decisions share a client: synchronous Client and asynchronous AsyncClient extend the OpenAI Python SDK with systemone.create(), preserving existing SDK interfaces, authentication, timeouts, and retries. Both request types can use the same client when the server supports both endpoints.
• Three backends: SGLang, Transformers, and MLX/MLX-VLM on Apple Silicon, available through offline Python inference or HTTP serving.
• Prefix reuse: staged candidate scoring with SGLang Radix Cache and explicit shared-prefix prefill with MLX.
• Per-request option rendering: include_choice_candidates=False hides the full Choice option list from each candidate prompt. The default is True; every candidate is still scored.

Get started

See the README for the quick start with one service and one client, or the usage guide for offline inference, asynchronous calls, and backend configuration.

from llm2jev import Client, Choice

client = Client(base_url="http://localhost:30000/v1", api_key="unused")
response = client.systemone.create(
    model="local-model",
    state="My card was charged twice.",
    questions={"department": Choice(criteria={
        "billing": "Charges and billing", "shipping": "Delivery",
    })},
)
print(response.choices["department"].probabilities)

Scope and validation

Model compatibility depends on the chosen backend and its support for the model's tokenizer, chat template, and image inputs. MLX and Transformers HTTP services currently expose /v1/systemone, /v1/models, and /health; shared chat and decision serving is available through SGLang. Existing SDK methods remain available on the client, but the server must implement the endpoint being called.

Probabilities and confidence are model-derived scores, not a guarantee of calibrated correctness or numerical equivalence to Jev's official service. Performance depends on the model, hardware, input length, candidate count, and cache state. See JevBench and prefix-cache benchmarks for measured results and conditions.

The implementation passed 261 tests, with 6 optional native MLX tests skipped, plus recursive syntax checks and source/wheel builds. Client tests use deterministic HTTP transports and fake model backends.

LLM2Jev is an independent open-source project and is not affiliated with or endorsed by Jev or TypeSafe.