Using multimodal data

LLM2Jev supports text and images in state or a question's instructions.

Supported image sources

Specify the image source in image_url.url. The following formats are supported:

Format	Example
HTTP(S) URL	https://example.com/photo.png
Local file URI	file:///data/photo.png
Local path	/data/photo.png or images/photo.png
Base64 data URL	data:image/png;base64,...

Building a request

Use an object marked with type: "multimodal" in state or instructions:

{
  "type": "multimodal",
  "content": [
    {"type": "text", "text": "Please inspect this image."},
    {"type": "image_url", "image_url": {"url": "file:///data/photo.png"}}
  ]
}

content is a nonempty list that can contain multiple text blocks and images, or only text or only images. Items are submitted in list order. A text block's text must be a string, and an image's url must be a nonempty string. Only the fields shown above are currently supported; detail, file_id, audio and video are not supported.

Keep the outer marker object. Passing the content list directly as state or instructions serializes it as ordinary JSON text and does not load images. Plain strings, unmarked JSON objects and arrays retain their existing behavior. Images are not interpreted in criteria.

The following example places an image in state so multiple questions can use it. The backend examples below use the model_path and request defined here:

from llm2jev import Choice, JevRequest, Noul

model_path = "/path/to/vlm"
request = JevRequest(
    model=model_path,
    state={"type": "multimodal", "content": [
        {"type": "text", "text": "Inspect the image"},
        {"type": "image_url", "image_url": {"url": "file:///data/photo.png"}},
    ]},
    questions={
        "person": Noul(instructions="Is a person visible in the image?"),
        "color": Choice(
            instructions="Which color dominates the image?",
            criteria={"red": "Red", "blue": "Blue", "green": "Green"},
        ),
    },
)

You can also place the text-and-image object in a question's instructions:

instructions = JevRequest(
    model=model_path,
    state="Inspect the attached image.",
    questions={
        "person": Noul(instructions={
            "type": "multimodal", "content": [
                {"type": "text", "text": "Is a person visible in the image?"},
                {"type": "image_url", "image_url": {"url": "file:///data/photo.png"}},
            ],
        }),
    },
)

SGLang backend

from llm2jev import LLM2Jev, SGLangBackend

if __name__ == "__main__":
    with SGLangBackend(model_path) as backend:
        response = LLM2Jev(backend=backend).evaluate(request)
        print(response.json)

Transformers backend

Set multimodal=True when creating the backend:

from llm2jev import LLM2Jev, TransformersBackend

with TransformersBackend(model_path, multimodal=True, dtype="bfloat16") as backend:
    response = LLM2Jev(backend=backend).evaluate(request)
    print(response.json)

MLX backend

Set multimodal=True when creating the backend:

from llm2jev import LLM2Jev, MLXBackend

with MLXBackend(model_path, multimodal=True, batch_size=8) as backend:
    response = LLM2Jev(backend=backend).evaluate(request)
    print(response.json)

HTTP API calls

Start SGLang following the Usage guide.

curl http://localhost:30000/v1/systemone \
  -H "Authorization: Bearer $LLM2JEV_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-vlm",
    "state": {
      "type": "multimodal",
      "content": [
        {"type": "text", "text": "Inspect the image"},
        {"type": "image_url", "image_url": {"url": "file:///data/photo.png"}}
      ]
    },
    "questions": {
      "person": {
        "type": "noul",
        "instructions": "Is a person visible in the image?"
      },
      "color": {
        "type": "choice",
        "instructions": "Which color dominates the image?",
        "criteria": {"red": "Red", "blue": "Blue", "green": "Green"}
      }
    }
  }'

Images can also be placed in a question's instructions, as shown earlier.

Choosing where to place images

Scenario	Suggested location	Usage
Multiple questions about the same image	state	Include the image once for all questions to share
An image relevant to only one question	That question's instructions	Attach the image to the question
The same question about different images	instructions is an option	Keep state and the question text fixed; build a separate request for each image
Comparing or evaluating multiple images together	The same content list	Arrange text and images in the desired order

Content in state appears before the question; content in instructions appears within that question. You control the order of text and images in each content list. When both locations contain images, the model considers images from both.

For prefix caching, placing reused content earlier is usually a useful starting point. Whether caching is possible or improves speed must be verified with your actual model and requests; image placement alone does not guarantee either.