Multimodal Input

Multimodal input allows models to process not only text but also images, audio, and other forms of content. RouteAPI supports passing multimodal content on both /v1/chat/completions (OpenAI format) and /v1/messages (Claude format), and automatically handles format conversion between different upstream protocols.

1. Multimodal Overview

Supported Modality Types

Modality Type	Description
Image (Vision)	PNG, JPEG, WebP, GIF formats for image understanding, OCR, chart analysis
Audio	Some models support audio input for speech understanding, transcription, etc.
Video	Some models support video frame input by splitting video into key frame sequences

Currently, image input has the widest support, with almost all mainstream vision models supporting it. Audio and video input depend on specific model capabilities.

Supported Vision Models

The following models support image input (non-exhaustive list):

Model Family	Typical Model ID
OpenAI GPT-4 Vision	gpt-4o, gpt-4-turbo, gpt-5.5
Claude Vision	claude-sonnet-4-5, claude-opus-4-5
Gemini Vision	gemini-2.0-flash, gemini-2.5-pro
Azure OpenAI	azure-gpt-4o

First use the models list endpoint to confirm the model is available in your account:

curl https://api.routeapi.ai/v1/models \
  -H "Authorization: Bearer $ROUTEAPI_KEY"

This endpoint does not return an image-input capability field. To determine vision support, rely on the provider’s documentation or one actual request with an image.

2. Image Input Basics

Supported Image Formats

Format	MIME Type	Description
PNG	image/png	Lossless format, suitable for screenshots and charts
JPEG	image/jpeg	Lossy compression, suitable for photos
WebP	image/webp	Modern format with small size and high quality
GIF	image/gif	Non-animated format (only first frame is used)

Image Size Limits

Different models have different image size limits. General recommendations:

Limit	Recommended Value	Description
Single image size	< 20 MB	Very large images increase processing time and cost
Image resolution	Long edge ≤ 2048px	High resolutions are automatically scaled or processed in chunks
Base64 encoded	< 32 MB	Total request size is limited by upstream service

It’s recommended to compress images before uploading, reducing resolution while maintaining readability to decrease both transmission time and token cost.

Resolution and Cost Considerations

Images are converted to tokens for billing. High-resolution images consume far more tokens than text:

• Low resolution mode (like OpenAI’s detail: "low"): Fixed ~85 tokens/image.
• High resolution mode (like detail: "high"): Split by image size; a 2048×2048 image may consume 800-1500 tokens.

In production, choose resolution mode based on actual needs. Use low mode when fine recognition isn’t required.

3. Image Delivery Methods

Images can be passed in two ways: URL and Base64 encoding.

URL Method

Pass a publicly accessible image URL for the upstream model service to fetch:

Advantages:

• Small request size, doesn’t use your upload bandwidth.
• Suitable for images already hosted on CDN.

Disadvantages:

• Image must be publicly accessible; upstream service IPs must be able to access it.
• If image loading fails (network issues, authentication, expired link), request will error.

Use Cases: Images already on public image hosts or CDN, no need for temporary upload.

Base64 Encoding Method

Read image as binary data, encode with Base64, and embed directly in request:

Advantages:

• No publicly accessible URL needed, suitable for private images.
• Self-contained request, doesn’t depend on external service availability.

Disadvantages:

• Base64 encoding increases data size by about 33%.
• Large request size, longer upload time.

Use Cases: User-uploaded private images, local files, temporary screenshots without public URLs.

Comparison of Both Methods

Comparison	URL Method	Base64 Method
Request size	Small (just URL string)	Large (Base64 ~1.33× original file)
Upload speed	Fast	Slow
Image accessibility	Must be publicly accessible	No requirements, private images OK
External dependencies	Depends on image server and upstream fetch	No external dependencies
Use cases	Public image hosts, CDN	User uploads, local files

4. OpenAI Format Image Input

In /v1/chat/completions, images are passed through the content array, where each element is a content block distinguished by type for text and images.

content Array Structure

content can be a string (plain text) or an array (multimodal):

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What's in this image?" },
    {
      "type": "image_url",
      "image_url": {
        "url": "https://example.com/image.jpg"
      }
    }
  ]
}

image_url Structure

Fields in the image_url content block:

Field	Type	Required	Description
type	string	Yes	Fixed as "image_url"
image_url.url	string	Yes	Image URL or Base64 Data URI
image_url.detail	string	No	Resolution mode: "low", "high", "auto" (default)

detail Parameter

detail controls image processing resolution and cost:

Value	Behavior	Token Consumption
"low"	Low resolution mode, image scaled to fixed size (e.g., 512×512)	Fixed ~85 tokens
"high"	High resolution mode, image processed in chunks, preserves details	By chunk count, typically hundreds to thousands of tokens
"auto"	Model automatically chooses (default)	Depends on model strategy

Cost Difference Example:

• A simple screenshot with "low" might only need 85 tokens (~$0.0001).
• The same image with "high" might consume 800 tokens (~$0.001).

For scenarios that don’t require recognizing small text or details (like “what animal is this” or “what’s the interface theme color”), "low" is sufficient. Use "high" only when you need OCR, reading chart values, or recognizing small objects.

URL Method Complete Example

{
  "model": "gpt-4o",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/320px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
            "detail": "high"
          }
        },
        { "type": "text", "text": "Describe the content of this image" }
      ]
    }
  ]
}

Base64 Method Complete Example

Base64 requires Data URI format: data:<mime_type>;base64,<encoded_data>.

{
  "model": "gpt-4o",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD..."
          }
        },
        { "type": "text", "text": "What text is in this image?" }
      ]
    }
  ]
}

Note that the Base64 string can be very long; the above example is truncated. In actual use, the complete encoding might be hundreds of KB to several MB.

5. Claude Format Image Input

In /v1/messages, images are passed through image type blocks in the content array, with structure significantly different from OpenAI format.

content Array Structure

{
  "role": "user",
  "content": [
    {
      "type": "image",
      "source": {
        "type": "base64",
        "media_type": "image/jpeg",
        "data": "/9j/4AAQSkZJRgABAQEAYABgAAD..."
      }
    },
    { "type": "text", "text": "Describe this image" }
  ]
}

source Structure

The source field specifies the image source in two ways:

Base64 Method

Field	Type	Required	Description
type	string	Yes	Fixed as "base64"
media_type	string	Yes	MIME type, such as image/jpeg, image/png
data	string	Yes	Base64 encoded image data (without data: prefix)

Note that Claude format Base64 does not need the Data URI prefix; pass the encoded string directly.

{
  "type": "image",
  "source": {
    "type": "base64",
    "media_type": "image/png",
    "data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
  }
}

URL Method

Field	Type	Required	Description
type	string	Yes	Fixed as "url"
url	string	Yes	Publicly accessible image URL

{
  "type": "image",
  "source": {
    "type": "url",
    "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/320px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
  }
}

Complete Request Example

{
  "model": "claude-sonnet-4-5",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "image",
          "source": {
            "type": "url",
            "url": "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg"
          }
        },
        { "type": "text", "text": "What insect is this?" }
      ]
    }
  ]
}

6. Gemini Format Image Input

Gemini native format uses a parts array instead of content, with quite different structure. RouteAPI handles conversion internally, so when calling Gemini models using OpenAI or Claude format, you don’t need to worry about native format details. The following is for reference only.

parts Array

{
  "contents": [
    {
      "role": "user",
      "parts": [
        { "text": "Describe this image" },
        {
          "inlineData": {
            "mimeType": "image/jpeg",
            "data": "/9j/4AAQSkZJRg..."
          }
        }
      ]
    }
  ]
}

inlineData (Base64)

Field	Description
inlineData.mimeType	MIME type
inlineData.data	Base64 encoded data

fileData (URI)

Gemini also supports referencing files uploaded to Google services via file URI:

{
  "fileData": {
    "mimeType": "image/jpeg",
    "fileUri": "gs://bucket-name/path/to/image.jpg"
  }
}

In practice, when calling Gemini models through RouteAPI, use OpenAI or Claude format; RouteAPI will automatically convert.

7. Multiple Image Input

All mainstream vision models support passing multiple images in a single request.

Multiple Images in Single Request

Place multiple image blocks in the content / parts array:

OpenAI Format:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Compare the differences between these two images" },
    {
      "type": "image_url",
      "image_url": { "url": "https://example.com/image1.jpg" }
    },
    {
      "type": "image_url",
      "image_url": { "url": "https://example.com/image2.jpg" }
    }
  ]
}

Claude Format:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What similarities and differences do these two images have?" },
    {
      "type": "image",
      "source": { "type": "url", "url": "https://example.com/before.jpg" }
    },
    {
      "type": "image",
      "source": { "type": "url", "url": "https://example.com/after.jpg" }
    }
  ]
}

Image Order and Reference

The model understands images in array order. If text refers to “the first image” or “the second image”, the model will correspond by appearance order. It’s recommended to place explanatory text before or after all images, not interspersed, for clearer semantics:

{
  "content": [
    { "type": "text", "text": "The first is a user interface screenshot, the second is a design mockup. Please compare the differences and provide modification suggestions." },
    { "type": "image_url", "image_url": { "url": "..." } },
    { "type": "image_url", "image_url": { "url": "..." } }
  ]
}

Mixed with Text

You can also alternate text and images for step-by-step explanation:

{
  "content": [
    { "type": "text", "text": "This is the original interface:" },
    { "type": "image_url", "image_url": { "url": "https://example.com/old.jpg" } },
    { "type": "text", "text": "This is the improved interface:" },
    { "type": "image_url", "image_url": { "url": "https://example.com/new.jpg" } },
    { "type": "text", "text": "Please summarize the improvements." }
  ]
}

Actual effectiveness depends on the model’s understanding of content block order; mainstream vision models usually handle this correctly.

8. Audio Input

Some models support audio input for speech understanding, transcription, sentiment analysis, etc. Current support is less widespread than images.

Supported Audio Formats

Depends on specific models, common formats include:

• WAV (audio/wav)
• MP3 (audio/mpeg)
• OGG (audio/ogg)
• FLAC (audio/flac)

Audio Delivery Method

Like images, audio supports both URL and Base64 methods. OpenAI format example (assuming model support):

{
  "role": "user",
  "content": [
    {
      "type": "input_audio",
      "input_audio": {
        "data": "<base64-encoded-audio>",
        "format": "wav"
      }
    },
    { "type": "text", "text": "Transcribe this audio and summarize key points" }
  ]
}

Actual field names and structure depend on model protocol. Before use, check the selected model’s documentation or verify the supports_audio_input field via the models list endpoint.

9. Complete Application Examples

The following examples show end-to-end image understanding implementation in curl, Python, and Node.js.

Image Understanding and Description

Given an image, have the model describe its content.

curl (OpenAI format, URL method):

curl https://api.routeapi.ai/v1/chat/completions \
  -H "Authorization: Bearer $ROUTEAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "image_url",
            "image_url": {
              "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/320px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
            }
          },
          { "type": "text", "text": "Describe the content and atmosphere of this image in detail" }
        ]
      }
    ]
  }'

Python (OpenAI SDK, Base64 method):

import base64
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ROUTEAPI_KEY"],
    base_url="https://api.routeapi.ai/v1",
)

# Read local image and encode to Base64
with open("image.jpg", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}"
                    },
                },
                {"type": "text", "text": "What's in this image?"},
            ],
        }
    ],
)

print(response.choices[0].message.content)

Node.js (openai package, URL method):

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.ROUTEAPI_KEY,
  baseURL: 'https://api.routeapi.ai/v1',
});

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'image_url',
          image_url: {
            url: 'https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/320px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg',
          },
        },
        { type: 'text', text: 'Summarize the theme of this image in one sentence' },
      ],
    },
  ],
});

console.log(response.choices[0].message.content);

Chart and Data Visualization Analysis

Upload a chart screenshot and have the model read data and analyze:

Python (Claude format, Base64):

import base64
import os
from anthropic import Anthropic

client = Anthropic(
    api_key=os.environ["ROUTEAPI_KEY"],
    base_url="https://api.routeapi.ai",
)

with open("chart.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": image_data,
                    },
                },
                {
                    "type": "text",
                    "text": "What trend does this chart show? Please extract key data points and provide analysis.",
                },
            ],
        }
    ],
)

print(message.content[0].text)

OCR Text Extraction

Extract text content from screenshots or photos:

curl (OpenAI format, high resolution):

curl https://api.routeapi.ai/v1/chat/completions \
  -H "Authorization: Bearer $ROUTEAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "image_url",
            "image_url": {
              "url": "https://example.com/document.jpg",
              "detail": "high"
            }
          },
          {
            "type": "text",
            "text": "Extract all text from the image, maintaining the original format and structure"
          }
        ]
      }
    ]
  }'

For OCR scenarios, use "detail": "high" to ensure recognition accuracy, especially for small or dense text.

Multi-Image Comparison Analysis

Compare differences between two or more images:

Python (OpenAI format, multiple images):

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["ROUTEAPI_KEY"],
    base_url="https://api.routeapi.ai/v1",
)

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Compare the following two images and find 5 major differences between them:",
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/before.jpg"},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/after.jpg"},
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

Mixed Image-Text Conversation

Mix images and text in multi-turn conversations:

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["ROUTEAPI_KEY"],
    base_url="https://api.routeapi.ai/v1",
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {"url": "https://example.com/product.jpg"},
            },
            {"type": "text", "text": "What are the main features of this product?"},
        ],
    }
]

response = client.chat.completions.create(model="gpt-4o", messages=messages)
messages.append(response.choices[0].message)
print("First turn:", response.choices[0].message.content)

# Continue asking (plain text)
messages.append({"role": "user", "content": "What type of user is it suitable for?"})
response = client.chat.completions.create(model="gpt-4o", messages=messages)
print("Second turn:", response.choices[0].message.content)

Images only need to be passed once in the first turn; in subsequent turns, the model will remember the image content (within the context window) without needing to re-upload.

10. Best Practices

Image Optimization

Perform necessary optimization before uploading images to reduce cost and improve response speed:

Optimization	Recommendation
Size	Keep long edge within 2048px unless you really need to recognize more details
Format	Use PNG for screenshots and charts, JPEG for photos, WebP for ultimate compression
Compression	JPEG quality 80-90% is sufficient, minimal visual difference but significantly smaller size
Cropping	Remove irrelevant areas (large blank spaces, watermarks, borders), keep only key content

Don’t compress images to the point of being unrecognizable to save tokens; if the model can’t recognize it, it’s actually wasteful.

Cost Considerations

Multimodal input cost mainly comes from images:

Scenario	Typical Token Consumption	Cost Estimate (GPT-4o)
Low resolution image (detail: "low")	~85 tokens	$0.0001
High resolution small image (500×500, detail: "high")	~200 tokens	$0.0003
High resolution large image (2000×2000, detail: "high")	~800 tokens	$0.0012
Multiple high resolution images (5 images, detail: "high")	~4000 tokens	$0.006

Specific rates depend on selected model; above are examples only. Production recommendations:

1. Default to detail: "auto" or "low", let model or user needs determine resolution.
2. Use "high" only when fine recognition is clearly needed (OCR, chart data, small object detection).
3. Log token usage for each request (usage field in response) to identify cost anomalies.

Error Handling

Multimodal input introduces additional failure points requiring targeted handling:

Unsupported Image Format

{
  "error": {
    "message": "Unsupported image format",
    "type": "invalid_request_error"
  }
}

Solution: Confirm MIME type is correct, or convert to PNG/JPEG.

Image Too Large

{
  "error": {
    "message": "Image size exceeds limit",
    "type": "invalid_request_error"
  }
}

Solution: Compress image or reduce resolution and retry.

URL Inaccessible

{
  "error": {
    "message": "Failed to fetch image from URL",
    "type": "invalid_request_error"
  }
}

Solution:

• Confirm URL is publicly accessible without authentication.
• Test if upstream service IPs can access the URL (firewall, geo-restrictions).
• Switch to Base64 method to avoid external service dependency.

Base64 Decode Failed

{
  "error": {
    "message": "Invalid base64 encoding",
    "type": "invalid_request_error"
  }
}

Solution: Check if Base64 encoding is complete and format is correct (OpenAI format needs data: prefix, Claude format doesn’t).

Security

Risk	Protection Measures
URL image leakage	Ensure URL points to image without sensitive info, or use authenticated temporary links
Base64 size attacks	Limit user upload image size cap (e.g., 20 MB) to prevent oversized requests
Injection attacks	Don’t directly concatenate user-uploaded image URLs into system commands or SQL
Model hallucinations	Image understanding results may be inaccurate; high-risk scenarios (medical, legal, financial) need human review

Compatibility Notes

• Vision capabilities, supported image formats, and maximum image count depend on selected model; test and verify before production.
• The detail parameter is only meaningful in OpenAI format; Claude format has no corresponding parameter.
• Different models have different resolution processing strategies; the same image may consume significantly different tokens across models.
• In streaming responses, image content processing results are typically returned all at once early or late in the stream, not streamed character by character.
• Log request ID, model ID, status code, and token usage for each request to troubleshoot cost and quality issues. See Errors and Debugging for details.