Vision

Vision lets you pass images, PDFs, and other media to vision-capable models. Use the multiModal() helper to combine text prompts with one or more image sources.

import {
  multiModal,
  imageUrl,
  imageFile,
  imageBuffer,
} from 'confused-ai';

---

Pass a remote image

import { createAgent, multiModal, imageUrl } from 'confused-ai';

const agent = createAgent({
  name: 'vision-agent',
  instructions: 'Analyse the images provided by the user.',
  model: 'gpt-4o',
  apiKey: process.env.OPENAI_API_KEY!,
});

const result = await agent.run(
  multiModal(
    'What is in this image?',
    imageUrl('https://example.com/photo.jpg'),
  ),
);

---

Pass a local file

import { multiModal, imageFile } from 'confused-ai';

// imageFile() is async — loads and base64-encodes the file
const result = await agent.run(
  multiModal(
    'Describe this chart.',
    await imageFile('./chart.png'),
  ),
);

---

Pass a buffer (canvas, upload, fetch response)

import { multiModal, imageBuffer } from 'confused-ai';

const response = await fetch('https://example.com/diagram.png');
const buffer = await response.arrayBuffer();

const result = await agent.run(
  multiModal(
    'What does this architecture diagram show?',
    imageBuffer(buffer, 'image/png'),
  ),
);

---

Multiple images in one message

const result = await agent.run(
  multiModal(
    'Compare these two screenshots and explain the differences.',
    imageUrl(beforeUrl),
    imageUrl(afterUrl),
  ),
);

---

Image detail level

Control quality vs. speed with the detail option:

imageUrl('https://example.com/photo.jpg', { detail: 'high' })
imageUrl('https://example.com/thumbnail.jpg', { detail: 'low' })
// 'auto' (default) — model decides

---

ImageSource types

Type	Factory	Description
ImageUrl	imageUrl(url, opts?)	HTTPS or data URI
ImageFile	await imageFile(path, opts?)	Local file — loaded at call time (Node.js only)
ImageBuffer	imageBuffer(data, mimeType, opts?)	Raw ArrayBuffer / Uint8Array

---

Supported formats

Images: jpg, jpeg, png, gif, webp, bmp, svg, tiff, heic
Audio: mp3, wav, ogg, m4a, flac, webm
Video: mp4, webm, mov, avi, mkv

---

Where to go next

• Voice — speech input and output.
• Video — process and summarise video content.
• Agents — agent.run() multiModal option.