Skip to content
Search
TechniquesGlossary

Multimodal Prompting

Many current models accept images and documents, and some accept audio, in the same request as your text. That lets you ask about a chart, a photographed receipt, a screenshot of an error or a scanned form. The Prompt Report survey, which is mostly about text, also catalogues 40 prompting techniques for other modalities, a sign of how quickly this area has grown.

Most text prompting advice still applies, and a few habits matter more. Say what each file is and what to look at ("the second chart, the bar for Q3"). Ask the model to transcribe or describe what it sees before it reasons about it, so you can catch misreadings. Ask for structured output when you're extracting fields from documents. And check numbers read from images against the source, because a misread digit looks just as confident as a correct one (hallucination). A prompt template keeps this consistent across many files.

Example

The attached image is a photo of a paper receipt.
First, transcribe every line exactly as printed, including prices.
Then return JSON: {"merchant", "date", "line_items": [{"description", "amount"}], "tax", "total"}.
If any value is unreadable, use null and list it under "unreadable". Don't guess.

When to use it

  • Pulling data out of scans, photos, receipts and forms
  • Asking about charts, diagrams or screenshots
  • Describing images for alt text or catalog entries

When not to use it

  • A text version of the document is available; text is cheaper and more reliable
  • Exact figures that will be used without checking