Multimodal input

How it works

Multimodal input means a model accepts more than text in a request: images, PDFs, audio or video, depending on the model. These inputs are converted into tokens, so they count toward the context window and the input price.

Support varies a lot. Some models take images but not audio; some read a PDF as page images plus extracted text; video may be sampled as frames. Each maker documents limits on file size, resolution, number of files and how many tokens each input uses.

Check the exact input types a model lists before designing around it, and test on your own material, such as dense tables, handwriting or small UI text, where quality varies most.

Worked example

Illustrative: if a maker counts a page image as about 1,500 tokens, a 40-page scanned PDF is about 60,000 input tokens. At $3 per 1M input tokens that is $0.18 per read, before any text you add.

Common mistakes

• 1Sending full-resolution images when a smaller size gives the same result for fewer tokens.
• 2Assuming 'multimodal' covers audio or video when the model only accepts images and PDFs.
• 3Trusting numbers read from charts or scans without spot-checking them by hand.