Streaming
Receiving the response piece by piece as it is generated, usually as server-sent events, instead of waiting for the complete response.
Why it matters. It makes long answers feel fast and lets you show progress, but changes how you handle errors, usage counts and tool calls.
How it works
With streaming on, the API sends a series of events: text deltas, fragments of tool-call arguments, and at the end a stop reason and token usage. Most provider APIs use server-sent events over HTTP; some offer WebSocket or realtime connections, mainly for voice.
Streaming doesn't make generation faster: total time is the same, but users see text after the time to first token instead of after the whole response. Event formats differ between providers, and SDKs offer helpers that assemble the final message. Some APIs recommend or require streaming for long outputs to avoid HTTP timeouts.
Build for partial data: tool arguments and structured output are valid JSON only once complete, moderation or post-processing may need the full text, and an error or disconnect can arrive mid-stream after text is already on screen.
Worked example
Illustrative: with 0.6 s to first token at 40 tokens per second, an 800-token answer completes in 0.6 + 800 / 40 = 20.6 s either way. Streamed, the user starts reading after 0.6 s; unstreamed, the screen stays blank for 20.6 s.
Common mistakes
- Parsing streamed tool-call arguments or JSON before the stream has finished.
- Ignoring the final usage event, so token counts and costs go unlogged for streamed requests.
- Not handling errors or disconnects mid-stream, leaving half an answer on screen with no retry or notice.
See it on a real card: Build with it on Llama 4 Maverick.