Stream LLM Responses with FastAPI SSE: A Working Python Tutorial (2026)

A language model takes 8 to 25 seconds to write a long answer. If your API waits for the full response before sending anything, your user stares at a spinner for 20 seconds and then gets 800 words dumped on screen at once. That is the single biggest reason AI apps feel slow — and it has nothing to do with how fast the model is.

Streaming fixes this. Instead of waiting for the whole completion, you forward each token to the browser the moment it arrives. This tutorial builds a production-shaped streaming endpoint with FastAPI and Server-Sent Events, covering three providers, the frontend glue, and the five mistakes that break streaming in production.

1. Why Streaming Changes Everything

Three numbers explain why every serious LLM product streams:

• Time to first token (TTFT) on a hosted model is typically 400–1,500 ms. That is your perceived responsiveness, not total generation time.
• Perceived speed. Users tolerate a 20-second answer if text starts appearing in 1 second. They abandon a blank loading state after roughly 3 seconds.
• Timeouts and proxies. Many load balancers and CDNs drop idle connections after 30–60 seconds. A stream that emits a packet every 50 ms never trips that idle timer; a blocking request routinely does.

In practice we have seen support bots with identical models rated as "fast" or "unusable" purely based on whether the first token arrives before the user finishes blinking.

2. SSE vs WebSocket: Which One?

Both are valid. SSE is the default choice for one-directional text generation, which is most chat UIs.

Rule of thumb: if the client's only job is to receive text, use SSE. It is simpler, survives proxies better, and reconnects for free. Reach for WebSocket when the client also needs to push binary or high-frequency data upstream.

3. Setup

pip install fastapi uvicorn openai

# Project layout
# app.py
# requirements.txt

Nothing special is required for streaming — StreamingResponse ships in Starlette, already a FastAPI dependency. Verify your version supports async generators (anything modern does):

python -c "import fastapi; print(fastapi.__version__)"  # 0.115.x or newer

4. Your First Streaming Endpoint

The core pattern is an async generator that yields bytes, wrapped in StreamingResponse with the text/event-stream media type:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio

app = FastAPI()

async def token_generator():
    words = ["Streaming", " responses", " feel", " instant", "."]
    for word in words:
        # SSE frame: "data: <payload>\n\n"
        yield f"data: {word}\n\n".encode("utf-8")
        await asyncio.sleep(0.15)   # simulate model latency

@app.post("/chat")
async def chat():
    return StreamingResponse(
        token_generator(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",   # disables nginx buffering
        },
    )

Run it and watch tokens arrive one at a time:

uvicorn app:app --reload
curl -N -X POST http://localhost:8000/chat

# data: Streaming
# data:  responses
# data:  feel
# ...

The -N flag on curl disables its own buffering. Without it you will see everything arrive at once and wrongly conclude your stream is broken.

5. Streaming From a Real LLM Provider

In reality your generator forwards chunks from an upstream SDK rather than inventing words. The three most common cases:

OpenAI SDK

from openai import AsyncOpenAI

client = AsyncOpenAI()  # reads OPENAI_API_KEY from env

async def openai_stream(prompt: str):
    stream = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    async for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            yield f"data: {delta}\n\n".encode("utf-8")
    yield b"data: [DONE]\n\n"

Anthropic SDK

from anthropic import AsyncAnthropic

anthropic = AsyncAnthropic()

async def anthropic_stream(prompt: str):
    async with anthropic.messages.stream(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        async for text in stream.text_stream:
            yield f"data: {text}\n\n".encode("utf-8")
    yield b"data: [DONE]\n\n"

Ollama (local models)

If you already run models locally, Ollama exposes an OpenAI-compatible endpoint, so one code path covers both — see our Ollama local LLM guide for the setup:

client = AsyncOpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

async def local_stream(prompt: str):
    stream = await client.chat.completions.create(
        model="llama3.2",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            yield f"data: {chunk.choices[0].delta.content}\n\n".encode("utf-8")

Note that you must use the Async client. The synchronous OpenAI client blocks the event loop, which stops Uvicorn from flushing chunks and silently destroys streaming for every concurrent user on the server.

6. Consuming the Stream in the Browser

The native EventSource API only issues GET requests. Since chat endpoints need a POST body, use fetch with a readable stream reader:

async function streamChat(prompt, outputEl) {
  const res = await fetch("/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt }),
  });

  const reader = res.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    const text = decoder.decode(value, { stream: true });
    for (const line of text.split("\n")) {
      if (!line.startsWith("data: ")) continue;
      const payload = line.slice(6);
      if (payload === "[DONE]") return;
      outputEl.textContent += payload;   // or render markdown incrementally
    }
  }
}

If your endpoint is a plain GET (for example, a progress bar or a notification feed), EventSource is less code and reconnects automatically:

const es = new EventSource("/chat?prompt=hello");
es.onmessage = (e) => { outputEl.textContent += e.data; };
es.onerror   = () => es.close();

7. Handling Client Disconnects

Users close tabs, hit stop, and navigate away mid-stream. Without cleanup your server keeps generating tokens — and paying for them — after the client is gone. Check request state inside the loop:

from fastapi import Request

@app.post("/chat")
async def chat(request: Request):
    async def gen():
        async for delta in openai_stream("Write a haiku about caching"):
            if await request.is_disconnected():
                print("Client gone — cancelling upstream generation")
                break
            yield f"data: {delta}\n\n".encode("utf-8")

    return StreamingResponse(gen(), media_type="text/event-stream")

Breaking out of the loop also closes the underlying HTTP connection to the provider, stopping token billing. For strictly correct cancellation, open the upstream stream inside an async with block so the SDK closes it explicitly.

8. Common Mistakes

Mistake 1: Defining the Generator as a Normal Function

A plain def generator runs in a threadpool and cannot await inside the loop. Model SDKs are async, so this fails with TypeError: object async_generator can't be used in 'await' expression:

# Wrong
def gen():
    async for chunk in stream:   # SyntaxError / TypeError territory
        yield chunk.content

# Right
async def gen():
    async for chunk in stream:
        yield chunk.content.encode("utf-8")

Mistake 2: Forgetting to Encode to Bytes

StreamingResponse wants bytes, not str. Python raises AssertionError or silently mangles UTF-8 for non-ASCII text. Always .encode("utf-8") — this is what breaks Japanese or emoji output in production while English tests pass.

Mistake 3: Letting nginx Buffer the Response

nginx, Cloudflare, and most reverse proxies buffer upstream responses by default. Your app streams perfectly, and the user still sees everything at once. Set the header (already included above) and configure nginx:

proxy_buffering off;
proxy_cache off;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_read_timeout 3600s;

Mistake 4: Streaming Without Any Heartbeat

Some providers pause for seconds between tokens on long reasoning chains. Idle proxies then kill the connection. Send a comment frame every 15 seconds to keep it warm:

async def with_heartbeat(gen, interval=15):
    import asyncio
    task = asyncio.create_task(gen.__anext__(), name="next-token")
    while True:
        try:
            chunk = await asyncio.wait_for(asyncio.shield(task), timeout=interval)
        except asyncio.TimeoutError:
            yield b": ping\n\n"          # SSE comment — ignored by the browser
            continue
        except StopAsyncIteration:
            break
        yield chunk
        task = asyncio.create_task(gen.__anext__())

Mistake 5: Ignoring Backpressure

An async generator that yields faster than the client consumes will balloon server memory. Streaming usually self-regulates because TCP applies backpressure, but if you buffer tokens into a list before yielding (a common pattern when you also need to save the full message to a database), cap the buffer size or flush incrementally.

Bottom Line

Streaming is the cheapest possible latency win for an LLM app — it changes no model, costs nothing extra, and turns a 20-second wait into a 1-second response. The pattern is small enough to memorize: an async generator that yields data: {chunk}\n\n, wrapped in StreamingResponse with text/event-stream and X-Accel-Buffering: no.

Get those four things right and most of the remaining work is ordinary engineering: handle disconnects, add a heartbeat for slow models, and test through your actual proxy rather than through curl alone. Start with the skeleton in section 4, swap in whichever provider from section 5 you already pay for, and measure your time to first token before and after — that number is what your users actually feel.