LLM Observability in Python: Trace Every LLM Call with OpenTelemetry (2026)

A user says your AI assistant gave a wrong answer. You open your logs and find one line: POST /chat 200 in 4.2s. That tells you nothing. Which of the six LLM calls in that request was slow? How many tokens did the retrieval step inject? Did the model call the right tool, or did it hallucinate a function name and get retried three times before failing silently?

This is the gap LLM observability fills. Instead of one log line per request, you get a tree of spans — retrieval, each model call, each tool execution — with prompts, token counts, latency, and costs attached to each. This guide builds that in Python using OpenTelemetry, the vendor-neutral standard that most LLM tooling now speaks.

1. What LLM Observability Actually Means

Three signals, and people routinely confuse them:

• Traces show one request end to end as a tree of timed spans. This is what you use to debug a single bad answer — it answers "what exactly happened here?"
• Metrics are aggregates over time: p95 latency, tokens per minute, cost per day, error rate. This is what you alert on.
• Logs are unstructured events. Still useful, still not what tells you why a chain took nine seconds.

LLM apps need traces far more than conventional services do, for one specific reason: a single user request fans out into many model calls, and each one is slow, expensive, and nondeterministic. A conventional endpoint does one database query and returns. An agent loop might make fifteen calls, any of which can be the bottleneck or the source of the error, and the aggregate latency number hides all of it.

The OpenTelemetry terminology map is small. A span is one timed operation. A trace is the tree of spans sharing a trace ID. Attributes are key-value data attached to a span — this is where prompts, model names, and token counts go. Semantic conventions are agreed attribute names, so your Langfuse backend and your Jaeger backend show the same field.

2. The Minimum Viable Span

Before any tooling, decide what every LLM span must carry. Six fields cover nearly every debugging session:

3. Manual Tracing with OpenTelemetry

Start here even if you plan to use auto-instrumentation, because you will write manual spans for your own business logic regardless. Install the SDK and an exporter:

pip install opentelemetry-sdk opentelemetry-exporter-otlp

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm-app")

Now wrap a call. The pattern that matters is start_as_current_span as a context manager — it records duration automatically and, critically, sets the span as the parent for anything nested inside it:

from opentelemetry.trace import Status, StatusCode
import time

def ask(messages, model="gpt-4o", user_id=None):
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("user.id", user_id or "anonymous")

        t0 = time.perf_counter()
        try:
            resp = client.chat.completions.create(model=model, messages=messages)
        except Exception as e:
            span.record_exception(e)
            span.set_status(Status(StatusCode.ERROR, str(e)))
            raise

        out = resp.choices[0].message.content
        u = resp.usage
        span.set_attribute("gen_ai.usage.input_tokens", u.prompt_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", u.completion_tokens)
        span.set_attribute("gen_ai.response.finish_reason",
                           resp.choices[0].finish_reason or "")
        span.set_attribute("llm.first_token_ms",
                           int((time.perf_counter() - t0) * 1000))
        return out

Two details people miss. record_exception plus an error status is what makes failures visible in the trace UI rather than just appearing as a short span. And finish_reason catches silent truncation — a response cut off at the token limit returns HTTP 200 and looks completely healthy in your logs.

One more: always attach attributes after the call when they depend on the response, but set the model and user ID before. If the call throws, you still want to know which model and which user were involved.

4. Auto-Instrumentation in One Line

Writing that wrapper for every provider and framework is tedious, and the libraries already did it. OpenLIT instruments OpenAI, Anthropic, LangChain, LlamaIndex, and most vector DBs with standard attribute names:

pip install openlit

import openlit

openlit.init(
    otlp_endpoint="http://localhost:4318",
    application_name="support-bot",
    capture_message_content=True,   # prompts + completions; see redaction note above
    environment="production",
    disable_metrics=False,
)
# import your app AFTER init
from myapp import create_app

Order matters: openlit.init() must run before the SDK clients are constructed, because instrumentation works by wrapping the client methods. If you import and instantiate your OpenAI client at module top and call init() at the bottom of the file, you get zero spans and no error.

OpenInference (Arize) is the other major option and is what Arize Phoenix expects; both emit compatible gen_ai.* attributes. Pick based on your backend, not on features — they cover roughly the same surface.

Either way you still write manual spans for your own code: the retrieval function, the business rule that decided which documents to fetch, the post-processing step. Auto-instrumentation covers vendor calls; it knows nothing about your application.

5. Tracing Agent Loops and RAG

This is where tracing earns its keep. A flat list of spans cannot express "the agent called search, then called it again with a reworded query, then gave up." A nested tree can:

def run_agent(question, user_id):
    with tracer.start_as_current_span("agent.run") as root:
        root.set_attribute("user.id", user_id)
        root.set_attribute("agent.question", question)

        for step in range(MAX_STEPS):
            with tracer.start_as_current_span(f"agent.step.{step}") as step_span:
                resp = ask(build_messages(history), user_id=user_id)  # nested llm.call
                step_span.set_attribute("agent.step.index", step)

                if tool_call := parse_tool_call(resp):
                    with tracer.start_as_current_span(
                        f"tool.{tool_call.name}"
                    ) as tool_span:
                        tool_span.set_attribute("tool.name", tool_call.name)
                        tool_span.set_attribute("tool.args", tool_call.args)
                        result = TOOLS[tool_call.name](**tool_call.args)
                        tool_span.set_attribute("tool.result_len", len(str(result)))
                    history.append(tool_result_message(result))
                else:
                    step_span.set_attribute("agent.terminated", True)
                    break
        return final_answer(history)

Because ask() uses start_as_current_span, its span automatically becomes a child of whatever span is active — no explicit parent wiring needed. That is the whole reason to use the context-manager form instead of tracer.start_span().

Once you have this tree, the diagnostic questions become trivial. Which tool is slow? Sort child spans by duration. Why did it loop five times? Look at the tool results — usually a tool returned an empty string and the model politely retried with a different phrasing. How much did this conversation cost? Sum tokens across descendants.

For RAG, put the retrieval in its own span and record what actually came back, not just that retrieval happened:

with tracer.start_as_current_span("rag.retrieve") as s:
    s.set_attribute("rag.top_k", TOP_K)
    s.set_attribute("rag.query", question)
    docs = vectorstore.similarity_search(question, k=TOP_K)
    s.set_attribute("rag.num_docs", len(docs))
    s.set_attribute("rag.doc_ids", [d.metadata.get("id", "?") for d in docs])
    s.set_attribute("rag.max_score", max(d.score for d in docs) if docs else 0.0)

rag.max_score is the single most useful RAG metric you can add. When a low maximum similarity score coincides with a bad answer, the model was not the problem — retrieval was, and you can prove it instead of arguing about it.

6. Per-User Cost Attribution

Token counts on spans are only useful if you can roll them up. Because every span carries user.id, you can compute cost per user directly from trace data rather than building a separate billing pipeline:

# Price table: USD per 1M tokens. Check your provider's pricing page —
# these move, and hardcoding stale prices produces confidently wrong invoices.
PRICES = {
    "gpt-4o":      {"in": 2.50, "out": 10.00},
    "gpt-4o-mini": {"in": 0.15, "out": 0.60},
}

def cost_usd(model, in_tok, out_tok):
    p = PRICES.get(model)
    if not p:
        return None           # unknown model: record it, do not guess
    return (in_tok * p["in"] + out_tok * p["out"]) / 1_000_000

# Span processor that accumulates cost into a metric as spans finish
from opentelemetry.sdk.trace.export import SpanProcessor, SpanExporter

class CostProcessor(SpanProcessor):
    def on_end(self, span):
        attrs = span.attributes or {}
        model = attrs.get("gen_ai.request.model")
        c = cost_usd(model,
                     attrs.get("gen_ai.usage.input_tokens", 0),
                     attrs.get("gen_ai.usage.output_tokens", 0))
        if c is None:
            return
        llm_cost.add(c, {"model": model, "user": attrs.get("user.id", "anon")})

provider.add_span_processor(CostProcessor())

Returning None for unknown models rather than defaulting to zero is deliberate. A missing price silently under-reports cost forever, and you will trust the dashboard right up until the invoice disagrees.

Set a budget alert on this within your first week. The most common production surprise in LLM apps is not a crash — it is one user triggering an agent loop that makes 200 calls and quietly costs more than the rest of the day's traffic combined.

7. Choosing a Backend

Your traces have to land somewhere. The honest summary:

Start with Phoenix locally — pip install arize-phoenix, run phoenix serve, point the OTLP endpoint at localhost:4318 — and you have a working trace UI in about two minutes with no account. Move to Langfuse or your existing APM stack when you need retention, alerting, and a shared team view.

The point of using OpenTelemetry rather than a vendor SDK is that this migration is a config change. Swap the exporter endpoint; keep every span you wrote.

8. Connecting Traces to Evaluation

Traces and evals are the same data viewed differently. A trace is one production request; an eval is a curated set of requests you re-run against your changes. If your tracing already captures input, output, retrieved document IDs, and tool calls, you can export real failing traces into an eval dataset instead of inventing test cases:

# Pull slow or errored traces, turn them into regression cases
def harvest_regression_cases(trace_store, min_latency_ms=5000, limit=50):
    cases = []
    for t in trace_store.query(min_duration_ms=min_latency_ms, limit=limit):
        llm = next(s for s in t.spans if s.name == "llm.call")
        cases.append({
            "input": llm.attributes["gen_ai.prompt"],
            "expected_contains": None,     # fill in by hand — this is the human step
            "note": f"slow ({t.duration_ms}ms), model={llm.attributes['gen_ai.request.model']}",
        })
    return cases

The human step matters: an eval without an expected answer is just a snapshot. Use traces to find candidates, then write the assertion yourself. Full setup is covered in our LLM evaluation guide with pytest.

9. Common Mistakes

Mistake 1: Logging Prompts Without a Redaction Policy

The default in most instrumentation libraries is to capture everything. Prompts carry customer emails, internal document text, and API-shaped secrets. Write a redaction hook before you enable content capture in production, not after the first support ticket.

Mistake 2: One Flat Span Per Request

A single span named chat with a 6-second duration reproduces the log line you started with. Nest the model calls, tool executions, and retrieval. Depth is the entire value.

Mistake 3: No Sampling Strategy, Then Bill Shock

At high volume, exporting 100% of traces — each carrying a multi-kilobyte prompt — gets expensive fast. Use tail sampling: keep all errors and all slow traces, sample the healthy ones at 5–10%. You keep every trace you would actually investigate.

Mistake 4: Forgetting That Streaming Changes Timing

With streaming, the provider response object often does not include token usage until the stream completes. If you set token attributes immediately after the first chunk, you record zero. Accumulate usage in the stream's completion handler, and record time-to-first-token separately from total duration — they fail for different reasons.

Mistake 5: Hardcoding Prices

Model pricing changes several times a year. Put the price table in config or a database with an effective_date, and make unknown models report as unknown rather than zero.

Mistake 6: Tracing in Production Only

Developers debug locally with print statements and then cannot read production traces because the span structure is unfamiliar. Run the same tracing in dev against Phoenix; the skill transfers.

Mistake 7: Confusing Trace IDs With Conversation IDs

A trace covers one request. A conversation spans many. If you want to see a whole chat session, you need your own session.id attribute threaded through every request — the trace backend will not infer it.

Bottom Line

LLM observability is not a dashboard project. It is the difference between "the bot was wrong" and "retrieval returned one weakly-matching document, the model called search_orders with a misspelled argument, retried twice, and answered from a stale chunk." You cannot fix the first; you can fix the second in an afternoon.

Do it in this order: stand up Phoenix locally and get auto-instrumentation emitting spans (30 minutes). Add manual spans around your retrieval, tools, and agent steps (a couple of hours). Attach user.id to everything and start accumulating per-user cost (30 minutes). Set a sampling policy before you go to production volume, not after the first storage bill.