LLM Session Tracing with OpenTelemetry

What is LLM Session Tracing with OpenTelemetry?

LLM session tracing with OpenTelemetry captures, analyzes, and visualizes the execution paths of generative AI applications using a standardized observability format. Unlike traditional software architectures where requests follow predictable synchronous routes through microservices, AI systems, especially agentic architectures, multi-prompt chains, and retrieval-augmented generation (RAG) pipelines, often feature non-deterministic, long-running, and highly branched execution flows.

OpenTelemetry (OTel), an open-source observability framework developed by the Cloud Native Computing Foundation (CNCF), is the standard for generating, collecting, and exporting telemetry data (metrics, logs, and traces) for cloud-native applications. With the rise of large language models (LLMs), the OpenTelemetry community introduced GenAI semantic conventions. These conventions standardize how AI-specific telemetry data, such as prompt text, completion outputs, token usage, model parameters, and generation latencies, is recorded and structured across distributed systems.

LLM session tracing binds these operations into a "session." A session encapsulates the entire user interaction lifecycle, which may span multiple discrete traces. For instance, a single user chat session might trigger an intent classification prompt, a vector database retrieval query, a reranking step, and a streaming generation prompt. By combining OpenTelemetry's distributed tracing capabilities with GenAI semantic attributes, engineering teams can visualize these nested operations. This visibility supports debugging hallucinations, optimizing prompt latency, identifying inefficient retrieval steps, and attributing token costs to specific user actions or system components.

OpenTelemetry GenAI Semantic Conventions

To ensure interoperability across different observability backends (like Datadog, Honeycomb, or purpose-built AI platforms such as Klu.ai, often reached through an AI gateway), OpenTelemetry established the GenAI Semantic Conventions. These conventions dictate a standardized vocabulary for span attributes, events, and metrics when instrumenting LLM applications, so that regardless of the underlying model provider (OpenAI, Anthropic, Google) or orchestration framework (LangChain, LlamaIndex), the telemetry data remains consistent and queryable. The attribute registry is the authoritative reference for every attribute name below; it changes often enough that instrumentations should be checked against it directly rather than against older blog posts or SDK examples.

One important caveat before adopting them: as of mid-2026, every GenAI span, metric, event, and attribute still carries "Development" stability status; none is marked Stable. In June 2026 the conventions moved out of the main semantic-conventions repository (deprecated there in v1.42.0, published June 12, 2026) into a dedicated open-telemetry/semantic-conventions-genai repository. The vocabulary is the industry's convergence point, but it is still moving, so instrumentations should pin convention versions and expect renames.

The conventions categorize data into several key areas:

Provider and Model Attributes

Attributes such as gen_ai.provider.name (e.g., openai, anthropic) and gen_ai.request.model define the foundational context of the span, with gen_ai.response.model capturing the exact model version the provider actually served. This allows teams to filter traces by provider or model version to compare performance and latency regressions across deployments. Note the rename: gen_ai.provider.name replaced the deprecated gen_ai.system in semantic conventions v1.37.0 (August 2025), and telemetry from older framework instrumentations still emits the old attribute, so queries during migration should match both.

Prompt and Completion Context

Standardized attributes like gen_ai.request.temperature, gen_ai.request.top_p, and gen_ai.request.max_tokens capture the hyperparameter configuration for each call. Payload capture has changed generations: older instrumentations logged prompts and completions as span events (gen_ai.content.prompt and gen_ai.content.completion), while the current conventions record structured gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions attributes, gated behind the OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental environment variable. Due to privacy concerns and payload size limits, full-text capture is opt-in in most SDKs.

Token Usage and Economics

Cost management is one of the core aspects of LLM observability. The registry defines gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (renamed from the earlier prompt_tokens/completion_tokens fields), plus finer-grained attributes such as gen_ai.usage.cache_read.input_tokens and gen_ai.usage.reasoning.output_tokens for billed thinking tokens on reasoning models. There is no separate total_tokens attribute in the registry; a combined total is something a backend computes by summing input and output itself, not a value instrumentations set directly. When propagated through the OpenTelemetry pipeline, these attributes allow for cost attribution down to the individual span, trace, or session level, enabling per-token cost tracking for AI features.

Error Handling and Finish Reasons

When generation fails or halts unexpectedly, OpenTelemetry conventions standardize the reporting of errors. The gen_ai.response.finish_reasons attribute is a string array, one entry per generation received (e.g., ["stop"], ["length"], ["content_filter"]), and provides immediate context on why the model stopped generating. Standard span error statuses map directly to HTTP/RPC error codes, so LLM timeouts or rate limits are treated as first-class citizens in existing APM alerts. Tracked over time, these attributes also feed drift and degradation monitoring, surfacing shifts in finish reasons or latency before they affect users.

Agent, Tool, and Evaluation Conventions

The conventions have expanded beyond single completion calls to cover agentic systems. Agent spans (gen_ai.agent.name, gen_ai.agent.id, and since v1.41.0 in April 2026, separate client and internal spans for agent invocation) capture multi-step agent runs; tool-execution spans record function calls and their arguments; and draft conventions for Model Context Protocol operations trace MCP tool calls across process boundaries. The gen_ai.evaluation.result event (added in v1.38.0, October 2025) links evaluation scores, such as an LLM-as-judge verdict, to the exact span that produced the evaluated output, closing the loop between tracing and LLM evaluation.

Hierarchical Tracing Architecture: Sessions -> Traces -> Spans -> Events

Effective LLM observability requires thinking beyond single HTTP requests. It requires a hierarchical architecture that contextualizes granular events within broader workflows. OpenTelemetry natively supports traces, spans, and events, but for AI systems, this hierarchy adds the logical concept of a "Session" as the highest-level grouping:

Session > Trace > Span > Event

1. Sessions: A session represents a contiguous user interaction, such as a multi-turn chat conversation or a complex asynchronous document processing job. Sessions aggregate multiple traces, allowing developers to see how conversational state evolves over time and how earlier context impacts later generations.
2. Traces: A trace represents a single, end-to-end operation within a session, such as processing a single user message. It acts as the root container, defining the total latency of the request from the moment the user clicks "send" to the moment the final token is rendered.
3. Spans: Spans are the individual units of work within a trace. In a RAG pipeline, you might have discrete spans for intent extraction, vector database embedding, vector search, and the final LLM generation call. Spans measure the latency of these specific operations and contain the GenAI semantic attributes discussed earlier.
4. Events: Events are timestamped logs attached to a specific span. In LLM tracing, events measure streaming performance. For example, logging a first_token_received event allows teams to calculate the Time to First Token (TTFT), a key metric for perceived application responsiveness.

LLM Tracing vs. Traditional APM Metrics

Traditional Application Performance Monitoring (APM) tools measure database query latencies, CPU utilization, and HTTP error rates well, but they often fall short when applied out of the box to GenAI workloads. LLM applications introduce performance bottlenecks and failure modes that traditional APM was not built to capture.

Traditional APM treats an LLM API call as a slow external HTTP request. LLM session tracing, using OpenTelemetry GenAI conventions, opens that black box, showing why the request was slow (a large input context, for example), how much it cost, and what was actually generated.

Instrumentation Code Example

Implementing LLM session tracing involves utilizing the OpenTelemetry SDK. While manual instrumentation is possible, many modern frameworks provide auto-instrumentation libraries. Below is an example demonstrating manual instrumentation using the Python OpenTelemetry SDK, explicitly applying GenAI semantic conventions to trace a simple completion call.

import os
from opentelemetry import trace
from opentelemetry.trace.status import Status, StatusCode
from openai import OpenAI

# Initialize the OpenTelemetry Tracer
tracer = trace.get_tracer('klu.llm_tracer', '1.0.0')

client = OpenAI(api_key=os.environ.get('OPENAI_API_KEY'))

def generate_answer(user_query, session_id):
    # Start a new trace span for the LLM operation
    with tracer.start_as_current_span('llm.generate_response') as span:

        # Set standard GenAI Semantic Attributes
        # (older instrumentations emit 'gen_ai.system' instead of provider.name)
        span.set_attribute('gen_ai.provider.name', 'openai')
        span.set_attribute('gen_ai.request.model', 'gpt-5')
        span.set_attribute('gen_ai.request.temperature', 0.7)

        # Custom attribute for Session tracking
        span.set_attribute('app.session.id', session_id)

        # Record the prompt as an event (or attribute depending on backend limits)
        span.add_event('gen_ai.content.prompt', {'content': user_query})

        try:
            response = client.chat.completions.create(
                model='gpt-5',
                messages=[
                    {'role': 'system', 'content': 'You are a helpful assistant.'},
                    {'role': 'user', 'content': user_query}
                ],
                temperature=0.7,
            )

            completion_text = response.choices[0].message.content
            usage = response.usage

            # Record Token Usage (no separate "total_tokens" attribute exists;
            # sum input_tokens + output_tokens if a backend needs a total)
            span.set_attribute('gen_ai.usage.input_tokens', usage.prompt_tokens)
            span.set_attribute('gen_ai.usage.output_tokens', usage.completion_tokens)

            # Record Finish Reason (registry defines this as a string array)
            span.set_attribute('gen_ai.response.finish_reasons', [response.choices[0].finish_reason])

            # Record the Completion text
            span.add_event('gen_ai.content.completion', {'content': completion_text})

            span.set_status(Status(StatusCode.OK))
            return completion_text

        except Exception as e:
            # Record errors properly in the trace
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.record_exception(e)
            raise e

# Example usage linking to a distinct session
session_id = 'sess_987654321'
answer = generate_answer('Explain OpenTelemetry tracing in one sentence.', session_id)

This instrumentation tracks every AI generation, measures its performance, and costs it accurately, outputting telemetry data that any OpenTelemetry-compatible observability backend can read.

Frequently Asked Questions

Are the OpenTelemetry GenAI semantic conventions stable? No. As of mid-2026 every GenAI convention remains in "Development" status, and the conventions moved to a dedicated repository (open-telemetry/semantic-conventions-genai) in June 2026 with no stabilization timeline committed. In practice this means attribute renames still happen (as gen_ai.system to gen_ai.provider.name did in 2025), different frameworks emit different convention generations, and robust pipelines normalize both old and new attribute names at ingestion rather than assuming a single vocabulary.

Does capturing prompts and completions in OpenTelemetry violate data privacy? It can, which is why payload capture should always be configurable. Many organizations implement redaction pipelines or PII (Personally Identifiable Information) scrubbing before exporting OpenTelemetry data to their observability backend. The GenAI semantic conventions allow you to disable prompt and completion logging while still retaining telemetry like token counts and latency.

How does LLM tracing handle streaming responses? Streaming is handled using Span Events. While the main Span measures the total duration of the request, an event is logged the moment the first chunk is received (Time to First Token). Subsequent chunks can be aggregated, and when the stream closes, the total token count and final assembled completion are appended to the Span attributes before it is closed.

Can OpenTelemetry connect my frontend user sessions to backend LLM calls? Yes. OpenTelemetry relies on context propagation (usually via HTTP headers like traceparent). By instrumenting your frontend application, passing the Trace ID to your backend API, and continuing the trace, you can achieve a complete end-to-end visualization of the user's journey from browser click to backend database retrieval to LLM generation.

Are there automated ways to instrument LLM apps without writing manual code? Yes. The OpenTelemetry ecosystem includes auto-instrumentation libraries for popular AI frameworks and SDKs, such as OpenAI, LangChain, and LlamaIndex. These libraries hook into the underlying HTTP requests and inject the required GenAI semantic attributes, reducing the boilerplate code needed for observability.