Trace LLM Application

This guide introduces tracing a customer-support chatbot with the middleware-llmobs SDK, then connects request-level tracing with evaluation and production monitoring. It helps you inspect the operations behind a response and identify where quality or performance needs attention.

By the end, you will have:

• OpenAI calls traced automatically as LLM spans.
• Application steps grouped into a single trace.
• Conversations grouped by session and attributed to users.
• Evaluation results attached to production traffic.
• A workflow for identifying low-quality responses and improving prompts or model behavior.

The setup follows this flow:

Instrument → Trace → Add Context → Evaluate → Monitor → Improve

1. Get your first trace#

The simplest way to start is to call register(auto_instrument=True), make a standard OpenAI request, and view the resulting LLM trace in Middleware.

Install the Middleware SDK, the OpenAI instrumentation, and the OpenAI client:

pip install middleware-llmobs openinference-instrumentation-openai openai

Export your Middleware endpoint, key, and OpenAI key. Replace <MW_UID> and <MW_API_KEY> with the values from your Middleware account:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<MW_UID>.middleware.io:443"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=<MW_API_KEY>,X-Trace-Source=openinference"
export OPENAI_API_KEY="<your-openai-key>"

from middleware.llmobs import register
import openai

# Register once at startup. auto_instrument=True picks up the installed
# OpenAI instrumentation, so every chat.completions.create() becomes an LLM span.
providers = register(service_name="support-chatbot", auto_instrument=True)
client = openai.OpenAI()

# AUTO: no decorator, no manual span. Auto-instrumentation spans this call
# on its own, capturing prompt, response, model, and token usage.
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a concise technical writer."},
        {"role": "user", "content": "Summarise OpenTelemetry in one sentence."},
    ],
    temperature=0.2,
)
print(response.choices[0].message.content)

# Flush so the span leaves the process before the script exits.
providers.tracer.force_flush()

In Middleware, open LLM Observability and select the support-chatbot service. You'll see an LLM span carrying the model name, input messages, assistant response, and prompt/completion token usage, with no manual instrumentation at all.

2. Structure the chatbot into spans#

A real chatbot does more than one call. Break it into named steps so each shows up as its own span under one trace. You get a tracer from the providers object that register() returns, then decorate your functions:

• @tracer.chain marks a sequential workflow or orchestration step. Use it for the top-level pipeline.
• @task marks a generic step inside it.

Both are for your own (non-model) code. The OpenAI call is spanned automatically by auto-instrumentation, you don't decorate it.

from middleware.llmobs import register, task
import openai

providers = register(service_name="support-chatbot", auto_instrument=True)
tracer = providers.tracer.get_tracer(__name__)   # OpenInference tracer: .chain / .agent / .tool
client = openai.OpenAI()

SYSTEM_PROMPT = (
    "You are a helpful customer support agent for Acme Corp. "
    "Answer concisely, friendly, and accurately. If you don't know, say so."
)

@task(name="generate_response")   # MANUAL: @task spans this step
def generate_response(user_message: str) -> str:
    # AUTO: auto-instrumentation spans this OpenAI call, nested under generate_response
    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    return completion.choices[0].message.content

@tracer.chain   # MANUAL: chain span for the overall workflow; everything below nests under it
def support_chatbot(user_message: str) -> str:
    return generate_response(user_message)

print(support_chatbot("How do I reset my password?"))
providers.tracer.force_flush()

Calling support_chatbot nests the spans into one trace. The chain and task decorators open your two named spans; auto-instrumentation adds the LLM span underneath, no manual span needed:

chain: support_chatbot           (manual, @tracer.chain)   root span
  └── task: generate_response    (manual, @task)
        └── chat completion       (auto)                    model, messages, response, token usage

Open a trace to see that structure as a waterfall, with the chain span over the task span over the auto-instrumented LLM call:

3. Add session and user context#

To follow a multi-turn conversation and attribute traffic to a customer, wrap a turn in using_session, using_user, and using_metadata. Every span created inside inherits that context.

from middleware.llmobs import using_session, using_user, using_metadata

def handle_turn(user_message: str, customer_id: str, thread_id: str) -> str:
    with using_session(session_id=thread_id), \
         using_user(user_id=customer_id), \
         using_metadata({"feature": "support_chat"}):
        return support_chatbot(user_message)

# Two turns sharing one thread are grouped into a single session in the UI.
handle_turn("How do I reset my password?", customer_id="user_123", thread_id="conv_abc")
handle_turn("It still won't let me in.",   customer_id="user_123", thread_id="conv_abc")

All spans from a thread share a session ID, so you can open a conversation and follow every turn in order. Filtering by user ID shows one customer's traffic, latency, and cost across sessions. The feature metadata is what you'll scope evaluations to in the next step.

4. Evaluate the responses#

Tracing shows you what happened. Evaluation tells you whether it was any good. Two ways to score this chatbot in Middleware:

Server-side, no code. Create an LLM-as-a-judge evaluator in the UI that scores every support response automatically. In LLM Traces → Evaluation → Add Evaluator, write a judge that rates support quality (answers the question, friendly tone, accurate), set a Score output with a passing threshold, and scope it with an Evaluate On filter so it only runs on this chatbot's spans. See Server-side evaluations.

In code. Score a response yourself and submit it with submit_evaluation, or run your own judge with LLMJudge:

from middleware.llmobs import submit_evaluation

@tracer.chain
def support_chatbot(user_message: str) -> str:
    answer = generate_response(user_message)
    submit_evaluation(
        label="answered",
        value=bool(answer.strip()),     # bool -> boolean
        assessment="pass" if answer.strip() else "fail",
    )
    return answer

For a quality judge that grades the answer with a model, or for deterministic checks, see Client-side Evaluations.

5. Monitor and iterate#

Once a server-side evaluator is published, scores attach to spans as they arrive. Your ongoing loop:

• Watch the trend. Evaluation scores export as metrics (gen_ai.evaluations.*); chart the pass rate and alert when it drops.
• Review the failures. Filter traces by gen_ai.evaluation.verdict = fail (or a low score) and read the judge's reasoning next to the input and output that produced it.
• Fix the cause. Adjust the system prompt, model, or retrieval, then watch the pass rate on the next batch of traffic.

Production data feeds evaluation, evaluation surfaces what to fix, and the fix deploys back to production.