---
title: "Middleware SDK"
description: "Install the middleware-llmobs Python SDK to trace LLM apps with one register() call, auto-instrument OpenAI, Anthropic and LangChain, and run evaluations."
section: "LLM Observability"
breadcrumb: "LLM Observability > Middleware SDK"
canonical: "https://docs.middleware.io/features/llm/middleware-sdk"
---
# Middleware SDK

`middleware-llmobs` is Middleware’s first-party LLM Observability SDK for Python, built on OpenInference and OpenTelemetry. It does two things:

- **Tracing** — a one-line `register()` configures an OpenTelemetry `TracerProvider` that auto-instruments your AI/ML libraries and exports OpenInference LLM spans to your Middleware collector.
- **Evaluations** — attach scores, verdicts, and LLM-as-judge results to your spans and traces, exported as OTel logs and metrics so you can monitor quality and cost alongside performance.

> **Note:** **Transport:** The SDK exports over **OTLP/HTTP (HTTP/protobuf) only**. gRPC is intentionally disabled — requesting it raises `NotImplementedError`.

Looking for end-to-end, copy-paste examples? See the [Cookbooks](/features/llm) for recipes covering tracing, tool/agent calls, RAG, sessions, and evaluations.

## Before you begin

You’ll need:

- A Python LLM app you can modify. Python `>=3.10, <3.15` is required.
- Your **Middleware UID** (for the tenant ingest endpoint) and **Middleware API key** (for the `Authorization` header).
- One or more **OpenInference instrumentation packages** for the providers and frameworks you use. The SDK has **zero LLM-provider dependencies** — it never imports `openai`, `anthropic`, etc. You bring your own provider client and install the matching instrumentation (see Step 2 below).

## 1. Install the SDK

```bash
pip install middleware-llmobs
```

## 2. Install instrumentation for your models and frameworks

Auto-instrumentation is **opt-in per library**: `register(auto_instrument=True)` only traces a library when its OpenInference instrumentation package is installed. Install one package for **each** LLM provider, vector store, and agent framework you want traced.

The packages follow the naming pattern `openinference-instrumentation-<library>`. Install one for each LLM provider and agent framework you use — for example, to trace OpenAI calls made from a LangChain app, install both:

```bash
# LLM provider
pip install openinference-instrumentation-openai

# Agent / orchestration framework
pip install openinference-instrumentation-langchain
```

A few more examples (Anthropic, Bedrock, LlamaIndex, CrewAI):

```bash
pip install openinference-instrumentation-anthropic
pip install openinference-instrumentation-bedrock
pip install openinference-instrumentation-llama-index
pip install openinference-instrumentation-crewai
```

OpenInference ships instrumentation for many more LLM providers (Anthropic, Bedrock, VertexAI, Gemini, Mistral, Groq, LiteLLM, …) and agent frameworks (LlamaIndex, CrewAI, Haystack, DSPy, AutoGen, Google ADK, OpenAI Agents, Pydantic AI, smolagents, MCP, …). For the full, up-to-date list of packages, see the [OpenInference instrumentation directory](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation).

## 3. Configure your credentials

`register()` reads standard OTLP environment variables. The cleanest setup is to export your endpoint and key, and pass the service name in code:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<MW_UID>.middleware.io:443"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=<MW_API_KEY>,X-Trace-Source=openinference"
export OTEL_SERVICE_NAME="my-llm-app"
```

`Authorization` carries your Middleware API key. `X-Trace-Source=openinference` tells Middleware these spans come from the OpenInference-based SDK so it routes them correctly. The paths `/v1/traces`, `/v1/logs`, and `/v1/metrics` are appended to the endpoint automatically — pass only the base URL.

## 4. Register and auto-instrument

Call `register()` once, as early as possible in your app’s startup, **before** you import or call your LLM libraries:

```python
from middleware.llmobs import register

providers = register(auto_instrument=True)  # batch + HTTP by default
# providers.tracer / providers.logger / providers.meter — hold onto this to flush at shutdown.
```

That single call:

- Builds an OpenTelemetry tracer, logger, and meter provider configured for your Middleware endpoint, and installs them as the OTel globals.
- With `auto_instrument=True`, discovers every installed OpenInference instrumentor and attaches it with GenAI semantic conventions enabled (so spans carry `gen_ai.*` attributes).

Now every instrumented call is traced automatically:

```python
import openai

client = openai.OpenAI()
response = client.chat.completions.create(   # automatically traced
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}],
)
```

> **Note:** **Production tip:** `register()` uses a batch span processor by default (`batch=True`), which exports spans in the background so your application isn’t blocked. Keep batching on in production; use `batch=False` only for quick local verification.

### Explicit configuration

You can pass endpoint, headers, and service name directly instead of using environment variables:

```python
from middleware.llmobs import register

providers = register(
    endpoint="https://<MW_UID>.middleware.io:443",
    headers={"Authorization": "<MW_API_KEY>", "X-Trace-Source": "openinference"},
    service_name="my-llm-app",
    auto_instrument=True,
)
```

> **Warning:** Never hardcode your Middleware ingestion key in committed code. Prefer the `OTEL_EXPORTER_OTLP_HEADERS` environment variable.

### Manual instrumentor wiring

If you’d rather attach instrumentors yourself instead of using `auto_instrument`, register without it and instrument explicitly. This is useful when you want to enable only specific libraries or pass a custom `TraceConfig`:

```python
from middleware.llmobs import register
from openinference.instrumentation import TraceConfig
from openinference.instrumentation.openai import OpenAIInstrumentor

providers = register(service_name="my-llm-app")

OpenAIInstrumentor().instrument(
    tracer_provider=providers.tracer,
    config=TraceConfig(enable_genai_semconv=True),
)
```

### Configuration reference

Each setting can be passed to `register()` or set via an environment variable (the argument takes precedence):

| `register()` argument | Environment variable | Purpose |
| --- | --- | --- |
| `endpoint` | `OTEL_EXPORTER_OTLP_ENDPOINT` | Collector endpoint, e.g. `https://<MW_UID>.middleware.io:443`. |
| `headers` | `OTEL_EXPORTER_OTLP_HEADERS` | Export headers: the `Authorization` key, plus `X-Trace-Source=openinference`. As a dict `{"Authorization": "<key>", "X-Trace-Source": "openinference"}` or env string `Authorization=<key>,X-Trace-Source=openinference`. |
| `service_name` | `OTEL_SERVICE_NAME` | Service identity (defaults to `"default"`). |
| `project_name` | `MW_PROJECT_NAME` | Optional project name (defaults to the service name). |

Other flags accepted by `register()`:

- `auto_instrument` (bool) — discover and attach all installed OpenInference instrumentors.
- `batch` (bool, default `True`) — use a batch span processor; set `False` for a simple processor during local debugging.
- `verbose` (bool, default `True`) — print a configuration summary at startup.
- `set_global_tracer_provider` / `set_global_logger_provider` / `set_global_meter_provider` — each defaults to `True`; set `False` to manage the OTel globals yourself.
- Any extra keyword arguments pass through to the underlying `TracerProvider` (for example a custom `resource=`, `sampler=`, or `id_generator=`).

## 5. Naming traces and enriching spans

Auto-instrumentation adds an LLM child span for each provider call. Wrap your call in a parent span to give the whole trace a clear name:

```python
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("chat"):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Summarise OpenTelemetry in one sentence."}],
    )
```

### Session and user context

The SDK re-exports OpenInference context managers to add session, user, and metadata context to every span created inside them — useful for grouping a multi-turn conversation or attributing traffic to a user:

```python
from middleware.llmobs import using_session, using_user

with using_session(session_id="abc-123"), using_user(user_id="user-456"):
    response = client.chat.completions.create(...)
```

Also available: `using_attributes`, `using_metadata`, `using_tags`, `using_prompt_template`, and `suppress_tracing` (a context manager that temporarily disables tracing).

### Span decorators for pipeline steps

The SDK ships decorators that open a span around a function and tag it with the right GenAI operation. They work on both sync and async functions, and can be used bare (`@task`) or with arguments (`@task(name="...")`):

- `@task` — a generic step in your pipeline.
- `@retriever` — a retrieval (RAG) step. Call `annotate_rag()` inside it to record the query and the documents you got back.
- `@embedding` — an embedding call.

```python
from middleware.llmobs import task, retriever, embedding, annotate_rag

@task
def build_prompt(question: str) -> str:
    return f"Answer concisely: {question}"

@retriever
def get_relevant_docs(question: str):
    docs = [
        {"id": d.id, "score": d.score, "text": d.text}
        for d in vector_db.search(question)
    ]
    annotate_rag(query=question, documents=docs)
    return docs

@embedding
def embed(text: str):
    return embedding_client.create(model="text-embedding-3-small", input=text)
```

You can also call `annotate_rag(query=..., documents=...)` inside your own `start_as_current_span` block when a decorator doesn’t fit. Documents can be plain dicts or any object exposing `id` / `score` / `text` (or `content`) / `name` / `metadata`.

## 6. Evaluations

Evaluations attach a score or verdict to a span or trace, and ship as OTel logs and gauge metrics (`gen_ai.evaluations.*`) over the same HTTP transport. There are three ways in, from lowest to highest level.

### Submit a value you already computed

When you’ve already computed a result, `submit_evaluation` records it. By default it auto-binds to the **active span**, so call it inside your `start_as_current_span` block:

```python
from middleware.llmobs import submit_evaluation

with tracer.start_as_current_span("chat"):
    response = client.chat.completions.create(...)

    submit_evaluation(
        label="response_not_empty",     # must match ^[a-zA-Z][a-zA-Z0-9_]*$
        value=True,                     # int/float (score), bool (boolean), or str (categorical)
        metric_type="boolean",          # inferred from the value if omitted
        assessment="pass",              # optional: "pass" | "fail"
        reasoning="The model returned a non-empty answer.",
        tags={"feature": "chat"},
    )
```

`metric_type` is inferred from the value when omitted: `bool` → `boolean`, `int`/`float` → `score`, `str` → `categorical`. To attach an eval to a specific span instead of the active one, pass `span_id=` and `trace_id=` together (as hex strings), or use `join_on_tag=("key", "value")`. Use `export_current_span()` to capture the current IDs for later submission, and `submit_evaluation_error(label=..., error=...)` to record a failed evaluation so it stays visible in dashboards.

### LLM-as-judge

For quality checks scored by a model, build an `LLMJudge`. Because the SDK never imports a provider, you write a thin client adapter that calls your model and returns its text response. Use `format_schema_for_provider` to turn the SDK’s canonical JSON schema into your provider’s request kwargs:

```python
from openai import OpenAI
from middleware.llmobs import (
    LLMJudge,
    BooleanStructuredOutput,
    EvaluatorContext,
    evaluate_and_submit,
    format_schema_for_provider,
)

openai_client = OpenAI()

def openai_judge_client(messages, model, json_schema=None, model_params=None) -> str:
    kwargs = {"model": model, "messages": messages}
    if model_params:
        kwargs.update(model_params)
    if json_schema:
        kwargs.update(format_schema_for_provider(json_schema, "openai"))
    resp = openai_client.chat.completions.create(**kwargs)
    return resp.choices[0].message.content or ""

toxicity_judge = LLMJudge(
    client=openai_judge_client,
    model="gpt-4o-mini",
    name="toxicity",
    system_prompt="You are a strict content-safety classifier. Respond ONLY in the requested JSON shape.",
    user_prompt="Classify this message as toxic (true) or not.\n\nMessage:\n{{output}}",
    structured_output=BooleanStructuredOutput(
        description="True if the message is toxic.",
        reasoning=True,
        pass_when=False,        # the eval PASSES when the value is False (= not toxic)
    ),
    model_params={"temperature": 0.0, "max_tokens": 200},
)

ctx = EvaluatorContext(input=customer_msg, output=agent_reply)
result = evaluate_and_submit(toxicity_judge, ctx)   # runs the judge AND submits in one call
```

The `user_prompt` uses `{{field.path}}` placeholders rendered from the `EvaluatorContext`: `{{input}}`, `{{output}}`, `{{expected_output}}`, `{{metadata.foo}}`, and so on. `format_schema_for_provider` supports `"openai"`, `"azure_openai"`, `"anthropic"`, `"vertexai"`, and `"bedrock"`.

Choose a structured-output type to drive both the JSON schema and the pass/fail logic:

| Type | Judge returns | Pass logic |
| --- | --- | --- |
| `BooleanStructuredOutput(description, reasoning=False, pass_when=None)` | `true` / `false` | `pass_when` sets which value passes. |
| `ScoreStructuredOutput(description, min_score, max_score, min_threshold=None, max_threshold=None)` | a number | thresholds define the passing range. |
| `CategoricalStructuredOutput(categories={value: desc, ...}, pass_values=None)` | one category | `pass_values` lists the passing ones. |

For async judges, use `AsyncLLMJudge` with an `async def` client adapter and `await aevaluate_and_submit(judge, ctx)`. Running several async judges with `asyncio.gather(...)` on the same span attaches multiple evaluations to one trace.

### Custom evaluators

For non-LLM checks (regex, JSON-validity, similarity, thresholds), use the `@evaluator` decorator. It captures exceptions into the result instead of propagating them, and normalizes the return value:

```python
from middleware.llmobs import evaluator, EvaluatorContext, evaluate_and_submit

@evaluator                       # or @evaluator(name="custom_name")
def has_answer(ctx: EvaluatorContext):
    return bool(ctx.output.strip())   # bool→boolean, int/float→score, str→categorical, None→skip

evaluate_and_submit(has_answer, EvaluatorContext(input=question, output=answer))
```

For stateful or configurable evaluators, subclass `BaseEvaluator` (or `AsyncBaseEvaluator`) and implement `evaluate(self, ctx)`:

```python
from middleware.llmobs import BaseEvaluator, EvaluatorResult

class SimilarityEvaluator(BaseEvaluator):
    def __init__(self, threshold=0.8):
        super().__init__(name="similarity")
        self.threshold = threshold

    def evaluate(self, ctx):
        score = cosine_sim(ctx.output, ctx.expected_output)
        return EvaluatorResult(
            value=score,
            assessment="pass" if score >= self.threshold else "fail",
        )
```

> **Note:** `evaluate(...)` may run concurrently across threads or tasks — don’t mutate instance attributes inside it; use locals. Use `@async_evaluator` (with `aevaluate_and_submit`) for `async def` evaluators.

`EvaluatorContext` carries the fields your evaluators read: `input`, `output`, `expected_output`, `retrieved_contexts`, `metadata`, `tags`, `span_id`, and `trace_id`.

## 7. Flush before short-lived processes exit

For scripts and serverless functions, drain the processors before the process ends, or spans and evaluations may be lost:

```python
from middleware.llmobs import flush_evaluations

providers.tracer.force_flush()   # spans
flush_evaluations()              # eval logs + metrics (if you submitted any)
```

Long-running servers don’t need this on every request — flushing is handled by the batch processor and at shutdown.

## View your data in Middleware

Run your app and trigger at least one LLM request. Open the **LLM Observability** section in Middleware to see your traces, spans, and evaluation results. Drill into a trace to inspect prompts, responses, token usage, and any evaluations attached to it.

## Troubleshooting & Common Pitfalls

- **Nothing shows up in the UI** → Confirm `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` are correct, that `register()` runs before your LLM calls, and that you actually execute a request after startup.
- **No spans from a provider or framework** → `auto_instrument=True` only traces libraries whose `openinference-instrumentation-*` package is installed. Install the matching package ([Step 2](#2-install-instrumentation-for-your-models-and-frameworks)) for each one you use.
- **Spans missing from a short script** → Call `providers.tracer.force_flush()` (and `flush_evaluations()` if you submitted evals) before the process exits.
- **gRPC errors** → This SDK is HTTP/protobuf only. Don’t pass `protocol="grpc"` or a gRPC-style endpoint.

> **Note:** Need assistance or want to learn more about using the Middleware LLM SDK? Get in touch with us via our [Contact Us](https://middleware.io/contact-us/) or join our [Slack channel](https://middlewarepublic.slack.com/join/shared_invite/zt-39l76rpo6-ZNGfpA_4U_kYL6oTuYAgmQ).