---
title: "OpenTelemetry Python Instrumentation"
description: "Use the upstream OpenTelemetry Python auto-instrumentation to send traces (and optionally metrics and logs) from your Python services to Middleware."
section: "APM"
breadcrumb: "APM > Python > OpenTelemetry"
canonical: "https://docs.middleware.io/get-started/apm/python/otel"
---
# OpenTelemetry Python Instrumentation

This guide shows you how to instrument your Python application with OpenTelemetry and send traces to Middleware. The auto-instrumentation approach works with Django, Flask, FastAPI, Falcon, Celery, and most Python libraries out of the box.

## Prerequisites

- **Python 3.8 or newer**
- **Middleware account & project**: your Middleware **UID** (`https://<uid>.middleware.io`) and **API key** (`MW_API_KEY`)
- **Network access** to `https://<uid>.middleware.io:443` for OTLP/gRPC or OTLP/HTTP

For a deeper overview of Middleware's OTLP endpoints and headers, see the [OpenTelemetry getting started guide](/get-started/agent/otel).

## Quick start (Linux / macOS / VM)

The steps below cover a typical app-on-a-VM (or bare-metal) setup. Kubernetes, Docker, and Windows use the same core commands but different ways of setting environment variables.

### 1. Set environment variables

Choose a transport. gRPC is the most common default; HTTP/protobuf is useful when gRPC is blocked by proxies or network policy.

#### OTLP/gRPC

Configure OpenTelemetry to send data directly to Middleware using OTLP/gRPC:

```bash
export OTEL_SERVICE_NAME="my-python-service"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<uid>.middleware.io:443"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=<MW_API_KEY>"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

# optional: add custom resource attributes;
export OTEL_RESOURCE_ATTRIBUTES="mw.resource.type=custom"
# optional: disable metrics if needed;
export OTEL_METRICS_EXPORTER="none"
```

#### OTLP/HTTP (protobuf)

Configure OpenTelemetry to send data directly to Middleware using OTLP/HTTP:

```bash
export OTEL_SERVICE_NAME="my-python-service"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://<uid>.middleware.io:443/v1/traces"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=<MW_API_KEY>"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"

# optional: add custom resource attributes;
export OTEL_RESOURCE_ATTRIBUTES="mw.resource.type=custom"
# optional: disable metrics if needed;
export OTEL_METRICS_EXPORTER="none"
```

> **Warning:** If you use `/v1/traces`, you must use an OTLP/HTTP exporter (`http/protobuf`). For OTLP/gRPC exporters, use the root endpoint (`:443`) without `/v1/*`.

Replace:

- `<uid>` with your Middleware project UID.
- `<MW_API_KEY>` with your Middleware API key.
- `<service-name>` with something meaningful (for example, `payments-api`).

> **Note:** OpenTelemetry auto-instruments popular HTTP clients and other libraries. If metrics are enabled, every outgoing HTTP call can produce metric data. Starting with `OTEL_METRICS_EXPORTER=none` keeps volume low while you validate traces. You can switch it to `otlp` later when you are ready to send metrics.

Optional (recommended when you export logs): enable trace context injection into Python logs:

```bash
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
export OTEL_PYTHON_LOG_CORRELATION=true
```

### 2. Install OpenTelemetry packages

Install the core OTel distribution and OTLP exporter:

```bash
pip install opentelemetry-distro opentelemetry-exporter-otlp
```

### 3. Install instrumentation for your dependencies

Detect installed libraries (web frameworks, DB clients, HTTP clients, etc.) and add matching instrumentation packages:

```bash
opentelemetry-bootstrap --action=install
```

Run this after installing your application's dependencies. Only already-installed libraries are instrumented.

### 4. Run your application with auto-instrumentation

Wrap your normal run command with the OTel launcher:

```bash
opentelemetry-instrument <your_run_command>
```

Examples:

- `opentelemetry-instrument python app.py`
- `opentelemetry-instrument gunicorn -k uvicorn.workers.UvicornWorker main:app`

Framework-specific notes are in [Framework run commands](#framework-run-commands).

## Other deployment targets

The OpenTelemetry behaviour is the same everywhere; only how you declare environment variables changes.

#### Kubernetes (direct)

Add the same environment variables to your Deployment (or StatefulSet) manifest:

```yaml
env:
  - name: OTEL_SERVICE_NAME
    value: "my-python-service"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "https://<uid>.middleware.io:443"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    value: "authorization=<MW_API_KEY>"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"
  - name: OTEL_TRACES_EXPORTER
    value: "otlp"
  - name: OTEL_METRICS_EXPORTER
    value: "none"
```

Build your image with the OTel packages installed (see "Install OpenTelemetry packages" above) and run your app with:

```bash
opentelemetry-instrument <your_run_command>
```

For cluster-wide control, you can instead send data to a Kubernetes OpenTelemetry Collector and have the Collector export to Middleware (see [Optional: OpenTelemetry Collector](#optional:-opentelemetry-collector)).

#### Kubernetes (OTel Operator)

The OpenTelemetry Operator can inject Python auto-instrumentation into your pods without changing your application image.

1. **Install the Operator**
   Follow the upstream [OpenTelemetry Operator installation docs](https://opentelemetry.io/docs/platforms/kubernetes/operator/).
2. **Create an `Instrumentation` resource for Python**

```yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
spec:
  exporter:
    # Usually a cluster-local Collector; see "Optional: OpenTelemetry Collector" below.
    endpoint: http://otel-collector:4318
  propagators:
    - tracecontext
    - baggage
  env:
    - name: OTEL_METRICS_EXPORTER
      value: "none"
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
```

3. **Annotate your Deployment**

```yaml
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"
    instrumentation.opentelemetry.io/otel-python-platform: "glibc"  # or "musl" for Alpine
```

The Collector configured in "Optional: OpenTelemetry Collector" should then forward data to Middleware using OTLP.

#### Windows

Use the PowerShell syntax for environment variables:

```powershell
$env:OTEL_RESOURCE_ATTRIBUTES = "service.name=<service-name>"
$env:OTEL_EXPORTER_OTLP_ENDPOINT = "https://<uid>.middleware.io:443"
$env:OTEL_EXPORTER_OTLP_HEADERS = "authorization=<MW_API_KEY>"
$env:OTEL_EXPORTER_OTLP_PROTOCOL = "grpc"
$env:OTEL_TRACES_EXPORTER = "otlp"
$env:OTEL_METRICS_EXPORTER = "none"
```

Install packages and run your application exactly as in the "Quick start" section above:

```powershell
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install
opentelemetry-instrument <your_run_command>
```

#### Docker

You can bake configuration into your Dockerfile:

```dockerfile
ENV OTEL_RESOURCE_ATTRIBUTES=service.name=<service-name>
ENV OTEL_EXPORTER_OTLP_ENDPOINT=https://<uid>.middleware.io:443
ENV OTEL_EXPORTER_OTLP_HEADERS=authorization=<MW_API_KEY>
ENV OTEL_EXPORTER_OTLP_PROTOCOL=grpc
ENV OTEL_TRACES_EXPORTER=otlp
ENV OTEL_METRICS_EXPORTER=none
```

Or pass them at runtime:

```bash
docker run \
  -e OTEL_RESOURCE_ATTRIBUTES="service.name=<service-name>" \
  -e OTEL_EXPORTER_OTLP_ENDPOINT="https://<uid>.middleware.io:443" \
  -e OTEL_EXPORTER_OTLP_HEADERS="authorization=<MW_API_KEY>" \
  -e OTEL_EXPORTER_OTLP_PROTOCOL="grpc" \
  -e OTEL_TRACES_EXPORTER="otlp" \
  -e OTEL_METRICS_EXPORTER="none" \
  your-image:latest
```

Inside the container, ensure you start your app with:

```bash
opentelemetry-instrument <your_run_command>
```

## Framework run commands

Choose your framework below to see the run command. The setup steps above are the same for all frameworks. Auto-instrumentation works by starting your process through `opentelemetry-instrument`.

#### Django

**Prerequisites:** Set the `DJANGO_SETTINGS_MODULE` environment variable:

```bash
export DJANGO_SETTINGS_MODULE=myproject.settings
```

**Run command:**

```bash
opentelemetry-instrument python manage.py runserver --noreload
```

> **Warning:** Always use `--noreload` with Django. The auto-reload mechanism spawns child processes that break OpenTelemetry instrumentation.

**For Docker users:**

```dockerfile
CMD ["opentelemetry-instrument", "python", "manage.py", "runserver", "0.0.0.0:8000", "--noreload"]
```

#### Flask

**Run command:**

```bash
opentelemetry-instrument flask run --no-reload
```

Or if running directly:

```bash
opentelemetry-instrument python app.py
```

> **Warning:** Always use `--no-reload` with Flask. The reloader spawns a child process that breaks OpenTelemetry instrumentation. Also avoid `FLASK_ENV=development` as it enables the reloader.

**For Docker users:**

```dockerfile
CMD ["opentelemetry-instrument", "flask", "run", "--host=0.0.0.0", "--no-reload"]
```

Or:

```dockerfile
CMD ["opentelemetry-instrument", "python", "app.py"]
```

#### FastAPI

**Run command:**

```bash
opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000
```

> **Warning:** Do not use `--reload` with Uvicorn when instrumenting. The reload mode spawns new processes that break instrumentation.

> **Note:** Uvicorn's `--workers` flag is not supported with `opentelemetry-instrument`. Use Gunicorn with Uvicorn workers instead: `gunicorn -k uvicorn.workers.UvicornWorker main:app`

**For Docker users:**

```dockerfile
CMD ["opentelemetry-instrument", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
```

#### Falcon

**Run command (with Gunicorn):**

```bash
opentelemetry-instrument gunicorn app:api --bind 0.0.0.0:8000
```

Or with Waitress:

```bash
opentelemetry-instrument waitress-serve --port=8000 app:api
```

**For Docker users:**

```dockerfile
CMD ["opentelemetry-instrument", "gunicorn", "app:api", "--bind", "0.0.0.0:8000"]
```

#### Gunicorn

Gunicorn generally works without extra configuration:

```bash
opentelemetry-instrument gunicorn myproject.wsgi:application
```

> **Warning:** If you use `--preload`, the app loads before workers fork and spans can get stuck. Add a post-fork hook (see [OpenTelemetry fork-process-model](https://github.com/open-telemetry/opentelemetry-python/tree/main/docs/examples/fork-process-model)) or disable preload.

#### Celery

**Run command:**

```bash
opentelemetry-instrument celery -A tasks worker --loglevel=info
```

Replace `tasks` with your Celery app module name.

**For Docker users:**

```dockerfile
CMD ["opentelemetry-instrument", "celery", "-A", "tasks", "worker", "--loglevel=info"]
```

> **Note:** Celery instrumentation captures task execution spans, including task name, arguments, and status. Both the worker and the code that enqueues tasks should be instrumented for full trace propagation.

**Celery with prefork workers (advanced):** Celery uses the prefork worker model by default. The OpenTelemetry SDK is not fork-safe, so you must initialize it in each worker process using the `worker_process_init` signal. Add this to your Celery app file:

```python
from celery.signals import worker_process_init
from opentelemetry.instrumentation.celery import CeleryInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

@worker_process_init.connect(weak=False)
def init_celery_tracing(*args, **kwargs):
    CeleryInstrumentor().instrument()
    resource = Resource.create({})
    trace.set_tracer_provider(TracerProvider(resource=resource))
    span_processor = BatchSpanProcessor(OTLPSpanExporter())
    trace.get_tracer_provider().add_span_processor(span_processor)
```

This ensures each worker process creates its own tracer instance. Only required for workers, not for the code that enqueues tasks.

#### Hypercorn/Unicorn

> **Warning:** **Hypercorn** and **Unicorn** are **not supported** by OpenTelemetry auto-instrumentation due to fork-safety limitations.

**Why they don't work:** The OpenTelemetry SDK components (`BatchSpanProcessor`, `PeriodicExportingMetricReader`, `BatchLogProcessor`) spawn background threads and use locks, which are not fork-safe (see [Python issue #6721](https://bugs.python.org/issue6721)). Most ASGI servers work around this using `register_at_fork` hooks to reinitialize after forking. However, Hypercorn and Unicorn use the `spawn` method to start worker processes, which doesn't invoke these hooks—making the workaround ineffective.

**Recommended alternative:** Use Gunicorn with Uvicorn workers instead:

```bash
opentelemetry-instrument gunicorn -k uvicorn.workers.UvicornWorker main:app --bind 0.0.0.0:8000
```

**For Docker users:**

```dockerfile
CMD ["opentelemetry-instrument", "gunicorn", "-k", "uvicorn.workers.UvicornWorker", "main:app", "--bind", "0.0.0.0:8000"]
```

This gives you the same ASGI capabilities with proper OpenTelemetry support. See [Hypercorn issue #215](https://github.com/pgjones/hypercorn/issues/215) for updates on native Hypercorn support.

#### uWSGI

uWSGI loads the app before forking by default. Enable lazy app loading so each worker initialises OpenTelemetry after fork.

In `uwsgi.ini`:

```ini
lazy-apps = true
```

Then start uWSGI via:

```bash
opentelemetry-instrument uwsgi --ini uwsgi.ini
```

## Troubleshooting

#### No data in Middleware

**Check environment variables are set:**

```bash
echo $OTEL_EXPORTER_OTLP_ENDPOINT
echo $OTEL_SERVICE_NAME
echo $OTEL_EXPORTER_OTLP_HEADERS
```

If your service is running and env vars are correct, use the **Console exporter** tab to confirm spans are being created.

#### Console exporter

To verify that spans are being created locally, temporarily export to the console:

```bash
OTEL_TRACES_EXPORTER=console opentelemetry-instrument <your_run_command>
```

If you see JSON span output in your terminal but traces don't appear in Middleware, the issue is with export configuration (endpoint, auth, or network). If no output appears, the instrumentation isn't capturing your requests.

#### Workers and reload

Application servers that create worker processes (or use hot reload) need extra care.

**Why do multi-worker servers drop spans?** The OpenTelemetry SDK isn't fork-safe. Application servers that spawn multiple worker processes require special handling:

- **Uvicorn** with the `--workers` flag is not supported. Use Gunicorn with Uvicorn workers instead:

  ```bash
  opentelemetry-instrument gunicorn -k uvicorn.workers.UvicornWorker main:app
  ```
- **Hypercorn/Unicorn** are not supported due to fork-safety issues. See the [Hypercorn/Unicorn](#framework-run-commands) tab for details and workarounds.
- **Gunicorn with `--preload`** or **uWSGI** need extra config. See [Running with Gunicorn or uWSGI](#running-with-gunicorn-or-uwsgi) below.

**Hot reload breaks instrumentation:** Don't run your app in reloader/hot-reload mode. For Flask, avoid `FLASK_ENV=development`. For Django, use `--noreload`. For Uvicorn/FastAPI, don't use `--reload`.

#### gRPC install issues

If `grpcio` installation fails, use the HTTP exporter instead:

```bash
pip install opentelemetry-exporter-otlp-proto-http
```

Then set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` and use the per-signal HTTP endpoints (e.g. `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`) as described in the [OpenTelemetry getting started guide](/get-started/agent/otel).

#### Missing DB spans

Auto-instrumentation detects and instruments common database libraries. Ensure you've run `opentelemetry-bootstrap --action=install` after installing your database drivers.

**PostgreSQL note:** `psycopg2` is preferred over `psycopg2-binary` for auto-instrumentation. If you must use `psycopg2-binary`, you may need to use manual instrumentation with `Psycopg2Instrumentor().instrument(skip_dep_check=True)`.

Common database instrumentations installed by bootstrap:

- PostgreSQL: `opentelemetry-instrumentation-psycopg2`
- MySQL: `opentelemetry-instrumentation-pymysql` or `opentelemetry-instrumentation-mysql`
- MongoDB: `opentelemetry-instrumentation-pymongo`
- Redis: `opentelemetry-instrumentation-redis`
- SQLAlchemy: `opentelemetry-instrumentation-sqlalchemy`

Check [supported versions](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation) for compatibility with your library versions.

### Running with Gunicorn or uWSGI

#### Show details

**Gunicorn** works out of the box. No extra setup needed unless you use the `--preload` flag. If you do use `--preload`, see the [post_fork hook example](https://github.com/open-telemetry/opentelemetry-python/tree/main/docs/examples/fork-process-model).

**uWSGI** requires one of these options:

- Add `lazy-apps = true` to your uWSGI config (recommended, simplest fix)
- Or implement a post_fork hook (see [example](https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html))

> **Note:** OpenTelemetry's span exporter uses a background thread. When a server forks worker processes, this thread doesn't copy over correctly, causing spans to get stuck.
>
> **Gunicorn** loads your app in each worker after forking (so the thread starts fresh). With `--preload`, it loads before forking, causing the same issue.
>
> **uWSGI** loads your app before forking by default. Setting `lazy-apps = true` makes it load after forking instead.

## Optional: OpenTelemetry Collector

Instead of sending data straight from your app to Middleware, you can route everything through an OpenTelemetry Collector:

- **Applications -> Collector** using OTLP.
- **Collector -> Middleware** using OTLP/gRPC or OTLP/HTTP to `https://<uid>.middleware.io`.

Benefits:

- Central place for sampling, filtering, and redaction.
- Easier rollouts when you change exporters or destinations.
- Ability to mirror data to multiple backends if needed.

You can find end-to-end Collector examples in the [OpenTelemetry getting started guide](/get-started/agent/otel).

## More configuration and troubleshooting

### More configuration

For full agent configuration—CLI options, environment variable mapping, and Python-specific options (excluded URLs, request attributes, logging, disabling instrumentations)—see the official [OpenTelemetry Python agent configuration](https://opentelemetry.io/docs/zero-code/python/configuration/) docs.

### Troubleshooting

For common issues such as package installation, Flask debug/reloader behavior, pre-fork servers (Gunicorn, Uvicorn workers), and gRPC connectivity, see [Troubleshooting Python automatic instrumentation](https://opentelemetry.io/docs/zero-code/python/troubleshooting/) on the OpenTelemetry site.

## Next steps and references

- **Add custom spans and business attributes**: [Python manual instrumentation](/get-started/apm/python/manual-instrumentation)
- **Manual instrumentation (custom spans/attributes)**: OpenTelemetry Python docs at [opentelemetry.io](https://opentelemetry.io/docs/languages/python/).
- **Auto-instrumentation library list**: See the upstream instrumentation packages list in the OpenTelemetry Python contrib repo at [github.com](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation).
- **Prefer Middleware SDK features (profiling, Middleware options, Host Agent)**: use the main [Python guide](/get-started/apm/python).

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us page or join our Slack channel.