Python manual instrumentation

Auto-instrumentation is great for HTTP requests, database calls, and common libraries. Manual instrumentation is how you capture your important business steps (for example: orders.handle, inventory.reserve, shipment.dispatched, pricing.recalculate-discount) with the attributes you care about.

Prerequisites#

• Complete one of these setups first:
  • OpenTelemetry auto-instrumentation: Python (OpenTelemetry)
  • Middleware SDK: Python guide
• Install the OpenTelemetry API/SDK (if you don't already have them through another package):

1pip install opentelemetry-api opentelemetry-sdk

1. Create custom spans#

Use a tracer and wrap meaningful work inside spans. Child spans created inside a parent span automatically connect to the same trace.

1from opentelemetry import trace
2
3tracer = trace.get_tracer("orders-service")
4
5
6def handle_order(order_id: str, customer_id: str) -> None:
7    with tracer.start_as_current_span("orders.handle") as span:
8        span.set_attribute("order.id", order_id)
9        span.set_attribute("customer.id", customer_id)
10        span.set_attribute("order.source", "web")
11
12        validate_order(order_id)
13        reserve_items(order_id)

1from opentelemetry import trace
2
3tracer = trace.get_tracer("orders-service")
4
5
6def reserve_items(order_id: str) -> None:
7    with tracer.start_as_current_span("inventory.reserve") as span:
8        span.set_attribute("order.id", order_id)
9        span.set_attribute("inventory.region", "eu-central-1")
10        # ... call database / external services here ...

1from opentelemetry import trace
2
3tracer = trace.get_tracer("pricing")
4
5
6@tracer.start_as_current_span("pricing.recalculate-discount")
7def recalculate_discount(customer_tier: str, items_total: float) -> float:
8    # ... custom discount logic ...
9    return items_total * 0.9 if customer_tier == "gold" else items_total

Tips:

• Reuse tracer instances (create once per module), not per request.
• Pick span names that match business steps and stay stable over time.
• Keep span attributes low-cardinality (avoid IDs that explode the number of unique values unless needed for debugging).

2. Add attributes and events#

Attributes become filter/group dimensions. Events capture notable moments inside a span (retries, cache misses, queue waits).

1from opentelemetry import trace
2
3
4def log_shipment(order_id: str, carrier: str, region: str) -> None:
5    span = trace.get_current_span()
6
7    span.set_attribute("shipment.order_id", order_id)
8    span.set_attribute("shipment.carrier", carrier)
9    span.set_attribute("shipment.region", region)
10
11    span.add_event("shipment.label_created")
12
13    # ... call shipping provider ...
14
15    span.add_event(
16        "shipment.dispatched",
17        attributes={"status": "in_transit", "tracking.enabled": True},
18    )

If you want to follow OpenTelemetry naming conventions, use semantic attribute keys where possible (for example, http.request.method, url.full, db.system).

3. Record errors#

When something fails, record the exception and set the span status to ERROR. This makes failures easier to find in trace search and alerts.

1from opentelemetry import trace
2from opentelemetry.trace import Status, StatusCode
3
4
5def fetch_user_profile(user_id: str) -> dict:
6    span = trace.get_current_span()
7
8    try:
9        profile = load_profile_from_api(user_id)
10        span.set_status(Status(StatusCode.OK))
11        return profile
12    except TimeoutError as exc:
13        span.record_exception(exc)
14        span.set_status(Status(StatusCode.ERROR, "profile service timeout"))
15        raise
16    except Exception as exc:
17        span.record_exception(exc)
18        span.set_status(Status(StatusCode.ERROR, "unexpected profile failure"))
19        raise

4. Add span links (optional)#

Span links are useful when work is related but not parent/child (for example, message queues where a consumer links back to a producer span context).

1from opentelemetry import trace
2
3tracer = trace.get_tracer("analytics")
4
5
6def ingest_event(raw_event: dict) -> trace.SpanContext:
7    with tracer.start_as_current_span("pipeline.ingest") as span:
8        span.set_attribute("event.type", raw_event.get("type", "unknown"))
9        return span.get_span_context()
10
11
12def enrich_event(parent_ctx: trace.SpanContext, raw_event: dict) -> None:
13    link = trace.Link(parent_ctx)
14    with tracer.start_as_current_span("pipeline.enrich", links=[link]):
15        # ... enrich and forward event ...
16        pass

Validate#

1. Trigger the code paths that create manual spans.
2. In Middleware, open APM > Traces and search by:
   • service.name, and/or
   • your custom span names (for example, orders.handle).
3. Open a trace and verify attributes, events, and error status are present.

Troubleshooting#

Next steps#

• If you're using OpenTelemetry auto-instrumentation, keep this handy: Python (OpenTelemetry)
• For Middleware SDK features (profiling, Middleware options, Host Agent), see: Python guide