AWS Lambda
Traces | Metrics | App Logs | Custom Logs | Profiling |
---|---|---|---|---|
✅ | ✅ | ✅ | ✅ | ✖ |
Why does Lambda need a special setup?
- Early exits & freezing: Lambda can return and freeze the container quickly, so you need a configuration that flushes spans immediately.
- Layers by language: Auto-instrumentation is delivered via AWS-managed ADOT layers (or community OTel layers) and must be combined with a Collector layer. Order matters.
- X-Ray: Enabling Active Tracing gives better service maps and context propagation.

1. Prerequisites
- AWS account with permissions to manage Lambda, layers, and environment variables.
- Your Middleware account key.
- (Optional) Enable Active tracing in Lambda if you also want AWS X-Ray correlation.
Note on ARNs: The OpenTelemetry community layers are regional and published by AWS account 184161586896
. Always copy the latest ARNs for your Region and architecture (amd64/arm64) from the official opentelemetry-lambda Releases page.
Example template:
1arn:aws:lambda:<region>:184161586896:layer:opentelemetry-nodejs-0_16_0:1
1 Copy the ARN for the OpenTelemetry (OTel) Collector Lambda Layer:
- Navigate to the OpenTelemetry Collector Lambda Repository and locate the appropriate layer for your environment.
- Copy the Amazon Resource Name (ARN) and change the
<region>
and<amd|arm>
tags to the region and cpu architecture of your Lambda.1arn:aws:lambda:<region>:184161586896:layer:opentelemetry-collector-<amd64|arm64>-0_11_0:1
2 Copy the ARN for the OTel Auto Instrumentation Layer:
- Navigate to the OpenTelemetry Lambda Repository and locate the appropriate layer for your environment.
- Copy the ARN and and change the
<region>
tag to the region your Lambda is in.1arn:aws:lambda:<region>:184161586896:layer:opentelemetry-nodejs-0_9_0:4
3 In the AWS Console, navigate to Lambda → Functions → your specific Lambda function and create a Layer for each ARN above
The Layers above must be added in order - Collector, then Auto Instrumentation - or they will not work.
4 Review the added layer and ensure it matches the ARN. Finally, click Add to complete the process.
5 Set these environment variables:
If you are using *.mjs / *.ts
you may need to set NODE_OPTIONS=--import ./lambda-config.[mjs/ts]
1AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
2NODE_OPTIONS=--require ./lambda-config.js
3OTEL_SERVICE_NAME=your-service-name
4OTEL_EXPORTER_OTLP_ENDPOINT=https://<MW_UID>.middleware.io:443
5OTEL_RESOURCE_ATTRIBUTES=mw.account_key=<MW_API_KEY>
6OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION=true
7OTEL_PROPAGATORS=tracecontext
6 Create a file named lambda-config.js in your Lambda function's root directory with the following content:
If you are using *.mjs / *.ts
you may need to set NODE_OPTIONS=--import ./lambda-config.[mjs/ts]
1const { SimpleSpanProcessor } = require("@opentelemetry/sdk-trace-base");
2const {
3 OTLPTraceExporter,
4} = require("@opentelemetry/exporter-trace-otlp-proto");
5
6global.configureTracerProvider = (tracerProvider) => {
7 const spanProcessor = new SimpleSpanProcessor(new OTLPTraceExporter());
8 tracerProvider.addSpanProcessor(spanProcessor);
9};
(Optional) Custom spans in code
To capture logic specific to your business, you can use the OpenTelemetry API as follows:
1const { trace } = require("@opentelemetry/api");
2const { logs, SeverityNumber } = require("@opentelemetry/api-logs");
3
4// Returns a tracer from the global tracer provider
5// Use this to create spans
6const tracer = trace.getTracer("<MODULE-OR-FILE-NAME>", "1.0.0");
7
8/* Example simple logger */
9const logger = {
10 info: (message, attributes = {}) => {
11 // Create a log record of type INFO
12 const logRecord = {
13 timestamp: Date.now(),
14 severityText: "INFO",
15 severityNumber: SeverityNumber.INFO,
16 body: message,
17 attributes: {
18 "deployment.environment": process.env.ENVIRONMENT || "development",
19 ...attributes,
20 },
21 };
22 // Emit the log
23 logs.getLogger("<MODULE-OR-FILE-NAME>").emit(logRecord);
24 // Console log in tandem
25 console.log(message);
26 },
27};
28
29// Lambda handler
30exports.handler = async (event, context) => {
31 // Start a span with the name "lambdaHandler"
32 const span = tracer.startSpan("lambdaHandler");
33
34 try {
35 // Add a span attribute
36 span.setAttribute("event", JSON.stringify(event));
37
38 // Your Lambda function logic here
39 const result = "Hello from Lambda!";
40
41 // Use the logger created above to emit logs
42 // Logs will be automatically associated with the spans they are
43 // emitted in
44 logger.info("dummy log from aws lambda");
45 logger.info("dummy log from aws lambda with custom attributes", {
46 foo: "bar",
47 });
48 span.addEvent("Lambda execution completed");
49
50 return {
51 statusCode: 200,
52 body: JSON.stringify({ message: result }),
53 };
54 } catch (error) {
55 span.recordException(error);
56 return {
57 statusCode: 500,
58 body: JSON.stringify({ error: error.message }),
59 };
60 } finally {
61 span.end();
62 }
63};
1 Copy the ARN for the OpenTelemetry (OTel) Collector Lambda Layer:
- Navigate to the OpenTelemetry Collector Lambda Repository and locate the appropriate layer for your environment. Here is a list of Python layers
- Copy the Amazon Resource Name (ARN) and change the
<region>
and<amd|arm>
tags to the region and CPU architecture of your Lambda.1arn:aws:lambda:<region>:184161586896:layer:opentelemetry-collector-<amd64|arm64>-0_13_0:1
- Create a new file named collector.yaml with the following contents, along with your Lambda function code:
1receivers: 2 telemetryapi: {} 3 otlp: 4 protocols: 5 grpc: 6 endpoint: "localhost:9319" 7 http: 8 endpoint: "localhost:9320" 9 10exporters: 11 debug: 12 verbosity: detailed 13 otlp: 14 endpoint: 'https://your-initial-uid.middleware.io:443' 15 headers: 16 authorization: your-initial-token 17 18processors: 19 decouple: {} 20 resource: 21 attributes: 22 - action: upsert 23 key: service.name 24 value: my-service-name 25 26service: 27 pipelines: 28 traces: 29 receivers: 30 - telemetryapi 31 - otlp 32 processors: 33 - resource 34 - decouple 35 exporters: 36 - otlp 37 logs: 38 receivers: 39 - telemetryapi 40 - otlp 41 processors: 42 - resource 43 - decouple 44 exporters: 45 - otlp
2 Copy the ARN for the OTel Auto Instrumentation Layer:
- Navigate to the OpenTelemetry Lambda Repository and locate the appropriate layer for your environment.
- Copy the ARN and and change the
<region>
tag to the region your Lambda is in.1arn:aws:lambda:<region>:184161586896:layer:opentelemetry-python-0_12_0:1
- In the AWS Console navigate to Lambda → Functions → your specific Lambda function and create a Layer for each ARN above.
The Layers above must be added in order - Collector, then Auto Instrumentation - or they will not work.
- Review the added layer and ensure it matches the ARN. Finally, click Add to complete the process.
- Set the following environment variables in your Lambda function configuration:
1AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument 2OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/collector.yaml 3OPENTELEMETRY_EXTENSION_LOG_LEVEL=info 4OTEL_BSP_SCHEDULE_DELAY=500 5OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:9320 6OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION=true 7OTEL_PROPAGATORS=tracecontext
- Update Log format to
JSON
for your Lambda function by navigating to Lambda → Functions → your specific Lambda function → Configuration → Monitoring and operations tools. Click onEdit
andset log format to JSON
. - Import and set up a LoggerProvider to send log messages at desired log levels:
1import json 2import logging 3 4def lambda_handler(event, context): 5 # Set the LoggerProvider 6 logger1 = logging.getLogger("logger1") 7 logger1.setLevel(logging.DEBUG) 8 logger1.debug("This is a debug message from aws lambda") 9 logger1.info("This is an info message from aws lambda") 10 11 # Note: This to false or else logs will show up twice in Middleware 12 logger1.propagate = False 13 logging.shutdown() 14 return { 15 'statusCode': 200, 16 'body': json.dumps('Hello from Lambda!') 17 }
logger1.propagate
must be set to False
in your code. Otherwise, logs will show up twice in your Middleware account.
1 Copy the ARN for the OpenTelemetry (OTel) Collector Lambda Layer:
- Navigate to the OpenTelemetry Collector Lambda Repository and locate the appropriate layer for your environment. Here is a list of Java layers
- Copy the Amazon Resource Name (ARN) and change the
<region>
and<amd|arm>
tags to the region and CPU architecture of your Lambda.1arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-javaagent-0_9_0:1
2 Set the following environment variables in your Lambda function configuration:
1AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
2OTEL_EXPORTER_OTLP_ENDPOINT=https://<MW_UID>.middleware.io:443
3OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION=true
4OTEL_PROPAGATORS=tracecontext
5OTEL_RESOURCE_ATTRIBUTES=mw.account_key==<YOUR_MW_ACCOUNT_KEY>
6OTEL_SERVICE_NAME={JAVA-LAMBDA-SERVICE}
3 Add Custom Instrumentation (Optional)
Your Lambda function is automatically instrumented. If you need to add custom spans or attributes, you can use the Opentelemetry API:
Add the following as dependencies:
- Gradle:
1plugins { 2 id("java") 3 id("io.opentelemetry.instrumentation.muzzle-check") version "1.28.0" 4} 5dependencies { 6 implementation 'io.opentelemetry:opentelemetry-api:1.28.0' 7 implementation 'com.amazonaws:aws-lambda-java-core:1.2.1' 8} 9repositories { 10 mavenCentral() 11}
- Maven:
1<dependencies> 2 <!-- OpenTelemetry Dependencies --> 3 <dependency> 4 <groupId>io.opentelemetry</groupId> 5 <artifactId>opentelemetry-api</artifactId> 6 <version>1.28.0</version> 7 </dependency> 8 9 <!-- AWS Lambda Dependencies --> 10 <dependency> 11 <groupId>com.amazonaws</groupId> 12 <artifactId>aws-lambda-java-core</artifactId> 13 <version>1.2.1</version> 14 </dependency> 15</dependencies>
These dependencies add the OpenTelemetry API (for optional manual spans in your handler) and the AWS Lambda core interfaces (RequestHandler, Context). The ADOT Java agent layer handles auto-instrumentation and exporting, so you don’t need extra exporter SDKs here. Keep the opentelemetry-api
version compatible with your Lambda runtime and the ADOT agent layer in use.
The goal is instrument your Go Lambda with the official OTel Go wrapper and send traces directly to Middleware over OTLP/HTTP (no Lambda layer required for Go).
1 Install / import the required packages
Add these imports at the top of your Lambda project’s entry point:
1import (
2 "context"
3 "log"
4
5 "github.com/aws/aws-lambda-go/lambda"
6 "go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda"
7 "go.opentelemetry.io/otel"
8 "go.opentelemetry.io/otel/attribute"
9 "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
10 "go.opentelemetry.io/otel/sdk/resource"
11 "go.opentelemetry.io/otel/sdk/trace"
12)
Here,
otellambda.InstrumentHandler
is the official wrapper for Go Lambdas.- The HTTP OTLP exporter (
otlptracehttp
) is the correct exporter for sending spans via OTLP/HTTP.
2 Configure your Middleware account and tracer provider
Create a helper that builds a tracer provider and points the exporter directly to Middleware’s OTLP endpoint.
Important fix: with the HTTP exporter, use WithEndpointURL
and include the full /v1/traces
URL (or use the env var OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
).
1func initTracerProvider(ctx context.Context) (*trace.TracerProvider, error) {
2 // Export spans directly to Middleware (OTLP/HTTP).
3 // WithEndpointURL expects a full URL; include /v1/traces.
4 exp, err := otlptracehttp.New(ctx,
5 otlptracehttp.WithEndpointURL("https://ruplp.middleware.io/v1/traces"),
6 )
7 if err != nil {
8 return nil, err
9 }
10
11 // Add useful resource attributes (seen in Middleware APM service list/filters)
12 res := resource.NewSchemaless(
13 attribute.String("service.name", "{GO-LAMBDA-SERVICE}"),
14 attribute.String("mw.account_key", "<YOUR_MW_ACCOUNT_KEY>"),
15 attribute.String("library.language", "go"),
16 )
17
18 tp := trace.NewTracerProvider(
19 trace.WithBatcher(exp),
20 trace.WithResource(res),
21 )
22 otel.SetTracerProvider(tp)
23 return tp, nil
24}
Here,
WithEndpointURL
(or env vars) is the recommended way to set an HTTPS URL for the HTTP exporter. UsingWithEndpoint("host:port")
is meant for host:port only and does not take a scheme/path.- Alternative (env only): set
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ruplp.middleware.io/v1/traces
and callotlptracehttp.New(ctx)
with no options. The exporter reads the env var.
3 Add your handler (custom attributes optional)
1type MyEvent struct {
2 Name string `json:"name"`
3 Age int `json:"age"`
4}
5
6type MyResponse struct {
7 Message string `json:"message"`
8}
9
10func HandleLambdaEvent(ctx context.Context, event *MyEvent) (*MyResponse, error) {
11 tracer := otel.Tracer("lambda")
12 _, span := tracer.Start(ctx, "HandleLambdaEvent")
13 defer span.End()
14
15 // Add any domain attributes you want to see in traces
16 span.SetAttributes(
17 attribute.String("user.name", event.Name),
18 attribute.Int("user.age", event.Age),
19 )
20
21 return &MyResponse{Message: "ok"}, nil
22}
4 Wire the wrapper in main and ensure flush
1func main() {
2 ctx := context.Background()
3 tp, err := initTracerProvider(ctx)
4 if err != nil {
5 log.Fatalf("failed to init tracer provider: %v", err)
6 }
7 defer tp.Shutdown(ctx) // flush on shutdown
8
9 // Wrap your handler so spans are started/ended around each invocation.
10 // WithFlusher ensures batches flush before the runtime freezes.
11 lambda.Start(otellambda.InstrumentHandler(
12 HandleLambdaEvent,
13 otellambda.WithFlusher(tp),
14 ))
15}
InstrumentHandler is the supported wrapper; WithFlusher is designed for Lambda’s short-lived execution model and ensures spans are exported before the runtime freezes.
1 First, ensure that you have .NET 6 or later on your build machine. Verify with:
1dotnet --version
2 Add the Middleware agent package to your project:
1dotnet add package MW.APM
3 Add the following to your Program.cs. This wires MW.APM using the .NET 6 minimal hosting pattern and sets up console logging for quick validation.
1var builder = WebApplication.CreateBuilder(args);
2
3var configuration = new ConfigurationBuilder()
4 .SetBasePath(Directory.GetCurrentDirectory())
5 .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
6 .AddEnvironmentVariables()
7 .Build();
8
9builder.Services.ConfigureMWInstrumentation(configuration);
10
11builder.Logging.AddConfiguration(configuration.GetSection("Logging"));
12builder.Logging.AddConsole();
13builder.Logging.SetMinimumLevel(LogLevel.Debug);
14
15// Build the app
16var app = builder.Build();
17
18// Initialize MW.APM logger AFTER app is built
19Logger.Init(app.Services.GetRequiredService<ILoggerFactory>());
20
21// Define endpoints / handlers as needed
22// app.MapGet("/", () => "OK");
23
24app.Run();
4 Add your account key, target URL, and service name to appsettings.json:
1{
2 "MW": {
3 "ApiKey": "<YOUR_MW_ACCOUNT_KEY>",
4 "TargetURL": "<MW_UID>.middleware.io:443",
5 "ServiceName": "{SERVICE NAME}"
6 },
7 "Logging": {
8 "LogLevel": {
9 "Default": "Information",
10 "Microsoft.AspNetCore": "Warning"
11 }
12 }
13}
Verify Installation
To deploy and test your application, check your Middleware account for traces by navigating to APM → Services and finding the name of your function in the services list.

X-Ray Tracing
Ensure X-Ray tracing is enabled for your Lambda function to get the full benefit of the instrumentation.

Here’s a single, copy-friendly AWS Lambda – Environment Variables (Quick Reference) table (no extras, just the vars).
Variable | Purpose | Example value |
---|---|---|
AWS_LAMBDA_EXEC_WRAPPER | Starts auto-instrumentation before your handler | Node/Java: /opt/otel-handler • Python: /opt/otel-instrument |
OTEL_SERVICE_NAME | Service name shown in Middleware | {NODEJS-LAMBDA-SERVICE} / {PYTHON-LAMBDA-SERVICE} / {JAVA-LAMBDA-SERVICE} |
OTEL_PROPAGATORS | Context propagation format | tracecontext |
OTEL_RESOURCE_ATTRIBUTES | Resource metadata (incl. Middleware key) | mw.account_key=<YOUR_MW_ACCOUNT_KEY> |
OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION | Disable AWS/X-Ray propagation if desired | true |
NODE_OPTIONS | Preload bootstrap file (Node) | --require ./lambda-config.js |
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED | Auto-capture Python logs | true |
OTEL_LOGS_EXPORTER | Export logs via OTLP | otlp |
OTEL_BSP_SCHEDULE_DELAY | Span batcher flush cadence (ms) | 500 |
OTEL_EXPORTER_OTLP_ENDPOINT | Direct export base endpoint | https://ruplp.middleware.io |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | Direct export (traces) full URL (HTTP) | https://ruplp.middleware.io/v1/traces |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT | Direct export (metrics) full URL (HTTP) | https://ruplp.middleware.io/v1/metrics |
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT | Direct export (logs) full URL (HTTP) | https://ruplp.middleware.io/v1/logs |
OPENTELEMETRY_COLLECTOR_CONFIG_URI | If using the Collector layer, where it reads config | /var/task/collector.yaml (or s3://…, https://… ) |
OPENTELEMETRY_EXTENSION_LOG_LEVEL | Collector extension log level | info / debug |
Troubleshooting
1) “I don’t see any traces in Middleware”
Likely causes
- The Lambda wrapper isn’t running (Node/Java/Python only).
- You’re exporting to the wrong endpoint or missing the /v1/traces path for OTLP/HTTP (common in Go).
- The Collector layer (if used) isn’t reading your config.
- The function has no internet egress (VPC without NAT).
What to check
- Wrappers: In Node, Java, Python, confirm the wrapper env var is set exactly:
- Node/Java:
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
- Python:
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
- Node/Java:
- Endpoints: For OTLP/HTTP, prefer the per-signal env var with the full path (or exporter option). Example:
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ruplp.middleware.io/v1/traces
(Go/HTTP exporters need the/v1/traces
suffix).
- Collector (if using the Lambda Collector layer): package a config and point the extension at it:
- Add
collector.yaml
to your deployment. - Set
OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/collector.yaml
. - In the file, export to Middleware via OTLP/HTTP.
- Add
- Networking/Egress: If your function runs in a VPC, ensure it has internet access (NAT Gateway / routing) so it can reach
https://ruplp.middleware.io
. - Logs: Open CloudWatch Logs for the function to see wrapper/exporter/extension messages (Monitor → View CloudWatch logs).
2) “Traces appear sporadically or late”
Likely causes
- The BatchSpanProcessor is batching too long for short Lambda runs.
- The runtime is freezing before spans flush.
What to try
- Reduce the batcher delay for short invocations:
OTEL_BSP_SCHEDULE_DELAY=500
(ms) (tune with BSP env vars).
- Go: wrap your handler with the Lambda wrapper and force a flush at the end:
lambda.Start(otellambda.InstrumentHandler(Handle, otellambda.WithFlusher(tp)))
.
3) “Node.js: the wrapper runs, but my bootstrap file doesn’t load”
Likely cause
- Using
NODE_OPTIONS=--require ./lambda-config.js
with an ESM (module) project.
Fix
- For ESM projects, use
--import/a
loader instead of--require
(which is for CommonJS).
4) “Python: logs aren’t showing up”
Likely cause
- Auto-logging isn’t enabled for the Python OTel SDK.
Fix
- Add:
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
OTEL_LOGS_EXPORTER=otlp
(if exporting logs) These are standard OTel SDK knobs for Python logging & exporters.
5) “Using the Collector layer: I still see data going to X-Ray, not Middleware”
Likely causes
- The extension is not reading your config file, or your exporter block is missing/incorrect.
Fix
- Ensure
OPENTELEMETRY_COLLECTOR_CONFIG_URI
points to a file you ship (e.g.,/var/task/collector.yaml
). - In that file, define an OTLP/HTTP exporter with endpoint:
https://ruplp.middleware.io
and a traces pipeline that uses it. - Check the extension logs in CloudWatch to confirm it loaded your file.
6) “Java/Node/Python: wrapper set, still no traces”
Likely causes
- Wrong layer ARN (Region/architecture mismatch) or outdated layer.
OTEL_SERVICE_NAME/mw.account_key
not set.
Fix
- Re-add the correct community OTel Lambda layer ARN for your Region/arch from the latest releases (publisher 184161586896).
- Set
OTEL_SERVICE_NAME
and include your Middleware key viaOTEL_RESOURCE_ATTRIBUTES=mw.account_key=<KEY>
.
7) “Go: exporter configured, but nothing arrives”
Likely causes
- Using
OTEL_EXPORTER_OTLP_ENDPOINT
without a per-signal endpoint/path for HTTP. - Missing TLS/URL settings, or not batching/flushing.
Fix
- Prefer
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ruplp.middleware.io/v1/traces
or in code useotlptracehttp.WithEndpointURL("https://ruplp.middleware.io/v1/traces")
. - Keep
WithFlusher
.
8) “Function times out or gets killed before export”
Likely causes
- Tight timeout/memory + long batch interval.
- Exporter retries blocked by no egress.
Fix
- Increase Lambda timeout slightly and reduce BSP delay (e.g.,
OTEL_BSP_SCHEDULE_DELAY=500
). - Verify VPC egress (NAT).
9) “Spans don’t link across services”
Likely cause
- Mixed propagation formats (AWS vs W3C).
Fix
- Set
OTEL_PROPAGATORS=tracecontext
across functions to keep W3C propagation consistent. (If you need AWS/X-Ray interop, don’t disable AWS propagation.)
Need assistance or want to learn more about Middleware? Contact our support team at [email protected] or join our Slack channel.