OpenTelemetry¶

The OpenTelemetry plugin instruments a durable execution and emits distributed traces to any OTLP backend, such as Amazon CloudWatch. It builds on the plugin interface: you register it in your handler configuration, and it opens spans at the invocation, operation, and attempt boundaries as the execution runs.

A durable execution runs across many Lambda invocations. The plugin derives deterministic trace and span IDs from the execution ARN, and from the X-Ray trace header when one is present, so the spans from every invocation join a single trace. Without deterministic IDs, each invocation would produce its own disconnected trace.

Some names on this page keep the X-Ray label (for example the X-Ray trace header and the awsxray collector exporter), but your spans land in CloudWatch.

Install the plugin¶

npm install @aws/durable-execution-sdk-js-otel

pip install aws-durable-execution-sdk-python-otel

<dependency>
    <groupId>software.amazon.lambda.durable</groupId>
    <artifactId>aws-durable-execution-sdk-java-plugin-otel</artifactId>
    <version>${durable.sdk.version}</version>
</dependency>

The two plugins¶

The package ships two plugins. Both correlate a whole durable execution into a single trace with deterministic IDs, and both emit a Workflow root span for the execution. They differ in where they root the invocation view and when they export the root span. You register exactly one of them.

ExecutionOtelPlugin¶

ExecutionOtelPlugin opens a synthetic Workflow span as the trace root. Every invocation span and operation span nests under it. The plugin exports the Workflow span only when the execution reaches a terminal status, so an execution that is still running does not leave a dangling root span, but you also cannot watch the trace in progress, and a span held open for hours can exceed the ingestion limits of some observability platforms. Choose this plugin for short executions, or when you want one unified trace per execution with the workflow as the logical root.

import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js";
import { ExecutionOtelPlugin } from "@aws/durable-execution-sdk-js-otel";

export const handler = withDurableExecution(
  async (event: { id: string }, context: DurableContext) => {
    const data = await context.step("fetch-data", async () => event.id);
    await context.wait("cooldown", { seconds: 5 });
    return await context.step("process", async () => `processed ${data}`);
  },
  { plugins: [new ExecutionOtelPlugin()] },
);

from aws_durable_execution_sdk_python import DurableContext, durable_execution
from aws_durable_execution_sdk_python.config import Duration
from aws_durable_execution_sdk_python_otel import ExecutionOtelPlugin


@durable_execution(plugins=[ExecutionOtelPlugin()])
def handler(event: dict, context: DurableContext) -> str:
    data = context.step(lambda ctx: event["id"], name="fetch-data")
    context.wait(Duration.from_seconds(5), name="cooldown")
    return context.step(lambda ctx: f"processed {data}", name="process")

import java.time.Duration;
import java.util.Map;
import software.amazon.lambda.durable.DurableConfig;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.otel.ExecutionOtelPlugin;

public class ExecutionOtelExample extends DurableHandler<Map<String, String>, String> {
    @Override
    protected DurableConfig createConfiguration() {
        return DurableConfig.builder().withPlugins(new ExecutionOtelPlugin()).build();
    }

    @Override
    protected String handleRequest(Map<String, String> event, DurableContext context) {
        String data = context.step("fetch-data", String.class, ctx -> event.get("id"));
        context.wait("cooldown", Duration.ofSeconds(5));
        return context.step("process", String.class, ctx -> "processed " + data);
    }
}

The trace you see groups every invocation of the execution under one Workflow root:

Workflow                              (root, exported on terminal status)
├── Invocation #1
├── Invocation #2
├── Operation: fetch-data  (STEP)      -> link to Invocation #1
│   └── Attempt: fetch-data attempt 1  -> link to Invocation #1
├── Operation: cooldown    (WAIT)      -> link to Invocation #2
└── Operation: process     (STEP)      -> link to Invocation #2
    └── Attempt: process attempt 1     -> link to Invocation #2

Operation spans link to the invocation span that produced them, so you can tell which invocation ran each operation even though the operations root under the Workflow span.

InvocationOtelPlugin¶

InvocationOtelPlugin keeps the trace invocation-rooted. It opens a parentless Workflow root span in both provider modes, keyed to a deterministic ID from the execution ARN and ended only at the terminal invocation. The invocation span is not a child of that Workflow span. Each invocation span roots its own operation and attempt spans, and those spans carry a link to the Workflow span for execution-scoped correlation.

The plugin exports each invocation span as that invocation ends. A Lambda invocation lasts at most 15 minutes, so spans stay within platform limits and appear as the execution runs, which renders reliably across most platforms. Choose this plugin for long-running executions, or when you want per-invocation traces that still correlate to one workflow and to view an execution in progress.

import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js";
import { InvocationOtelPlugin } from "@aws/durable-execution-sdk-js-otel";

export const handler = withDurableExecution(
  async (event, context: DurableContext) => {
    return await context.step("process", async () => "done");
  },
  { plugins: [new InvocationOtelPlugin()] },
);

from aws_durable_execution_sdk_python import DurableContext, durable_execution
from aws_durable_execution_sdk_python_otel import InvocationOtelPlugin


@durable_execution(plugins=[InvocationOtelPlugin()])
def handler(event: dict, context: DurableContext) -> str:
    return context.step(lambda ctx: "done", name="process")

import software.amazon.lambda.durable.DurableConfig;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.otel.InvocationOtelPlugin;

public class InvocationOtelExample extends DurableHandler<Object, String> {
    @Override
    protected DurableConfig createConfiguration() {
        return DurableConfig.builder().withPlugins(new InvocationOtelPlugin()).build();
    }

    @Override
    protected String handleRequest(Object event, DurableContext context) {
        return context.step("process", String.class, ctx -> "done");
    }
}

The Workflow span is a parentless root that operation and attempt spans link to. Each Lambda invocation roots its own view. The example above suspends at the cooldown wait, so it runs across two invocations, and the wait produces a span in each:

Workflow                              (parentless root; deterministic ID; ended at the terminal invocation)

Invocation #1                         (per-invocation root; not nested under Workflow)
├── Operation: fetch-data  (STEP)      -> link to Workflow
│   └── Attempt: fetch-data attempt 1  -> link to Workflow
└── Operation: cooldown    (WAIT)      -> link to Workflow      (wait starts; execution suspends)

Invocation #2                         (per-invocation root; resumes after the wait)
├── Operation: cooldown    (WAIT)      -> link to the first cooldown span, and to Workflow
└── Operation: process     (STEP)      -> link to Workflow
    └── Attempt: process attempt 1     -> link to Workflow

An operation that completes in a later invocation, such as an invoke, wait, or callback, can appear as more than one span across invocations. Each later span carries a link back to the first span for that operation, so a retry or a resumed wait relates to its first appearance. CloudWatch does not yet visualize these links.

Choosing a plugin¶

The ExecutionOtelPlugin and InvocationOtelPlugin sections above cover when to choose each. This table summarizes the differences. For both plugins, your observability platform's quotas and limits apply.

Span structure and attributes¶

Both plugins emit a Workflow root span for the whole execution. Beneath it (ExecutionOtelPlugin) or linked to it (InvocationOtelPlugin) sit three levels of spans. An invocation span covers one Lambda invocation. Operation spans nest one per durable operation, such as a step, wait, or child invoke. Attempt spans nest one per try under a step or wait-for-condition operation, so retries appear as sibling spans.

durable.operation.type is one of STEP, WAIT, CONTEXT, CHAINED_INVOKE, or CALLBACK. A CONTEXT operation (a child context) gets an operation span but no attempt span, since attempt spans apply only to steps and wait-for-conditions.

Deploy with the ADOT layer¶

The AWS Distro for OpenTelemetry (ADOT) Lambda layer bundles OpenTelemetry auto-instrumentation and a collector extension. The collector listens on localhost:4318 and forwards spans to CloudWatch. Add the layer, enable active tracing so the runtime populates the _X_AMZN_TRACE_ID header the plugin reads, and grant the function's role the AWSXRayDaemonWriteAccess managed policy.

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Runtime: nodejs24.x
    Handler: index.handler
    Layers:
      - !Sub arn:aws:lambda:${AWS::Region}:<account>:layer:<adot-js-layer>:<version>
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
    Tracing: Active
    Policies:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
      - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

new ExecutionOtelPlugin({ useDefaultTracerProvider: true });
// or
new InvocationOtelPlugin({ useDefaultTracerProvider: true });

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Runtime: python3.14
    Handler: index.handler
    Layers:
      - !Sub arn:aws:lambda:${AWS::Region}:<account>:layer:<adot-python-layer>:<version>
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
    Tracing: Active
    Policies:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
      - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Runtime: java25
    Handler: com.example.ExampleHandler
    Layers:
      - !Sub arn:aws:lambda:${AWS::Region}:<account>:layer:<adot-java-layer>:<version>
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
        OTEL_JAVAAGENT_EXTENSIONS: /opt/otel-plugin-extension.jar
    Tracing: Active
    Policies:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
      - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

new ExecutionOtelPlugin();   // uses the agent's global provider
new InvocationOtelPlugin();  // uses the agent's global provider

Deploy with the community collector layer¶

The OpenTelemetry community collector layer runs a collector extension only, without auto-instrumentation. The plugin creates its own tracer provider and exports spans to the collector on localhost:4318. Do not set AWS_LAMBDA_EXEC_WRAPPER with this layer.

Include a collector.yaml in your function bundle and set OPENTELEMETRY_COLLECTOR_CONFIG_URI to its path:

receivers:
  otlp:
    protocols:
      http:
        endpoint: "localhost:4318"
exporters:
  awsxray:
    region: "${AWS_REGION}"
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray]

Routing spans through a collector also lets you export to a third-party platform such as Datadog, Honeycomb, or Grafana by changing the collector's exporter, without first sending them to CloudWatch. Check https://github.com/open-telemetry/opentelemetry-lambda/releases for the most recent Collector layer releases.

Layers:
  - !Sub arn:aws:lambda:${AWS::Region}:<account>:layer:<collector-layer>:<version>
Environment:
  Variables:
    OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml

new ExecutionOtelPlugin({ useDefaultTracerProvider: false });
new InvocationOtelPlugin({ useDefaultTracerProvider: false });

Layers:
  - !Sub arn:aws:lambda:${AWS::Region}:<account>:layer:<collector-layer>:<version>
Environment:
  Variables:
    OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml

Layers:
  - !Sub arn:aws:lambda:${AWS::Region}:<account>:layer:<collector-layer>:<version>
Environment:
  Variables:
    OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml

var exporter = OtlpHttpSpanExporter.getDefault();
var plugin = new ExecutionOtelPlugin(
        SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)));

Configuration¶

new InvocationOtelPlugin();
new InvocationOtelPlugin(tracerProviderBuilder);
new InvocationOtelPlugin(tracerProviderBuilder, contextExtractor);
new InvocationOtelPlugin(tracerProviderBuilder, contextExtractor, enableMdc);
new InvocationOtelPlugin(tracerProviderBuilder, contextExtractor, enableMdc, workflowSpanName);

Correlate logs with traces¶

The plugin stamps the active trace and span IDs onto your log records, so a log line in CloudWatch links to the span that emitted it. See Logging for the SDK logger.

Verify¶

Invoke a durable function that includes a wait or a resume, so the execution runs across more than one invocation. In the CloudWatch console, open Traces and confirm the invocation and operation spans appear under one trace ID. Check that your log entries carry traceId, spanId, and otelTraceSampled matching those spans.

When you use the community collector layer, enable CloudWatch Transaction Search in your account for traces to appear. If no traces show up, the collector layer is missing or its configuration variable is unset. If traces fragment across several IDs, active tracing is off. If some operation spans are missing, the sampling ratio is below 1.0.

See also¶

• Plugins
• Logging
• Steps