OpenTelemetry¶

The OpenTelemetry plugins trace durable operations across Lambda invocations. Create a factory for the execution view (the default) or select the invocation view explicitly. Configure an OpenTelemetry provider through a telemetry layer or application code. The provider owns exporters, resources, library instrumentation, and sampling policy.

The plugins are available for TypeScript, Python, and Java. Go, Rust, and C# do not yet provide this integration.

Install the plugin¶

These examples use the unified factory APIs available in the versions below. Upgrade the core and plugin together, including copies supplied by Lambda layers.

npm install @aws/durable-execution-sdk-js@^3.0.0 \
  @aws/durable-execution-sdk-js-otel@^2.1.0 \
  @opentelemetry/api @opentelemetry/core @opentelemetry/sdk-trace-node

pip install "aws-durable-execution-sdk-python-otel>=2.1.0,<3"

pip install "aws-durable-execution-sdk-python-otel[standalone]>=2.1.0,<3"

<dependency>
    <groupId>software.amazon.lambda.durable</groupId>
    <artifactId>aws-durable-execution-sdk-java-plugin-otel</artifactId>
    <version>3.1.0</version>
</dependency>

Choose and register a view¶

Register one factory using the unified entry point for your language. Omitting the mode selects the execution view. The SDK creates a separate plugin instance for each invocation, while the provider can be shared across invocations.

Choosing a view¶

For both views, your observability platform's quotas and limits apply.

Nested operations remain under their child-context operation span; attempts remain under their operation. Both views export Invocation spans as invocations end and Workflow only when the SDK observes execution success or failure. Workflow spans can cover long durations, so backend ingestion limits still apply to both views. Export and flush are best effort; a timeout, stop, or terminated runtime can leave spans missing.

The diagrams below show an execution with a valid remote backend parent and ambient Lambda spans in the same trace. See trace identity and sampling for other parent contexts. The tree shows parent-child relationships; arrows show span links.

Execution view¶

Top-level operation spans are children of Workflow and link to the current Invocation. The factory selects this view by default. An operation that suspends is exported when it completes, using its durable start and end times. Its span links to the invocation that exports the completion; it does not accumulate links to every earlier invocation.

import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js";
import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel";

export const handler = withDurableExecution(
  async (event: { id: string }, context: DurableContext) => {
    const data = await context.step("fetch-data", async () => event.id);
    await context.wait("cooldown", { seconds: 5 });
    return await context.step("process", async () => `processed ${data}`);
  },
  { plugins: [createOtelPluginFactory()] },
);

from aws_durable_execution_sdk_python import DurableContext, durable_execution
from aws_durable_execution_sdk_python.config import Duration
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory


@durable_execution(plugins=[create_otel_plugin_factory()])
def handler(event: dict, context: DurableContext) -> str:
    data = context.step(lambda ctx: event["id"], name="fetch-data")
    context.wait(Duration.from_seconds(5), name="cooldown")
    return context.step(lambda ctx: f"processed {data}", name="process")

import java.time.Duration;
import java.util.Map;
import software.amazon.lambda.durable.DurableConfig;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.otel.OtelPlugin;

public class ExecutionOtelExample extends DurableHandler<Map<String, String>, String> {
    @Override
    protected DurableConfig createConfiguration() {
        return DurableConfig.builder().withPlugins(OtelPlugin.factory()).build();
    }

    @Override
    public String handleRequest(Map<String, String> event, DurableContext context) {
        String data = context.step("fetch-data", String.class, ctx -> event.get("id"));
        context.wait("cooldown", Duration.ofSeconds(5));
        return context.step("process", String.class, ctx -> "processed " + data);
    }
}

For an execution that suspends at cooldown and resumes in a second invocation, the trace resembles:

Remote backend server span
|-- Workflow                              (exported on terminal status)
|   |-- Operation: fetch-data  (STEP)      -> link to Invocation #1
|   |   `-- Attempt: fetch-data attempt 1 -> link to Invocation #1
|   |-- Operation: cooldown    (WAIT)      -> link to Invocation #2
|   `-- Operation: process     (STEP)      -> link to Invocation #2
|       `-- Attempt: process attempt 1    -> link to Invocation #2
|-- Ambient Lambda span #1
|   `-- Invocation #1
`-- Ambient Lambda span #2
    `-- Invocation #2

The links to Invocation do not make invocation spans children of Workflow.

Invocation view¶

Top-level operation spans are children of the Invocation that emitted them and link to Workflow, so each invocation's spans export as it ends. Select invocation mode explicitly. Completed work uses checkpointed results on replay. A resumed operation or replayed context can produce another span segment without rerunning a completed step's body.

import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js";
import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel";

export const handler = withDurableExecution(
  async (event, context: DurableContext) => {
    return await context.step("process", async () => "done");
  },
  { plugins: [createOtelPluginFactory({ mode: "invocation" })] },
);

from aws_durable_execution_sdk_python import DurableContext, durable_execution
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory


@durable_execution(plugins=[create_otel_plugin_factory(mode="invocation")])
def handler(event: dict, context: DurableContext) -> str:
    return context.step(lambda ctx: "done", name="process")

import software.amazon.lambda.durable.DurableConfig;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.otel.OtelPlugin;

public class InvocationOtelExample extends DurableHandler<Object, String> {
    @Override
    protected DurableConfig createConfiguration() {
        return DurableConfig.builder().withPlugins(OtelPlugin.factory(OtelPlugin.Mode.INVOCATION)).build();
    }

    @Override
    public String handleRequest(Object event, DurableContext context) {
        return context.step("process", String.class, ctx -> "done");
    }
}

The same execution, using invocation view, produces this hierarchy:

Remote backend server span
|-- Workflow                               (exported on terminal status)
|-- Ambient Lambda span #1
|   `-- Invocation #1
|       |-- Operation: fetch-data  (STEP)  -> link to Workflow
|       |   `-- Attempt: fetch-data attempt 1
|       |       `-> link to Workflow
|       `-- Operation: cooldown    (WAIT)  -> link to Workflow
`-- Ambient Lambda span #2
    `-- Invocation #2
        |-- Operation: cooldown    (WAIT)  -> links to initial cooldown
        |                                      span and Workflow
        `-- Operation: process     (STEP)  -> link to Workflow
            `-- Attempt: process attempt 1 -> link to Workflow

An operation that crosses an invocation boundary can produce more than one span. A continuation segment uses a new span ID and links to the initial logical operation span, which uses the deterministic operation span ID. It also retains the Workflow link. CloudWatch does not yet visualize span links.

Configure a telemetry layer¶

A language auto-instrumentation layer registers the global provider used by the default factories. Select a layer compatible with your runtime, architecture, and region:

The following SAM snippets show the telemetry settings for an AWS::Serverless::Function resource. Merge the entries under Properties with your function's existing layers, environment variables, and policies. SAM applies Policies only when it creates the execution role. If you specify Role, attach the required permissions directly to that role.

ADOT layer¶

For ADOT with X-Ray export, enable Active tracing and grant AWSXRayDaemonWriteAccess alongside the durable execution permissions.

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Layers:
      - <adot-language-layer-arn>
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
    Tracing: Active
    Policies:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
      - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

Community language layer¶

For a community language layer exporting directly to an OTLP backend, use PassThrough tracing. Lambda passes the upstream X-Ray trace header to the function without emitting native Lambda X-Ray segments. The layer and durable plugin still provide application instrumentation.

Propagate the caller's trace context and sampling decision. Without an upstream header, Lambda generates a header with Sampled=0; the documented plugins honor that decision even with a local always_on sampler. See supported sampling modes.

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Layers:
      - <community-language-layer-arn>
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
        OTEL_TRACES_EXPORTER: otlp
        OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
        OTEL_EXPORTER_OTLP_ENDPOINT: https://<otlp-endpoint>
        OTEL_METRICS_EXPORTER: none
        OTEL_LOGS_EXPORTER: none
    Tracing: PassThrough
    Policies:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy

Exporting to CloudWatch: Enable Transaction Search in the destination account and Region, and confirm that Ingest OpenTelemetry spans shows Enabled. Otherwise, CloudWatch's OTLP endpoint rejects the spans. This setting is not required when exporting directly to another observability provider.

Configure OTEL_EXPORTER_OTLP_HEADERS if your backend requires authentication. Direct OTLP export to another observability provider does not require X-Ray write permissions. SAM still adds an X-Ray policy when it creates a role for Tracing: PassThrough.

With Active tracing, a durable span can reference a native Lambda parent sent only to X-Ray. The plugin preserves that parent without re-exporting it, so a shared trace ID does not guarantee a complete parent tree in an OTLP backend. If you need native Lambda platform spans, use an integration that receives both OTLP spans and their X-Ray ancestors.

PassThrough does not supply missing caller spans. For a complete upstream hierarchy, export the real caller span to the same backend and propagate its trace ID, span ID, and sampling decision through X-Amzn-Trace-Id when using the default plugin extractors. Preserve this propagation across downstream durable invokes.

You can optionally export telemetry through a Collector extension, which does not initialize the OpenTelemetry provider required by the plugin.

Provider initialization¶

These requirements apply to both layer choices.

OTEL_JAVAAGENT_EXTENSIONS=/opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-<version>.jar

If the global SDK provider or required durable ID integration is unavailable, the plugin logs a warning and disables its telemetry for that invocation. The next invocation retries initialization.

Load the factory without handler registration¶

DURABLE_EXECUTION_PLUGINS selects a default-configured OTel factory from an installed package or layer. These factories use the global OpenTelemetry tracer provider initialized by your telemetry layer or application. Choose one view through either code or environment configuration.

The value follows each SDK's discovery mechanism: an importable module specifier or a registered provider name.

Conflicting plugin factories otel-execution (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin) in exclusive group 'durable-otel-view'. Configure only one plugin from this group.

Configure an application-owned provider¶

This setup is for advanced use cases that require direct control over the OpenTelemetry provider. You do not need it when using a language auto-instrumentation layer: the layer initializes the global provider used by the plugin's default factories.

Configure the exporter and processors once, then register the resulting factory as in the handler examples above. The plugin flushes at invocation end but does not take over provider shutdown. These examples send OTLP HTTP to a collector listening on loopback port 4318.

npm install @opentelemetry/exporter-trace-otlp-http

import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { BatchSpanProcessor, NodeTracerProvider } from "@opentelemetry/sdk-trace-node";

const pluginFactory = createOtelPluginFactory({
  tracerProviderFactory: (createIdGenerator) => {
    const provider = new NodeTracerProvider({
      idGenerator: createIdGenerator(),
      spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter({
        url: "http://localhost:4318/v1/traces",
      }))],
    });
    provider.register();
    return provider;
  },
});

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from aws_durable_execution_sdk_python_otel import (
    create_otel_plugin_factory,
    OtelPluginConfig,
)

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
))
trace.set_tracer_provider(provider)
plugin_factory = create_otel_plugin_factory(
    OtelPluginConfig(tracer_provider=provider)
)

plugin_factory = create_otel_plugin_factory(
    OtelPluginConfig(tracer_provider=provider), mode="invocation"
)

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
    <version>1.67.0</version>
</dependency>

import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import software.amazon.lambda.durable.otel.OtelPlugin;

var exporter = OtlpHttpSpanExporter.builder()
        .setEndpoint("http://localhost:4318/v1/traces")
        .build();
var pluginFactory = OtelPlugin.factory(
        OtelPlugin.Mode.EXECUTION,
        SdkTracerProvider.builder()
                .addSpanProcessor(BatchSpanProcessor.builder(exporter).build()));

Plugin options¶

A custom extractor must return the durable execution's own context consistently across resumes. An unrelated or per-invocation trace ID splits the execution trace.

Trace identity and sampling¶

Both views keep one canonical trace ID per execution. A valid propagated X-Ray Root supplies that ID, and a valid Parent becomes the remote execution ancestor. Missing Sampled does not invalidate a complete remote parent. Workflow has a stable execution-scoped span ID and parents to that ancestor. Invocation uses an active ambient span, such as the Lambda span, only when it belongs to the same trace; otherwise it parents to the execution ancestor.

Execution ancestor
|-- Workflow
|-- Ambient Lambda span #1
|   `-- Invocation #1
`-- Ambient Lambda span #2
    `-- Invocation #2

When no complete remote parent is available, the SDK uses a synthetic DurableExecutionRoot. It retains a valid propagated trace ID, or derives one from the execution ARN and stable execution start time. With a supported provider and that start time, sampled invocations export a zero-duration root with stable identity to preserve ancestry before the execution completes. Resumes can export another copy. Workflow carries duration and outcome; the fallback root carries neither. Remote backend spans are owned by the backend and are never exported by the plugin.

Configure a supported sampling mode through the provider, or through OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG when using auto-configuration. Keep the plugin's sampler integration installed so durable spans share the resolved decision while other instrumentation retains its own policy.

Supported sampling modes¶

These rules apply to TypeScript core 3.0.0 / OTel 2.1.0, Python core 3.0.2 / OTel 2.1.0, and Java SDK/plugin 3.1.0, in both views. These versions include the unified factory APIs used by this guide.

• Explicit upstream decision: a valid Sampled=0 or Sampled=1 overrides the local sampler for durable spans. Thus Sampled=1 still exports durable spans with local always_off, and Sampled=0 drops them with local always_on. Active tracing alone does not imply Sampled=1.
• Otherwise: use a deterministic policy based on the canonical trace ID, such as always_on, always_off, or traceidratio with a fixed ratio. For traceidratio, OTEL_TRACES_SAMPLER_ARG=0.5 selects whole execution traces when the local policy is consulted.

Keep the policy, ratio, and any inherited upstream or ambient decision stable across retries, resumes, and processes. Random, time-, load-, or mutable-state-dependent policies are unsupported and can produce partial traces. The plugins do not checkpoint sampling decisions; caching or application persistence does not make another policy supported.

The provider or auto-configuration layer resolves the environment settings, defaults, and invalid values. They do not override an explicitly configured application provider. Keep the plugin's sampler integration installed; other application and ADOT spans retain their own policy.

Sampling behavior may change in future SDK/plugin releases. Issue #295 proposes letting the configured sampler decide whether to inherit upstream sampling, using standard direct and parentbased_* semantics. That proposal is not implemented in the versions listed above. Check the release notes and this section when upgrading.

Sampling does not guarantee export or a complete parent tree: a durable span may be sampled while its application parent is dropped, or its native X-Ray parent may be absent from OTLP. See the community layer setup for trace propagation and native X-Ray parents.

Span attributes and failures¶

Optional attributes such as durable.operation.name and durable.operation.subtype are emitted only when supplied.

Operation types include STEP, WAIT, CONTEXT, CHAINED_INVOKE, and CALLBACK. A context has an operation span without a separate attempt span. An invocation that will retry reports durable.invocation.status=RETRYING with OTel UNSET. A caught operation failure can have an ERROR span while the enclosing invocation and workflow succeed. Inspect operation status as well as exception events; a backend can report failure without diagnostic details.

Correlate logs and verify traces¶

Log correlation is enabled by default. traceId identifies the execution trace, spanId identifies the active span, and otelTraceSampled carries its sampled flag. Application-created spans on that trace can become the active context too.

npm install @aws-lambda-powertools/logger

import { Logger } from "@aws-lambda-powertools/logger";
import {
  createPowertoolsLogger,
  DurableContext,
  withDurableExecution,
} from "@aws/durable-execution-sdk-js";
import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel";

const logger = createPowertoolsLogger(new Logger({ serviceName: "orders" }));

export const handler = withDurableExecution(
  async (_event: unknown, context: DurableContext) => {
    context.configureLogger({ customLogger: logger });
    context.logger.info("Processing order");
    return "processed";
  },
  { plugins: [createOtelPluginFactory()] },
);

{
  "message": "Processing order",
  "traceId": "1234567890abcdef1234567890abcdef",
  "spanId": "1234567890abcdef",
  "otelTraceSampled": true
}

import logging

from aws_durable_execution_sdk_python import DurableContext, durable_execution
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory


root_logger = logging.getLogger()
root_logger.setLevel(logging.INFO)
formatter = logging.Formatter(
    "%(message)s traceId=%(traceId)s spanId=%(spanId)s "
    "otelTraceSampled=%(otelTraceSampled)s",
    defaults={"traceId": "-", "spanId": "-", "otelTraceSampled": "-"},
)
for log_handler in root_logger.handlers:
    log_handler.setFormatter(formatter)

root_logger.info("Function initialized")


@durable_execution(plugins=[create_otel_plugin_factory()])
def handler(event: dict, context: DurableContext) -> str:
    context.logger.info("Processing request")
    return "done"

Function initialized traceId=- spanId=- otelTraceSampled=-
Processing request traceId=1234567890abcdef1234567890abcdef spanId=1234567890abcdef otelTraceSampled=True

<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.6.6</version>
    <scope>runtime</scope>
</dependency>

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%msg traceId=%X{traceId:--} spanId=%X{spanId:--} otelTraceSampled=%X{otelTraceSampled:--}%n</pattern>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

import java.util.Map;
import org.slf4j.LoggerFactory;
import software.amazon.lambda.durable.DurableConfig;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.otel.OtelPlugin;

public class LogCorrelationExample extends DurableHandler<Map<String, String>, String> {
    static {
        LoggerFactory.getLogger(LogCorrelationExample.class).info("Function initialized");
    }

    @Override
    protected DurableConfig createConfiguration() {
        return DurableConfig.builder().withPlugins(OtelPlugin.factory()).build();
    }

    @Override
    public String handleRequest(Map<String, String> event, DurableContext context) {
        context.getLogger().info("Processing request");
        return context.step("process", String.class, stepContext -> {
            stepContext.getLogger().info("Processing step");
            return "done";
        });
    }
}

Function initialized traceId=- spanId=- otelTraceSampled=-
Processing request traceId=1234567890abcdef1234567890abcdef spanId=- otelTraceSampled=-
Processing step traceId=1234567890abcdef1234567890abcdef spanId=1234567890abcdef otelTraceSampled=true

Test with an execution that waits and resumes. Confirm that its spans share a trace ID, invocations appear as they finish, completed steps do not run again, and logs identify the active span. Workflow appears after terminal completion. Span links correlate the two views but do not change parent-child relationships; CloudWatch does not currently visualize these links.

If spans are missing, check provider warnings, exporter configuration, the layer's execution wrapper, and sampling. Workflow requires a terminal invocation to export it. An unresolved parent can belong to the remote backend or an undelivered fallback root.

See also¶

• Plugins
• Logging
• Durable Execution Telemetry
• Steps
• Feedback and discussion