Skip to content

[Feature]: Load plugins dynamically from Lambda layers #618

Description

@zhongkechen

What would you like?

I would like the SDK to support opt-in discovery and registration of
DurableInstrumentationPlugin implementations from runtime dependencies during
cold start.

The target Lambda workflow is:

  1. Publish a plugin package in a Lambda layer under python.
  2. Attach the layer to an existing function.
  3. Set configuration such as
    DURABLE_EXECUTION_PLUGINS=otel,example_audit.
  4. Have the SDK load and register those plugins without rebuilding or
    repackaging the function's application artifact.

Today plugins must be imported and passed explicitly through
@durable_execution(plugins=[...]). Making a package importable from a Lambda
layer is not sufficient to register it.

The dynamic path should:

  • Be explicitly enabled; an unset configuration must preserve current behavior.
  • Load plugins once when the decorated handler is initialized.
  • Preserve configured ordering and support multiple plugins.
  • Resolve providers from the normal application and Lambda-layer import paths.
  • Validate provider type and SDK/API compatibility with actionable failures.
  • Define duplicate handling when a plugin is both explicitly and dynamically
    registered.
  • Preserve the decorator's plugins argument for applications that prefer
    code-based registration.

Possible Implementation

Define a plugin provider/factory entry-point group, for example
aws_durable_execution.plugins, and discover providers with
importlib.metadata.entry_points(). Each provider could expose a stable name
and return a DurableInstrumentationPlugin; the environment variable would
select an explicit allow-list of provider names.

Direct module:factory references loaded with importlib could also be
supported, but named package entry points would provide a cleaner distribution
and compatibility contract.

Provider-specific settings could remain in namespaced environment variables.

Is this a breaking change?

No

Does this require an RFC?

Yes

Additional Context

The immediate use case is distributing observability integrations, including
OpenTelemetry, independently from the function artifact. The same capability is
useful for logging, metrics, auditing, and other instrumentation plugins.

Attaching a layer and updating environment variables is still a Lambda
configuration deployment, but it avoids rebuilding and republishing application
code solely to add or update instrumentation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions