Skip to content

feat: add itemNamer for map operation - #576

Closed
wangyb-A wants to merge 8 commits into
mainfrom
feature/item-namer-for-map
Closed

feat: add itemNamer for map operation#576
wangyb-A wants to merge 8 commits into
mainfrom
feature/item-namer-for-map

Conversation

@wangyb-A

@wangyb-A wangyb-A commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Description

Adds itemNamer functionality to MapConfig, allowing custom naming of map iterations. This feature provides parity with the Python SDK implementation (PR #387) while adapting to Java's existing API patterns.

Changes

  1. MapConfig.java:

    • Added itemNamer: BiFunction<Object, Integer, String> field
    • Added getter itemNamer() method
    • Updated builder to support itemNamer() configuration
    • Maintained backward compatibility
  2. MapOperation.java:

    • Updated to use custom iteration names when itemNamer is provided
    • Falls back to default naming ("map-iteration-N" or "name-iteration-N") when no itemNamer is set
  3. Unit Tests:

    • MapConfigItemNamerTest.java: 9 comprehensive tests covering all itemNamer functionality
    • MapOperationItemNamerTest.java: 3 integration tests for MapOperation construction
    • All 27 existing map tests continue to pass

Usage

MapConfig config = MapConfig.builder()
    .itemNamer((item, index) -> "process-" + item + "-iteration-" + index)
    .maxConcurrency(2)
    .build();

// Map iterations will be named "process-order-101-iteration-0", etc.
// Instead of default "map-iteration-0", "map-iteration-1"

Testing

  • ✅ All new unit tests pass
  • ✅ All existing map-related tests pass (backward compatibility)
  • ✅ Local integration test verified functionality
  • ✅ Feature matches Python SDK reference implementation

Notes

  • Java already has named parallel branches via branch(String name, ...), so no ParallelBranch wrapper was needed (unlike Python)
  • Implementation follows Java's builder pattern and existing API conventions
  • Resolves issue [Feature]: Item namer for map operation #528

@github-actions

This comment has been minimized.

Comment thread sdk/src/main/java/software/amazon/lambda/durable/config/MapConfig.java Outdated
Comment thread sdk/src/main/java/software/amazon/lambda/durable/operation/MapOperation.java Outdated
@github-actions

This comment has been minimized.

@wangyb-A
wangyb-A force-pushed the feature/item-namer-for-map branch from dc25639 to 3eec842 Compare July 31, 2026 21:05
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 21:05 — with GitHub Actions Inactive
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 21:05 — with GitHub Actions Inactive
Comment thread sdk/src/main/java/software/amazon/lambda/durable/config/MapConfig.java Outdated
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@wangyb-A
wangyb-A force-pushed the feature/item-namer-for-map branch from 3eec842 to 58be1d7 Compare July 31, 2026 21:14
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 21:14 — with GitHub Actions Inactive
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 21:14 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

Comment thread sdk/src/main/java/software/amazon/lambda/durable/operation/MapOperation.java Outdated
@github-actions

This comment has been minimized.

@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 21:32 — with GitHub Actions Inactive
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 21:32 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@wangyb-A
wangyb-A marked this pull request as ready for review July 31, 2026 21:46
@wangyb-A
wangyb-A requested a review from a team July 31, 2026 21:46
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 21:47 — with GitHub Actions Inactive
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 21:47 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

public NestingType nestingType;
private NestingType nestingType;
private Integer maxConcurrency;
private CompletionConfig completionConfig;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This narrows MapConfig.Builder.nestingType from public to private, an unrelated source-incompatible change bundled into this feature PR. Any external code that read/wrote the field directly (MapConfig.builder().nestingType = ...) will fail to compile against the new SDK. AGENTS.md states "Do not change public API signatures ... without instruction."

Encapsulating the field is the right long-term design (it matches how the other builder fields and ParallelConfig are declared), but since it is a breaking change orthogonal to itemNamer, consider either splitting it into its own change with a note in the changelog/release notes, or explicitly calling it out in the PR description so it is a deliberate, documented break rather than an incidental one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and reverted in 0d16426. Confirmed against origin/main, which has public NestingType nestingType; — this branch narrowed it to private. That was incidental tidying on my part with no connection to itemNamer, and you are right that it breaks already-compiled clients. Restored to public; whether that field should be public at all is a separate call, not something this PR should decide as a side effect.

Two other findings from the same round were raised outside this thread, so recording the outcome here for traceability:

Iteration names resolved after START (fixed in the same commit). execute() sends the START checkpoint and calls addAllItems() on the next line, so the ParameterValidator call added in d61dd05 threw after the map was STARTED and its start plugin hook had fired. If the caller caught the IllegalArgumentException, a permanently STARTED map and an unbalanced plugin lifecycle were left behind. Name resolution and validation now happen in the MapOperation constructor, before any checkpoint. Added testCaughtItemNamerValidationLeavesNoHalfOpenMap, which swallows the failure and asserts the rejected map was never checkpointed and that replay still succeeds — verified non-vacuous by running it against d61dd05, where it fails with failed map should not be checkpointed ... but was: TestOperation@f438904.

Parameterizing MapConfig by input type — not done here, deliberately. The WaitForConditionConfig<T> precedent is real and the ergonomic complaint is legitimate; casting Object for domain items is poor. Two reasons I have left it out of this PR rather than silently doing it: MapConfig appears in 28 files, so it turns a focused feature change into a public-API reshape reviewed under the wrong heading; and it sits in tension with the visibility finding above, where the standard applied was to avoid incidental API changes in a feature PR. Worth noting the change is less breaking than it appears — MapConfig<I> erases to MapConfig, so the four DurableContext.map(...) overloads stay binary compatible and existing source keeps compiling with raw-type warnings. Happy to do it as a follow-up, or here if you would rather it land together.

1112 sdk tests and 393 integration tests pass; spotless:check clean project-wide.

@github-actions

This comment has been minimized.

@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 22:31 — with GitHub Actions Inactive
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 22:31 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@wangyb-A
wangyb-A had a problem deploying to ai-pr-review-runtime July 31, 2026 22:45 — with GitHub Actions Failure
@wangyb-A
wangyb-A had a problem deploying to ai-pr-review-runtime July 31, 2026 22:45 — with GitHub Actions Failure
Alex Wang added 7 commits July 31, 2026 22:49
Adds itemNamer functionality to MapConfig, allowing custom naming of map
iterations. This feature provides parity with the Python SDK implementation
while adapting to Java's existing API patterns.

Changes:
- Add itemNamer: BiFunction<Object, Integer, String> field to MapConfig
- Update MapOperation to use custom iteration names when itemNamer is provided
- Add comprehensive unit tests for both MapConfig and MapOperation
- Maintain backward compatibility (all existing tests pass)

Usage:
  MapConfig config = MapConfig.builder()
      .itemNamer((item, index) -> "process-" + item + "-iteration-" + index)
      .build();

Resolves: #528
MapConfig.itemNamer makes requirement 9-13 (map with a custom item namer)
satisfiable for Java, so drop its NotImplemented declaration and add the
handler plus template resource.

Validated end to end against us-west-2: the java map suite reports 9-13
PASSED with 15 PASSED / 5 NOT_IMPLEMENTED / 0 FAILED, and the recorded
history carries the custom names (Map "named-items", iterations "item-1"
and "item-2"), matching the JS and Python results exactly.
Spotless: java.util.function.BiFunction was wedged between the two org.slf4j
imports, and MapConfig plus both new test files were missing trailing
newlines. Fixed via spotless:apply; mvn spotless:check now passes.

Docs: the getter and builder Javadoc claimed the default iteration name is
"map-item-N". The actual default from MapOperation.addUnskippedItems is
"<mapName>-iteration-N". Corrected both occurrences.

Tests: the existing unit tests only covered the config POJO and MapOperation
construction, so nothing executed the naming path. Added LocalDurableTestRunner
integration tests that run a map and assert the custom names reach the child
operations and survive replay without re-execution, plus a test pinning the
default naming so the Javadoc and runtime cannot drift again.

Verified the new test is discriminating by mutation: with the itemNamer branch
disabled it fails on "custom iteration name should be checkpointed".

Also pinned the duplicate-name and null-name behavior surfaced in review.
Both are tolerated and results stay correct, but a null from the namer is
passed through as the operation name rather than falling back to the default,
and duplicates make name-based lookup ambiguous.
…e custom names

A null from the itemNamer previously became the iteration's operation name,
suppressing the long-standing "<mapName>-iteration-N" default and leaving the
iteration unaddressable by name. It now falls back to the default, per item
rather than all-or-nothing.

Custom names also bypassed the validation that DurableContextImpl.mapAsync
applies to the map's own name, so an empty, over-long or non-ASCII name reached
the backend and failed at checkpoint time in the cloud. The namer result is now
passed through ParameterValidator.validateOperationName, which fails fast at
map time. Confirmed by mutation that an empty name previously succeeded.

Javadoc on both the getter and the builder now documents the null fallback, the
name constraints, and the determinism requirement that replay's name comparison
imposes.
…es before START

Builder.nestingType was public on main and this branch had narrowed it to
private, which breaks source and binary compatibility for already-compiled
clients. Restored to public; narrowing it is a separate decision, not a
side effect of adding itemNamer.

Iteration names are now resolved and validated in the MapOperation constructor
instead of in addUnskippedItems. addUnskippedItems runs immediately after the
map checkpoints START and fires its start plugin hook, so a namer that threw or
produced an invalid name left a permanently STARTED map with an unbalanced
plugin lifecycle whenever the caller caught the IllegalArgumentException.
Resolving in the constructor means the failure happens before any checkpoint.

Added testCaughtItemNamerValidationLeavesNoHalfOpenMap, which swallows the
validation failure and asserts the rejected map was never checkpointed and that
replay still succeeds. Verified against the previous commit, where it fails with
"failed map should not be checkpointed ... but was: TestOperation".
MapConfig is now MapConfig<I> with Builder<I>, and itemNamer is typed
BiFunction<I, Integer, String> instead of BiFunction<Object, Integer, String>,
following the existing WaitForConditionConfig<T> precedent. Callers naming
domain objects no longer need a cast:

    MapConfig.<Order>builder().itemNamer((order, i) -> "order-" + order.id())

The map APIs accept MapConfig<? super I> rather than MapConfig<I>. This matters
for source compatibility: MapConfig.builder().build() in an argument position
infers MapConfig<Object>, which does not match MapConfig<I>, so the stricter
signature broke every existing inline call site at compile time. The
consumer-position wildcard accepts both MapConfig<Object> from existing code and
an explicitly typed MapConfig<Order>. Confirmed by leaving all pre-existing call
sites in examples/ and conformance-tests/ untouched; they still compile.

MapConfigItemNamerTest declarations moved from the raw MapConfig type to
MapConfig<Object>, since a raw type erases itemNamer()'s return and made apply()
yield Object. Added coverage for the cast-free domain-typed path at both the
config level and end to end through map().
Not related to itemNamer. PR #574 (a46235f) merged plugin conformance handlers
that spotless had never checked, because check-spotless.yml filters on
sdk/**, sdk-testing/**, sdk-integration-tests/** and examples/** while running
spotless:check project-wide via --file pom.xml. A PR touching only
conformance-tests/** therefore never triggers the job, but the next PR that
touches sdk/** inherits the failure.

This is the second time on this branch: 570a195 left the same situation and was
resolved by rebasing because main had already been fixed. This time main is the
source, so the reformat has to land here to get CI green.

Reformatted with JDK 17 to match the CI runner and verified the result is stable
under JDK 21, so it will not flip-flop between local and CI runs. Only the six
inherited files changed; no sdk/ or sdk-integration-tests/ file was touched.
@wangyb-A
wangyb-A force-pushed the feature/item-namer-for-map branch from a9f68ac to 4c94a18 Compare July 31, 2026 22:51
When a completed map's result is small enough to replay from the checkpoint
payload, its children are enqueued but never executed, so the per-operation
validateReplay name comparison never runs. A namer whose output changed between
invocations therefore replayed silently under the old checkpointed names.

MapOperation now compares the regenerated iteration names against the
checkpointed child operations in the SUCCEEDED replay path and raises
NonDeterministicExecutionException on mismatch, matching how the per-operation
guard reports a name mismatch.

Added ExecutionManager.peekOperation for this: the existing
getOperationAndUpdateReplayState transitions REPLAY to EXECUTION as a side effect
when the looked-up operation is absent or non-terminal, which would corrupt
replay state if used for read-only validation.

Tests: testChangedItemNamerOnCachedReplayIsRejected covers the reported gap, and
testStableItemNamerReplaysCleanlyFromCache is the control proving an unchanged
namer still replays from cache. Verified by mutation that removing the new call
makes the first test fail with "expected: not equal but was: <SUCCEEDED>",
confirming the gap was real and that the guard is what closes it.

This also supersedes the caveat noted earlier on this PR, where a
non-deterministic namer was observed replaying without the guard firing.
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 23:09 — with GitHub Actions Inactive
@wangyb-A
wangyb-A temporarily deployed to ai-pr-review-runtime July 31, 2026 23:09 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

Codex AI review

  • [P1] Resolve names before consuming the operation IDsdk/src/main/java/software/amazon/lambda/durable/operation/MapOperation.java:87
    A caught namer or validation exception leaves an uncheckpointed ID hole. If the namer later succeeds on replay, new work can silently occupy that hole while subsequent operations replay normally. Resolve names before nextOperationId() and add a replay test covering this transition.

  • [P1] Validate names for flat-map replayssdk/src/main/java/software/amazon/lambda/durable/operation/MapOperation.java:126
    FLAT iteration operations are virtual, so peekOperation() always returns null and every comparison is skipped. A suspended map can therefore resume with changed context names and observable logging/plugin behavior without detecting nondeterminism. Persist names in the parent checkpoint or reject this combination, with a suspended FLAT replay test.

  • [P2] Preserve the existing DurableContext signaturesdk/src/main/java/software/amazon/lambda/durable/DurableContext.java:549
    Replacing the raw MapConfig parameter with MapConfig<? super I> is source-incompatible for downstream DurableContext implementations: their old override has the same erasure but no longer overrides this method. Keep the public signatures raw and adapt internally, or defer the signature change to a major release.

Reviewed commit 841bb2f2cb89ab1846616626ad9ce8ce12203a9c. Workflow run

@github-actions

Copy link
Copy Markdown

Claude AI review

Review: feat: add itemNamer for map operation (PR #576)

No actionable findings. This is a clean, well-tested change. I traced the full path — MapConfig<I> generification, the DurableContext/DurableContextImpl signature changes to MapConfig<? super I>, MapOperation.resolveIterationNames()/validateIterationNamesAgainstCheckpoint(), ExecutionManager.peekOperation(), and the existing name/replay validation in BaseDurableOperation and ChildContextOperation — and verified backward compatibility across all existing MapConfig call sites.

Verified correct:

  • Generic variance: MapConfig<? super I> correctly models config as an item consumer (PECS); itemNamer composes with MapOperation's BiFunction<? super I,Integer,String> and apply(items.get(i), i).
  • Backward compatibility: existing raw MapConfig.builder().build() sites still infer through the new MapConfig.<I>builder() defaults and the ? super I parameter; only unchecked warnings result, and no compiler config fails on warnings.
  • Fail-fast: resolving/validating names in the constructor — before START checkpoint and the start plugin hook — avoids a half-open STARTED map on namer throw/invalid name (confirmed by testCaughtItemNamerValidationLeavesNoHalfOpenMap).
  • Replay gap closed: small-result (cached-payload) maps markAlreadyCompleted() without executing children, so per-operation validateReplay never runs for iterations; validateIterationNamesAgainstCheckpoint() fills that gap via read-only peekOperation (no REPLAY→EXECUTION transition) and skips uncheckpointed/virtual children via the null-name guard.
  • ID stability: operation IDs stay counter-derived, so custom/duplicate/null names don't affect ordering or replay matching.

Test coverage is comprehensive across config, construction, typed items, null/partial-null fallback, invalid-name rejection, duplicate names, changed-vs-stable namer on cached replay, and half-open recovery.

Residual test risk (disclosed; read-only review, no build/run):

  • Could not execute mvn, so cannot confirm the suite passes or spotless formatting is clean — only that the changes are type-safe and logically consistent.
  • FLAT/virtual maps get no iteration-name replay validation (children aren't checkpointed by name). By design and consistent with prior behavior, but there is no negative test asserting a changed namer under NestingType.FLAT is not rejected (MapOperation.java:863 / validateIterationNamesAgainstCheckpoint).
  • The new mismatch check runs in replay() (after the map's potential fireOnOperationEnd continuation hook), later than the standard check in validateReplay(); termination there is safe, but plugin-hook balance around a mid-replay termination on the small-result path is not directly asserted.

Reviewed commit 841bb2f2cb89ab1846616626ad9ce8ce12203a9c. Workflow run

@wangyb-A wangyb-A closed this Jul 31, 2026
@wangyb-A
wangyb-A deleted the feature/item-namer-for-map branch July 31, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant