Operator extra links per attempt (draft B): the link opts in - #71521
Closed
1fanwang wants to merge 2 commits into
Closed
Operator extra links per attempt (draft B): the link opts in#715211fanwang wants to merge 2 commits into
1fanwang wants to merge 2 commits into
Conversation
Approach B for apache#71471: the link decides, rather than core keeping a row per attempt for every link. BaseOperatorLink gains keeps_a_link_per_attempt and xcom_key_for_try, and only links that set the flag get extra rows. The flag has to survive DAG serialization, since the api-server reads links through XComOperatorLink, which is rebuilt from {name: xcom_key} and knows nothing about the original class. That mapping now carries {key, per_attempt} for opted-in links and stays a bare string otherwise. Signed-off-by: 1fanwang <1fannnw@gmail.com>
2 tasks
Signed-off-by: 1fanwang <1fannnw@gmail.com>
This was referenced Aug 12, 2026
Contributor
Author
|
Going with #71522 instead: the task state store is where cross-retry state already lives, so the XCom clear stays absolute rather than gaining an exemption. Closing this one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR — operator extra links don't work per attempt (#71471). After a retry the button
under attempt 1 opens attempt 2's logs. It hits every bundled provider linking to per-attempt
logs: EMR, Glue, Databricks, Dataproc, Livy.
Draft B of three for the same bug — pick a shape and I'll finish it and close the others.
Recommended: C (#71522). See the issue for why; A is the fallback if this needs to ship on the 3.2 line.
How
BaseOperatorLinkgainskeeps_a_link_per_attemptandxcom_key_for_try. A link that setsthe flag gets a row per attempt, exempt from the retry clear; everything else behaves as it
does today.
The flag has to survive DAG serialization, and that is the cost. The api-server never sees a
provider's link class — it rebuilds every link as
XComOperatorLinkfrom a{name: xcom_key}mapping carrying a string and nothing else. That mapping now carries
{"key": ..., "per_attempt": true}for opted-in links and stays a bare string otherwise, soexisting serialized DAGs read back unchanged.
Testing
Read path against a real metastore on 3.2.2, rows present for both attempts, asking an
opted-in link and a plain one.
Raw output
Line 3 is the before state on the same path: without the flag, attempt 1 still returns the
latest attempt's URL. Lines 1 and 2 are the same read with it set.
Key derivation:
Serialization round-trip, the risky part: