[compare B] Operator extra links per attempt: the link opts in

[1fanwang]

TL;DR — operator extra links don't work per attempt (apache#71471). After a retry the button
under attempt 1 opens attempt 2's logs, which bites any provider linking to per-attempt logs
in EMR, Glue, Databricks, Dataproc or Livy. This is approach B of three: the link opts in,
rather than core keeping a row per attempt for every link.

One of three shapes for the same fix, drafted so they can be read side by side: A keeps a row
per attempt for every link (apache#71518), B lets the link opt in, C puts the row in
the task state store.

How

BaseOperatorLink gains keeps_a_link_per_attempt and xcom_key_for_try. Links that set the
flag get a row per attempt and are exempt from the retry clear; everything else behaves as it
does today.

The flag has to survive DAG serialization, and that is the cost. The api-server never sees a
provider's link class — it rebuilds every link as XComOperatorLink from a {name: xcom_key}
mapping that carries a string and nothing else. That mapping now carries
{"key": ..., "per_attempt": true} for opted-in links and stays a bare string otherwise.

Testing

Read path against a real metastore on 3.2.2, with rows present for both attempts, asking an
opted-in link and a plain one for each attempt.

Raw output

$ python b_test.py
opted-in  try 1 -> https://logs/attempt-1
opted-in  try 2 -> https://logs/attempt-2
not opted try 1 -> https://logs/attempt-2

Line 3 is the before state on the same path: a link that does not opt in still returns the
latest attempt's URL for attempt 1, which is today's behaviour and today's bug. Lines 1 and 2
are the same read with the flag set.

Key derivation on the same runtime:

default  try1 -> _link_X                 | try2 -> _link_X
opted-in try1 -> _link_attempt_1__link_X | try2 -> _link_attempt_2__link_X

Serialization round-trip, the risky part:

encoded: {'plain': '_link_P', 'opted': {'key': '_link_O', 'per_attempt': True}}
  plain: key=_link_P per_attempt=False  (legacy string, unchanged)
  opted: key=_link_O per_attempt=True

An existing serialized DAG reads back unchanged.

Versus A

Same per-attempt key and same purge exemption underneath, plus an opt-in. Links whose URL can
be recomputed from try_number write nothing extra, which A cannot express. The cost is a
serialization format change and one more concept in the public link API.

[1fanwang]

Reopened against apache/airflow as a draft so all three options sit on the issue.