Skip to content

clientv3/concurrency: preserve leadership state on failed Resign - #22124

Open
AkashKumar7902 wants to merge 1 commit into
etcd-io:mainfrom
AkashKumar7902:agent/retry-failed-election-resign
Open

clientv3/concurrency: preserve leadership state on failed Resign#22124
AkashKumar7902 wants to merge 1 commit into
etcd-io:mainfrom
AkashKumar7902:agent/retry-failed-election-resign

Conversation

@AkashKumar7902

@AkashKumar7902 AkashKumar7902 commented Jul 19, 2026

Copy link
Copy Markdown

Refs #22123.

What changed

  • Preserve an election's local leadership state when Resign returns a transaction error.
  • Add an integration regression proving that a failed resignation can be retried successfully.

Why

Election.Resign previously cleared leaderKey and leaderSession even when the delete transaction returned an error. If the request never reached etcd, the server-side leader key remained while the client forgot how to remove it.

The existing create-revision comparison makes a retry safe: it cannot delete a newer incarnation of the same key. A successful transaction response still retires the local state, including when the comparison does not match.

Impact

Callers can retry Resign after an RPC error. Successful resignation behavior and the exported API are unchanged.

This is complementary to #21165 and distinct from #19598 and #21128. Those changes concern Campaign cleanup or waiting-session expiration rather than errors returned by Resign itself.

Backport candidate: release-3.7 and release-3.6.

Testing

  • Confirmed the regression fails on upstream main because the failed call clears Election.Key().
  • cd tests && go test ./integration/clientv3/concurrency -run '^TestElectionResignCanRetryAfterError$' -count=20
  • cd tests && go test ./integration/clientv3/concurrency -count=1
  • cd tests && go test -race ./integration/clientv3/concurrency -run '^TestElectionResignCanRetryAfterError$' -count=1
  • cd client/v3 && go test ./concurrency -count=1
  • cd client/v3 && go vet ./concurrency
  • make verify-lint
  • git diff --check

The aggregate make verify cannot complete faithfully on this Darwin/arm64 host: the local BOM generator excludes the amd64-only Antithesis SDK and therefore differs from the checked-in Linux BOM. All non-BOM verify targets were run separately.

Previously, `Election.Resign` cleared its local leadership state even
when the underlying delete transaction failed. This left the leader key
in etcd, but made subsequent `Resign` calls no-ops. This prevented
immediate retries and delayed leadership handoff until the lease
expired.

This change fixes the issue by:
- Preserving `leaderKey` and `leaderSession` if the transaction returns
  an error.
- Clearing the leadership state only after receiving a successful response.
- Adding an integration test that cancels a resignation, verifies the
  key remains, retries, and confirms successful deletion.

Retries remain safe because the existing create-revision comparison
prevents the deletion of a newer leader key. There are no changes to
the exported API.

Fixes etcd-io#22123

Signed-off-by: Akash Kumar <meakash7902@gmail.com>
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: AkashKumar7902
Once this PR has been reviewed and has the lgtm label, please assign spzala for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

Copy link
Copy Markdown

Hi @AkashKumar7902. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

1 participant