[release-4.20] OCPBUGS-85271: fsync static pod cert and manifest writes for crash durability by sanchezl · Pull Request #2207 · openshift/library-go

sanchezl · 2026-05-07T19:28:20Z

This is a cherry-pick of #2176.

pkg/operator/staticpod/internal/atomicdir contains internal helpers for performing atomic operations on directories. This patch contains only a standalone swap function, which will be subsequenty used for synchronizing directory with the given state.

The function can be used to atomically sync a directory with the desired state. This uses atomicdir.swap implemented earlier.

Use atomicdir.Sync to write target secret/configmap directories to be synchronized with the relevant objects. Added unit tests, but the coverage is not complete. Particularly filesystem operations failing are not being tested.

atomicdir.Sync writes files to a staging directory, atomically swaps it with the target directory via renameat2(RENAME_EXCHANGE), then deletes the old data. Without fsync, file data lives only in the kernel page cache. On ungraceful shutdown the journal replays the swap and deletion (metadata), but the file data was never flushed, leaving truncated or empty files. Introduce an fsutil package with WriteFileFsync (write + fsync file + fsync parent directory) and Fsync (fsync a path) primitives. Use WriteFileFsync for all file writes so each file is individually durable, and fsync both parent directories after the swap to persist which inode each directory name points to.

writePod uses bare os.WriteFile plus a delete-then-write pattern for kubelet manifests. On ungraceful shutdown, the delete is journaled but the new file data may not have reached disk, leaving the manifest missing. Replace os.WriteFile with fsutil.WriteFileFsync, which writes, fsyncs the file, and fsyncs the parent directory in a single call, ensuring both the resource copy and the kubelet manifest are durable before the function returns.

openshift-ci · 2026-05-07T19:28:24Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

coderabbitai · 2026-05-07T19:28:27Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4fd218bc-fd3f-44d6-8ebb-18179ae54973

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci-robot · 2026-05-07T19:31:42Z

@sanchezl: This pull request references Jira Issue OCPBUGS-85271, which is invalid:

expected dependent Jira Issue OCPBUGS-85270 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Add atomicdir package

operator/staticpod/internal/atomicdir/swap_other: add missing import

Add atomicdir.Sync function

atomicdir: Set permission for each file separately

atomicdir: Split testing utils into a helper pkg

certsync+installer: Write object dirs atomically

atomicdir: fsync files and directories for crash durability

installerpod: fsync pod manifest writes for crash durability

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

sanchezl · 2026-05-07T19:47:58Z

/jira refresh

openshift-ci-robot · 2026-05-07T19:48:03Z

@sanchezl: This pull request references Jira Issue OCPBUGS-85271, which is invalid:

expected dependent Jira Issue OCPBUGS-85270 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

sanchezl · 2026-05-08T14:52:38Z

/retest

openshift-ci · 2026-05-08T15:04:49Z

@sanchezl: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

sanchezl · 2026-05-08T21:04:57Z

/payload-job periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-upgrade-fips

sanchezl · 2026-05-08T21:04:59Z

/payload-job periodic-ci-openshift-release-main-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-upgrade

sanchezl · 2026-05-12T18:55:24Z

/jira refresh

openshift-ci-robot · 2026-05-12T18:55:30Z

@sanchezl: This pull request references Jira Issue OCPBUGS-85271, which is invalid:

expected dependent Jira Issue OCPBUGS-85270 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rh-roman · 2026-05-14T18:44:23Z

/lgtm

openshift-ci · 2026-05-14T18:44:41Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rh-roman, sanchezl
Once this PR has been reviewed and has the lgtm label, please assign p0lyn0mial for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

pkg/operator/staticpod/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tchap and others added 8 commits May 7, 2026 15:20

Add atomicdir package

1e604b0

pkg/operator/staticpod/internal/atomicdir contains internal helpers for performing atomic operations on directories. This patch contains only a standalone swap function, which will be subsequenty used for synchronizing directory with the given state.

operator/staticpod/internal/atomicdir/swap_other: add missing import

43445d0

Add atomicdir.Sync function

56e39a3

The function can be used to atomically sync a directory with the desired state. This uses atomicdir.swap implemented earlier.

atomicdir: Set permission for each file separately

9e3c633

atomicdir: Split testing utils into a helper pkg

0210cf0

certsync+installer: Write object dirs atomically

95d966c

Use atomicdir.Sync to write target secret/configmap directories to be synchronized with the relevant objects. Added unit tests, but the coverage is not complete. Particularly filesystem operations failing are not being tested.

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 7, 2026

sanchezl changed the title ~~cherry pick 2176 to release 4.20~~ [release-4.20] OCPBUGS-85271: fsync static pod cert and manifest writes for crash durability May 7, 2026

sanchezl marked this pull request as ready for review May 7, 2026 19:31

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 7, 2026

openshift-ci Bot requested review from dgrisonnet and p0lyn0mial May 7, 2026 19:32

sanchezl mentioned this pull request May 7, 2026

[release-4.20] OCPBUGS-85271: fsync static pod cert and manifest writes for crash durability openshift/cluster-kube-apiserver-operator#2145

Open

openshift-ci Bot assigned rh-roman May 14, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label May 14, 2026

Conversation

sanchezl commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented May 7, 2026

Uh oh!

coderabbitai Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

openshift-ci-robot commented May 7, 2026

Uh oh!

sanchezl commented May 7, 2026

Uh oh!

openshift-ci-robot commented May 7, 2026

Uh oh!

sanchezl commented May 8, 2026

Uh oh!

openshift-ci Bot commented May 8, 2026

Uh oh!

sanchezl commented May 8, 2026

Uh oh!

sanchezl commented May 8, 2026

Uh oh!

sanchezl commented May 12, 2026

Uh oh!

openshift-ci-robot commented May 12, 2026

Uh oh!

rh-roman commented May 14, 2026

Uh oh!

openshift-ci Bot commented May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sanchezl commented May 7, 2026 •

edited

Loading

coderabbitai Bot commented May 7, 2026 •

edited

Loading