fix(runtimes): set MPI SSH auth secret volume default mode to 0640

[harxhist]

What this PR does?:

• Set DefaultMode to 0640 on the MPI SSH auth Secret volume in the MPI plugin so mounted secret files (private key, authorized_keys) have restricted permissions.
• Update unit, framework, and integration test expectations to include DefaultMode: 0640 for MPI SSH auth volumes.
• In the framework component-builder test, add cmpopts.IgnoreFields(corev1.SecretVolumeSource{}, "DefaultMode") so the generic comparison focuses on Secret data; DefaultMode is asserted in MPI-specific tests.

Why we need it?

Explicit default mode improves security for MPI SSH keys (owner read/write, group read, others none) and avoids relying on cluster default behavior.

Which issue this PR fixes

Fixes 3262

Checklist:

• Unit tests passed
• Integration tests passed
• E2E tests passed
• Linter & fmt ran

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
• Our team will review your PR soon! cc @kubeflow/kubeflow-trainer-team

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Sridhar1030]

Thanks for the PR. The change to set explicit defaultMode 0640 on the MPI SSH auth Secret volume matches the issue and is a sensible hardening step. Test updates look consistent with the behavior change.

just a small non-blocking nit
the global IgnoreFields(..., "DefaultMode") in the framework component-builder test could mask defaultMode on other Secret volumes someday; spelling out 0640 on the expected Secret volumes and dropping the ignore would make the test stricter.

[google-oss-prow]

@Sridhar1030: changing LGTM is restricted to collaborators

Details

In response to this:

Thanks for the PR. The change to set explicit defaultMode 0640 on the MPI SSH auth Secret volume matches the issue and is a sensible hardening step. Test updates look consistent with the behavior change.

just a small non-blocking nit
the global IgnoreFields(..., "DefaultMode") in the framework component-builder test could mask defaultMode on other Secret volumes someday; spelling out 0640 on the expected Secret volumes and dropping the ignore would make the test stricter.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[google-oss-prow]

@Sridhar1030: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jiayuzhao05]

I think there is one important behavioral risk to verify before merge.

DefaultMode is applied to the entire MPI SSH auth secret volume, which means the same 0640 permission will be used for the private key, public key, and authorized_keys. That may be problematic for the private key specifically. In many SSH/OpenSSH environments, private keys are expected to be more restrictive, and group-readable permissions can trigger “permissions are too open” style errors.

So while the patch correctly updates the generated volume spec and the related tests, it does not yet prove that SSH runtime behavior remains valid with 0640 for the private key. I would recommend setting per-item modes instead of a single volume-wide DefaultMode.

[google-oss-prow]

@jiayuzhao05: changing LGTM is restricted to collaborators

Details

In response to this:

I think there is one important behavioral risk to verify before merge.

DefaultMode is applied to the entire MPI SSH auth secret volume, which means the same 0640 permission will be used for the private key, public key, and authorized_keys. That may be problematic for the private key specifically. In many SSH/OpenSSH environments, private keys are expected to be more restrictive, and group-readable permissions can trigger “permissions are too open” style errors.

So while the patch correctly updates the generated volume spec and the related tests, it does not yet prove that SSH runtime behavior remains valid with 0640 for the private key. I would recommend setting per-item modes instead of a single volume-wide DefaultMode.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[harxhist]

I think there is one important behavioral risk to verify before merge.

DefaultMode is applied to the entire MPI SSH auth secret volume, which means the same 0640 permission will be used for the private key, public key, and authorized_keys. That may be problematic for the private key specifically. In many SSH/OpenSSH environments, private keys are expected to be more restrictive, and group-readable permissions can trigger “permissions are too open” style errors.

So while the patch correctly updates the generated volume spec and the related tests, it does not yet prove that SSH runtime behavior remains valid with 0640 for the private key. I would recommend setting per-item modes instead of a single volume-wide DefaultMode.

@jiayuzhao05

Thanks for catching this, that makes sense.

I’ll switch from using a single defaultMode to setting specific modes per key. The private key will use a stricter permission (0600), while the public key and authorized_keys will use 0644.

I’ll update the PR accordingly 👍

[jiayuzhao05]

This looks good to me now. This revision fixes previous concern.

[google-oss-prow]

@jiayuzhao05: changing LGTM is restricted to collaborators

Details

In response to this:

This looks good to me now. This revision fixes previous concern.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[harxhist]

@jinchihe @kuizhiqing
Please review this PR.

[Sridhar1030]

looks good to me too