Trainer registers cluster-wide ConfigMap and Secret causing excessive memory consumption

[sutaakar]

What happened?

The MPI plugin's ReconcilerBuilders() in pkg/runtime/framework/plugins/mpi/mpi.go registers cluster-wide watches on ConfigMap and Secret with no
label selector or namespace filter. While EnqueueRequestForOwner filters reconcile events to owner-referenced objects, the underlying informer
cache still lists and watches all ConfigMaps and Secrets cluster-wide, causing excessive memory consumption on clusters with many ConfigMaps.

Reproducer:

1. Install trainer v2.2.0 on KinD
2. Create 700 large ConfigMaps (200KB each)
3. Add a memory limit for Trainer operator and observe OOMKill

The controller OOMKills within seconds and restarts repeatedly. Without a memory limit, memory consumption reaches 415Mi with 700×200KB ConfigMaps in a fresh KinD cluster.

What did you expect to happen?

The controller memory consumption should not grow with the number of unrelated ConfigMaps and Secrets in the cluster.

Environment

Kubernetes version: v1.32.2 (KinD)
Kubeflow Trainer version: v2.2.0

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-trainer-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[beep-boopp]

Took a quick look at this. The event filtering with EnqueueRequestForOwner is working fine, but the informer cache is still pulling down every ConfigMap and Secret cluster-wide, hence the OOM.

Proposed fix: Stamp a trainer.kubeflow.org/trainjob-name label on buildHostFileConfigMap and buildSSHAuthSecret at creation, and drop a label-selector predicate on the watches in ReconcilerBuilders(). That way the cache only ever bothers with MPI-owned objects.

Happy to pick this up!
/assign