Skip to content

remove default affinity and toleration #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
n0rad merged 1 commit into from
Apr 19, 2025
Merged

remove default affinity and toleration #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions base/ci/advanced-result.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,12 +133,6 @@ spec:
runAsNonRoot: true
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down Expand Up @@ -171,11 +165,6 @@ spec:
maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
tolerations:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"
containers:
- env:
- name: HTTPS_PORT
Expand Down
11 changes: 0 additions & 11 deletions base/ci/basic-result.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,12 +196,6 @@ spec:
runAsNonRoot: true
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down Expand Up @@ -234,11 +228,6 @@ spec:
maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
tolerations:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"
containers:
- env:
- name: DD_SERVICE
Expand Down
11 changes: 0 additions & 11 deletions base/ci/inferenceService-result.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,12 +86,6 @@ spec:
runAsUser: 1000
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down Expand Up @@ -124,11 +118,6 @@ spec:
maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
tolerations:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"
volumes:
- hostPath:
path: /var/run/datadog
Expand Down
22 changes: 0 additions & 22 deletions base/ci/multiple-workloads-main-result.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,12 +193,6 @@ spec:
runAsNonRoot: true
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down Expand Up @@ -231,11 +225,6 @@ spec:
maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
tolerations:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"
containers:
- image: there:42
imagePullPolicy: IfNotPresent
Expand Down Expand Up @@ -319,12 +308,6 @@ spec:
runAsNonRoot: true
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down Expand Up @@ -357,11 +340,6 @@ spec:
maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
tolerations:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"
containers:
- image: test:43
imagePullPolicy: IfNotPresent
Expand Down
22 changes: 0 additions & 22 deletions base/ci/multiple-workloads-result.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,12 +193,6 @@ spec:
runAsNonRoot: true
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down Expand Up @@ -231,11 +225,6 @@ spec:
maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
tolerations:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"
containers:
- image: test:43
imagePullPolicy: IfNotPresent
Expand Down Expand Up @@ -319,12 +308,6 @@ spec:
runAsNonRoot: true
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down Expand Up @@ -357,11 +340,6 @@ spec:
maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
tolerations:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"
containers:
- image: there:42
imagePullPolicy: IfNotPresent
Expand Down
11 changes: 0 additions & 11 deletions base/ci/test-result.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,12 +117,6 @@ spec:
runAsNonRoot: true
dnsPolicy: ClusterFirst
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down Expand Up @@ -155,11 +149,6 @@ spec:
maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
tolerations:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"
containers:
- image: ghcr.io/mendhak/http-https-echo:31
imagePullPolicy: IfNotPresent
Expand Down
36 changes: 1 addition & 35 deletions base/templates/features/workloads/_scheduling_infra.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,32 +5,12 @@


{{- if include "base.lib.utils.isEnabled" (dig "scheduling" dict $workload ) -}}
{{- $nodepools := (list "common" "bursty" "volatile") }}
{{- if not (has $workload.scheduling.nodepool $nodepools) }}
{{- fail (printf "workload scheduling must be one of %v(workload: %s)" $nodepools $workloadId) }}
{{- end }}
resources:
controllers:
{{$workloadId}}:
pod:
{{- if or (eq $workload.scheduling.nodepool "volatile") (eq $workload.scheduling.nodepool "bursty") (include "base.lib.utils.isEnabled" $workload.scheduling.podSpreadingForHA) }}
{{- if include "base.lib.utils.isEnabled" $workload.scheduling.podSpreadingForHA }}
affinity:
{{- if or (eq $workload.scheduling.nodepool "volatile") (eq $workload.scheduling.nodepool "bursty") }}
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
{{- if eq $workload.scheduling.nodepool "volatile" }}
- key: cloud.blabla.io/k8s-volatile
operator: Exists
{{- else if eq $workload.scheduling.nodepool "bursty" }}
- key: dedicated
operator: In
values:
- bursty
{{- end }}
{{- end }}

{{- if include "base.lib.utils.isEnabled" $workload.scheduling.podSpreadingForHA }}
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
Expand Down Expand Up @@ -68,19 +48,5 @@ resources:
{{- include "base.lib.metadata.selectorLabels" $rootContext | nindent 14 }}
{{- end }}

{{- if or (eq $workload.scheduling.nodepool "volatile") (eq $workload.scheduling.nodepool "bursty") }}
tolerations:
- effect: NoSchedule
operator: Equal
{{- if eq $workload.scheduling.nodepool "volatile" }}
key: cloud.blabla.io/k8s-volatile
value: "true"
{{- else if eq $workload.scheduling.nodepool "bursty" }}
key: dedicated
value: "bursty"
{{- end }}
{{- end }}


{{- end -}}
{{- end }}
110 changes: 0 additions & 110 deletions base/templates/features/workloads/_scheduling_infra_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,113 +72,3 @@ tests:
path: spec.template.spec.topologySpreadConstraints
- notExists:
path: spec.template.spec.affinity.podAntiAffinity

- it: nodepool is mandatory
set:
workloads.main.scheduling.nodepool: ""
asserts:
- failedTemplate:
errorMessage: "workload scheduling must be one of [common bursty volatile](workload: main)"

- it: nodepool must be in enum
set:
workloads.main.scheduling.nodepool: toto
asserts:
- failedTemplate:
errorMessage: "workload scheduling must be one of [common bursty volatile](workload: main)"

- it: default nodepool is volatile
documentSelector:
path: kind
value: Deployment
asserts:
- equal:
path: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution
value:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
- equal:
path: spec.template.spec.tolerations[0]
value:
effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"

- it: nodepool can be common
set:
workloads.main.scheduling.nodepool: common
asserts:
- notExists:
path: spec.template.spec.affinity.nodeAffinity
- notExists:
path: spec.template.spec.tolerations

- it: nodepool can be bursty
set:
workloads.main.scheduling.nodepool: bursty
documentSelector:
path: kind
value: Deployment
asserts:
- equal:
path: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution
value:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values:
- bursty
- equal:
path: spec.template.spec.tolerations[0]
value:
effect: NoSchedule
operator: Equal
key: dedicated
value: "bursty"

- it: common nodepool with spread disabled
set:
workloads.main.scheduling:
nodepool: common
podSpreadingForHA:
enabled: false
documentSelector:
path: kind
value: Deployment
asserts:
- notExists:
path: spec.template.spec.affinity
- notExists:
path: spec.template.spec.tolerations
- notExists:
path: spec.template.spec.topologySpreadConstraints

- it: override toleration from workload is working
set:
resources.controllers.main.pod.tolerations:
- effect: NoSchedule
key: test
operator: Equal
value: "true"
documentSelector:
path: kind
value: Deployment
asserts:
- equal:
path: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution
value:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
- equal:
path: spec.template.spec.tolerations[0]
value:
effect: NoSchedule
key: test
operator: Equal
value: "true"
6 changes: 0 additions & 6 deletions base/templates/resources/pods/fields/field_affinity_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,6 @@ tests:
- equal:
path: spec.template.spec.affinity
value:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: cloud.blabla.io/k8s-volatile
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
Expand Down
7 changes: 1 addition & 6 deletions base/templates/resources/pods/fields/field_tolerations_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,8 @@ tests:
path: kind
value: Deployment
asserts:
- equal:
- isNull:
path: spec.template.spec.tolerations
value:
- effect: NoSchedule
key: cloud.blabla.io/k8s-volatile
operator: Equal
value: "true"

- it: resourcesDefault should pass
set:
Expand Down