Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DaemonSet should respect Pod Affinity and Pod AntiAffinity #29276

Closed
lukaszo opened this issue Jul 20, 2016 · 26 comments
Closed

DaemonSet should respect Pod Affinity and Pod AntiAffinity #29276

lukaszo opened this issue Jul 20, 2016 · 26 comments
Labels
area/workload-api/daemonset lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.

Comments

@lukaszo
Copy link
Contributor

lukaszo commented Jul 20, 2016

This is a split from #22205 where only node affinity was added to DaemonSets. Pod Affinity and Pod AntiAffinity are still missing in DaemonSets.

It can be implemented in two ways:

  1. Add InterPodAffinityMatches predicate check to nodeShouldRunDaemonPod in daemoncontroller.go
  2. Add InterPodAffinityMatches predicate to GeneralPredicates which is used by daemon set.

cc @bgrant0607
@lukaszo lukaszo mentioned this issue Jul 20, 2016
Closed
@lukaszo lukaszo mentioned this issue Aug 17, 2016
Closed
@ivan4th ivan4th mentioned this issue Aug 22, 2016
Closed
@bgrant0607 bgrant0607 added sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. and removed team/control-plane (deprecated - do not use) labels Mar 21, 2017
@bgrant0607 bgrant0607 mentioned this issue Mar 21, 2017
Closed
@0xmichalis
Copy link
Contributor

@lukaszo is this issue fixed?

@lukaszo
Copy link
Contributor Author

lukaszo commented May 21, 2017

@Kargakis nope

@davidopp davidopp mentioned this issue Jul 8, 2017
Closed
@kow3ns
Copy link
Member

kow3ns commented Jul 14, 2017

@lukaszo @Kargakis @davidopp

DeamonSet is meant to run one copy of Pod on every node that matches the NodeSelector. What would PodAffinity PodAntiAffinity do here?

I can image that hard PodAntiAffinity might be used to indicate "put a Pod on every Node that matches your NodeSelector except for Nodes that have Pod x", but users could achieve the same functionality with a more restrictive NodeSelector and a more granular labeling scheme.

Have we put thought into the semantics of the other forms of Pod Affinity/AntiAffinity?

@lukaszo
Copy link
Contributor Author

lukaszo commented Jul 14, 2017

@kow3ns some use cases are described in the PR #31136

@davidopp
Copy link
Member

@kow3ns it's not an unreasonable question. Thinking out loud here, I could imagine you might want to run one daemon per rack, e.g. to control some rack-level hardware resource. (Pod affinity is harder to justify.)

@antoniaklja antoniaklja mentioned this issue Nov 8, 2017
Merged
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 31, 2017
@bgrant0607
Copy link
Member

/remove-lifecycle stale

@bsalamat has been looking at this

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 23, 2018
@dblackdblack
Copy link

dblackdblack commented Jan 23, 2018 •
edited
Loading

+1 . This used to work back when affinity was indicated via the scheduler.alpha.kubernetes.io/affinity pod annotation. We used this feature for daemonsets but it is now broken.

In other words, this is a regression.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 30, 2018
@dblackdblack
Copy link

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 30, 2018
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 29, 2018
@dblackdblack
Copy link

dblackdblack commented Jul 29, 2018 via email

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 29, 2018
@guevara guevara mentioned this issue Apr 29, 2019
Open
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 7, 2019
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dblackdblack
Copy link

/reopen

@k8s-ci-robot
Copy link
Contributor

@dblackdblack: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@yf2008 yf2008 mentioned this issue May 8, 2020
Closed
@mukgupta
Copy link

mukgupta commented Jun 12, 2020 •
edited
Loading

I wanted to mention my use case for this feature. We use New Relic APM for monitoring some of our applications. NR APM charges per-host, not per-pod/per-container. To reduce costs, we would like to run the APM agent only on the nodes where the relevant pods are running (i.e. the pods running applications that use NR APM). One (theoretical) way of doing this would be to add a pod affinity to the New Relic APM DaemonSet to match pods labeled as using the APM; however, that feature obviously isn't supported.

A workaround that I thought of (but not yet implemented) would be to have a custom controller that labels nodes with new-relic-apm: active whenever relevant pods are scheduled on that nodes, and removes the label after all the relevant pods are removed from the node. Then, we would add a node selector to the DaemonSet to match that label. Therefore, the DaemonSet would still hold the responsibility for creating the APM agent pods and performing updates and we wouldn't need to implement our own pod controller.

@rabbitfang I have the exact same requirement but for datadog. Did you manage to solve it?

@dblackdblack
Copy link

dblackdblack commented Jun 12, 2020 via email

@andyxning
Copy link
Member

I wanted to mention my use case for this feature. We use New Relic APM for monitoring some of our applications. NR APM charges per-host, not per-pod/per-container. To reduce costs, we would like to run the APM agent only on the nodes where the relevant pods are running (i.e. the pods running applications that use NR APM). One (theoretical) way of doing this would be to add a pod affinity to the New Relic APM DaemonSet to match pods labeled as using the APM; however, that feature obviously isn't supported.

A workaround that I thought of (but not yet implemented) would be to have a custom controller that labels nodes with new-relic-apm: active whenever relevant pods are scheduled on that nodes, and removes the label after all the relevant pods are removed from the node. Then, we would add a node selector to the DaemonSet to match that label. Therefore, the DaemonSet would still hold the responsibility for creating the APM agent pods and performing updates and we wouldn't need to implement our own pod controller.

We have somewhat the same requirements about scheduling the daemonset pods with affinity with app pods and anti-affinity with nodes not running the app pods after app pods are deleted from the nodes. Any suggestion?

@ellistarn ellistarn mentioned this issue Jan 25, 2022
Closed
ijsong added a commit to kakao/varlog that referenced this issue Aug 5, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
5b6248c 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](https://jira.daumkakao.com/browse/VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 8, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
ba43299 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](https://jira.daumkakao.com/browse/VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
6b439d6 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
1f6f8b1 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
cc282b5 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
109895e 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
1347a54 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
6ce32ae 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
12054b6 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
b079410 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
08c66fc 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 9, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
f2defac 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 10, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
1e1d275 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 10, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
aa1b354 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
ijsong added a commit to kakao/varlog that referenced this issue Aug 10, 2022
@ijsong
*: some fixes for e2e test on krane-based k8s (#443)
850af70 
**change resources for `deploy/k8s/dev`**
To test the Varlog in a Krane-based cluster, the size of the cluster
should be at least 20 nodes. Recommended numbers are as follows: the
number of MRs is three and the number of SNs is 3 or 4 and the
replication factor is 3.

**daemonset doesn't respect podAntiAffinity**
See kubernetes/kubernetes#29276.
We should attach label `varlog-type=telemetry` to nodes running
jaeger, prometheus, otel-collector and grafana. Daemonset for MR and SN
won't be deployed to those nodes. The e2e testing module ignores nodes
labed with `varlog-type=telemetry`.

Resolves [#VARLOG-509](VARLOG-509).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/workload-api/daemonset lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement Pod Affinity and AntiAffinity for DaemonSets ivan4th/kubernetes