/milestone v1.10

this also fails in upgrade suite
xref #60764

looks like should restart all nodes and ensure all nodes and pods recover is now green. guess it's still flaky

@dims not from http://k8s-testgrid.appspot.com/sig-release-master-blocking#gci-gce-serial

oh, probably pasted wrong link in the issue body? my bad

ah cool.

Mar 7 16:52:36.943: At least one pod wasn't running and ready or succeeded at test start.

^ The preconditions are not met at the start of the test... and the pods state shows as pending.
Always seems to be a fluentd pod on the master node doesn't run. Are things tainted properly?

/cc @yujuhong & @mbforbes as they are the test authors.

/assign @yujuhong
/assign @mbforbes

@timothysc: GitHub didn't allow me to assign the following users: mbforbes.

Note that only kubernetes members and repo collaborators can be assigned.

FWIW, I didn't author the test, and the test didn't even run since it failed the precondition....

/unassign
/assign @bmoyles0117
/assign @crassirostris

fluentd pods are pending on the master nodes.

@yujuhong: GitHub didn't allow me to assign the following users: bmoyles0117.

Note that only kubernetes members and repo collaborators can be assigned.

any solution here?

@k82cn @janetkuo fluentd is scheduled on the master node, which is marked unschedulable (spec.unschedulable: true). Is this working as intended?

fluentd is scheduled on the master node, which is marked unschedulable (spec.unschedulable: true). Is this working as intended?

Yes, assuming fluentd is a DaemonSet pod. #60386 is nothing new; it's introduced to fix a regression in 1.10 (#60163).

As stated in DaemonSet doc:

The unschedulable field of a node is not respected by the DaemonSet controller.

The changes lets the fluentd pod to be scheduled on the master, but there is no capacity on the master node to run this pod.

If fluentd Daemon pod isn't supposed to be scheduled on the master, taints should be added to the master; if instead it should be scheduled on the master, we should add more capacity to the master node.

fluentd's asking more cpu resources than 1.9 (the regression caused by #60613 masked the issue until now). Seems like this could be related to the introduction of fluentd-gcp-scaler.
/assign @crassirostris

@yujuhong

fluentd's asking more cpu resources than 1.9

That's unexpected, thanks for noticing

/assign @x13n

Daniel, please take a look

fluentd-gcp in both 1.9 and 1.10 asks for the same amount of resources: 100m cpu request, 200Mi memory request (and 300Mi memory limit). Introducing fluentd-gcp-scaler didn't change these values.
@yujuhong Do you mean that CPU request increased? I see it is 100m now (as it was for 1.9), what values did you see for 1.9 and now?

any updates?

The fluentd pod on the master from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/3153:
https://gist.github.com/yujuhong/8044d7daaf169b5cd6b0c3d587579246

Both the fluentd-gcp and the prometheus-to-sd-exporter containers request 100m cpu

                        "requests": {
                            "cpu": "100m",
                            "memory": "200Mi"
                        }

This amounts to 200m cpu. In v1.9, prometheus-to-sd-exporter did not have any cpu request IIRC. You can double check that.

Thanks! Looks like there is a bug in the scaler, so it sets resources on all containers instead of fluentd-gcp only. Scaler fix is in GoogleCloudPlatform/k8s-stackdriver#130, I will create a PR with version bump once this is merged.

Actually, maybe better to
/reopen
until testgrid becomes green.

ACK. In progress
ETA: 19/03/2018

[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process

@crassirostris @krzyzacy @x13n

Note: This issue is marked as priority/critical-urgent, and must be updated every 1 day during code freeze.

Example update:

ACK.  In progress
ETA: DD/MM/YYYY
Risks: Complicated fix required

https://k8s-testgrid.appspot.com/sig-release-master-blocking#gci-gce-serial is green now