Skip to content

Kubernetes v1.6.4 -> v1.7.0 update breaks statefulsets #48327

Closed
@CallMeFoxie

Description

/kind bug

What happened:
Updated from Kubernetes v1.6.4 environment to v1.7.0 environment with existing statefulsets. After seeing some errors regarding to scheduling I noticed repeated event:

8m 17s 32 statefulset Warning FailedUpdate update Pod affinity-test-0 in StatefulSet affinity-test failed error: Pod "affinity-test-0" is invalid: spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations (only additions to existing tolerations)

It seems that the statefulset is trying to update all its pods but with something that is immutable. Trying to figure out what and my current best guess is serviceAccount (which was not there previously when it was created).

What you expected to happen:
Delete and re-create the pods since it is changing immutable fields

How to reproduce it (as minimally and precisely as possible):

  • create statefulset in 1.6.4 with some features missing (SA?)
  • update to v1.7.0

I have logs from controller-manager but they are rather spammy at --v=8 so if you have something to look for - here is what I guessed should be relevant (private info removed): https://gist.github.com/CallMeFoxie/353971434e20a3048b368d3fc835ab76

Environment:

  • Kubernetes version (use kubectl version): v1.6.4 -> v1.7.0
  • Cloud provider or hardware configuration**: self-hosted
  • Install tools: manual

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.sig/appsCategorizes an issue or PR as relevant to SIG Apps.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions