Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible Bug: Error syncing deployment, replica set already exists #26673

Closed
osterman opened this issue Jun 2, 2016 · 10 comments
Closed

Possible Bug: Error syncing deployment, replica set already exists #26673

osterman opened this issue Jun 2, 2016 · 10 comments
Labels
area/workload-api/deployment priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@osterman
Copy link

osterman commented Jun 2, 2016

We’re using Deployment resources. Last night, one of our k8s clusters started spewing a lot of logs - to the tune of 70K messages per hour (~1gb/hr). The same message was repeated over and over again.

image

The errors look like this:

deployment_controller.go:395] Error syncing deployment sagan/supernova: error creating replica set supernova: replicasets "supernova-3273314060" already exists

image

This is a long-lived deployment. It seems that after several hours the problem resolved itself.

Our deployment process runs (roughly) the following commands synchronously:
(1) kubectl apply -f supernova-deployment.yml
(2) kubectl annotate deployment supernova 'kubernetes.io/change-cause=something'

Maybe has something to do with it?

Kubernetes Version:

Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}

On CoreOS 899.17.0

Any tips on how to triage this? Not sure where to begin. kubectl list replicasets does not reveal the problem set.
@osterman
Copy link
Author

osterman commented Jun 3, 2016

After doing some more testing, I am able to reproduce deployment_controller.go:395] Error syncing deployment sagan/help: deployments.extensions "help" cannot be updated: the object has been modified; please apply your changes to the latest version and try again by triggering two back-to-back deployments in rapid succession.

@osterman osterman mentioned this issue Jun 3, 2016
Closed
@janetkuo
Copy link
Member

@kubernetes/deployment

@pwittrock
Copy link
Member

@osterman

triggering two back-to-back deployments in rapid succession.

What are the back-to-back deployments? Is this just 2 kubectl apply calls? Does the apply operation complete (return) before the second is started?

please apply your changes to the latest version and try again

This error message indicates that the deployment was updated between reading the deployment and writing it. If you are using kubectl apply and get this, it indicates that the apply command either read a stale version or was updated between the read/write performed by apply. If this is the issue, PR #26557 will reduce the occurrences by retrying patches the fail due to optimistic lock failures.

@pwittrock pwittrock added -1 - Triage priority/backlog Higher priority than priority/awaiting-more-evidence. labels Jun 10, 2016
@pwittrock
Copy link
Member

Marking as P2 since this may already be addressed

@osterman
Copy link
Author

@pwittrock thanks for your response!

What are the back-to-back deployments? Is this just 2 kubectl apply calls? Does the apply operation complete (return) before the second is started?

Yes, back-to-back apply calls, though we throw in an annotate after each apply.

If this is the issue, PR #26557 will reduce the occurrences by retrying patches the fail due to optimistic lock failures.

That sounds very promising!

@pwittrock
Copy link
Member

Closing issue since the PR to address has been merged. Re-open if this continues to be an issue for kubectl version 1.3+

@osterman
Copy link
Author

We've not seen a repeat of this since upgrading to 1.3. It's been about 10 days so far...

@jkemp101
Copy link

I believe I am getting this same error on 1.3 running AWS. I was running some test deployments using the deployment below. The {{ version }} gets set to the current timestamp. I had made a change of maxSurge from 1 to 2 and redeployed using pykube which I think does a patch. The new replicaset was never created. The log gets many of the already exists entries. I just redeployed again and it worked itself out.

{
  "kind": "Deployment",
  "spec": {
    "replicas": 3,
    "revisionHistoryLimit": 2,
    "minReadySeconds": 2,
    "selector": {
      "matchLabels": {
        "app": "nginx-test"
      }
    },
    "template": {
      "spec": {
        "containers": [
          {
            "image": "nginx:1.11",
            "args": [
              ],

            "name": "nginx",
            "ports": [
              {
                "containerPort": 80
              }
            ]
          }
        ]
      },
      "metadata": {
        "labels": {
          "app": "nginx-test",
          "version": "{{ version }}"
        }
      }
    },
    "strategy": {
      "rollingUpdate": {
        "maxSurge": 2,
        "maxUnavailable": 1
      },
      "type": "RollingUpdate"
    }
  },
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "nginx-test",
    "labels": {
      "version": "{{ version }}",
        "app": "nginx-test"

    }
  }
}

kube-controller-manager.log log file:

I0724 19:54:58.158428       5 event.go:216] Event(api.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"nginx-test", UID:"0d764cc5-51d7-11e6-a181-02abebde0669", APIVersion:"extensions", ResourceVersion:"54408", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set nginx-test-569949374 to 0
I0724 19:54:58.158852       5 replica_set.go:463] Too many "default"/"nginx-test-569949374" replicas, need 0, deleting 1
E0724 19:54:58.164265       5 deployment_controller.go:400] Error syncing deployment default/nginx-test: Operation cannot be fulfilled on deployments.extensions "nginx-test": the object has been modified; please apply your changes to the latest version and try again
I0724 19:54:58.169148       5 event.go:216] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"nginx-test-569949374", UID:"73eb5bd0-51d7-11e6-a181-02abebde0669", APIVersion:"extensions", ResourceVersion:"54416", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: nginx-test-569949374-r2tr0
E0724 19:57:09.945581       5 deployment_controller.go:400] Error syncing deployment default/nginx-test: error creating replica set nginx-test: replicasets "nginx-test-3675045042" already exists
E0724 19:57:09.947542       5 deployment_controller.go:400] Error syncing deployment default/nginx-test: error creating replica set nginx-test: replicasets "nginx-test-3675045042" already exists
E0724 19:57:09.949969       5 deployment_controller.go:400] Error syncing deployment default/nginx-test: error creating replica set nginx-test: replicasets "nginx-test-3675045042" already exists
E0724 19:57:09.951971       5 deployment_controller.go:400] Error syncing deployment default/nginx-test: error creating replica set nginx-test: replicasets "nginx-test-3675045042" already exists
E0724 19:57:09.953838       5 deployment_controller.go:400] Error syncing deployment default/nginx-test: error creating replica set nginx-test: replicasets "nginx-test-3675045042" already exists

@gigaroby
Copy link

gigaroby commented Jul 26, 2016 •
edited
Loading

I hit the same problem as well after the call to kubectl apply -f folder/ where folder contained 3 deployment manifests.
All three manifests suffered from the problem afterwards.
I am running on AWS as well.

Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.615754   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.557964   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.509632   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.456868   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.409374   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.357072   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.306676   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.257030   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.206504   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.157741   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.112898   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.059652   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:27 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:27.007476   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.956218   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.910275   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.857240   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.806429   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.756885   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.706639   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.656417   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.607126   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.557562   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.506025   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.456588   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.422213   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.356907   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.313353   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.257953   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.209894   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.160954   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.107205   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.056387   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists
Jul 26 12:42:26 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:26.006022   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets "application-celery-worker-unstable-3295993310" already exists
Jul 26 12:42:25 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:25.956830   17798 deployment_controller.go:400] Error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets "application-unstable-900752437" already exists
Jul 26 12:42:25 ip-x-x-x-x kube-controller-manager[17798]: E0726 12:42:25.906450   17798 deployment_controller.go:400] Error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets "application-celery-unstable-1891916216" already exists

@gigaroby gigaroby mentioned this issue Jul 28, 2016
Closed
@jsravn
Copy link
Contributor

jsravn commented Oct 13, 2016

I just ran into this as well, on 1.3.7. I noticed controller-manager pegs at 100% cpu and spams the already mentioned message. I'm not sure yet what triggered it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/workload-api/deployment priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@osterman @jsravn @gigaroby @jkemp101 @janetkuo @Random-Liu @a-robinson @pwittrock