k8s 1.16 custom object resource version changes on status patch #86771

apoorvak-citrix · 2020-01-02T07:39:27Z

What happened:
In kubernetes 1.16.2 patching a status subresource of a custom object with a body that effectively does nothing increases the resource version of the object and triggers modify events.

I see that a bug was already raised and its fixed(#67541 might be a regression.

What you expected to happen:
Applying a patch that does not change the resource the object resource version should not change and no modify event be triggered.

How to reproduce it (as minimally and precisely as possible):
Applying a status patch with "Content-Type: application/json-patch+json" with the following patch.json is incrementing the resource Version
[{"op": "replace","path": "/status","value": {"state":"ERROR", "status_message":"done"}}]
Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Kernel (e.g. uname -a):
Linux highend-node1 4.4.0-116-generic PreStart and PostStop event hooks #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

k8s-ci-robot · 2020-01-02T09:45:49Z

@apoorva-05: The label(s) sig/api-mechinery cannot be applied, because the repository doesn't have them

In response to this:

/sig api-mechinery

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2020-01-02T09:49:07Z

@apoorva-05: The label(s) sig/api-mechinery cannot be applied, because the repository doesn't have them

In response to this:

/sig api-mechinery

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

apoorvak-citrix · 2020-01-02T09:49:27Z

/sig api-machinery

apoorvak-citrix · 2020-01-02T09:49:29Z

/sig api-machinery

liggitt · 2020-01-02T22:11:44Z

I'm unable to reproduce this

status.yaml:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: foos.example.com
spec:
  group: example.com
  scope: Namespaced
  versions:
  - name: v1
    storage: true
    served: true
  - name: v2
    storage: false
    served: true
  subresources:
    status: {}
  names:
    singular: foo
    plural: foos
    kind: Foo
    listKind: FooList
---
apiVersion: example.com/v1
kind: Foo
metadata:
  name: foo
spec:
  specValue: true
status:
  specValue: true

kubectl version

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.5-beta.1.37+5a9078c6025b30", GitCommit:"5a9078c6025b300dcd13dc80aae3332f9946d4cc", GitTreeState:"clean", BuildDate:"2020-01-02T21:58:26Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.5-beta.1.37+5a9078c6025b30", GitCommit:"5a9078c6025b300dcd13dc80aae3332f9946d4cc", GitTreeState:"clean", BuildDate:"2020-01-02T21:58:26Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}

kubectl create -f status.yaml

customresourcedefinition.apiextensions.k8s.io/foos.example.com created
foo.example.com/foo created

kubectl get foo/foo -o yaml

apiVersion: example.com/v2
kind: Foo
metadata:
  creationTimestamp: "2020-01-02T22:07:41Z"
  generation: 1
  name: foo
  namespace: default
  resourceVersion: "317"
  selfLink: /apis/example.com/v2/namespaces/default/foos/foo
  uid: 8a7d29f7-2051-4a89-beb5-f6c74605dce7
spec:
  specValue: true

curl -X PATCH \
  -H "Content-Type: application/json-patch+json" \
  --data-binary '[{"op": "replace","path": "/status","value": {"state":"ERROR", "status_message":"done"}}]' \
  http://localhost:8080/apis/example.com/v2/namespaces/default/foos/foo/status
kubectl get foo/foo -o yaml

apiVersion: example.com/v2
kind: Foo
metadata:
  creationTimestamp: "2020-01-02T22:07:41Z"
  generation: 1
  name: foo
  namespace: default
  resourceVersion: "331"
  selfLink: /apis/example.com/v2/namespaces/default/foos/foo
  uid: 8a7d29f7-2051-4a89-beb5-f6c74605dce7
spec:
  specValue: true
status:
  state: ERROR
  status_message: done

curl -X PATCH \
  -H "Content-Type: application/json-patch+json" \
  --data-binary '[{"op": "replace","path": "/status","value": {"state":"ERROR", "status_message":"done"}}]' \
  http://localhost:8080/apis/example.com/v2/namespaces/default/foos/foo/status
kubectl get foo/foo -o yaml

apiVersion: example.com/v2
kind: Foo
metadata:
  creationTimestamp: "2020-01-02T22:07:41Z"
  generation: 1
  name: foo
  namespace: default
  resourceVersion: "331"
  selfLink: /apis/example.com/v2/namespaces/default/foos/foo
  uid: 8a7d29f7-2051-4a89-beb5-f6c74605dce7
spec:
  specValue: true
status:
  state: ERROR
  status_message: done

liggitt · 2020-01-02T22:12:56Z

can you start a kubectl watch -o json in a separate window on the resource being patched and capture the updates to it while you are submitting the no-op patches?

apoorvak-citrix · 2020-01-03T04:43:30Z

@liggitt is this the correct command? kubectl watch -o json. Doesn't seem to be working.
Anyway i tried with your yamls. Its output is mentioned below.

Created the yamls

root@highend-node1:~/bug# kubectl create -f crd_def.yaml 
customresourcedefinition.apiextensions.k8s.io/foos.example.com created
root@highend-node1:~/bug# kubectl create -f crd_obj.yaml 
foo.example.com/foo created

Resource version of foo object before status update

root@highend-node1:~/bug# kubectl get foo/foo -o yaml
apiVersion: example.com/v2
kind: Foo
metadata:
  creationTimestamp: "2020-01-03T04:12:29Z"
  generation: 1
  name: foo
  namespace: default
  resourceVersion: "7643297"
  selfLink: /apis/example.com/v2/namespaces/default/foos/foo
  uid: 540e4579-186f-4986-be44-8f73dc3124e0
spec:
  specValue: true

Applied the patch

root@highend-node1:~/bug# curl -X PATCH  --data-binary '[{"op": "replace","path": "/status","value": {"state":"ERROR", "status_message":"done"}}]' $APISERVER/apis/example.com/v2/namespaces/default/foos/foo/status  --header "Authorization: Bearer $TOKEN" -H "Content-Type: application/json-patch+json" --cacert /tmp/ca.crt 
{"apiVersion":"example.com/v2","kind":"Foo","metadata":{"creationTimestamp":"2020-01-03T04:12:29Z","generation":1,"name":"foo","namespace":"default","resourceVersion":"7643374","selfLink":"/apis/example.com/v2/namespaces/default/foos/foo/status","uid":"540e4579-186f-4986-be44-8f73dc3124e0"},"spec":{"specValue":true},"status":{"state":"ERROR","status_message":"done"}}

Output of foo object after patch

root@highend-node1:~/bug# 
root@highend-node1:~/bug# kubectl get foo/foo -o yaml
apiVersion: example.com/v2
kind: Foo
metadata:
  creationTimestamp: "2020-01-03T04:12:29Z"
  generation: 1
  name: foo
  namespace: default
  resourceVersion: "7643374"
  selfLink: /apis/example.com/v2/namespaces/default/foos/foo
  uid: 540e4579-186f-4986-be44-8f73dc3124e0
spec:
  specValue: true
status:
  state: ERROR
  status_message: done

Not sure what i am missing. Is there any cluster level changes/flags which needs to be enabled for the status subresource to work?

apoorvak-citrix · 2020-01-03T08:07:07Z

I'm unable to reproduce this

status.yaml:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: foos.example.com
spec:
  group: example.com
  scope: Namespaced
  versions:
  - name: v1
    storage: true
    served: true
  - name: v2
    storage: false
    served: true
  subresources:
    status: {}
  names:
    singular: foo
    plural: foos
    kind: Foo
    listKind: FooList
---
apiVersion: example.com/v1
kind: Foo
metadata:
  name: foo
spec:
  specValue: true
status:
  specValue: true

kubectl version

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.5-beta.1.37+5a9078c6025b30", GitCommit:"5a9078c6025b300dcd13dc80aae3332f9946d4cc", GitTreeState:"clean", BuildDate:"2020-01-02T21:58:26Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.5-beta.1.37+5a9078c6025b30", GitCommit:"5a9078c6025b300dcd13dc80aae3332f9946d4cc", GitTreeState:"clean", BuildDate:"2020-01-02T21:58:26Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}

kubectl create -f status.yaml

customresourcedefinition.apiextensions.k8s.io/foos.example.com created
foo.example.com/foo created

kubectl get foo/foo -o yaml

apiVersion: example.com/v2
kind: Foo
metadata:
  creationTimestamp: "2020-01-02T22:07:41Z"
  generation: 1
  name: foo
  namespace: default
  resourceVersion: "317"
  selfLink: /apis/example.com/v2/namespaces/default/foos/foo
  uid: 8a7d29f7-2051-4a89-beb5-f6c74605dce7
spec:
  specValue: true

curl -X PATCH \
  -H "Content-Type: application/json-patch+json" \
  --data-binary '[{"op": "replace","path": "/status","value": {"state":"ERROR", "status_message":"done"}}]' \
  http://localhost:8080/apis/example.com/v2/namespaces/default/foos/foo/status
kubectl get foo/foo -o yaml

apiVersion: example.com/v2
kind: Foo
metadata:
  creationTimestamp: "2020-01-02T22:07:41Z"
  generation: 1
  name: foo
  namespace: default
  resourceVersion: "331"
  selfLink: /apis/example.com/v2/namespaces/default/foos/foo
  uid: 8a7d29f7-2051-4a89-beb5-f6c74605dce7
spec:
  specValue: true
status:
  state: ERROR
  status_message: done

curl -X PATCH \
  -H "Content-Type: application/json-patch+json" \
  --data-binary '[{"op": "replace","path": "/status","value": {"state":"ERROR", "status_message":"done"}}]' \
  http://localhost:8080/apis/example.com/v2/namespaces/default/foos/foo/status
kubectl get foo/foo -o yaml

apiVersion: example.com/v2
kind: Foo
metadata:
  creationTimestamp: "2020-01-02T22:07:41Z"
  generation: 1
  name: foo
  namespace: default
  resourceVersion: "331"
  selfLink: /apis/example.com/v2/namespaces/default/foos/foo
  uid: 8a7d29f7-2051-4a89-beb5-f6c74605dce7
spec:
  specValue: true
status:
  state: ERROR
  status_message: done

I see that in your case the resource Version changed from "317" to "331" after the PATCH of the status. Does this mean, that this behavior is expected?

liggitt · 2020-01-03T15:51:15Z

I see that in your case the resource Version changed from "317" to "331" after the PATCH of the status. Does this mean, that this behavior is expected?

Yes, whenever the status update actually results in a change, the resourceVersion is expected to change. Note that the second time the identical patch was submitted, it was a no-op, and the resourceVersion did not change.

apoorvak-citrix added the kind/bug Categorizes issue or PR as related to a bug. label Jan 2, 2020

k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jan 2, 2020

k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Jan 2, 2020

k8s-ci-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jan 2, 2020

liggitt added triage/not-reproducible Indicates an issue can not be reproduced as described. kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jan 2, 2020

liggitt closed this as completed Jan 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

k8s 1.16 custom object resource version changes on status patch #86771

k8s 1.16 custom object resource version changes on status patch #86771

apoorvak-citrix commented Jan 2, 2020

k8s-ci-robot commented Jan 2, 2020

k8s-ci-robot commented Jan 2, 2020

apoorvak-citrix commented Jan 2, 2020

apoorvak-citrix commented Jan 2, 2020

liggitt commented Jan 2, 2020

liggitt commented Jan 2, 2020

apoorvak-citrix commented Jan 3, 2020

apoorvak-citrix commented Jan 3, 2020

liggitt commented Jan 3, 2020 •

edited

Loading

k8s 1.16 custom object resource version changes on status patch #86771

k8s 1.16 custom object resource version changes on status patch #86771

Comments

apoorvak-citrix commented Jan 2, 2020

k8s-ci-robot commented Jan 2, 2020

k8s-ci-robot commented Jan 2, 2020

apoorvak-citrix commented Jan 2, 2020

apoorvak-citrix commented Jan 2, 2020

liggitt commented Jan 2, 2020

liggitt commented Jan 2, 2020

apoorvak-citrix commented Jan 3, 2020

Created the yamls

Resource version of foo object before status update

Applied the patch

Output of foo object after patch

apoorvak-citrix commented Jan 3, 2020

liggitt commented Jan 3, 2020 • edited Loading

liggitt commented Jan 3, 2020 •

edited

Loading