-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intermittent error when doing kubectl apply -f for a HorizontalPodAutoscaler: 'unable to find api field in struct HorizontalPodAutoscalerSpec for the json field "scaleTargetRef"' #34413
Comments
Tried deleting the hpa using |
Does this work consistently if the the hpa is in a different file? |
@pwittrock No, it's the same thing. Every other time it seems to work. |
@ymqytw Would you look into this one as well? |
@pwittrock Yeah, I will look into this issue. |
Getting this issue too on 1.4.3
yaml
|
It happens on 1.4.0 and HEAD. I think there is something wrong with the encoder genertaed by codec. |
It's possible that we are not forcing the stored version into a stable apiVersion/group, and so when the HPA controller runs it mutates the object via one name (extensions?) and when the user runs it updates via (autoscaling), and so the user action fails. There is code in the storage codec (storage_factory.go) that should force objects from either type into a single version when stored in etcd. That's probably the first place to look. If that's not stable then multiple clients can cause this problem. |
This busted our CD system for any repos that contain HPA manifests. Details:
|
I will prioritize this issue. |
The root cause is bestMatch(). |
@caesarxuchao Is the codec library owned by the api-machinery sig? |
Yes. This is an api-machinery issue. |
Lemme try to first reproduce this and I'll see what I can do. |
It looks like the same problem as in #35149, goes all the way down the exact same method. Working on it now... |
Thx |
@soltysh any updates on this? |
@pwittrock Giving this P1 priority and targeting to 1.5 for now. Should this be backported to 1.4? |
@smarterclayton @liggitt is this issue fixed by #38406? |
It is possible this is already fixed on master/head. @ymqytw is trying to verify if this is the case now, and create a patch for 1.5 if this is the case. |
…d_obj Automatic merge from submit-queue make apply use the correct versioned obj Cherrypick [part of the changes](https://github.com/kubernetes/kubernetes/pull/38406/files#diff-b3aa1e4377838f45d595f8ecca3c2619) from #38406. Make `kubectl apply` in `v1.5` works for #34413. ``` release-note Give apply the versioned struct that generated from the type defined in the restmapping. ``` cc: @pwittrock @matchstick @liggitt
If #38982 does in fact fix this issue, does anyone know if it'll make it into the 1.5.2 release? This issue is really frustrating in a CI/CD pipeline! |
@munnerz I believe #38982 will be in 1.5.2. |
Yes, this will be in 1.5.2
…On Wed, Jan 4, 2017 at 10:47 AM, Mengqi Yu ***@***.***> wrote:
@munnerz <https://github.com/munnerz> I believe #38982
<#38982> will be in 1.5.2.
@pwittrock <https://github.com/pwittrock> Can you confirm?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#34413 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHhIAv2iKjM9i4CAsEoLKNrWDoOvjPxlks5rO-kqgaJpZM4KR_cS>
.
|
I'm still hitting this on an 1.5.2 cluster, although much fewer times. Both master and node version are 1.5.2. |
do you have any pre-1.5.2 kubectl clients performing apply? |
@liggitt Thanks for reminding! We were running pre-1.5.2 |
Automatic merge from submit-queue (batch tested with PRs 39223, 40260, 40082, 40389) make kubectl generic commands work with unstructured objects part of making apply, edit, label, annotate, and patch work with third party resources fixes #35149 fixes #34413 prereq of: #35496 #40096 related to: #39906 #40119 kubectl is currently decoding any resource it doesn't have compiled-in to a ThirdPartyResourceData struct, which means it computes patches using that struct, and would try to send a ThirdPartyResourceData object to the API server when running `apply` This PR removes the behavior that decodes unknown objects into ThirdPartyResourceData structs internally, and fixes up the following generic commands to work with unstructured objects - [x] apply - [x] decode into runtime.Unstructured objects - [x] successfully use `--record` with unregistered objects - [x] patch - [x] decode into runtime.Unstructured objects - [x] successfully use `--record` with unregistered objects - [x] describe - [x] decode into runtime.Unstructured objects - [x] implement generic describer - [x] fix other generic kubectl commands to work with unstructured objects - [x] label - [x] annotate follow-ups for pre-existing issues: - [ ] `explain` doesn't work with unregistered resources - [ ] remove special casing of federation group in clientset lookups, etc - [ ] `patch` - [ ] doesn't honor output formats when persisting to server (`kubectl patch -f svc.json --type merge -p '{}' -o json` doesn't output json) - [ ] --local throws exception (`kubectl patch -f svc.json --type merge -p '{}' --local`) - [ ] `apply` - [ ] fall back to generic JSON patch computation if no go struct is registered for the target GVK (e.g. #40096) - [ ] ensure subkey deletion works in CreateThreeWayJSONMergePatch - [ ] ensure type stomping works in CreateThreeWayJSONMergePatch - [ ] lots of tests for generic json patch computation - [ ] prevent generic apply patch computation among different versions - [ ] reconcile treatment of nulls with #35496 - [ ] `edit` - [ ] decode into runtime.Unstructured objects - [ ] fall back to generic JSON patch computation if no go struct is registered for the target GVK
I have this issue with a 1.5.2 cluster with a 1.5.4 client, anyone else seeing it? |
I'm seeing the same/similar issue with a 1.5.2 cluster and 1.6.3 client |
seeing the same issue with v1.8.0 How to overcome |
apiVersion: autoscaling/v1
|
What keywords did you search in Kubernetes issues before filing this one?
Also searched google/stackoverflow
Is this a BUG REPORT or FEATURE REQUEST?
Bug report
Kubernetes version (use
kubectl version
):v1.4.0
Environment:
What happened:
Do
kubectl apply -f my-service.yaml
Intermittently, since upgrading to 1.4.0 I have been getting
As you can see, it seems that the "current" metadata specifies "apiVersion" as extensions/v1beta1, that might be correct, but it seems suspicious to me...
I have the feeling it works every second time I do apply, but hard to say from my small sample size.
What you expected to happen:
The apply to work every time.
How to reproduce it
I used this yaml file (cleaned)
Anything else do we need to know:
GET on clusters using https://cloud.google.com/container-engine/reference/rest/v1/projects.zones.clusters/get reveals:
To solve this I have temporarily moved the HPA section out of the file into a separate file, but I prefer having the complete configuration in one place.
Am I doing something wrong?
Thanks!
The text was updated successfully, but these errors were encountered: