cc @kubernetes/sig-scheduling-misc @aveshagarwal

Created a deployment on 1.6.4 that included pod affinity annotations.

Is there any reason you are still using annotations with 1.6 and not fields? In general, I'd say it would be better if you could have migrated from annotations to fields before upgrading to 1.7. That said there was an issue (#44360) that we fixed in 1.7 which we missed in 1.6. I am still looking if your issue is related to this or not.

@andor44 Could you please provide your pull deployment json/yaml file?

I think I can reproduce your issue by upgrading 1.6 to 1.7 and I see the same error. And it is indeed happening due to the fix to the issue #44360 .

Is there any reason you are still using annotations with 1.6 and not fields?

None, other than annotations coming first so that's how these deployments were written originally.

In general, I'd say it would be better if you could have migrated from annotations to fields before upgrading to 1.7.

The 1.7 release notes did not indicate that the annotations would stop working or break in some way so we assumed it was safe to upgrade. The note about the feature gate did not concern us as we had never enabled that feature gate. It seems to me that this is either an undocumented breaking change or a bug, hence this issue.

Here's one of the affected deployments. Be aware that this wasn't the only one.

@aveshagarwal I am not clear about the problem, why the topologyKey is empty? I see the key have values:
"topologyKey": "failure-domain.beta.kubernetes.io/zone"

Could you give me some pointers?

@wanghaoran1988 my guess is that they might have updated topology key to something (so that its not empty now) during upgrade to 1.7.
The issue is that if the topology key is empty in 1.6 for pod anti affinity PreferredDuringSchedulingIgnoredDuringExecution, and then its upgraded to 1.7, it would fail because now 1.7, incorrectly, it requires the topology key to be not empty for pod anti affinity PreferredDuringSchedulingIgnoredDuringExecution when AffinityInAnnotation is disabled.

@aveshagarwal Thanks, that make sense !

@wanghaoran1988 @aveshagarwal no, it was not updated, it was set incorrectly. topologyKey needs to be under another object podAffinityTerm, but it was one level up. Since before 1.7 it was not mandatory it was silently ignored. We updated to 1.7 which made it mandatory, hence the failure.

To clarify, preferredDuringSchedulingIgnoredDuringExecution should have been:

{
    "weight": 80,
    "labelSelector": {
        "matchExpressions": [
        {
            "key": "app",
            "operator": "In",
            "values": ["foobar"]
        }
        ]
    },
    "podAffinityTerm": {
        "topologyKey": "failure-domain.beta.kubernetes.io/zone"
    }
}

but it was:

{
    "weight": 80,
    "labelSelector": {
        "matchExpressions": [
        {
            "key": "app",
            "operator": "In",
            "values": ["foobar"]
        }
        ]
    },
    "topologyKey": "failure-domain.beta.kubernetes.io/zone"
}

Omitting topologyKey is no longer permitted in 1.7, therefore the second one became invalid.

@andor44 After further discussion with sig scheduling group, it is intended to not allow empty topology key with soft pod anti affinity (preferredDuringSchedulingIgnoredDuringExecution pod anti affinity). So it is working as expected. So i think this bug could be closed.

@aveshagarwal I realize at this point that the name of the issue is probably misleading. My issue is not that the topology key is required. I understand that it is required, I am not reporting that as the bug. The bug is that I had an object already Kubernetes that became invalid after the upgrade. Once it became invalid it stayed in Kubernetes and it could not be deleted. Delete requests returned a success return code, but the object didn't go away. It could only be deleted once the object was manually edited to conform to the spec. This behavior is very counter-intuitive and I do not see the reason for disallowing object deletion of invalid objects.

Should I edit the title, or open a new bug and reference this one?

@andor44 ok then you could keep it open.

In my case, having an old replicaset with a bad configuration doesn't let me create new replicasets so deploy new versions as old replicaset still has a desired number of pods that can't run due configuration errors

Even if deployment has no affinity configuration, doing a get deployment -o yaml shows that annotation kubectl.kubernetes.io/last-applied-configuration has that affinity config and status shows this failure

status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2017-08-25T00:37:32Z
    lastUpdateTime: 2017-08-25T00:37:32Z
    message: 'unable to create pods: Pod "php-2221189365-kzxvk" is invalid: metadata.annotations.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey:
      Required value: can not be empty'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure

is there a quick workaround at least to be able to deploy and delete old replica set?

UPDATE: from the dashboard, I can delete the replicaset without any problem....

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@andor44 - Can you please change the labels as this issue is not related to sig-scheduling anymore? I think adding appropriate labels might help in getting the attention of sig's

I don't know what sig is appropriate, maybe API machinery?

Just run into this issue after upgrading k8s from 1.6.6 to 1.7.12.
Deployments working in 1.6.6 stopped working after upgrade. We have pods stuck in Terminating state that can't be deleted.

Hit this also, 1.6.13 to 1.7.12.

In my case however we were not using the older annotations still. It seems this affects both the old and new affinity formats. Something should definitely be done, a previously valid deployment/replicaset/podspec shouldnt become invalid during an upgrade due to a spec change.

I found an easy enough work around for the issues though.

For those still affected:
TLDR; Fix the deployment if you haven't already, then edit the latest (pre upgrade) replicaset (it should be the 2nd newest, with the newest being the one just created). Once the replicaset is correct, the rollout is happy and cleans things up

The reason editing the deployment doesn't fix it (and leaves old pods in a strange state) is the old replicaset.

When the deployment is edited in an attempt to add the topologykey, it makes a new replicaset and scales down the old.
As the old one is missing the key, the internal rollout process fails to scale down the number of replicas as the replicaset is invalid.
From here it all just gets stuck with little explanation as the deployment is now referring to the new correct replicaset

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close