-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add EgressRule to NetworkPolicy #51351
Add EgressRule to NetworkPolicy #51351
Conversation
pkg/apis/networking/types.go
Outdated
@@ -55,6 +55,16 @@ type NetworkPolicySpec struct { | |||
// solely to ensure that the pods it selects are isolated by default) | |||
// +optional | |||
Ingress []NetworkPolicyIngressRule | |||
|
|||
// List of egress rules to be applied to the selected pods. Outgoing traffic is | |||
//allowed unbounded if there are no NetworkPolicies selecting the pod |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missing space after "//", and the word "unbounded" doesn't seem necessary, and also we need to add the thing discussed in https://groups.google.com/d/msg/kubernetes-sig-network/nWyAL-8vKV8/DDRZ9Jx8BwAJ so that all pods with ingress rules don't suddenly become default-deny to egress
pkg/apis/networking/types.go
Outdated
// List of egress rules to be applied to the selected pods. Outgoing traffic is | ||
//allowed unbounded if there are no NetworkPolicies selecting the pod | ||
// (and cluster policy otherwise allows the traffic), OR if the traffic source is | ||
// the pod's local node, OR if the traffic matches at least one egress rule |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"if the traffic source is the pod's local node" is meaningless here. You might have meant to change it to "if the traffic destination is the pod's local node", but I don't think we need that exception here. (It exists in ingress rules to make sure that the kubelet can do health checks on pods.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree we do not need the excption, unless someone can say exactly why we need it.
pkg/apis/networking/types.go
Outdated
// the pod's local node, OR if the traffic matches at least one egress rule | ||
// across all of the NetworkPolicy objects whose podSelector matches the pod. If | ||
// this field is empty then this NetworkPolicy does not limit any outgoing traffic | ||
// (and serves solely to ensure that the pods it selects are isolated by default) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does not allow any outgoing traffic
pkg/apis/networking/types.go
Outdated
@@ -77,6 +87,26 @@ type NetworkPolicyIngressRule struct { | |||
From []NetworkPolicyPeer | |||
} | |||
|
|||
// NetworkPolicyEgressRule describes a particular set of traffic that pods can egress to | |||
// matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and to. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"NetworkPolicyEgressRule describes a particular set of traffic that is allowed out of pods matched by a NetworkPolicySpec's podSelector."
pkg/apis/networking/types.go
Outdated
// matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and to. | ||
type NetworkPolicyEgressRule struct { | ||
// List of destination ports for outgoing traffic. | ||
//Each item in this list is combined using a logical OR. If this field is |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(missing space)
pkg/apis/networking/types.go
Outdated
//Each item in this list is combined using a logical OR. If this field is | ||
// empty or missing, this rule matches all ports (traffic not restricted by port). | ||
// If this field is present and contains at least one item, then this rule does note | ||
// constrain outgoing traffic. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list". (Just copied from NetworkPolicyIngressRule.Ports)
pkg/apis/networking/types.go
Outdated
// source). If this field is present and contains at least on item, this rule | ||
// allows traffic only if the traffic matches at least one item in the from list. | ||
// +optional | ||
To []NetworkPolicyPeer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// List of sources destinations for outgoing traffic of pods selected for this rule.
// Items in this list are combined using a logical OR operation. If this field is
// empty or missing, this rule matches all sources destinations (traffic not restricted by
// source destination). If this field is present and contains at least on one item, this rule
// allows traffic only if the traffic matches at least one item in the from to list.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in the to list.
-> in the To list.
// Validate egress rules | ||
for i, egress := range spec.Egress { | ||
egressPath := fldPath.Child("egress").Index(i) | ||
for i, port := range egress.Ports { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably should split out ValidateNetworkPolicyPort and ValidateNetworkPolicyPeer functions to share between ingress and egress validation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(You "+1"ed this but then didn't actually do it.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good overall
pkg/apis/networking/types.go
Outdated
// List of egress rules to be applied to the selected pods. Outgoing traffic is | ||
//allowed unbounded if there are no NetworkPolicies selecting the pod | ||
// (and cluster policy otherwise allows the traffic), OR if the traffic source is | ||
// the pod's local node, OR if the traffic matches at least one egress rule |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree we do not need the excption, unless someone can say exactly why we need it.
pkg/apis/networking/types.go
Outdated
// this field is empty then this NetworkPolicy does not limit any outgoing traffic | ||
// (and serves solely to ensure that the pods it selects are isolated by default) | ||
// +optional | ||
Egress []NetworkPolicyEgressRule |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have a comment indicating 'beta' or 'alpha' ? @lavalamp since machinery team seems to want people to just use fields rather than annotations, shouldn't the generated docs at least say "beta" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't we decide we need a field to discriminate whether this policy was applying ingress vs egress?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes @danwinship mentioned the additional PolicyType above. Adding now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes it should be beta, but I was not sure if the format was supposed to be similar to the feature gate note or not.
// owner: @cmluciano
// beta: v1.8
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've noticed this doesn't mention local node. What's the behavior for if the traffic destination is the pod's local node
?
// source). If this field is present and contains at least on item, this rule | ||
// allows traffic only if the traffic matches at least one item in the from list. | ||
// +optional | ||
To []NetworkPolicyPeer `json:"to,omitempty" protobuf:"bytes,2,rep,name=to"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
port/to is kind of awkward in this direction. ingress.ports and ingress.from makes sense. I'm not sure I have a better answer off the top of my head, and breaking symmetry would be bad, but ... better ideas?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does Out
sound any better than To
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for ports/out
Are we declaring the beta? |
@thockin @danwinship @caseydavenport I've been pondering a bit more on the "how to enable default deny" case. From the Group posting we landed on:
Instead of adding something to NetworkPolicySpec, should we augment the Egress rule with a field that resembles the default? Then we can consider deprecating the current Ingress defaults over a few releases in favor of a similar change. Ex:
|
We can't have "DefaultPolicy" in NetworkPolicyEgressRule, because what if there were two rules with opposite default policies? But maybe there could be something like a "Nothing" option in NetworkPolicyEgressRule/NetworkPolicyIngressRule so you can explicitly write a rule that doesn't match anything:
and that combines fine with other policies without contradiction. Although "Nothing" probably isn't the right name, and it can't be just an empty field because that would get lost in serialization. |
2a2cd2b
to
6885dfb
Compare
/retest |
I like the option of adding to the Peer group, however I'm also lost on what to name it. |
Is there a design/proposal doc for egress rules in networkpolicy? Or is the comment #51351 (comment) sufficient enough to capture the gist of this work? |
@ahmetb The summary of proposals is in a google doc. SIG-Network Mailing List Post: egress network policy requirements |
So actually, wait, the problem is not "how to enable default deny", it's "how to not (accidentally) enable default deny". We told people to use a policy like this for ingress default-deny in 1.7:
But that will now implicitly have an empty My suggestion had been to fix that by adding a way to specify that egress is ignored for a given policy (and make that the default behavior for existing policies). I guess the other way would be to say that empty/non-existent |
I'm fine with the PolicyType field for now if we want to maintain the Ingress behavior. I figured it was worth bringing up since the Default code that I started writing looked messy due to all of the edge-cases. |
@@ -77,6 +86,26 @@ type NetworkPolicyIngressRule struct { | |||
From []NetworkPolicyPeer | |||
} | |||
|
|||
// NetworkPolicyEgressRule describes a particular set of traffic that is allowed out of pods | |||
// matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and to. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small nit: The traffic must match both Ports and To
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cmluciano this comment is still pending it looks like
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
JSON names are correct
pkg/apis/networking/types.go
Outdated
// source). If this field is present and contains at least on item, this rule | ||
// allows traffic only if the traffic matches at least one item in the from list. | ||
// +optional | ||
To []NetworkPolicyPeer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in the to list.
-> in the To list.
This egress policy is enforced on connections made from the pod but what about on reply packets? Example:
Based on the egress policy installed, can |
We can't really deprecate the ingress defaults until we rev the API to v2{alpha,beta,}. It is part of the API semantic. Unfortunately, lack of We might be able to slightly rectify the situation, though. I'm going to think out loud: We have IPBlock now, which can unambiguously indicate "nothing" as
If user specifies nothing, they get the default (no ingress). If the user specifies If the user specifies If the use specifies The only thing that is irregular is the defaulting logic. Does this break anything? I am trying to work through it. If a user creates this on v1.8, then rolls back to v1.7, the ipBlock will get dropped, leaving an empty Starting over: We could just leave the empty ingress to be irregular. Can we only make it irregular iff If the user specifies nothing, they get today's behavior (no ingress). If the user specifies If the user specifies If the use specifies Would this break anything? If a user had an empty Starting over: What if we change the defaulting for Starting over: We're stuck. We can't even do what Dan suggested. I think we have to respect that any NP object that has no Someone check my math or convince me I am wrong? |
Reply packets are always allowed, just like with ingress policy. (ie, if you have an ingress policy for app=database to receive requests from app=frontend, then that also allows replies from the database pod to the frontend pod, even though it doesn't allow the database pod to initiate connections to the frontend pod.) |
Didn't you just contradict yourself? "any NP object that has no ingress block means 'default-deny ingress'" vs "The only way it can not affect ingress is if the user specifies just 'Egress'" ? So like:
Does that default-deny ingress or not? If the concern is only with rollback, then it seems a little weird to worry about the ingress behavior of that policy after rolling back from 1.8 to 1.7, since regardless of the interpretation of ingress, the policy as a whole is guaranteed to not have the behavior the user wanted anyway... If we need to make it possible to roll back and not have egress-only policies turn into deny-ingress policies then I guess the only way we can do that is to say that there's no such thing as an egress-only policy. So the fully foward-and-backward-compatible rule would be:
Actually, there's another way around the problem, which is if we say that
or if you want to be more paranoid/explicit:
either of which is also terrible, but maybe someone can come up with a less terrible version? (Or maybe we just let v1 be terrible but plan to move to a less-terrible v2 soon?) |
@danwinship @thockin @caseydavenport Take a look at f058d5d . The tests will probably fail since I have not added this to v1beta1, but I wanted to see if this behavior works. |
I don't think so. I just wasn't clear. If
It would have no effect on Ingress. But you're right - upon a rollback, this breaks the cluster. Both
This is something I think I never really considered about one-of blocks. You can't add a new option and then allow rollbacks. Any IPBlock user who rolls back to 1.7 will be left with an object that fails validation. I need to think about how we do this elsewhere. It may be that we're setting the bar too high here. |
@thockin @danwinship @caseydavenport This should pass all tests now. TODO:
Can we have these as TODOs post-freeze? |
I suspect we can make the validation tweaks post-freeze if we really need to. I'd like to get a conclusion on this comment though: https://github.com/kubernetes/kubernetes/pull/51351/files#r136650232 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly nits or code layout, not semantics.
@@ -133,4 +133,14 @@ func SetDefaults_NetworkPolicy(obj *extensionsv1beta1.NetworkPolicy) { | |||
} | |||
} | |||
} | |||
|
|||
if obj.Spec.PolicyTypes == nil || len(obj.Spec.PolicyTypes) == 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, fixup in followup OK
|
||
if obj.Spec.PolicyTypes == nil || len(obj.Spec.PolicyTypes) == 0 { | ||
// Default for undefined PolicyTypes with defined Egress rules | ||
if obj.Spec.Egress != nil || len(obj.Spec.Egress) != 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same, just len()
if obj.Spec.PolicyTypes == nil || len(obj.Spec.PolicyTypes) == 0 { | ||
// Default for undefined PolicyTypes with defined Egress rules | ||
if obj.Spec.Egress != nil || len(obj.Spec.Egress) != 0 { | ||
obj.Spec.PolicyTypes = []extensionsv1beta1.PolicyType{extensionsv1beta1.PolicyTypeIngress, extensionsv1beta1.PolicyTypeEgress} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes. I would restructure it as:
if len(obj.Spec.PolicyTypes) == 0 {
// Any policy that does not specify policyTypes implies at least "Ingress".
obj.Spec.PolicyTypes = []extensionsv1beta1.PolicyType{extensionsv1beta1.PolicyTypeIngress}
if len(obj.Spec.Egress) != 0 {
obj.Spec.PolicyTypes = append(obj.Spec.PolicyTypes, extensionsv1beta1.PolicyTypeEgress)
}
}
pkg/apis/networking/types.go
Outdated
// allowed if there are no NetworkPolicies selecting the pod (and cluster policy | ||
// otherwise allows the traffic), OR if the traffic matches at least one egress rule | ||
// across all of the NetworkPolicy objects whose podSelector matches the pod. If | ||
// this field is empty then this NetworkPolicy does not limit any outgoing traffic. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did we agree that empty hear means allow-all while empty ingress means deny-all? I can't recall why? It should be the same or well-documented here, why not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we did - this should be the same as for ingress.
pkg/apis/networking/types.go
Outdated
// (whether or not they contain an Ingress section) are assumed to affect Ingress. | ||
// If you want to write an egress-only policy, you must explicitly specify PolicyTypes [ "Egress" ]. | ||
// Likewise, if you want to write a policy that specifies that no egress is allowed, | ||
// you must specify a PolicyTypes values that include "Egress" (since such a policy would not include |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/values/value
s/PolicyType/policyType - JSON name
@@ -77,6 +86,26 @@ type NetworkPolicyIngressRule struct { | |||
From []NetworkPolicyPeer | |||
} | |||
|
|||
// NetworkPolicyEgressRule describes a particular set of traffic that is allowed out of pods | |||
// matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and to. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
JSON names are correct
if obj.Spec.PolicyTypes == nil || len(obj.Spec.PolicyTypes) == 0 { | ||
// Default for undefined PolicyTypes with defined Egress rules | ||
if obj.Spec.Egress != nil || len(obj.Spec.Egress) != 0 { | ||
obj.Spec.PolicyTypes = []extensionsv1beta1.PolicyType{extensionsv1beta1.PolicyTypeIngress, extensionsv1beta1.PolicyTypeEgress} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@caseydavenport If we don't assume Ingress, then any policy that is a default-deny-ingress today would change meaning by adding an egress block. That's not compatible.
} | ||
} | ||
// Validate PolicyTypes | ||
if len(spec.PolicyTypes) > 2 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not convert it to a sets.String, and check that the set only contains known items? I don't like coding in numbers like 2.
for i, pType := range spec.PolicyTypes { | ||
policyPath := fldPath.Child("policyTypes").Index(i) | ||
if pType != networking.PolicyTypeIngress && pType != networking.PolicyTypeEgress { | ||
allErrs = append(allErrs, field.NotSupported(policyPath, pType, []string{string("Ingress"), string("Egress")})) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should use teh const values.
allowed := sets.NewString(PolicyTypeIngress, PolicyTypeEgress)
for _, p := range spec.PolicyTypes {
if !allowed.Has(string(p)) {
// error
/lgtm |
/retest Review the full test history for this PR. |
3 similar comments
/retest Review the full test history for this PR. |
/retest Review the full test history for this PR. |
/retest Review the full test history for this PR. |
Signed-off-by: Christopher M. Luciano <cmluciano@us.ibm.com>
Signed-off-by: Christopher M. Luciano <cmluciano@us.ibm.com>
e97db92
to
84290ce
Compare
Since I needed to rebase and fix some golint stuff, I added the comment and validation suggestions as well. Will need another LGTM |
/test pull-kubernetes-unit |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cmluciano, thockin Associated issue: 50453 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
/test pull-kubernetes-e2e-kops-aws |
@cmluciano: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Automatic merge from submit-queue (batch tested with PRs 51984, 51351, 51873, 51795, 51634) |
What this PR does / why we need it:
Add EgressRule to NetworkPolicy
Which issue this PR fixes: fixes #50453
Special notes for your reviewer:
Release note: