-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Paramaterize stickyMaxAgeMinutes
for service in API
#49850
Paramaterize stickyMaxAgeMinutes
for service in API
#49850
Conversation
Hi @m1093782566. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
829d7be
to
822d765
Compare
/assign @thockin |
stickyMaxAgeMinutes
for service in APIstickyMaxAgeMinutes
for service in API
@k8s-bot ok to test |
pkg/api/types.go
Outdated
@@ -2573,6 +2573,13 @@ const ( | |||
ServiceAffinityNone ServiceAffinity = "None" | |||
) | |||
|
|||
type ClientIPAffinityConfig struct { | |||
// TimeoutSeconds specifies the seconds of ClientIP type session sticky time. | |||
// The value should be specified if ServiceAffinity == "ClientIP". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
comment that it must be > 0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
pkg/api/types.go
Outdated
@@ -2700,6 +2707,10 @@ type ServiceSpec struct { | |||
// +optional | |||
SessionAffinity ServiceAffinity | |||
|
|||
// Optional: Represents the configurations of Client IP based session affinity. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you want a struct, for future safety.
type ServiceAffinityConfig struct {
// only one of the following may be set
ClientIPConfig *ServiceAffinityConfigClientIP
}
type ServiceAffinityConfigClientIP struct {
TimeoutSeconds int
}
Then embed ServiceAffinityConfig
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should see if there is a precedent for "Config" vs "Params"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems that there is no precedent for "Config" or "Params". Does it matter?
pkg/api/validation/validation.go
Outdated
return allErrs | ||
} | ||
|
||
allErrs = append(allErrs, ValidateNonnegativeField(int64(config.TimeoutSeconds), fldPath.Child("timeoutSeconds"))...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need positive, not just non-negative. 0 is no good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed. Validate (value >0 && value <= maxTimeout)
pkg/api/validation/validation.go
Outdated
@@ -2844,6 +2854,8 @@ func ValidateService(service *api.Service) field.ErrorList { | |||
allErrs = append(allErrs, field.NotSupported(specPath.Child("sessionAffinity"), service.Spec.SessionAffinity, supportedSessionAffinityType.List())) | |||
} | |||
|
|||
allErrs = append(allErrs, validateClientIPAffinityConfig(service.Spec.ClientIPAffinityConfig, specPath.Child("clientIPAffinityConfig"))...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to check which sessionAffinity setting is in use. If it is None, don't verify.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and if it is ClientIP, require that the config is non-nil.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also we should probably document and enforce an upper bound. 86400 for 1 day? 604800 for 1 week?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to check which sessionAffinity setting is in use. If it is None, don't verify.
I check the sessionAffinity setting inside validateClientIPAffinityConfig()
. PTAL. THANKS!
and if it is ClientIP, require that the config is non-nil.
If it is ClientIP, I create a default config. Is that ok?
also we should probably document and enforce an upper bound. 86400 for 1 day? 604800 for 1 week?
Current upper bound is 86400 for 1 day. Is that ok?
|
||
func TestValidateClientIPAffinityConfig(t *testing.T) { | ||
validTimeoutSeconds := []int32{ | ||
0, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
0 can't be valid. what does it mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed. 0 is invalid.
} | ||
errorCases := []*api.ClientIPAffinityConfig{} | ||
for _, timeout := range invalidTimeoutSeconds { | ||
config := &api.ClientIPAffinityConfig{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also test nil.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also test too big
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also test too big
Fixed.
also test nil.
Currently I will create a default config if user-provided config is nil. So, I test nil config in success cases. Please take a look. Please let me know if there is problem. Thanks!
pkg/proxy/iptables/proxier.go
Outdated
@@ -135,6 +135,9 @@ func (lkct LinuxKernelCompatTester) IsCompatible() error { | |||
const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" | |||
const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" | |||
|
|||
// stickyMaxAgeSeconds default value is 180 minutes | |||
const defaultStickyMaxAgeSeconds int = 180 * 60 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should have a default here. document and default it in the API code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
pkg/proxy/userspace/proxier.go
Outdated
@@ -450,12 +452,16 @@ func (proxier *Proxier) mergeService(service *api.Service) sets.String { | |||
info.loadBalancerStatus = *helper.LoadBalancerStatusDeepCopy(&service.Status.LoadBalancer) | |||
info.nodePort = int(servicePort.NodePort) | |||
info.sessionAffinityType = service.Spec.SessionAffinity | |||
// Use session affinity timeout value | |||
if service.Spec.ClientIPAffinityConfig != nil && service.Spec.ClientIPAffinityConfig.TimeoutSeconds > 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
API code should ensure that by the time we get here, we already can assume that if affinity == ClientIP, the config structure is non-nil
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, I agree. Fixed.
type ClientIPAffinityConfig struct { | ||
// TimeoutSeconds specifies the seconds of ClientIP type session sticky time. | ||
// The value should be specified if ServiceAffinity == "ClientIP". | ||
// +optional |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All the same comments from internal code, plus:
If this is optional, it must be a pointer, and you need to document the default value
Must be greater than 0.
Must be less than whatever we decide for limit.
No "should" in the comments. If this struct is provided, this field must be specified.
2cab86f
to
b7aa674
Compare
b7aa674
to
257e54c
Compare
a278a4c
to
5514f32
Compare
Thanks for your quick reply. I fixed all comments now. Please take a look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The defaults should be set in the defaulting code - ./pkg/api/v1/defaults.go
pkg/api/types.go
Outdated
// TimeoutSeconds specifies the seconds of ClientIP type session sticky time. | ||
// The value must be >0 && <=86400(for 1 day) if ServiceAffinity == "ClientIP". | ||
// Default value is 86400 for 1 day. | ||
// +optional |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have to decide if this is optional or not. If it is optional, that means a user could specify:
serviceAffinityConfig:
clientIPConfig: {}
Is that OK? Or do we want to require this field if you ever specify the config at all?
If it is optional, it has to be a pointer, because convention.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your good example.
Currently, I changed TimeoutSeconds
to be a required field when clientIPConfig is not nil.
pkg/api/types.go
Outdated
@@ -2700,6 +2725,10 @@ type ServiceSpec struct { | |||
// +optional | |||
SessionAffinity ServiceAffinity | |||
|
|||
// ServiceAffinityConfig contains the configurations of session affinity. | |||
// +optional | |||
ServiceAffinityConfig *ServiceAffinityConfig |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should think about this from the API point of view. Do we need 2 levels of nesting? Or should this be embedded.
e.g.
sessionAffinity: ClientIP
serviceAffinityConfig:
clientIPConfig:
timeoutSeconds: 60
vs
sessionAffinity: ClientIP
clientIPAffinityConfig:
timeoutSeconds: 60
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree we should consider it more.
In my previous implementaion, it was
sessionAffinity: ClientIP
clientIPAffinityConfig:
timeoutSeconds: 60
And from your previous comment,
I think you want a struct, for future safety.
type ServiceAffinityConfig struct {
// only one of the following may be set
ClientIPConfig *ServiceAffinityConfigClientIP
}
type ServiceAffinityConfigClientIP struct {
TimeoutSeconds int
}
Then embed ServiceAffinityConfig
I guess you think clientIPAffinityConfig
was not future-safe enough. So, I introduced a 2-level nesting.
sessionAffinity: ClientIP
serviceAffinityConfig:
clientIPConfig:
timeoutSeconds: 60
Am I missing the point?
I think the difference is serviceAffinityConfig
vs. clientIPAffinityConfig
. For example, if we introduce a new sessionAffinity type, such as "UserName", in the future. Perhaps we will to create a new type userNameAffinityConfig
which has a timeoutSeconds
field.
2 levels of nesting would be
sessionAffinity: UserName
serviceAffinityConfig:
userNameAffinityConfig:
timeoutSeconds: 60
embedded would be:
sessionAffinity: UserName
userNameAffinityConfig:
timeoutSeconds: 60
It seems embedded way looks more clean and it seems serviceAffinityConfig
is unnecessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently, I adopt the embedded way, and the API definition is
type ServiceSpec {
...
ClientIPAffinityConfig *ClientIPAffinityConfig
}
// ClientIPAffinityConfig represents the configurations of Client IP based session affinity.
type ClientIPAffinityConfig struct {
TimeoutSeconds int32
}
What do you think about it? Thanks!
pkg/proxy/iptables/proxier.go
Outdated
externalIPs: make([]string, len(service.Spec.ExternalIPs)), | ||
loadBalancerSourceRanges: make([]string, len(service.Spec.LoadBalancerSourceRanges)), | ||
onlyNodeLocalEndpoints: onlyNodeLocalEndpoints, | ||
} | ||
if info.sessionAffinityType == api.ServiceAffinityClientIP { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
extract this before, and use the result in the struct assignment. Don't use the API default here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay. Fixed.
pkg/proxy/userspace/proxier.go
Outdated
sessionAffinityType: api.ServiceAffinityNone, // default | ||
stickyMaxAgeMinutes: 180, // TODO: parameterize this in the API. | ||
sessionAffinityType: api.ServiceAffinityNone, // default | ||
stickyMaxAgeSeconds: int(api.ServiceAffinityClientIPDefaultSeconds), // default |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as other - don't use the default here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay. Fixed.
pkg/proxy/userspace/roundrobin.go
Outdated
ttlMinutes = 180 //default to 3 hours if not specified. Should 0 be unlimited instead???? | ||
func (lb *LoadBalancerRR) newServiceInternal(svcPort proxy.ServicePortName, affinityType api.ServiceAffinity, ttlSeconds int) *balancerState { | ||
if ttlSeconds == 0 { | ||
ttlSeconds = int(api.ServiceAffinityClientIPDefaultSeconds) //default to 3 hours if not specified. Should 0 be unlimited instead???? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as other - don't use the default here. do we need this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need this?
if ttlSeconds == 0 {
ttlSeconds = int(api.ServiceAffinityClientIPDefaultSeconds) //default to 3 hours if not specified. Should 0 be unlimited instead????
I'm afraid we may need this.
Since the function func (lb *LoadBalancerRR) newServiceInternal()
will be called by OnEndpointsAdd(), see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/userspace/roundrobin.go#L278-L282
and OnEndpointsUpdate()
, see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/userspace/roundrobin.go#L312-L316
They both set "0" as the ttlSeconds for newServiceInternal()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We may need to keep it in order to not breaking the existing logic? What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Those are just placeholders until a real Service is propagated, right? Been a while since I touched that,
pkg/proxy/winuserspace/roundrobin.go
Outdated
ttlMinutes = 180 //default to 3 hours if not specified. Should 0 be unlimited instead???? | ||
func (lb *LoadBalancerRR) newServiceInternal(svcPort proxy.ServicePortName, affinityType api.ServiceAffinity, ttlSeconds int) *balancerState { | ||
if ttlSeconds == 0 { | ||
ttlSeconds = int(api.ServiceAffinityClientIPDefaultSeconds) //default to 3 hours if not specified. Should 0 be unlimited instead???? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as other - don't use the default here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar to userspace, the function func (lb *LoadBalancerRR) newServiceInternal()
will be called by OnEndpointsAdd()
, see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/winuserspace/roundrobin.go#L268-L272
and OnEndpointsUpdate()
, see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/winuserspace/roundrobin.go#L302-L306
They both set "0" as the ttlSeconds for newServiceInternal()
. So, we may need to set a default value here.
pkg/api/validation/validation.go
Outdated
} | ||
if affinity == api.ServiceAffinityClientIP { | ||
// If ServiceAffinityConfig is nil, set the default value | ||
if config == nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Defaulting happens in API defaulting code, not validation.
./pkg/api/v1/defaults.go
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your tips. It's fixed now.
b4d5404
to
ad73fe6
Compare
/test pull-kubernetes-unit |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: m1093782566, thockin Associated issue: 49831 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
/test pull-kubernetes-e2e-gce-etcd3 |
/test pull-kubernetes-e2e-gce-bazel |
/retest |
Automatic merge from submit-queue (batch tested with PRs 49850, 47782, 50595, 50730, 51341) |
I am seeing panic errors after this PR.
I run somewhat of a odd setup of kubernetes for development because I don't use
|
Crashes with
|
I think crash is caused by the service:
Still trying to figure out why this service doesn't have |
@gnufied Thanks for reporting. Yes, currently I will work with you together to find out why :) Please feel free to update your information here. |
发件人: DuJun [mailto:notifications@github.com]
发送时间: 2017年8月29日 10:50
收件人: kubernetes/kubernetes <kubernetes@noreply.github.com>
抄送: dujun (D) <dujun5@huawei.com>; Your activity <your_activity@noreply.github.com>
主题: Re: [kubernetes/kubernetes] Paramaterize `stickyMaxAgeMinutes` for service in API (#49850)
For service apiserver, it's created by apiserver controller.go?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#49850 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEuXarxYbH4ApiL-bOGFoDniHQcAZ1Utks5sc3xcgaJpZM4OnpTf>.
|
Okay, I think deleting |
I think we can fix it by:
1. check nil pointer when use sessionAffinityConfig
2.update sessionAffinityConfig when update kubernetes service in pkg/master/controller.go
wdyt?
…________________________________
杜军
M:+86-13732245813<tel:+86-13732245813>
E:dujun5@huawei.com<mailto:dujun5@huawei.com>
产品与解决方案-PaaS服务产品部
Products & Solutions-PaaS Product Dept
发件人: Hemant Kumar
收件人: kubernetes/kubernetes<kubernetes@noreply.github.com<mailto:kubernetes@noreply.github.com>>
抄送: dujun (D)<dujun5@huawei.com<mailto:dujun5@huawei.com>>;Mention<mention@noreply.github.com<mailto:mention@noreply.github.com>>
主题: Re: [kubernetes/kubernetes] Paramaterize `stickyMaxAgeMinutes` for service in API (#49850)
时间: 2017-08-29 11:00:55
Okay, I think deleting kubernetes service via k delete service kubernetes and restarting kubernetes components has fixed the problem. I see that, there was a change made to add sessionAffinityConfig in service that gets started via controller. I am wondering, how will this affect users who are simply upgrading and already have an etcd database.
―
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#49850 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEuXavoFWuB1Radkp_5n0yFNAUuZDbaVks5sc37jgaJpZM4OnpTf>.
|
But I think any old service with Client IP type session affinity may has the same issue. So, I prefer solution 1. |
solution 3, which maybe better I think, add defaulting when create/update service. |
Automatic merge from submit-queue (batch tested with PRs 51819, 51706, 51761, 51818, 51500) fix kube-proxy panic because of nil sessionAffinityConfig **What this PR does / why we need it**: fix kube-proxy panic because of nil sessionAffinityConfig **Which issue this PR fixes**: closes #51499 **Special notes for your reviewer**: I apology that this bug is introduced by #49850 :( @thockin @smarterclayton @gnufied **Release note**: ```release-note NONE ```
As per @thockin's comment, I'm cc'ing @steveperry-53 for the doc and @radhikapc and @Bradamant3 for the release note. |
@chenopis Thanks for doing that. Please ping me when PR is ready, I would be very happy to review :) |
What this PR does / why we need it:
Currently I find
stickyMaxAgeMinutes
for a session affinity type service is hard code to 180min. There is a TODO comment, seehttps://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L205
I think the seesion sticky max time varies from service to service and users may not aware of it since it's hard coded in all proxier.go - iptables, userspace and winuserspace.
Once we parameterize it in API, users can set/get the values for their different services.
Perhaps, we can introduce a new field
api.ClientIPAffinityConfig
inapi.ServiceSpec
.There is an initial discussion about it in sig-network group. See,
https://groups.google.com/forum/#!topic/kubernetes-sig-network/i-LkeHrjs80
Which issue this PR fixes:
fixes #49831
Special notes for your reviewer:
Release note: