-
Notifications
You must be signed in to change notification settings - Fork 40.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubeadm: Make the CLI arguments for the control plane overridable #41772
kubeadm: Make the CLI arguments for the control plane overridable #41772
Conversation
Hi @xilabao. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
@xilabao thanks for your PR! Do you have a specific use case for why you want to implement this change? We want to try and avoid API bloat and I'd like to understand why you need it. |
@dmmcquay It's very helpful to allow user to add extra admission controls. E.g. https://kubernetes.io/docs/user-guide/pod-security-policy/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@xilabao Thanks for this PR!
In order to not make a field on the kubeadm config file for every option on apiserver and controller-manager (leads to bloat), I think we should make the solution more generic.
Let's start by doing this for the API Server. Then we might want to do it for controller-manager as well
@@ -40,6 +40,7 @@ type MasterConfiguration struct { | |||
KubernetesVersion string | |||
CloudProvider string | |||
AuthorizationMode string | |||
AdmissionControl string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you rename this to ExtraAPIServerArgs map[string]string
?
@@ -30,6 +30,7 @@ type MasterConfiguration struct { | |||
KubernetesVersion string `json:"kubernetesVersion"` | |||
CloudProvider string `json:"cloudProvider"` | |||
AuthorizationMode string `json:"authorizationMode"` | |||
AdmissionControl string `json:"admissionControl"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also
ExtraAPIServerArgs map[string]string `json:"extraAPIServerArgs"`
@@ -27,6 +27,7 @@ const ( | |||
DefaultAPIBindPort = 6443 | |||
DefaultDiscoveryBindPort = 9898 | |||
DefaultAuthorizationMode = "RBAC" | |||
DefaultAdmissionControl = "NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this can move to constants.go
cmd/kubeadm/app/master/manifests.go
Outdated
@@ -305,7 +305,7 @@ func getAPIServerCommand(cfg *kubeadmapi.MasterConfiguration, selfHosted bool) [ | |||
|
|||
command = append(getComponentBaseCommand(apiServer), | |||
"--insecure-bind-address=127.0.0.1", | |||
"--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota", | |||
"--admission-control="+cfg.AdmissionControl, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you do this instead here:
for k, v := range cfg.ExtraAPIServerArgs {
command = append(command, fmt.Sprintf("--%s=%s", k, v))
}
// These arguments are defaults that can be overridden by ExtraAPIServerArgs
configurableDefaultCommands := map[string]string{
"admission-control": kubeadmconstants.DefaultAdmissionControl,
"insecure-bind-address": "127.0.0.1",
"allow-privileged": "true",
}
for k, v := range configurableDefaultCommands {
if _, overrideExists := cfg.ExtraAPIServerArgs[k]; !overrideExists {
command = append(command, fmt.Sprintf("--%s=%s", k, v))
}
}
the logic I outlined here should probably be a separate function that takes cfg.ExtraAPIServerArgs
and configurableDefaultCommands
as inputs and outputs a []string
bb49b1d
to
ce036e5
Compare
@luxas PTAL |
KubernetesVersion string `json:"kubernetesVersion"` | ||
CloudProvider string `json:"cloudProvider"` | ||
AuthorizationMode string `json:"authorizationMode"` | ||
ExtraAPIServerArgs map[string]string `json:"extraAPIServerArgs"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would name this "APIServerExtraFlags"
Put API server first so that as we have other stuff it sorts.
Flags explains why it is a map. Still not clear how to do singleton flags or remove flags but I think that is okay.
Also, we should start documenting these structures (ala types.go everywhere else). Explain that this is a way to add and override flags that get set on the API server.
For things like the admission controller this still isn't 100% as there is no way to add/remove/modify the list. But it is good enough to get started for now and at least it is possible to modify this stuff.
Might also be good to support this for the KCM and the scheduler. But that can come in other PRs.
cmd/kubeadm/app/master/manifests.go
Outdated
"--allow-privileged", | ||
"--storage-backend=etcd3", | ||
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname", | ||
constantDefaultCommands := map[string]string{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not let users override all of this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, let's rename this to defaultArguments
or similar
cmd/kubeadm/app/master/manifests.go
Outdated
// These arguments are defaults that can be overridden by ExtraAPIServerArgs | ||
configurableDefaultCommands := map[string]string{ | ||
"admission-control": kubeadmconstants.DefaultAdmissionControl, | ||
"insecure-bind-address": "127.0.0.1", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we going to turn this off at some point?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, much better 👍
Still some comments
cmd/kubeadm/app/master/manifests.go
Outdated
"--allow-privileged", | ||
"--storage-backend=etcd3", | ||
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname", | ||
constantDefaultCommands := map[string]string{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, let's rename this to defaultArguments
or similar
cmd/kubeadm/app/master/manifests.go
Outdated
"allow-privileged": "true", | ||
} | ||
|
||
getConstantParameters := func() []string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This func can be removed
cmd/kubeadm/app/master/manifests.go
Outdated
return command | ||
} | ||
|
||
getExtraParameters := func(args map[string]string) []string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be a standalone function that takes two params: defaults
and overrides
. The logic could/should look something like this:
command := []string{}
for k, v := range overrides {
command = append(command, fmt.Sprintf("--%s=%s", k, v))
}
for k, v := range defaults {
if _, overrideExists := overrides[k]; !overrideExists {
command = append(command, fmt.Sprintf("--%s=%s", k, v))
}
}
it just takes what's in overrides right away and appends the rest of the defaults that don't exist already
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There must be an unit test for that function
cmd/kubeadm/app/master/manifests.go
Outdated
"--allow-privileged", | ||
"--storage-backend=etcd3", | ||
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname", | ||
constantDefaultCommands := map[string]string{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a similar field for the scheduler and controller-manager and implement this for them as well.
Due to the code freeze, we must get this today/tomorrow or wait a month.
Comments would be appreciated but I wouldn't block on that
e4ea00c
to
cc520de
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This starts to look really good! Keep up the good work.
This still needs to:
- be rebased
- have controller-manager and scheduler args added in the same manner
- have all args configurable
Then this LGTM.
Quick note -- if you want this into the v1.6 release, it has to be ready tomorrow (in ~24 hours), due to the code freeze. If you can make it, then it would be great, if you can't; it would ok as well, then we'll just target it for v1.7
KubernetesVersion string | ||
CloudProvider string | ||
AuthorizationMode string | ||
APIServerExtraArgs map[string]string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add the controller-manager and scheduler args here as well
cmd/kubeadm/app/master/manifests.go
Outdated
} | ||
|
||
// These arguments are defaults that can be canceled by APIServerExtraArgs | ||
configurableDefaultArguments := map[string]string{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please make all args configurable and call them just defaultArguments
, same for controller-manager and scheduler
cmd/kubeadm/app/master/manifests.go
Outdated
|
||
command = getComponentBaseCommand(apiServer) | ||
for k, v := range defaultArguments { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this becomes unnecessary when having only one map of defaults
actual[i], | ||
) | ||
} | ||
sort.Strings(actual) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Many thanks for fixing this. Has been annoying but haven't got to fixing it 👍
cc520de
to
8654217
Compare
Thanks for your help. @luxas PTAL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me!
Thanks for your work with this PR!
@k8s-bot ok to test /lgtm |
[APPROVALNOTIFIER] This PR is APPROVED The following people have approved this PR: luxas, xilabao Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
Automatic merge from submit-queue |
No description provided.