Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm: Make the CLI arguments for the control plane overridable #41772

Conversation

xilabao
Copy link
Contributor

@xilabao xilabao commented Feb 21, 2017

No description provided.

@k8s-ci-robot
Copy link
Contributor

Hi @xilabao. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 21, 2017
@k8s-reviewable
Copy link

This change is Reviewable

@k8s-github-robot k8s-github-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. release-note-label-needed labels Feb 21, 2017
@dmmcquay
Copy link
Contributor

@xilabao thanks for your PR! Do you have a specific use case for why you want to implement this change? We want to try and avoid API bloat and I'd like to understand why you need it.

@xilabao
Copy link
Contributor Author

xilabao commented Feb 22, 2017

@dmmcquay It's very helpful to allow user to add extra admission controls. E.g. https://kubernetes.io/docs/user-guide/pod-security-policy/

Copy link
Member

@luxas luxas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xilabao Thanks for this PR!

In order to not make a field on the kubeadm config file for every option on apiserver and controller-manager (leads to bloat), I think we should make the solution more generic.

Let's start by doing this for the API Server. Then we might want to do it for controller-manager as well

@@ -40,6 +40,7 @@ type MasterConfiguration struct {
KubernetesVersion string
CloudProvider string
AuthorizationMode string
AdmissionControl string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you rename this to ExtraAPIServerArgs map[string]string?

@@ -30,6 +30,7 @@ type MasterConfiguration struct {
KubernetesVersion string `json:"kubernetesVersion"`
CloudProvider string `json:"cloudProvider"`
AuthorizationMode string `json:"authorizationMode"`
AdmissionControl string `json:"admissionControl"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also

ExtraAPIServerArgs map[string]string `json:"extraAPIServerArgs"`

@@ -27,6 +27,7 @@ const (
DefaultAPIBindPort = 6443
DefaultDiscoveryBindPort = 9898
DefaultAuthorizationMode = "RBAC"
DefaultAdmissionControl = "NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can move to constants.go

@@ -305,7 +305,7 @@ func getAPIServerCommand(cfg *kubeadmapi.MasterConfiguration, selfHosted bool) [

command = append(getComponentBaseCommand(apiServer),
"--insecure-bind-address=127.0.0.1",
"--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota",
"--admission-control="+cfg.AdmissionControl,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you do this instead here:

for k, v := range cfg.ExtraAPIServerArgs {
    command = append(command, fmt.Sprintf("--%s=%s", k, v))
}
// These arguments are defaults that can be overridden by ExtraAPIServerArgs
configurableDefaultCommands := map[string]string{
    "admission-control": kubeadmconstants.DefaultAdmissionControl,
    "insecure-bind-address": "127.0.0.1",
    "allow-privileged": "true",
}
for k, v := range configurableDefaultCommands {
    if _, overrideExists := cfg.ExtraAPIServerArgs[k]; !overrideExists {
        command = append(command, fmt.Sprintf("--%s=%s", k, v))
    }
}

the logic I outlined here should probably be a separate function that takes cfg.ExtraAPIServerArgs and configurableDefaultCommands as inputs and outputs a []string

@xilabao xilabao force-pushed the add-admission-control-option-to-config branch from bb49b1d to ce036e5 Compare February 24, 2017 07:56
@k8s-github-robot k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 24, 2017
@xilabao
Copy link
Contributor Author

xilabao commented Feb 24, 2017

@luxas PTAL

KubernetesVersion string `json:"kubernetesVersion"`
CloudProvider string `json:"cloudProvider"`
AuthorizationMode string `json:"authorizationMode"`
ExtraAPIServerArgs map[string]string `json:"extraAPIServerArgs"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would name this "APIServerExtraFlags"

Put API server first so that as we have other stuff it sorts.

Flags explains why it is a map. Still not clear how to do singleton flags or remove flags but I think that is okay.

Also, we should start documenting these structures (ala types.go everywhere else). Explain that this is a way to add and override flags that get set on the API server.

For things like the admission controller this still isn't 100% as there is no way to add/remove/modify the list. But it is good enough to get started for now and at least it is possible to modify this stuff.

Might also be good to support this for the KCM and the scheduler. But that can come in other PRs.

"--allow-privileged",
"--storage-backend=etcd3",
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
constantDefaultCommands := map[string]string{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not let users override all of this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, let's rename this to defaultArguments or similar

// These arguments are defaults that can be overridden by ExtraAPIServerArgs
configurableDefaultCommands := map[string]string{
"admission-control": kubeadmconstants.DefaultAdmissionControl,
"insecure-bind-address": "127.0.0.1",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to turn this off at some point?

Copy link
Member

@luxas luxas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, much better 👍
Still some comments

"--allow-privileged",
"--storage-backend=etcd3",
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
constantDefaultCommands := map[string]string{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, let's rename this to defaultArguments or similar

"allow-privileged": "true",
}

getConstantParameters := func() []string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This func can be removed

return command
}

getExtraParameters := func(args map[string]string) []string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a standalone function that takes two params: defaults and overrides. The logic could/should look something like this:

command := []string{}
for k, v := range overrides {
   command = append(command, fmt.Sprintf("--%s=%s", k, v))
}
for k, v := range defaults {
    if _, overrideExists := overrides[k]; !overrideExists {
         command = append(command, fmt.Sprintf("--%s=%s", k, v))
    }
}

it just takes what's in overrides right away and appends the rest of the defaults that don't exist already

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There must be an unit test for that function

"--allow-privileged",
"--storage-backend=etcd3",
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
constantDefaultCommands := map[string]string{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a similar field for the scheduler and controller-manager and implement this for them as well.
Due to the code freeze, we must get this today/tomorrow or wait a month.

Comments would be appreciated but I wouldn't block on that

@xilabao xilabao force-pushed the add-admission-control-option-to-config branch 5 times, most recently from e4ea00c to cc520de Compare February 25, 2017 14:29
@xilabao
Copy link
Contributor Author

xilabao commented Feb 25, 2017

@jbeda @luxas PTAL

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 26, 2017
Copy link
Member

@luxas luxas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This starts to look really good! Keep up the good work.
This still needs to:

  • be rebased
  • have controller-manager and scheduler args added in the same manner
  • have all args configurable

Then this LGTM.

Quick note -- if you want this into the v1.6 release, it has to be ready tomorrow (in ~24 hours), due to the code freeze. If you can make it, then it would be great, if you can't; it would ok as well, then we'll just target it for v1.7

KubernetesVersion string
CloudProvider string
AuthorizationMode string
APIServerExtraArgs map[string]string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the controller-manager and scheduler args here as well

}

// These arguments are defaults that can be canceled by APIServerExtraArgs
configurableDefaultArguments := map[string]string{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please make all args configurable and call them just defaultArguments, same for controller-manager and scheduler


command = getComponentBaseCommand(apiServer)
for k, v := range defaultArguments {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this becomes unnecessary when having only one map of defaults

actual[i],
)
}
sort.Strings(actual)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for fixing this. Has been annoying but haven't got to fixing it 👍

@luxas luxas added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed labels Feb 26, 2017
@luxas luxas changed the title add admission control option to config kubeadm: Make the CLI arguments to the control plane overridable Feb 26, 2017
@luxas luxas changed the title kubeadm: Make the CLI arguments to the control plane overridable kubeadm: Make the CLI arguments for the control plane overridable Feb 26, 2017
@xilabao xilabao force-pushed the add-admission-control-option-to-config branch from cc520de to 8654217 Compare February 27, 2017 00:58
@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 27, 2017
@xilabao
Copy link
Contributor Author

xilabao commented Feb 27, 2017

Thanks for your help. @luxas PTAL

Copy link
Member

@luxas luxas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me!
Thanks for your work with this PR!

@luxas luxas self-assigned this Feb 27, 2017
@luxas
Copy link
Member

luxas commented Feb 27, 2017

@k8s-bot ok to test

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 27, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

The following people have approved this PR: luxas, xilabao

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 27, 2017
@enisoc enisoc added this to the v1.6 milestone Feb 28, 2017
@k8s-github-robot
Copy link

Automatic merge from submit-queue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants