deprecate insecure http flags and remove already deprecated flags #59018

hzxuzhonghu · 2018-01-30T04:13:19Z

What this PR does / why we need it:

deprecate insecure-bind-address insecure-port flags
remove flags public-address-override address port They are mark deprecated in move parts of the mega generic run struct out #36604, which is more than a year ago.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #58951

Special notes for your reviewer:

Release note:

Deprecate insecure flags `--insecure-bind-address`, `--insecure-port` and remove  `--public-address-override`.

hzxuzhonghu · 2018-01-30T04:13:31Z

/assign @deads2k @sttts

hzxuzhonghu · 2018-01-30T06:17:43Z

/assign @gmarek

hzxuzhonghu · 2018-01-30T06:19:15Z

cluster/gce/gci/configure-helper.sh

@@ -1518,7 +1518,6 @@ function start-kube-apiserver {

  # Calculate variables and assemble the command line.
  local params="${API_SERVER_TEST_LOG_LEVEL:-"--v=2"} ${APISERVER_TEST_ARGS:-} ${CLOUD_CONFIG_OPT}"
-  params+=" --address=127.0.0.1"


--address is removed, insecure-bind-address is deprecated.

hzxuzhonghu · 2018-01-30T06:21:15Z

/assign @bowei

k8s-ci-robot · 2018-01-30T06:48:32Z

@hzxuzhonghu: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gke	9152862a4a61205ed127f47aa3ed46dd9695d29c	link	`/test pull-kubernetes-e2e-gke`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

hzxuzhonghu · 2018-01-30T08:00:30Z

kops is using --address, should not execute until kops stop using --address.

remove flags public-address-override address port They are mark deprecated in move parts of the mega generic run struct out #36604, which is more than a year ago.

…dress-override

hzxuzhonghu · 2018-01-30T08:09:55Z

/unassign @bowei @gmarek

sttts · 2018-01-30T10:26:27Z

kops is using --address, should not execute until kops stop using --address.

Are there issues in those projects to remove the deprecated use?

sttts · 2018-01-30T10:27:43Z

/lgtm

k8s-ci-robot · 2018-01-30T10:27:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hzxuzhonghu, sttts

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cmd/kube-apiserver/OWNERS~~ [sttts]
~~pkg/kubeapiserver/OWNERS~~ [sttts]
~~staging/src/k8s.io/apiserver/OWNERS~~ [sttts]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

hzxuzhonghu · 2018-01-30T10:45:42Z

Are there issues in those projects to remove the deprecated use?

Yes, I create one kubernetes/kops#4355

k8s-github-robot · 2018-01-30T10:59:31Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-01-30T11:43:36Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

justinsb · 2018-02-02T14:26:30Z

Do we have a solution for #43784 ?

deads2k · 2018-02-05T15:27:30Z

Do we have a solution for #43784 ?

Health checks? /healthz isn't confidential and default policy allows anonymous access.

As of Kubernetes 1.10, the insecure flags will be deprecated: kubernetes/kubernetes#59018 Currently, there is no other way to allow unauthenticated health checks (requests on kube-apiserver's /healthz endpoint) other than allowing anonymous requests (which we do not want). Related issue: kubernetes/kubernetes#43784 We are now going to use the basic authentication credentials which the kubelet will use to reach the /healthz endpoint.

liggitt · 2018-03-28T18:37:21Z

pkg/kubeapiserver/options/serving.go


 	fs.IntVar(&s.BindPort, "insecure-port", s.BindPort, ""+
 		"The port on which to serve unsecured, unauthenticated access. It is assumed "+
 		"that firewall rules are set up such that this port is not reachable from outside of "+
 		"the cluster and that port 443 on the cluster's public address is proxied to this "+
-		"port. This is performed by nginx in the default setup. Set to zero to disable")
+		"port. This is performed by nginx in the default setup. Set to zero to disable.")
+	fs.MarkDeprecated("insecure-port", "This flag will be removed in a future version.")


marking this deprecated actually omits it from the help, which means the fact that it is enabled by default and you need to "Set to zero to disable." is invisible to the end user. That is not good.

Never understood this behaviour.

mwthink · 2018-04-13T19:56:49Z

The API server still serves on localhost:8080 by default. Is this expected to be the case going forward, or is insecure access being removed entirely?

If the latter, it should be disabled by default, but if that's the case, what about health endpoints? Am I going to need a sidecar container to validate the TLS and proxy /healthz over HTTP?

Edit - For anyone coming along later, check out #43784

deads2k · 2018-04-16T12:34:16Z

The API server still serves on localhost:8080 by default. Is this expected to be the case going forward, or is insecure access being removed entirely?

Nothing should be relying on the insecure port. It is a target for future removal.

javefang · 2018-04-16T13:24:05Z

@deads2k we are running the apiserver as a static pod on master nodes, which currently requires the apiserver to listen on an insecure port 8080 on localhost for kubelet to be able to talk to it. All other worker nodes' kubelet uses the "--bootstrap-kubeconfig" to perform TLS bootstrapping once the apiserver is up.

I tried to use the same "--bootstrap-kubeconfig" for kubelet running on master, but it complains that it can't reach the apiserver (for obvious reason) and crash, without starting the apiserver/controller-manager/scheduler pods defined in the manifests.

Is there a proper way for kubelet on master to talk to the apiserver without utilising the --insecure-port and --insecure-bind-address?

Environment:
OS: Centos 7.3
Docker: 1.12.6
Kubernetes 1.10.1

deads2k · 2018-04-16T13:45:33Z

Short lived client certificates are a good option since the kubelet running the kube-apiserver is effectively root on the cluster already.

* Although the --insecure-port flag is deprecated, apiserver continues to default to listening on 127.0.0.1:8080 * Explicitly disable insecure local listener since its unused * kubernetes/kubernetes#59018 (comment) * 5f3546b

mrmm · 2018-10-02T15:34:55Z

Update, thanks @justinsb implemented this for Kops (PR kubernetes/kops#5234)
Cheers 🍺

Though the flags are deprecated, the deprecation process revealed that the flag is important for health checks in more secure environments. Add a link to kubernetes#43784 to help developers understand why the flags have not yet been removed. Issue kubernetes#43784 Issue kubernetes#74172 Issue kubernetes#59018

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 30, 2018

k8s-ci-robot assigned deads2k and sttts Jan 30, 2018

k8s-ci-robot requested review from derekwaynecarr, lavalamp and liggitt January 30, 2018 04:13

k8s-ci-robot assigned gmarek Jan 30, 2018

hzxuzhonghu commented Jan 30, 2018

View reviewed changes

k8s-ci-robot assigned bowei Jan 30, 2018

deprecate insecure http flags and remove already deprecated public-ad…

24c687f

…dress-override

hzxuzhonghu force-pushed the deprecate-http branch from 9152862 to 24c687f Compare January 30, 2018 08:06

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 30, 2018

k8s-ci-robot unassigned bowei and gmarek Jan 30, 2018

hzxuzhonghu mentioned this pull request Jan 30, 2018

Should stop using --address kube-apiserver flag kubernetes/kops#4355

Closed

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 30, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 30, 2018

k8s-github-robot merged commit 2f175bc into kubernetes:master Jan 30, 2018

hzxuzhonghu deleted the deprecate-http branch January 30, 2018 11:46

liggitt reviewed Mar 28, 2018

View reviewed changes

Evalle mentioned this pull request May 11, 2018

Liveness probes for kube-apiserver pod are failing with --anonymous-auth=false in place kubernetes/kubeadm#798

Closed

dghubble mentioned this pull request Jun 28, 2018

Update Kubernetes from v1.10.5 to v1.11.x poseidon/terraform-render-bootstrap#68

Merged

chadswen mentioned this pull request Sep 11, 2018

Revert "Remove insecure-port and insecure-bind-address when possible" kubernetes-sigs/kubespray#3288

Merged

WanLinghao mentioned this pull request Feb 17, 2019

Remove DEPRECATED insecure flags in kube-apiserver #74172

Closed

justinsb mentioned this pull request Feb 18, 2019

Add code comments to guard inadvertent removal of insecure-bind-address #74217

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deprecate insecure http flags and remove already deprecated flags #59018

deprecate insecure http flags and remove already deprecated flags #59018

hzxuzhonghu commented Jan 30, 2018 •

edited by sttts

Loading

hzxuzhonghu commented Jan 30, 2018

hzxuzhonghu commented Jan 30, 2018

hzxuzhonghu Jan 30, 2018

hzxuzhonghu commented Jan 30, 2018

k8s-ci-robot commented Jan 30, 2018 •

edited

Loading

hzxuzhonghu commented Jan 30, 2018

hzxuzhonghu commented Jan 30, 2018

sttts commented Jan 30, 2018

sttts commented Jan 30, 2018

k8s-ci-robot commented Jan 30, 2018

hzxuzhonghu commented Jan 30, 2018

k8s-github-robot commented Jan 30, 2018

k8s-github-robot commented Jan 30, 2018

justinsb commented Feb 2, 2018

deads2k commented Feb 5, 2018

liggitt Mar 28, 2018

sttts Mar 29, 2018

mwthink commented Apr 13, 2018 •

edited

Loading

deads2k commented Apr 16, 2018

javefang commented Apr 16, 2018 •

edited

Loading

deads2k commented Apr 16, 2018

mrmm commented Oct 2, 2018 •

edited

Loading

deprecate insecure http flags and remove already deprecated flags #59018

deprecate insecure http flags and remove already deprecated flags #59018

Conversation

hzxuzhonghu commented Jan 30, 2018 • edited by sttts Loading

hzxuzhonghu commented Jan 30, 2018

hzxuzhonghu commented Jan 30, 2018

hzxuzhonghu Jan 30, 2018

Choose a reason for hiding this comment

hzxuzhonghu commented Jan 30, 2018

k8s-ci-robot commented Jan 30, 2018 • edited Loading

hzxuzhonghu commented Jan 30, 2018

hzxuzhonghu commented Jan 30, 2018

sttts commented Jan 30, 2018

sttts commented Jan 30, 2018

k8s-ci-robot commented Jan 30, 2018

hzxuzhonghu commented Jan 30, 2018

k8s-github-robot commented Jan 30, 2018

k8s-github-robot commented Jan 30, 2018

justinsb commented Feb 2, 2018

deads2k commented Feb 5, 2018

liggitt Mar 28, 2018

Choose a reason for hiding this comment

sttts Mar 29, 2018

Choose a reason for hiding this comment

mwthink commented Apr 13, 2018 • edited Loading

deads2k commented Apr 16, 2018

javefang commented Apr 16, 2018 • edited Loading

deads2k commented Apr 16, 2018

mrmm commented Oct 2, 2018 • edited Loading

hzxuzhonghu commented Jan 30, 2018 •

edited by sttts

Loading

k8s-ci-robot commented Jan 30, 2018 •

edited

Loading

mwthink commented Apr 13, 2018 •

edited

Loading

javefang commented Apr 16, 2018 •

edited

Loading

mrmm commented Oct 2, 2018 •

edited

Loading