Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use lowercase hostnames for node names #65487

Merged
merged 1 commit into from
Jun 29, 2018

Conversation

dshcherb
Copy link
Contributor

@dshcherb dshcherb commented Jun 26, 2018

What this PR does / why we need it:

Uppercase hostnames used in charms result in (lowercase) node name lookup errors. This happens when /etc/hostname contains uppercase characters and gethostname or getfqdn return those characters.

Special notes for your reviewer:

Discovered in a field deployment where hostnames are all uppercase.

Release note:

Hostnames are now converted to lowercase before being used for node lookups in the kubernetes-worker charm.

Usage of names containing uppercase characters returned by calls to
gethostname and getfqdn in requests to apiserver related to nodes
results in 404 errors. Node names are lowercase in K8s itself so charms
should make sure to use lowercase names well as it results in errors.

pkg/util/node/node.go has code to convert hostnames to lowercase in
GetHostname and that function is used to form node names.
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 26, 2018
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jun 26, 2018
@Cynerva
Copy link
Contributor

Cynerva commented Jun 26, 2018

/ok-to-test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 26, 2018
@@ -1058,9 +1058,9 @@ def get_node_name():
elif is_state('endpoint.openstack.ready'):
cloud_provider = 'openstack'
if cloud_provider == 'aws':
return getfqdn()
return getfqdn().lower()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this one. It looks to me like the node name is only lower-cased when no cloud provider is set.

Here's where the hostname gets lower-cased:

// GetHostname returns OS's hostname if 'hostnameOverride' is empty; otherwise, return 'hostnameOverride'.
func GetHostname(hostnameOverride string) string {
hostname := hostnameOverride
if hostname == "" {
nodename, err := os.Hostname()
if err != nil {
glog.Fatalf("Couldn't determine hostname: %v", err)
}
hostname = nodename
}
return strings.ToLower(strings.TrimSpace(hostname))
}

Here's what kubelet does with it:

hostname := nodeutil.GetHostname(hostnameOverride)
// Query the cloud provider for our node name, default to hostname
nodeName := types.NodeName(hostname)
cloudIPs := []net.IP{}
cloudNames := []string{}
if kubeDeps.Cloud != nil {
var err error
instances, ok := kubeDeps.Cloud.Instances()
if !ok {
return nil, fmt.Errorf("failed to get instances from cloud provider")
}
nodeName, err = instances.CurrentNodeName(context.TODO(), hostname)

In short: if there's a cloud provider, kubelet will get the node name from that instead of from GetHostname.

For the AWS cloud provider, I don't see it being lower-cased:

func mapInstanceToNodeName(i *ec2.Instance) types.NodeName {
return types.NodeName(aws.StringValue(i.PrivateDnsName))
}

I don't actually know if it's possible for AWS's PrivateDnsName to contain uppercase characters, but I would like to err on the side of caution and assume it might.

@dshcherb can we reduce the scope of this change to only apply when no cloud-provider is set? I would feel safest with something like this instead:

if cloud_provider == '':
    return gethostname().lower()
elif cloud_provider == 'aws':
    return getfqdn()
else:
    return gethostname()

Copy link
Contributor Author

@dshcherb dshcherb Jun 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The K8s docs seem to be pretty clear on conventions:

https://kubernetes.io/docs/concepts/overview/working-with-objects/names/
"By convention, the names of Kubernetes resources should be up to maximum length of 253 characters and consist of lower case alphanumeric characters, -, and ., but certain resources have more specific restrictions."
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/identifiers.md - this has a bit vague references to resources specifically but mentions DNS labels and lowercase nevertheless.

I have also followed some code paths (from "node create" to the request validation) - looks like an uppercase node (resource) name would not pass ValidateNodeName check as it requires a node name to be have the following regex:
const dns1035LabelFmt string = "[a-z]([-a-z0-9]*[a-z0-9])?"

https://gist.github.com/dshcherb/1ba9b9cb62f8753de210bc459115c07a

I could reduce the scope but I don't think this is necessary due to the above.

Thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dshcherb. You're right - it looks like the apiserver does adhere strictly to that convention:

$ cat test-node.yaml 
apiVersion: v1
kind: Node
metadata:
  name: TEST-NODE

$ kubectl create -f test-node.yaml 
The Node "TEST-NODE" is invalid: metadata.name: Invalid value: "TEST-NODE": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

I'm +1 to this as-is then - the charm should always convert it to lower case. Thanks for setting me straight.

@Cynerva
Copy link
Contributor

Cynerva commented Jun 28, 2018

/lgtm

Good work, thanks 👍

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 28, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Cynerva, dshcherb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 28, 2018
@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 60150, 65467, 65487, 65595, 65374). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit 4859645 into kubernetes:master Jun 29, 2018
@marpaia
Copy link
Contributor

marpaia commented Jul 2, 2018

/kind bug

@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 2, 2018
@neolit123
Copy link
Member

/sig node
adding sig label for release notes tools to fetch.
please adjust if needed.

@k8s-ci-robot k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Aug 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants