-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
use lowercase hostnames for node names #65487
Conversation
Usage of names containing uppercase characters returned by calls to gethostname and getfqdn in requests to apiserver related to nodes results in 404 errors. Node names are lowercase in K8s itself so charms should make sure to use lowercase names well as it results in errors. pkg/util/node/node.go has code to convert hostnames to lowercase in GetHostname and that function is used to form node names.
/ok-to-test |
@@ -1058,9 +1058,9 @@ def get_node_name(): | |||
elif is_state('endpoint.openstack.ready'): | |||
cloud_provider = 'openstack' | |||
if cloud_provider == 'aws': | |||
return getfqdn() | |||
return getfqdn().lower() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure about this one. It looks to me like the node name is only lower-cased when no cloud provider is set.
Here's where the hostname gets lower-cased:
kubernetes/pkg/util/node/node.go
Lines 45 to 56 in c005b9d
// GetHostname returns OS's hostname if 'hostnameOverride' is empty; otherwise, return 'hostnameOverride'. | |
func GetHostname(hostnameOverride string) string { | |
hostname := hostnameOverride | |
if hostname == "" { | |
nodename, err := os.Hostname() | |
if err != nil { | |
glog.Fatalf("Couldn't determine hostname: %v", err) | |
} | |
hostname = nodename | |
} | |
return strings.ToLower(strings.TrimSpace(hostname)) | |
} |
Here's what kubelet does with it:
kubernetes/pkg/kubelet/kubelet.go
Lines 372 to 384 in c005b9d
hostname := nodeutil.GetHostname(hostnameOverride) | |
// Query the cloud provider for our node name, default to hostname | |
nodeName := types.NodeName(hostname) | |
cloudIPs := []net.IP{} | |
cloudNames := []string{} | |
if kubeDeps.Cloud != nil { | |
var err error | |
instances, ok := kubeDeps.Cloud.Instances() | |
if !ok { | |
return nil, fmt.Errorf("failed to get instances from cloud provider") | |
} | |
nodeName, err = instances.CurrentNodeName(context.TODO(), hostname) |
In short: if there's a cloud provider, kubelet will get the node name from that instead of from GetHostname
.
For the AWS cloud provider, I don't see it being lower-cased:
kubernetes/pkg/cloudprovider/providers/aws/aws.go
Lines 4294 to 4296 in c005b9d
func mapInstanceToNodeName(i *ec2.Instance) types.NodeName { | |
return types.NodeName(aws.StringValue(i.PrivateDnsName)) | |
} |
I don't actually know if it's possible for AWS's PrivateDnsName
to contain uppercase characters, but I would like to err on the side of caution and assume it might.
@dshcherb can we reduce the scope of this change to only apply when no cloud-provider
is set? I would feel safest with something like this instead:
if cloud_provider == '':
return gethostname().lower()
elif cloud_provider == 'aws':
return getfqdn()
else:
return gethostname()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The K8s docs seem to be pretty clear on conventions:
https://kubernetes.io/docs/concepts/overview/working-with-objects/names/
"By convention, the names of Kubernetes resources should be up to maximum length of 253 characters and consist of lower case alphanumeric characters, -, and ., but certain resources have more specific restrictions."
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/identifiers.md - this has a bit vague references to resources specifically but mentions DNS labels and lowercase nevertheless.
I have also followed some code paths (from "node create" to the request validation) - looks like an uppercase node (resource) name would not pass ValidateNodeName check as it requires a node name to be have the following regex:
const dns1035LabelFmt string = "[a-z]([-a-z0-9]*[a-z0-9])?"
https://gist.github.com/dshcherb/1ba9b9cb62f8753de210bc459115c07a
I could reduce the scope but I don't think this is necessary due to the above.
Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @dshcherb. You're right - it looks like the apiserver does adhere strictly to that convention:
$ cat test-node.yaml
apiVersion: v1
kind: Node
metadata:
name: TEST-NODE
$ kubectl create -f test-node.yaml
The Node "TEST-NODE" is invalid: metadata.name: Invalid value: "TEST-NODE": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
I'm +1 to this as-is then - the charm should always convert it to lower case. Thanks for setting me straight.
/lgtm Good work, thanks 👍 |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Cynerva, dshcherb The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Automatic merge from submit-queue (batch tested with PRs 60150, 65467, 65487, 65595, 65374). If you want to cherry-pick this change to another branch, please follow the instructions here. |
/kind bug |
/sig node |
What this PR does / why we need it:
Uppercase hostnames used in charms result in (lowercase) node name lookup errors. This happens when /etc/hostname contains uppercase characters and gethostname or getfqdn return those characters.
Special notes for your reviewer:
Discovered in a field deployment where hostnames are all uppercase.
Release note: