-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
e2e tests specific Network/DNS related : add Ipv6 capability #59894
Conversation
/assign @danehans |
/assign @leblancd |
@fturib: GitHub didn't allow me to assign the following users: leblancd. Note that only kubernetes members and repo collaborators can be assigned. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/cc @leblancd |
@fturib: GitHub didn't allow me to request PR reviews from the following users: leblancd. Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
test/e2e/network/dns.go
Outdated
probeCmd += fmt.Sprintf(`test -n "$$(dig +tcp +noall +answer +search $${podARec} A)" && echo OK > /results/%s;`, podARecByTCPFileName) | ||
podARecByUDPFileName := fmt.Sprintf("%s_udp@Pod%sRecord", fileNamePrefix, typeA) | ||
podARecByTCPFileName := fmt.Sprintf("%s_tcp@Pod%sRecord", fileNamePrefix, typeA) | ||
probeCmd += fmt.Sprintf(`podARec="$$(hostname -i| awk -F. '{print $$1"-"$$2"-"$$3"-"$$4".%s.pod.cluster.local"}')";`, namespace) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is an error here.
I did not take the right file. should call the ReverseAddr function
This PR needs rebase. Currently an error of deployment in the docker-in-docker tool I use for testing is preventing to go ahead. I am waiting on this PR : kubernetes-retired/kubeadm-dind-cluster#85 |
@pmichali : since 2 days I tried to run my D-in-D and it runs properly for IPv4, but not for IPv6. Looks like there is a piece missing behind the services. Any idea what can go wrong here ? (It prevent me to re-verify the test after rebasing). |
@fturib - In IPv6 mode, when you try to curl kubernetes API service IP from a node, does it matter from which node you try? My guess is that maybe ip6tables aren't getting set up correctly for some reason. Can you check 'ip6tables -t nat -L | grep kube'. You can IM on Slack (my ID is 'dane' in kubernetes) if you prefer. |
PR is up to date.
For all of them, the usual DNS tests are passing. These failing tests are:
|
@fturib Does the test for "DNS [It] should provide DNS for services [Conformance]" pass, when using an IPv6 cluster in kube-dns and coredns mode? I was seeing that test failing (without any changes applied). One concern that I had, was it seemed that during test setup (the Before part), it was trying to access the API to check nodes and it was using 127.0.0.1:8080, which was failing (should be [::1]:8080?) |
@pmichali : without these changes, the tests : "DNS ... Should provide ..." will not work for IPv6, at least because the DNS query is "A" instead of "AAAA". with the changes of this PR, you will need to select to run either the IPv4 test, either IPV6 tests ... about "127.0.0.1:8080" I cannot say ... I had no need to change anything there, only about the DNS queries. |
@fturib Yes, I am running the test on an IPv6 cluster, and using the ginkgo focus to include this test. It is failing in the Before step, being unable to access the API, and is not even getting to the actual test. Did you happen to run this test in IPv6 mode for kube-dns? For coredns? Do you see this error (or success in accessing the API)? |
@pmichali : if am a little confused by the name of the test you are talking about: with the current PR, you can only have:
If you have exactly : "DNS [It] should provide DNS for services [Conformance]" - it means you are running WITHOUT this PR and in that case the test is working only for IPV4 |
@fturib Right, I'm running without the PR, and I see a failure in IPv6 mode. I know that it is not doing the right probe command, but it seemed (when I ran it) that it was failing before that point, and not able to communicate with the API. I was wondering whether or not the test works in IPv6 with kube-dns, when using this PR (I'm assuming it works with coredns). I'll try to apply this PR to my environment and run the test (once I get my IPv6 test bed working again). |
@pmichali : I am running in IPv6 environment, with kube-dns server, I get the following errors in 'kubedns' pod:
Other pods seems in better shape:
|
Result of test for kube-dns in IPv6 environment: command launched (run all DNS e2e test, except the one specific to IPv4):
Summarizing 6 Failures:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for doing this, just few comments
test/e2e/network/dns.go
Outdated
f := framework.NewDefaultFramework("dns") | ||
|
||
/* | ||
Release : v1.9 | ||
Testname: DNS, cluster | ||
Testname: DNS, cluster, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Why do we need this Testname?
f := framework.NewDefaultFramework("dns") | ||
|
||
/* | ||
Release : v1.9 | ||
Testname: DNS, cluster | ||
Testname: DNS, cluster, | ||
Description: When a Pod is created, the pod MUST be able to resolve cluster dns entries such as kubernetes.default via DNS and /etc/hosts. | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment of line 39-43 is for conformance test.
We need to move the comment to line 74 if IPv4's one only is a conformance test, I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fturib this still needs to be addressed
@@ -96,8 +104,7 @@ var _ = SIGDescribe("DNS", func() { | |||
Testname: DNS, services | |||
Description: When a headless service is created, the service MUST be able to resolve all the required service endpoints. When the service is created, any pod in the same namespace must be able to resolve the service by all of the expected DNS names. | |||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fturib this as well
@oomichi : are you sure about the comments of the tests ? I am seeing it is used to process all tests with [Conformance] flag and transform into a ItConformance(..) kind if code. |
I wanted to be smart and share the sync with 2 machines ... but looks like I am wrong and need to resync properly. |
test/e2e/network/dns_common.go
Outdated
arr[i], arr[j] = arr[j], arr[i] | ||
const hexDigit = "0123456789abcdef" | ||
|
||
func ReverseAddr(addr string) (arpa string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
doesn't seem like named return is being used
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right. removed.
test/e2e/network/dns_common.go
Outdated
return "" | ||
} | ||
if ip.To4() != nil { | ||
return strconv.Itoa(int(ip[15])) + "." + strconv.Itoa(int(ip[14])) + "." + strconv.Itoa(int(ip[13])) + "." + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if ip4 := ip.To4(); ip4 != nil {
return fmt.Sprintf("%d.%d.%d.%d.in-addr.arpa", ip4[3], ip4[2], ip4[1], ip4[0])
}
test/e2e/network/dns_common.go
Outdated
strconv.Itoa(int(ip[12])) + ".in-addr.arpa." | ||
} | ||
// Must be IPv6 | ||
buf := make([]byte, 0, len(ip)*4+len("ip6.arpa.")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just do var buf string
; don't over optimize
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code updated.
test/e2e/network/dns_common.go
Outdated
// Add it, in reverse, to the buffer | ||
for i := len(ip) - 1; i >= 0; i-- { | ||
v := ip[i] | ||
buf = append(buf, hexDigit[v&0xF]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
v := ip[i]
buf += fmt.Sprintf("%x%.x.", v&0xf, v>>4)
}
return buf + "ip4.arpa."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code updated.
} | ||
} | ||
|
||
var _ = SIGDescribe("DNS [IPv4]", func() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feature:Networking-IPv4?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here we cannot set [Feature:Networking-IPv4], because "Feature" is a keyword used in the e2e tests, and are most often just skipped.
We want to flag IPv4 - but keep the usual information, especially "Conformance".
/retest |
@fturib: Cannot trigger testing until a trusted user reviews the PR and leaves an In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@bowei : Looks like I lack a "ok-to-test" label now ... could you add it ? |
/ok-to-test |
@bowei : i updated the code following review comments. |
/test pull-kubernetes-e2e-gce |
@bowei : I rebased after changes (I guess linked to windowss integration in the dns.go tests). I think I replied all concerns and comments. |
1.14 code freeze is coming up in 3 days. @kubernetes/sig-network-pr-reviews can someone do a final review and merge? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
couple of nits but overall looks good. Please be sure to address @bowei 's remaining comments
test/e2e/network/dns.go
Outdated
}) | ||
} | ||
|
||
// only IPv4 is conformance for now |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: let's list this as a TODO:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you sure.
the TODO would be to run IPv6 Conformance tests .. it is a k8s-wide tests issue. Not only this one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed that comment.
test/e2e/network/dns.go
Outdated
Expect(f.WaitForPodRunning(testUtilsPod.Name)).NotTo(HaveOccurred(), "failed to wait for pod %s to be running", testUtilsPod.Name) | ||
|
||
By("Verifying customized DNS option is configured on pod...") | ||
// TODO: Figure out a better way other than checking the actual resolv,conf file. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// TODO: Figure out a better way other than checking the actual resolv,conf file. | |
// TODO: Figure out a better way other than checking the actual resolv.conf file. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok.
@cmluciano : I think I already addressed the comment of @bowei no ? Which one are not included ? |
@aanandr - Thanks to look if this PR can be useful for the work on dual stack Ipv4/Ipv6. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/hold
I appreciate how old this PR is and how much work has been put in to get it this far, but I have a number of objections to it landing as-is, mainly around the conformance related stuff.
Given where we are with our CI Signal (through no fault of your own) I feel like this should be removed from the milestone at this point.
I will leave this in for now, if you are willing to put in a super quick turnaround on the requested changes
f := framework.NewDefaultFramework("dns") | ||
|
||
/* | ||
Release : v1.9 | ||
Testname: DNS, cluster | ||
Testname: DNS, cluster, | ||
Description: When a Pod is created, the pod MUST be able to resolve cluster dns entries such as kubernetes.default via DNS and /etc/hosts. | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fturib this still needs to be addressed
@@ -96,8 +104,7 @@ var _ = SIGDescribe("DNS", func() { | |||
Testname: DNS, services | |||
Description: When a headless service is created, the service MUST be able to resolve all the required service endpoints. When the service is created, any pod in the same namespace must be able to resolve the service by all of the expected DNS names. | |||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fturib this as well
if isIPv6 { | ||
It("should provide DNS for the cluster", testDNSforCluster) | ||
} else { | ||
framework.ConformanceIt("should provide DNS for the cluster", testDNSforCluster) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comments @oomichi pointed out need to live here
if isIPv6 { | ||
It("should provide /etc/hosts entries for the cluster [LinuxOnly]", testHostEntries) | ||
} else { | ||
framework.ConformanceIt("should provide /etc/hosts entries for the cluster [LinuxOnly]", testHostEntries) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here as well
if isIPv6 { | ||
It("should provide DNS for services", testDNSService) | ||
} else { | ||
framework.ConformanceIt("should provide DNS for services", testDNSService) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And here
@@ -444,14 +446,20 @@ func createDNSPod(namespace, wheezyProbeCmd, jessieProbeCmd, podHostName, servic | |||
return dnsPod | |||
} | |||
|
|||
func createProbeCommand(namesToResolve []string, hostEntries []string, ptrLookupIP string, fileNamePrefix, namespace, dnsDomain string) (string, []string) { | |||
func createProbeCommand(isIpv6 bool, namesToResolve []string, hostEntries []string, ptrLookupIP string, fileNamePrefix, namespace, dnsDomain string) (string, []string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this also remains unresolved
probeCmd += fmt.Sprintf(`test -n "$$(getent %s %s)" && echo OK > /results/%s;`, getentdb, name, fileName) | ||
} | ||
|
||
podARecByUDPFileName := fmt.Sprintf("%s_udp@Pod%sRecord", fileNamePrefix, typeA) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Has this been resolved? Because I agree it should be possible for a cluster to be Conformant without using a specific DNS implementation (CoreDNS vs. kube-dns). Given that this function ends up being used in conformance tests I view this as a non-starter
/area conformance |
/remove-priority critical-urgent /milestone clear |
@spiffxp : Thanks for your review. I should have closed it last Thursday already. |
What this PR does / why we need it:
Support of IPV6 for e2e network/dns tests.
Use the flag "[Feature:Networking-IPv6]" to run in an IPv6 context only
Use the flag "[Feature:Networking-IPv4]" to run in an IPv4 context only
NOTE: the existing [Conformance] still apply to the same IPv4 tests only.
Which issue(s) this PR fixes *
None, but is related to : coredns/coredns#1236
Special notes for your reviewer
I ran these tests for Ipv6 and for Ipv4 in dedicated environment "Docker-in-Docker" that can propose IPv6 stack.
the IPv4 version is already running part of e2e tests of Kubernetes.
NOTE: I ran the IPv6 ONLY for the "coreDNS" version of DNS. Which is now BETA for v.1.10 and will be GA for v1.11
Release note: