Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make test consistently fails on plugin/pkg/auth/authenticator/token/oidc #20242

Closed
timothysc opened this issue Jan 27, 2016 · 20 comments
Closed

Comments

@timothysc
Copy link
Member

This issue has existed for some time now on my f23 env.

https://paste.fedoraproject.org/315507/14539328/
go1.5.3

/cc @yifan-gu @liggitt (also seen on latest origin rebase as well)

@yifan-gu
Copy link
Contributor

@timothysc Thanks for reporting.
/cc @ericchiang

@ericchiang
Copy link
Contributor

Maybe something to do with these test's use of os.TempDir here?

Wasn't able to replicate timeouts like in the pasted code, but need to tweak the tests before running stress.

$ godep go test -c -race 
$ stress ./oidc.test -test.run=TestOIDCAuthentication
2 runs so far, 0 failures
4 runs so far, 0 failures
8 runs so far, 0 failures

/tmp/go-stress886503499
I0127 23:33:18.673952   13625 oidc.go:98] Fetched provider config from https://127.0.0.1:35067: oidc.ProviderConfig{Issuer:"https://127.0.0.1:35067", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:35067/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0127 23:33:18.815125   13625 oidc.go:98] Fetched provider config from https://127.0.0.1:35067: oidc.ProviderConfig{Issuer:"https://127.0.0.1:35067", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:35067/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0127 23:33:18.987013   13625 oidc.go:98] Fetched provider config from https://127.0.0.1:35067: oidc.ProviderConfig{Issuer:"https://127.0.0.1:35067", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:35067/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
E0127 23:33:18.997738   13625 oidc.go:94] Failed to fetch provider config, trying again in 3s: Get https://127.0.0.1:35067/.well-known/openid-configuration: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "127.0.0.1@1453966398")
E0127 23:33:22.001041   13625 oidc.go:94] Failed to fetch provider config, trying again in 3s: Get https://127.0.0.1:35067/.well-known/openid-configuration: x509: certificate signed by unknown authority (possibly because of "crypto/r

/tmp/go-stress890593582
I0127 23:33:18.413839   13632 oidc.go:98] Fetched provider config from https://127.0.0.1:37527: oidc.ProviderConfig{Issuer:"https://127.0.0.1:37527", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:37527/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0127 23:33:18.560412   13632 oidc.go:98] Fetched provider config from https://127.0.0.1:37527: oidc.ProviderConfig{Issuer:"https://127.0.0.1:37527", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:37527/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0127 23:33:18.732062   13632 oidc.go:98] Fetched provider config from https://127.0.0.1:37527: oidc.ProviderConfig{Issuer:"https://127.0.0.1:37527", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:37527/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0127 23:33:18.850003   13632 oidc.go:98] Fetched provider config from https://127.0.0.1:37527: oidc.ProviderConfig{Issuer:"https://127.0.0.1:37527", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:37527/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0127 23:33:19.055062   13632 oidc.go:98] Fetched provider config from https
11 runs so far, 2 failures
13 runs so far, 2 failures
$ git diff oidc_test.go
diff --git a/plugin/pkg/auth/authenticator/token/oidc/oidc_test.go b/plugin/pkg/auth/authenticator/token/oidc/oidc_test.go
index 348ae19..528c68c 100644
--- a/plugin/pkg/auth/authenticator/token/oidc/oidc_test.go
+++ b/plugin/pkg/auth/authenticator/token/oidc/oidc_test.go
@@ -202,11 +202,14 @@ func TestOIDCDiscoveryNoKeyEndpoint(t *testing.T) {
        var err error
        expectErr := fmt.Errorf("OIDC provider must provide 'jwks_uri' for public key discovery")

-       cert := path.Join(os.TempDir(), "oidc-cert")
-       key := path.Join(os.TempDir(), "oidc-key")
+       tempdir, err := ioutil.TempDir("", "")
+       if err != nil {
+               t.Fatal(err)
+       }
+       defer os.RemoveAll(tempdir)

-       defer os.Remove(cert)
-       defer os.Remove(key)
+       cert := path.Join(tempdir, "oidc-cert")
+       key := path.Join(tempdir, "oidc-key")

        generateSelfSignedCert(t, "127.0.0.1", cert, key)
$ godep go test -c -race 
$ stress ./oidc.test -test.run=TestOIDCAuthentication
2 runs so far, 0 failures
8 runs so far, 0 failures
11 runs so far, 0 failures
16 runs so far, 0 failures
20 runs so far, 0 failures
25 runs so far, 0 failures
28 runs so far, 0 failures
33 runs so far, 0 failures
37 runs so far, 0 failures
41 runs so far, 0 failures
45 runs so far, 0 failures
50 runs so far, 0 failures
54 runs so far, 0 failures
59 runs so far, 0 failures
63 runs so far, 0 failures
67 runs so far, 0 failures
71 runs so far, 0 failures
75 runs so far, 0 failures
79 runs so far, 0 failures

@timothysc
Copy link
Member Author

repro on f23 w/go1.5.3 is 100%

@ericchiang
Copy link
Contributor

@timothysc by f23 you mean fedora 23? Odd, that's what I'm using.

I'll try to parse the stack trace and figure out what's going on.

@yifan-gu
Copy link
Contributor

@timothysc What's the command you use to run the test?

@timothysc
Copy link
Member Author

@yifan-gu make test

@ericchiang
Copy link
Contributor

These pass for me when running through the Makefile.

$ make test WHAT=plugin/pkg/auth/authenticator/token/oidc
hack/test-go.sh plugin/pkg/auth/authenticator/token/oidc 
Running tests for APIVersion: v1,extensions/v1beta1,metrics/v1alpha1 with etcdPrefix: registry
+++ [0128 10:03:39] Running tests without code coverage
ok      k8s.io/kubernetes/plugin/pkg/auth/authenticator/token/oidc  7.275s
Running tests for APIVersion: v1,extensions/v1beta1,metrics/v1alpha1 with etcdPrefix: kubernetes.io/registry
+++ [0128 10:03:52] Running tests without code coverage
ok      k8s.io/kubernetes/plugin/pkg/auth/authenticator/token/oidc  7.455s

Logs of a full make test here

@timothysc
Copy link
Member Author

@ericchiang - what version of golang are you running?

@ericchiang
Copy link
Contributor

@timothysc sorry should have specified that

$ go version
go version go1.5.3 linux/amd64

@dcbw
Copy link
Member

dcbw commented Feb 4, 2016

Hitting this on two different Fedora 23 machines consistently...

golang-1.5.3-1.fc23.x86_64

@ericchiang
Copy link
Contributor

@dcbw does the error look the same as the original comment? Can you paste the output of

$ cd $GOPATH/src/k8s.io
$ godep go test -v ./plugin/pkg/auth/authenticator/token/oidc

Are you installing Go via dnf or through the binary distributions from golang.org?

@dcbw
Copy link
Member

dcbw commented Feb 4, 2016

@ericchiang it's a similar error, though not exactly the same. But perhaps similar enough to be the same root cause?

$ make test WHAT=plugin/pkg/auth/authenticator/token/oidc GOFLAGS=-v
hack/test-go.sh plugin/pkg/auth/authenticator/token/oidc 
Running tests for APIVersion: v1,extensions/v1beta1,metrics/v1alpha1 with etcdPrefix: registry
+++ [0204 11:58:59] Running tests without code coverage
=== RUN   TestOIDCDiscoveryTimeout
I0204 11:59:02.662823    5418 oidc.go:72] No x509 certificates provided, will use host's root CA set
E0204 11:59:02.712165    5418 oidc.go:94] Failed to fetch provider config, trying again in 1s: Get https://foo/bar/.well-known/openid-configuration: dial tcp: lookup foo: No address associated with hostname
E0204 11:59:03.748918    5418 oidc.go:94] Failed to fetch provider config, trying again in 1s: Get https://foo/bar/.well-known/openid-configuration: dial tcp: lookup foo: No address associated with hostname
E0204 11:59:04.759155    5418 oidc.go:94] Failed to fetch provider config, trying again in 1s: Get https://foo/bar/.well-known/openid-configuration: dial tcp: lookup foo: No address associated with hostname
--- PASS: TestOIDCDiscoveryTimeout (3.10s)
=== RUN   TestOIDCDiscoveryNoKeyEndpoint
I0204 11:59:06.224165    5418 oidc.go:98] Fetched provider config from https://127.0.0.1:41106: oidc.ProviderConfig{Issuer:"https://127.0.0.1:41106", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
--- PASS: TestOIDCDiscoveryNoKeyEndpoint (0.46s)
=== RUN   TestOIDCDiscoverySecureConnection
E0204 11:59:06.560479    5418 oidc.go:94] Failed to fetch provider config, trying again in 1s: Get https://127.0.0.1:42249/.well-known/openid-configuration: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "127.0.0.1@1454608746")
E0204 11:59:07.561300    5418 oidc.go:94] Failed to fetch provider config, trying again in 1s: Get https://127.0.0.1:42249/.well-known/openid-configuration: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "127.0.0.1@1454608746")
E0204 11:59:08.562491    5418 oidc.go:94] Failed to fetch provider config, trying again in 1s: Get https://127.0.0.1:42249/.well-known/openid-configuration: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "127.0.0.1@1454608746")
--- PASS: TestOIDCDiscoverySecureConnection (3.34s)
=== RUN   TestOIDCAuthentication
I0204 11:59:10.281694    5418 oidc.go:98] Fetched provider config from https://127.0.0.1:39713: oidc.ProviderConfig{Issuer:"https://127.0.0.1:39713", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:39713/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0204 11:59:10.289408    5418 oidc.go:98] Fetched provider config from https://127.0.0.1:39713: oidc.ProviderConfig{Issuer:"https://127.0.0.1:39713", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:39713/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0204 11:59:10.297004    5418 oidc.go:98] Fetched provider config from https://127.0.0.1:39713: oidc.ProviderConfig{Issuer:"https://127.0.0.1:39713", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:39713/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0204 11:59:10.305314    5418 oidc.go:98] Fetched provider config from https://127.0.0.1:39713: oidc.ProviderConfig{Issuer:"https://127.0.0.1:39713", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:39713/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0204 11:59:10.314927    5418 oidc.go:98] Fetched provider config from https://127.0.0.1:39713: oidc.ProviderConfig{Issuer:"https://127.0.0.1:39713", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:39713/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
I0204 11:59:10.322470    5418 oidc.go:98] Fetched provider config from https://127.0.0.1:39713: oidc.ProviderConfig{Issuer:"https://127.0.0.1:39713", AuthEndpoint:"", TokenEndpoint:"", KeysEndpoint:"https://127.0.0.1:39713/keys", ResponseTypesSupported:[]string(nil), GrantTypesSupported:[]string(nil), SubjectTypesSupported:[]string(nil), IDTokenAlgValuesSupported:[]string(nil), TokenEndpointAuthMethodsSupported:[]string(nil), ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}

The last test eventually times out.

I'm installing Go via dnf; since bare 'go' and 'godep' commands don't work (they don't use the environment set up by make and the hack/* scripts) I'm using this instead.

$ make test WHAT=plugin/pkg/auth/authenticator/token/oidc GOFLAGS=-v

@ericchiang
Copy link
Contributor

@dcbw ah I'm installing via the binary distros. Maybe the dnf installs are slightly different? Will investigate.

@dcbw
Copy link
Member

dcbw commented Feb 4, 2016

@ericchiang full failure is at http://people.redhat.com/dcbw/oidc.log

@dcbw
Copy link
Member

dcbw commented Feb 4, 2016

@ericchiang the test hangs on client.Close() for the last testcase:

    {
        "sub",
        op.generateExpiredToken(t, srv.URL, "client-foo", "client-foo", "sub", "user-foo"),
        nil,
        false,
        "oidc: JWT claims invalid: token is expired",
    },

@dcbw
Copy link
Member

dcbw commented Feb 4, 2016

@ericchiang I lied; it's actually hanging on the httptest.Server Close() from the defer srv.Close(). Trying to track that down...

@ericchiang
Copy link
Contributor

Either way, maybe a lack of dial timeouts in the transport might be causing the hang?

@dcbw
Copy link
Member

dcbw commented Feb 5, 2016

@ericchiang looks like it's actually golang/go#12262 and there are a couple other places in oidc_test.go that comment out srv.Close() due to this. Doing that fixes it for me too...

@dcbw
Copy link
Member

dcbw commented Feb 5, 2016

@ericchiang the original bug is probably different, so I'll drop off this report and handle this in #20752 instead.

@timothysc
Copy link
Member Author

I have not seen this in a long time, closing.

openshift-publish-robot pushed a commit to openshift/kubernetes that referenced this issue Jul 12, 2018
upstream machinery picks

Origin-commit: ea877acae817559712c8d42ef2c5caee880d319d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants