Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spawner options error(401) #784

Closed
xieydd opened this issue May 10, 2018 · 11 comments
Closed

Spawner options error(401) #784

xieydd opened this issue May 10, 2018 · 11 comments

Comments

@xieydd
Copy link
Member

xieydd commented May 10, 2018 •
edited
Loading

When i use Jupyter ,i choose some option and click Spawn ,error occurred.

Error: (401) Reason: error HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 10 May 2018 07:54:23 GMT', 'Content-Length': '13'}) HTTP response body: Unauthorized

I have the notebook image

[root@00-e0-81-ee-82-5b ~]# docker images
REPOSITORY                                                    TAG                                  IMAGE ID            CREATED             SIZE
gcr.io/kubeflow-images-public/tensorflow-1.7.0-notebook-cpu   v20180419-0ad94c4e                   128a584ad3c1        2 months ago        907MB

and i check the container log

[E2018-05-10 07:50:16.549 JupyterHub spawner:923] Failed for {'api_version': 'v1',
     'kind': 'Pod',
     'metadata': {'annotations': None,
                  'cluster_name': None,
                  'creation_timestamp': None,
                  'deletion_grace_period_seconds': None,
                  'deletion_timestamp': None,
                  'finalizers': None,
                  'generate_name': None,
                  'generation': None,
                  'initializers': None,
                  'labels': {'app': 'jupyterhub',
                             'component': 'singleuser-server',
                             'heritage': 'jupyterhub',
                             'hub.jupyter.org/username': 'aaa'},
                  'name': 'jupyter-aaa',
                  'namespace': None,
                  'owner_references': None,
                  'resource_version': None,
                  'self_link': None,
                  'uid': None},
     'spec': {'active_deadline_seconds': None,
              'affinity': None,
              'automount_service_account_token': None,
              'containers': [{'args': ['start-singleuser.sh',
                                       '--ip="0.0.0.0"',
                                       '--port=8888',
                                       '--allow-root'],
                              'command': None,
                              'env': [{'name': 'JUPYTERHUB_API_TOKEN',
                                       'value': 'f735996ef76049a688ea882921859ad9',
                                       'value_from': None},
                                      {'name': 'JPY_API_TOKEN',
                                       'value': 'f735996ef76049a688ea882921859ad9',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_CLIENT_ID',
                                       'value': 'user-aaa',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_HOST',
                                       'value': '',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL',
                                       'value': '/user/aaa/oauth_callback',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_USER',
                                       'value': 'aaa',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_API_URL',
                                       'value': 'http://tf-hub-0:8081/hub/api',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_BASE_URL',
                                       'value': '/',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_SERVICE_PREFIX',
                                       'value': '/user/aaa/',
                                       'value_from': None},
                                      {'name': 'MEM_GUARANTEE',
                                       'value': '1Gi',
                                       'value_from': None},
                                      {'name': 'CPU_GUARANTEE',
                                       'value': '500m',
                                       'value_from': None}],
                              'env_from': None,
                              'image': 'gcr.io/kubeflow/tensorflow-notebook-cpu',
                              'image_pull_policy': 'IfNotPresent',
                              'lifecycle': {},
                              'liveness_probe': None,
                              'name': 'notebook',
                              'ports': [{'container_port': 8888,
                                         'host_ip': None,
                                         'host_port': None,
                                         'name': 'notebook-port',
                                         'protocol': None}],
                              'readiness_probe': None,
                              'resources': {'limits': {},
                                            'requests': {'cpu': '500m',
                                                         'memory': '1Gi'}},
                              'security_context': None,
                              'stdin': None,
                              'stdin_once': None,
                              'termination_message_path': None,
                              'termination_message_policy': None,
                              'tty': None,
                              'volume_mounts': [{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
                                                 'name': 'no-api-access-please',
                                                 'read_only': True,
                                                 'sub_path': None}],
                              'working_dir': None}],
              'dns_policy': None,
              'host_aliases': None,
              'host_ipc': None,
              'host_network': None,
              'host_pid': None,
              'hostname': None,
              'image_pull_secrets': None,
              'init_containers': [],
              'node_name': None,
              'node_selector': None,
              'restart_policy': None,
              'scheduler_name': None,
              'security_context': {'fs_group': 0,
                                   'run_as_non_root': None,
                                   'run_as_user': 0,
                                   'se_linux_options': None,
                                   'supplemental_groups': None},
              'service_account': None,
              'service_account_name': None,
              'subdomain': None,
              'termination_grace_period_seconds': None,
              'tolerations': None,
              'volumes': [{'aws_elastic_block_store': None,
                           'azure_disk': None,
                           'azure_file': None,
                           'cephfs': None,
                           'cinder': None,
                           'config_map': None,
                           'downward_api': None,
                           'empty_dir': {},
                           'fc': None,
                           'flex_volume': None,
                           'flocker': None,
                           'gce_persistent_disk': None,
                           'git_repo': None,
                           'glusterfs': None,
                           'host_path': None,
                           'iscsi': None,
                           'name': 'no-api-access-please',
                           'nfs': None,
                           'persistent_volume_claim': None,
                           'photon_persistent_disk': None,
                           'portworx_volume': None,
                           'projected': None,
                           'quobyte': None,
                           'rbd': None,
                           'scale_io': None,
                           'secret': None,
                           'storageos': None,
                           'vsphere_volume': None}]},
     'status': None}
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/site-packages/kubespawner/spawner.py", line 917, in start
        pod
      File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
        return self.__get_result()
      File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
        raise self._exception
      File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.6/site-packages/kubespawner/spawner.py", line 888, in asynchronize
        return method(*args, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 6561, in create_namespaced_pod
        (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 6651, in create_namespaced_pod_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 335, in call_api
        _preload_content, _request_timeout)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 393, in request
        body=body)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 287, in POST
        body=body)
      File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (401)
    Reason: Unauthorized
    HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 10 May 2018 07:50:16 GMT', 'Content-Length': '13'})
    HTTP response body: Unauthorized
[E 2018-05-10 07:50:16.556 JupyterHub user:427] Unhandled error starting aaa's server: (401)
    Reason: Unauthorized
    HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 10 May 2018 07:50:16 GMT', 'Content-Length': '13'})
    HTTP response body: Unauthorized

@pineking
@xieydd xieydd closed this as completed May 10, 2018
@xieydd xieydd reopened this May 10, 2018
@jlewi
Copy link
Contributor

jlewi commented May 10, 2018

It looks like the JupyterHub spawner doesn't have sufficient permissions to create a POD for the notebook.

What service account is the JupyterHub spawner pod using? Was the appropriate role binding correctly created for this service account?

@xieydd
Copy link
Member Author

xieydd commented May 11, 2018 •
edited
Loading

I closed RBAC authorized
There are some useful information

$ kubectl get pod -n kubeflow-xyd -o wide
NAME                                READY     STATUS    RESTARTS   AGE       USED-GPU   IP           NODE
ambassador-6bd9cc864-lsqg9          2/2       Running   0          41m       0/0        10.1.81.10   00-e0-81-ee-82-5b
ambassador-6bd9cc864-x9h8q          2/2       Running   0          41m       0/0        10.1.51.5    00-25-90-c0-f7-c8
ambassador-6bd9cc864-zgprf          2/2       Running   0          41m       0/0        10.1.9.3     00-25-90-c0-f7-88
tf-hub-0                            1/1       Running   1          41m       0/0        10.1.81.5    00-e0-81-ee-82-5b
tf-job-dashboard-8666f858cb-r58g2   1/1       Running   0          41m       0/0        10.1.81.12   00-e0-81-ee-82-5b
tf-job-operator-59c5d76886-vbmdr    1/1       Running   0          41m       0/0        10.1.81.6    00-e0-81-ee-82-5b

$ kubectl get svc -n kubeflow-xyd -o wide
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE       SELECTOR
ambassador         ClusterIP      10.100.0.3     <none>        80/TCP         42m       service=ambassador
ambassador-admin   ClusterIP      10.100.0.244   <none>        8877/TCP       42m       service=ambassador
k8s-dashboard      ClusterIP      10.100.0.213   <none>        443/TCP        42m       k8s-app=kubernetes-dashboard
tf-hub-0           ClusterIP      None           <none>        8000/TCP       42m       app=tf-hub
tf-hub-lb          LoadBalancer   10.100.0.85    <pending>     80:32719/TCP   42m       app=tf-hub
tf-job-dashboard   ClusterIP      10.100.0.51    <none>        80/TCP         42m       name=tf-job-dashboard

$ kubectl get role -n kubeflow-xyd -o wide
NAME           AGE
ambassador     42m
jupyter-role   42m

$ kubectl get rolebinding -n kubeflow-xyd -o wide
NAME           AGE       ROLE                USERS     GROUPS    SERVICEACCOUNTS
ambassador     43m       Role/ambassador                         kubeflow-xyd/ambassador
jupyter-role   43m       Role/jupyter-role                       kubeflow-xyd/jupyter-hub

$ kubectl get secret -n kubeflow-xyd -o wide
NAME                           TYPE                                  DATA      AGE
ambassador-token-8hlff         kubernetes.io/service-account-token   3         43m
default-token-978lz            kubernetes.io/service-account-token   3         2d
jupyter-hub-token-5rmjb        kubernetes.io/service-account-token   3         43m
tf-job-dashboard-token-gwzfx   kubernetes.io/service-account-token   3         43m
tf-job-operator-token-8hj8s    kubernetes.io/service-account-token   3         43m

$ kubectl get serviceaccount -n kubeflow-xyd
NAME               SECRETS   AGE
ambassador         1         52m
default            1         2d
jupyter-hub        1         52m
tf-job-dashboard   1         52m
tf-job-operator    1         52m

$ kubectl describe rolebinding jupyter-role -n kubeflow-xyd
Name:         jupyter-role
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  Role
  Name:  jupyter-role
Subjects:
  Kind            Name         Namespace
  ----            ----         ---------
  ServiceAccount  jupyter-hub  kubeflow-xyd

$ kubectl describe role jupyter-role -n kubeflow-xyd 
Name:         jupyter-role
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  *.*        []                 []              [*]

$ kubectl describe serviceaccount jupyter-hub -n kubeflow-xyd
Name:                jupyter-hub
Namespace:           kubeflow-xyd
Labels:              app=jupyter-hub
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   jupyter-hub-token-5rmjb
Tokens:              jupyter-hub-token-5rmjb
Events:              <none>

@jlewi
Copy link
Contributor

jlewi commented May 11, 2018

This looks like jupyterhub/kubespawner#155

It looks like killing the pod might help. Can you try that?

@xieydd
Copy link
Member Author

xieydd commented May 14, 2018

Thanks,it works;But I meet this problem #78 ;I don`t find the specific method in this issue?

@jlewi
Copy link
Contributor

jlewi commented May 14, 2018

@xieydd You're last message isn't clear; what's working and what is the new problem?

Per #78 are you hitting problems with kube-dns not being configured correctly?

@xieydd
Copy link
Member Author

xieydd commented May 15, 2018 •
edited
Loading

@jlewi Thanks , when i rebuild kubeflow-core ，i didn`t got the 401 error,a container named jupyter-username was build but i got 500 error just like #78 description, below is my logs

[E 2018-05-14 01:54:27.342 SingleUserNotebookApp log:114] {
      "X-Forwarded-Host": "10.10.15.98:8000",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "192.168.6.120",
      "Cookie": "user-aaa-oauth-state=\"2|1:0|10:1526262867|20:user-aaa-oauth-state|156:ZXlKMWRXbGtJam9nSW1VeU1XUmpPRGxtWlRoa1pUUTRPV1ppTVRZMU4yWmhZemt3T0RSbU5ERmhJaXdnSW01bGVIUmZkWEpzSWpvZ0lpOTFjMlZ5TDJGaFlTOTBjbVZsUDNKbFpHbHlaV04wY3oweEluMA==|0f9ee60b98aae1fc5755aed4ca7fed6d26b43eedf7a9fdf5e1e7e0182197162b\"; _xsrf=2|091224d4|c3fcc62d95145d081574808a26f57f47|1526262385",
      "Accept-Language": "zh-CN,zh;q=0.9",
      "Accept-Encoding": "gzip, deflate",
      "Referer": "http://10.10.15.98:8000/hub/spawn?next=%2Fhub%2Fuser%2Faaa%2F",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Cache-Control": "max-age=0",
      "Connection": "close",
      "Host": "10.10.15.98:8000"
    }
[E 2018-05-14 01:54:27.342 SingleUserNotebookApp log:122] 500 GET /user/aaa/oauth_callback?code=df667e39-4855-4db0-9786-20d453a6307c&state=eyJ1dWlkIjogImUyMWRjODlmZThkZTQ4OWZiMTY1N2ZhYzkwODRmNDFhIiwgIm5leHRfdXJsIjogIi91c2VyL2FhYS90cmVlP3JlZGlyZWN0cz0xIn0 (@192.168.6.120) 42.54ms

[E 2018-05-14 01:54:55.781 SingleUserNotebookApp singleuser:354] Failed to connect to my Hub at http://tf-hub-0:8081/hub/api (attempt 5/5). Is it running?
    Traceback (most recent call last):
      File "/home/jovyan/.conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 351, in check_hub_version
        resp = yield client.fetch(self.hub_api_url)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/stack_context.py", line 315, in wrapped
        ret = fn(*args, **kwargs)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 293, in <lambda>
        future, lambda future: callback(future.result()))
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
        yielded = self.gen.throw(*exc_info)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/tcpclient.py", line 226, in connect
        addrinfo = yield self.resolver.resolve(host, port, af)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
        yielded = self.gen.throw(*exc_info)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/netutil.py", line 364, in resolve
        None, _resolve_addr, host, port, family)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/home/jovyan/.conda/lib/python3.6/concurrent/futures/thread.py", line 56, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/netutil.py", line 349, in _resolve_addr
        addrinfo = socket.getaddrinfo(host, port, family, socket.SOCK_STREAM)
      File "/home/jovyan/.conda/lib/python3.6/socket.py", line 745, in getaddrinfo
        for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -3] Temporary failure in name resolution

I might be my kube-dns error ,i will check ,and return back

@xieydd xieydd mentioned this issue May 15, 2018
Closed
@jlewi
Copy link
Contributor

jlewi commented May 17, 2018

Any luck?

[E 2018-05-14 01:54:55.781 SingleUserNotebookApp singleuser:354] Failed to connect to my Hub at http://tf-hub-0:8081/hub/api (attempt 5/5). Is it running?
    Traceback (most recent call last):

This looks like a cluster networking issue or something is going wrong with JupyterHub.

My suggestion would be to try running a container (e.g. busybox) in the cluster and seeing if you can access tf-hub-0:8081

@xieydd
Copy link
Member Author

xieydd commented May 17, 2018 •
edited
Loading

@jlewi I try it , and i am sure my kube-dns is not working,but the status is running;

$ po kube-system -o wide
NAME                                   READY     STATUS              RESTARTS   AGE       USED-GPU   IP          NODE
kube-dns-v20-r67mz                     3/3       Running             0          57d       0/0        10.1.51.3   00-25-90-c0-f7-c8

[root@00-e0-81-ee-82-5b ~]# docker exec -it f7de7f869635 ping default.svc.cluster.local
ping: bad address 'default.svc.cluster.local'
[root@00-e0-81-ee-82-5b ~]# docker exec -it f7de7f869635 nslookup kubernetes
Server:    10.100.0.10
Address 1: 10.100.0.10

nslookup: can't resolve 'kubernetes'

[root@00-e0-81-ee-82-5b ~]# docker exec -it f7de7f869635 nslookup default.svc.cluster.local
Server:    10.100.0.10
Address 1: 10.100.0.10

nslookup: can't resolve 'default.svc.cluster.local'

[root@00-e0-81-ee-82-5b ~]# docker exec -it f7de7f869635 nslookup tf-hub-0:8081
Server:    10.100.0.10
Address 1: 10.100.0.10

nslookup: can't resolve 'tf-hub-0:8081'

one question , why 3 ambassador ? Each of tf-hub tf-job-dashboard and tf-job-operator has one ?
is there some thing i can get the frame of kubeflow?

@jlewi
Copy link
Contributor

jlewi commented May 21, 2018

is there some thing i can get the frame of kubeflow?

I didn't understand what you are asking.

We run 3 ambassadors because that just happens to be the default and we haven't changed it.

If the problem is kube-dns your best bet would probably to try the Kubernetes mailing list for help. You can try looking at the logs of kube-dns.

@xieydd
Copy link
Member Author

xieydd commented May 23, 2018

@jlewi Sorry , I konw about it.
And I solve 401 error, my kube-dns is not working，I replace it with coredns;
But I meet a new error 500 , tf-hub-0 service has Endpoints

Failed to connect to Hub API at 'http://tf-hub-0:8081/hub/api'.  Is the Hub accessible at this URL (from host: jupyter-aaa)?

usermod: no changes
Execute the command as jovyan
[W 2018-05-23 06:36:42.420 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2018-05-23 06:36:42.699 SingleUserNotebookApp extension:53] JupyterLab beta preview extension loaded from /home/jovyan/.conda/lib/python3.6/site-packages/jupyterlab
[I 2018-05-23 06:36:42.699 SingleUserNotebookApp extension:54] JupyterLab application directory is /home/jovyan/.conda/share/jupyter/lab
[I 2018-05-23 06:36:42.709 SingleUserNotebookApp singleuser:365] Starting jupyterhub-singleuser server version 0.8.1
[E 2018-05-23 06:36:42.716 SingleUserNotebookApp singleuser:354] Failed to connect to my Hub at http://tf-hub-0:8081/hub/api (attempt 1/5). Is it running?
    Traceback (most recent call last):
      File "/home/jovyan/.conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 351, in check_hub_version
        resp = yield client.fetch(self.hub_api_url)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/stack_context.py", line 315, in wrapped
        ret = fn(*args, **kwargs)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 293, in <lambda>
        future, lambda future: callback(future.result()))
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
        yielded = self.gen.throw(*exc_info)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/tcpclient.py", line 226, in connect
        addrinfo = yield self.resolver.resolve(host, port, af)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
        yielded = self.gen.throw(*exc_info)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/netutil.py", line 364, in resolve
        None, _resolve_addr, host, port, family)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "/home/jovyan/.conda/lib/python3.6/concurrent/futures/thread.py", line 56, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/home/jovyan/.conda/lib/python3.6/site-packages/tornado/netutil.py", line 349, in _resolve_addr
        addrinfo = socket.getaddrinfo(host, port, family, socket.SOCK_STREAM)
      File "/home/jovyan/.conda/lib/python3.6/socket.py", line 745, in getaddrinfo
        for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -5] No address associated with hostname
[I 2018-05-23 06:36:43.056 SingleUserNotebookApp log:122] 302 GET /user/aaa/ → /user/aaa/tree? (@10.99.33.69) 0.95ms
[I 2018-05-23 06:36:43.179 SingleUserNotebookApp log:122] 302 GET /user/aaa/?redirects=1 → /user/aaa/tree?redirects=1 (@192.168.6.120) 0.64ms
[I 2018-05-23 06:36:43.285 SingleUserNotebookApp log:122] 302 GET /user/aaa/tree?redirects=1 → /hub/api/oauth2/authorize?client_id=user-aaa&redirect_uri=%2Fuser%2Faaa%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjFkNjlmODM3ZTAyZTQxZGY5MzZlYzQzYTNkNTg3ZTg3IiwgIm5leHRfdXJsIjogIi91c2VyL2FhYS90cmVlP3JlZGlyZWN0cz0xIn0 (@192.168.6.120) 4.19ms
[E 2018-05-23 06:36:43.528 SingleUserNotebookApp auth:249] Error connecting to http://tf-hub-0:8081/hub/api: HTTPConnectionPool(host='tf-hub-0', port=8081): Max retries exceeded with url: /hub/api/oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f078ae057b8>: Failed to establish a new connection: [Errno -2] Name or service not known',))
[W 2018-05-23 06:36:43.528 SingleUserNotebookApp web:1618] 500 GET /user/aaa/oauth_callback?code=6b6a5b50-abc8-40f5-8d99-302f08e8df60&state=eyJ1dWlkIjogIjFkNjlmODM3ZTAyZTQxZGY5MzZlYzQzYTNkNTg3ZTg3IiwgIm5leHRfdXJsIjogIi91c2VyL2FhYS90cmVlP3JlZGlyZWN0cz0xIn0 (192.168.6.120): Failed to connect to Hub API at 'http://tf-hub-0:8081/hub/api'.  Is the Hub accessible at this URL (from host: jupyter-aaa)?
[E 2018-05-23 06:36:43.566 SingleUserNotebookApp log:114] {
      "X-Forwarded-Host": "10.10.15.50:8000",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "192.168.6.120",
      "Cookie": "user-aaa-oauth-state=\"2|1:0|10:1527057403|20:user-aaa-oauth-state|156:ZXlKMWRXbGtJam9nSWpGa05qbG1PRE0zWlRBeVpUUXhaR1k1TXpabFl6UXpZVE5rTlRnM1pUZzNJaXdnSW01bGVIUmZkWEpzSWpvZ0lpOTFjMlZ5TDJGaFlTOTBjbVZsUDNKbFpHbHlaV04wY3oweEluMA==|06f3ea9912fa218e838ea9e4dd925313a01bdd9584685a5d1a8d1f6ec31b798b\"; _xsrf=2|1ef5bada|8369134fb8227bc8f53bcbd5eef49db5|1526270027",
      "Accept-Language": "zh-CN,zh;q=0.9",
      "Accept-Encoding": "gzip, deflate",
      "Referer": "http://10.10.15.50:8000/hub/spawn?next=%2Fhub%2Fuser%2Faaa%2F",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Cache-Control": "max-age=0",
      "Connection": "close",
      "Host": "10.10.15.50:8000"
    }
[E 2018-05-23 06:36:43.566 SingleUserNotebookApp log:122] 500 GET /user/aaa/oauth_callback?code=6b6a5b50-abc8-40f5-8d99-302f08e8df60&state=eyJ1dWlkIjogIjFkNjlmODM3ZTAyZTQxZGY5MzZlYzQzYTNkNTg3ZTg3IiwgIm5leHRfdXJsIjogIi91c2VyL2FhYS90cmVlP3JlZGlyZWN0cz0xIn0 (@192.168.6.120) 46.51ms


$ kubectl describe svc tf-hub-0 -n kubeflow
Name:              tf-hub-0
Namespace:         kubeflow
Labels:            app=tf-hub
Annotations:       <none>
Selector:          app=tf-hub
Type:              ClusterIP
IP:                None
Port:              hub  8000/TCP
TargetPort:        8000/TCP
Endpoints:         10.99.33.69:8000
Session Affinity:  None
Events:            <none>

@xieydd
Copy link
Member Author

xieydd commented May 24, 2018

I sloved thr problem , i Setting error parameters for coredns.

@xieydd xieydd closed this as completed May 24, 2018
yanniszark pushed a commit to arrikto/kubeflow that referenced this issue Feb 15, 2021
@johnugeorge @k8s-ci-robot
Adding algorithm deployment status to Suggestion status (kubeflow#784)
f26f7b3 
* Adding deployment status to Suggestion status

* Adding running status immediately when set
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jlewi @xieydd