Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster teardown and bring up for GCE e2e not working #5451

Closed
satnam6502 opened this issue Mar 13, 2015 · 19 comments
Closed

Cluster teardown and bring up for GCE e2e not working #5451

satnam6502 opened this issue Mar 13, 2015 · 19 comments
Assignees
@zmerlynn
Labels
priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.

Comments

@satnam6502
Copy link
Contributor 

$ go run hack/e2e.go -v -down ; go run hack/e2e.go -v -up ; hack/ginkgo-e2e.sh -t Cluster
2015/03/13 12:04:50 Running: teardown
Project: kubernetes-satnam
Zone: us-central1-b
Shutting down test cluster in background.
ERROR: (gcloud.compute.firewall-rules.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-minion-e2e-test-satnam-http-alt' was not found

Bringing down cluster using provider: gce
Project: kubernetes-satnam
Zone: us-central1-b
ERROR: (gcloud.compute.firewall-rules.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-fluentd-elasticsearch-logging' was not found

current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl delete services elasticsearch-logging
F0313 12:04:53.869278   28975 delete.go:67] Client error processing command: Get http://localhost:8080/api/v1beta1/services/elasticsearch-logging?namespace=default: dial tcp 127.0.0.1:8080: connection refused
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl delete services kibana-logging
F0313 12:04:54.035822   29018 delete.go:67] Client error processing command: Get http://localhost:8080/api/v1beta1/services/kibana-logging?namespace=default: dial tcp 127.0.0.1:8080: connection refused
Bringing down cluster
Managed instance group e2e-test-satnam-minion-group is being deleted. Operation: operation-1426273494669-74fdb756-843d-403c-85b3-00b33b8faa48
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/instanceTemplates/e2e-test-satnam-minion-template].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
ERROR: (gcloud.compute.disks.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/zones/us-central1-b/disks/e2e-test-satnam-master-pd' was not found

Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-master-https].
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-minion-all].
Deleting routes e2e-test-satnam-minion-jo0q e2e-test-satnam-minion-qhgy
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-jo0q].
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-qhgy].
ERROR: (gcloud.compute.addresses.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/regions/us-central1/addresses/e2e-test-satnam-master-ip' was not found

current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl config unset clusters.kubernetes-satnam_e2e-test-satnam
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl config unset users.kubernetes-satnam_e2e-test-satnam
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl config unset contexts.kubernetes-satnam_e2e-test-satnam
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl config view -o template --template={{ index . "current-context" }}
Cleared config for kubernetes-satnam_e2e-test-satnam from /usr/local/google/home/satnam/.kube/.kubeconfig
Done
2015/03/13 12:06:16 Running: get status
Project: kubernetes-satnam
Zone: us-central1-b
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl --match-server-version version
F0313 12:06:17.432473   29475 version.go:34] Client error processing command: couldn't read version from server: Get http://localhost:8080/version: dial tcp 127.0.0.1:8080: connection refused
2015/03/13 12:06:17 Error running get status: exit status 255
2015/03/13 12:06:17 Running: up
Project: kubernetes-satnam
Zone: us-central1-b
Starting cluster using provider: gce
... calling verify-prereqs
... calling kube-up
Project: kubernetes-satnam
Zone: us-central1-b
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl config view -o template --template={{$dot := .}}{{with $ctx := index $dot "current-context"}}{{$user := index $dot "contexts" $ctx "user"}}{{index $dot "users" $user "username"}}{{end}}
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl config view -o template --template={{$dot := .}}{{with $ctx := index $dot "current-context"}}{{$user := index $dot "contexts" $ctx "user"}}{{index $dot "users" $user "password"}}{{end}}
+++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e/devel
Starting master and configuring firewalls
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#pdperformance.
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-master-https].
NAME                         NETWORK SRC_RANGES RULES   SRC_TAGS TARGET_TAGS
e2e-test-satnam-master-https e2e     0.0.0.0/0  tcp:443          e2e-test-satnam-master
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/e2e-test-satnam-master-pd].
NAME                      ZONE          SIZE_GB TYPE        STATUS
e2e-test-satnam-master-pd us-central1-b 10      pd-standard READY
+++ Logging using Fluentd to elasticsearch
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-minion-all].
NAME                       NETWORK SRC_RANGES    RULES                    SRC_TAGS TARGET_TAGS
e2e-test-satnam-minion-all e2e     10.245.0.0/16 tcp,udp,icmp,esp,ah,sctp          e2e-test-satnam-minion
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
NAME                   ZONE          MACHINE_TYPE INTERNAL_IP    EXTERNAL_IP     STATUS
e2e-test-satnam-master us-central1-b g1-small     10.240.245.162 162.222.177.110 RUNNING
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
Creating minions.
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#pdperformance.
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/instanceTemplates/e2e-test-satnam-minion-template].
NAME                            MACHINE_TYPE CREATION_TIMESTAMP
e2e-test-satnam-minion-template g1-small     2015-03-13T12:07:21.482-07:00
Managed instance group e2e-test-satnam-minion-group is being created. Operation: operation-1426273658633-1bd5031d-7a9b-457f-ad91-c5efe7ee1722
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 1 out of 2 running. Retrying.
MINION_NAMES=e2e-test-satnam-minion-dd48 e2e-test-satnam-minion-wn3q
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
ERROR: (gcloud.compute.routes.create) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/routes/e2e-test-satnam-master' already exists

Attempt 1 failed to create route e2e-test-satnam-master. Retrying.
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-minion-wn3q].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-minion-dd48].
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
ERROR: (gcloud.compute.routes.create) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/routes/e2e-test-satnam-master' already exists

Attempt 2 failed to create route e2e-test-satnam-master. Retrying.
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
ERROR: (gcloud.compute.routes.create) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/routes/e2e-test-satnam-master' already exists

Attempt 3 failed to create route e2e-test-satnam-master. Retrying.
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
ERROR: (gcloud.compute.routes.create) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/routes/e2e-test-satnam-master' already exists

Attempt 4 failed to create route e2e-test-satnam-master. Retrying.
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
ERROR: (gcloud.compute.routes.create) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/routes/e2e-test-satnam-master' already exists

Attempt 5 failed to create route e2e-test-satnam-master. Retrying.
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-wn3q].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-dd48].
NAME                        NETWORK DEST_RANGE    NEXT_HOP                                            PRIORITY
e2e-test-satnam-minion-wn3q e2e     10.245.2.0/24 us-central1-b/instances/e2e-test-satnam-minion-wn3q 1000
NAME                        NETWORK DEST_RANGE    NEXT_HOP                                            PRIORITY
e2e-test-satnam-minion-dd48 e2e     10.245.1.0/24 us-central1-b/instances/e2e-test-satnam-minion-dd48 1000
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
ERROR: (gcloud.compute.routes.create) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/routes/e2e-test-satnam-master' already exists

Attempt 6 failed to create route e2e-test-satnam-master. Retrying.
NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY
ERROR: (gcloud.compute.routes.create) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/routes/e2e-test-satnam-master' already exists

Failed to create route e2e-test-satnam-master
@satnam6502 satnam6502 added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Mar 13, 2015
@satnam6502
Copy link
Contributor Author

@zmerlynn @jlowdermilk
And I assume this is because of the context. However, if should fail error with a better error message?

@zmerlynn
Copy link
Member

This is really cc @brendandburns, but are you synced to past #5448?

@bgrant0607 bgrant0607 added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label Mar 13, 2015
@satnam6502
Copy link
Contributor Author

Building from HEAD and trying again I get stuck at the dots -- I've done this N times.

Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/instanceTemplates/e2e-test-satnam-minion-template].
NAME                            MACHINE_TYPE CREATION_TIMESTAMP
e2e-test-satnam-minion-template g1-small     2015-03-13T15:53:36.015-07:00
Managed instance group e2e-test-satnam-minion-group is being created. Operation: operation-1426287228110-5cf2665c-674a-45a5-9b53-8a9fb6833509
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 1 out of 2 running. Retrying.
Waiting for minions to run. 1 out of 2 running. Retrying.
MINION_NAMES=e2e-test-satnam-minion-sdjx e2e-test-satnam-minion-y4fw
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-minion-sdjx].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-minion-y4fw].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-master].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-sdjx].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-y4fw].
NAME                   NETWORK DEST_RANGE    NEXT_HOP                                       PRIORITY
e2e-test-satnam-master e2e     10.246.0.0/24 us-central1-b/instances/e2e-test-satnam-master 1000
NAME                        NETWORK DEST_RANGE    NEXT_HOP                                            PRIORITY
e2e-test-satnam-minion-sdjx e2e     10.245.1.0/24 us-central1-b/instances/e2e-test-satnam-minion-sdjx 1000
NAME                        NETWORK DEST_RANGE    NEXT_HOP                                            PRIORITY
e2e-test-satnam-minion-y4fw e2e     10.245.2.0/24 us-central1-b/instances/e2e-test-satnam-minion-y4fw 1000
Using master: e2e-test-satnam-master (external IP: 162.222.177.110)
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/e2e-test-satnam-master-ip].
NAME                      REGION      ADDRESS         STATUS
e2e-test-satnam-master-ip us-central1 162.222.177.110 IN_USE
Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

............................................................................................

@satnam6502
Copy link
Contributor Author

@roberthbailey @a-robinson apologies -- this is blocking my progress on addressing your kind review comments for the e2e cluster level logging test #5269

@j3ffml
Copy link
Contributor

j3ffml commented Mar 13, 2015

If you have the full kube-up output, can you dump it in a gist?

@satnam6502
Copy link
Contributor Author

I do get:

Use "kubectl help [command]" for more information about a command.
Error: unknown command "--match-server-version"
Run 'help' for usage.

@satnam6502
Copy link
Contributor Author

But here you go:

$ go run hack/e2e.go -v -down ; go run hack/e2e.go -v -up ; hack/ginkgo-e2e.sh -t Cluster
2015/03/13 16:09:39 Running: teardown
Project: kubernetes-satnam
Zone: us-central1-b
Shutting down test cluster in background.
ERROR: (gcloud.compute.firewall-rules.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-minion-e2e-test-satnam-http-alt' was not found

Bringing down cluster using provider: gce
Project: kubernetes-satnam
Zone: us-central1-b
ERROR: (gcloud.compute.firewall-rules.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-fluentd-elasticsearch-logging' was not found

current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl delete services elasticsearch-logging
F0313 16:09:43.321812   22722 delete.go:67] Client error processing command: Get http://localhost:8080/api/v1beta1/services/elasticsearch-logging?namespace=default: dial tcp 127.0.0.1:8080: connection refused
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl delete services kibana-logging
F0313 16:09:43.500610   22765 delete.go:67] Client error processing command: Get http://localhost:8080/api/v1beta1/services/kibana-logging?namespace=default: dial tcp 127.0.0.1:8080: connection refused
Bringing down cluster
Managed instance group e2e-test-satnam-minion-group is being deleted. Operation: operation-1426288184141-b597577f-6f41-4fe3-9e9a-9fb42b7433e6
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/instanceTemplates/e2e-test-satnam-minion-template].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
ERROR: (gcloud.compute.disks.delete) Some requests did not succeed:
 - The resource 'projects/kubernetes-satnam/zones/us-central1-b/disks/e2e-test-satnam-master-pd' was not found

Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-master-https].
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-minion-all].
Deleting routes e2e-test-satnam-minion-sdjx e2e-test-satnam-minion-y4fw e2e-test-satnam-master
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-sdjx].
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-y4fw].
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-master].
Deleted [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/e2e-test-satnam-master-ip].
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl config unset clusters.kubernetes-satnam_e2e-test-satnam
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl config unset users.kubernetes-satnam_e2e-test-satnam
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl config unset contexts.kubernetes-satnam_e2e-test-satnam
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl config view -o template --template={{ index . "current-context" }}
Cleared config for kubernetes-satnam_e2e-test-satnam from /usr/local/google/home/satnam/.kube/.kubeconfig
Done
2015/03/13 16:11:17 Running: get status
Project: kubernetes-satnam
Zone: us-central1-b
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl --match-server-version version

Usage: 
  kubectl [flags]
  kubectl [command]
Available Commands: 
  version       Print the client and server version information.
  clusterinfo   Display cluster info
  proxy         Run a proxy to the Kubernetes API server
  get           Display one or many resources
  describe      Show details of a specific resource
  create        Create a resource by filename or stdin
  update        Update a resource by filename or stdin.
  delete        Delete a resource by filename, stdin, or resource and ID.
  config        config modifies .kubeconfig files
  namespace     SUPERCEDED: Set and view the current Kubernetes namespace
  log           Print the logs for a container in a pod.
  rollingupdate Perform a rolling update of the given ReplicationController.
  resize        Set a new size for a Replication Controller.
  exec          Execute a command in a container.
  port-forward  Forward 1 or more local ports to a pod.
  run-container Run a particular image on the cluster.
  stop          Gracefully shut down a resource by id or filename.
  expose        Take a replicated application and expose it as Kubernetes Service
  label         Update the labels on a resource
  help          Help about any command

Flags:
      --alsologtostderr=false: log to standard error as well as files
      --api-version="": The API version to use when talking to the server
  -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https.
      --certificate-authority="": Path to a cert. file for the certificate authority.
      --client-certificate="": Path to a client key file for TLS.
      --client-key="": Path to a client key file for TLS.
      --cluster="": The name of the kubeconfig cluster to use
      --context="": The name of the kubeconfig context to use
  -h, --help=false: help for kubectl
      --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
      --kubeconfig="": Path to the kubeconfig file to use for CLI requests.
      --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace
      --log_dir=: If non-empty, write log files in this directory
      --log_flush_frequency=5s: Maximum number of seconds between log flushes
      --logtostderr=true: log to standard error instead of files
      --match-server-version=false: Require server version to match client version
      --namespace="": If present, the namespace scope for this CLI request.
      --password="": Password for basic authentication to the API server.
  -s, --server="": The address and port of the Kubernetes API server
      --stderrthreshold=2: logs at or above this threshold go to stderr
      --token="": Bearer token for authentication to the API server.
      --user="": The name of the kubeconfig user to use
      --username="": Username for basic authentication to the API server.
      --v=0: log level for V logs
      --validate=false: If true, use a schema to validate the input before sending it
      --vmodule=: comma-separated list of pattern=N settings for file-filtered logging


Use "kubectl help [command]" for more information about a command.
Error: unknown command "--match-server-version"
Run 'help' for usage.

kubectl: invalid command [`--match-server-version` `version`]
Run 'kubectl help' for usage
2015/03/13 16:11:18 Error running get status: exit status 1
2015/03/13 16:11:18 Running: up
Project: kubernetes-satnam
Zone: us-central1-b
Starting cluster using provider: gce
... calling verify-prereqs
... calling kube-up
Project: kubernetes-satnam
Zone: us-central1-b
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl config view -o template --template={{$root := .}}{{with index $root "current-context"}}{{with index $root "contexts" .}}{{with index . "user"}}{{with index $root "users" .}}{{index . "username"}}{{end}}{{end}}{{end}}{{end}}
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl config view -o template --template={{$root := .}}{{with index $root "current-context"}}{{with index $root "contexts" .}}{{with index . "user"}}{{with index $root "users" .}}{{index . "password"}}{{end}}{{end}}{{end}}{{end}}
+++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e/devel
Starting master and configuring firewalls
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#pdperformance.
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/e2e-test-satnam-master-pd].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-master-https].
NAME                      ZONE          SIZE_GB TYPE        STATUS
e2e-test-satnam-master-pd us-central1-b 10      pd-standard READY
NAME                         NETWORK SRC_RANGES RULES   SRC_TAGS TARGET_TAGS
e2e-test-satnam-master-https e2e     0.0.0.0/0  tcp:443          e2e-test-satnam-master
+++ Logging using Fluentd to elasticsearch
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-minion-all].
NAME                       NETWORK SRC_RANGES    RULES                    SRC_TAGS TARGET_TAGS
e2e-test-satnam-minion-all e2e     10.245.0.0/16 tcp,udp,icmp,esp,ah,sctp          e2e-test-satnam-minion
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
NAME                   ZONE          MACHINE_TYPE INTERNAL_IP   EXTERNAL_IP    STATUS
e2e-test-satnam-master us-central1-b g1-small     10.240.62.187 104.154.81.134 RUNNING
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
Creating minions.
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#pdperformance.
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/instanceTemplates/e2e-test-satnam-minion-template].
NAME                            MACHINE_TYPE CREATION_TIMESTAMP
e2e-test-satnam-minion-template g1-small     2015-03-13T16:12:19.593-07:00
Managed instance group e2e-test-satnam-minion-group is being created. Operation: operation-1426288344032-fe59f9ba-4554-491d-b617-d21a7efc2b31
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 1 out of 2 running. Retrying.
MINION_NAMES=e2e-test-satnam-minion-iydx e2e-test-satnam-minion-j9nf
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-minion-j9nf].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-minion-iydx].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-iydx].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-master].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-j9nf].
NAME                        NETWORK DEST_RANGE    NEXT_HOP                                            PRIORITY
e2e-test-satnam-minion-iydx e2e     10.245.1.0/24 us-central1-b/instances/e2e-test-satnam-minion-iydx 1000
NAME                   NETWORK DEST_RANGE    NEXT_HOP                                       PRIORITY
e2e-test-satnam-master e2e     10.246.0.0/24 us-central1-b/instances/e2e-test-satnam-master 1000
NAME                        NETWORK DEST_RANGE    NEXT_HOP                                            PRIORITY
e2e-test-satnam-minion-j9nf e2e     10.245.2.0/24 us-central1-b/instances/e2e-test-satnam-minion-j9nf 1000
Using master: e2e-test-satnam-master (external IP: 104.154.81.134)
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/e2e-test-satnam-master-ip].
NAME                      REGION      ADDRESS        STATUS
e2e-test-satnam-master-ip us-central1 104.154.81.134 IN_USE
Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

.........................

@satnam6502
Copy link
Contributor Author

It could be me. My brain is mush after being call for two days.

@j3ffml
Copy link
Contributor

j3ffml commented Mar 13, 2015

How long did you wait before killing it? Could just be gce slowness.

@satnam6502
Copy link
Contributor Author

Various durations including one > 20 minutes.

On Fri, Mar 13, 2015 at 4:24 PM, Jeff Lowdermilk notifications@github.com
wrote:

How long did you wait before killing it? Could just be gce slowness.

—
Reply to this email directly or view it on GitHub
#5451 (comment)
.

@satnam6502 satnam6502 changed the title Cluster teardown for GCE not working Cluster teardown and bring up for GCE e2e not working Mar 13, 2015
@satnam6502
Copy link
Contributor Author

How many more dotes do you want to see?

e2e-test-satnam-master-ip us-central1 104.154.81.134 IN_USE
Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

..................................................................................................................................................................................................

@satnam6502
Copy link
Contributor Author

Switching to a different GCE project makes everything work.

@zmerlynn
Copy link
Member

You probably have some resource that's screwed up and not getting torn down
correctly. (Maybe something that teardown didn't find because it was in a
different zone?)

On Fri, Mar 13, 2015 at 5:02 PM, Satnam Singh notifications@github.com
wrote:

Switching to a different GCE project makes everything work.

—
Reply to this email directly or view it on GitHub
#5451 (comment)
.

@satnam6502
Copy link
Contributor Author

Several times I went into Dev Console and deleted all stray resources manually -- it did not help.

@zmerlynn
Copy link
Member

I'm disturbed there were stray resources after kube-down...

On Fri, Mar 13, 2015 at 5:16 PM, Satnam Singh notifications@github.com
wrote:

Several time I went into Dev Console and deleted all stray resources
manually -- it did not help.

—
Reply to this email directly or view it on GitHub
#5451 (comment)
.

@satnam6502 satnam6502 mentioned this issue Mar 14, 2015
Closed
@satnam6502
Copy link
Contributor Author

I deleted all of my clusters.
I deleted my ~/.kube directory.
I build from HEAD.
I tried to make a fresh new e2e cluster.
Still to no avail.

$ go run hack/e2e.go -v -up ; hack/ginkgo-e2e.sh -t Cluster
2015/03/16 13:40:20 Running: get status
Project: kubernetes-satnam
Zone: us-central1-b
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl --match-server-version version
F0316 13:40:21.883362    7715 version.go:34] Client error processing command: couldn't read version from server: Get http://localhost:8080/version: dial tcp 127.0.0.1:8080: connection refused
2015/03/16 13:40:21 Error running get status: exit status 255
2015/03/16 13:40:21 Running: up
Project: kubernetes-satnam
Zone: us-central1-b
Starting cluster using provider: gce
... calling verify-prereqs
... calling kube-up
Project: kubernetes-satnam
Zone: us-central1-b
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl config view -o template --template={{$root := .}}{{with index $root "current-context"}}{{with index $root "contexts" .}}{{with index . "user"}}{{with index $root "users" .}}{{index . "username"}}{{end}}{{end}}{{end}}{{end}}
current-context: ""
Running: /usr/local/google/home/satnam/gocode/src/github.com/GoogleCloudPlatform/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl config view -o template --template={{$root := .}}{{with index $root "current-context"}}{{with index $root "contexts" .}}{{with index . "user"}}{{with index $root "users" .}}{{index . "password"}}{{end}}{{end}}{{end}}{{end}}
+++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e/devel
Starting master and configuring firewalls
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#pdperformance.
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-master-https].
NAME                         NETWORK SRC_RANGES RULES   SRC_TAGS TARGET_TAGS
e2e-test-satnam-master-https e2e     0.0.0.0/0  tcp:443          e2e-test-satnam-master
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/e2e-test-satnam-master-pd].
NAME                      ZONE          SIZE_GB TYPE        STATUS
e2e-test-satnam-master-pd us-central1-b 10      pd-standard READY
+++ Logging using Fluentd to elasticsearch
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-minion-all].
NAME                       NETWORK SRC_RANGES    RULES                    SRC_TAGS TARGET_TAGS
e2e-test-satnam-minion-all e2e     10.245.0.0/16 tcp,udp,icmp,esp,ah,sctp          e2e-test-satnam-minion
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
NAME                   ZONE          MACHINE_TYPE INTERNAL_IP    EXTERNAL_IP   STATUS
e2e-test-satnam-master us-central1-b g1-small     10.240.100.148 104.154.69.40 RUNNING
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
Creating minions.
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#pdperformance.
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/instanceTemplates/e2e-test-satnam-minion-template].
NAME                            MACHINE_TYPE CREATION_TIMESTAMP
e2e-test-satnam-minion-template g1-small     2015-03-16T13:41:28.890-07:00
Managed instance group e2e-test-satnam-minion-group is being created. Operation: operation-1426538496534-948f6ba4-fcdb-4a04-9eb2-a53123be3876
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
Waiting for minions to run. 0 out of 2 running. Retrying.
MINION_NAMES=e2e-test-satnam-minion-2dkr e2e-test-satnam-minion-anxf
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-minion-2dkr].
Updated [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-minion-anxf].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-anxf].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-master].
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/routes/e2e-test-satnam-minion-2dkr].
NAME                        NETWORK DEST_RANGE    NEXT_HOP                                            PRIORITY
e2e-test-satnam-minion-anxf e2e     10.245.2.0/24 us-central1-b/instances/e2e-test-satnam-minion-anxf 1000
NAME                   NETWORK DEST_RANGE    NEXT_HOP                                       PRIORITY
e2e-test-satnam-master e2e     10.246.0.0/24 us-central1-b/instances/e2e-test-satnam-master 1000
NAME                        NETWORK DEST_RANGE    NEXT_HOP                                            PRIORITY
e2e-test-satnam-minion-2dkr e2e     10.245.1.0/24 us-central1-b/instances/e2e-test-satnam-minion-2dkr 1000
Using master: e2e-test-satnam-master (external IP: 104.154.69.40)
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/e2e-test-satnam-master-ip].
NAME                      REGION      ADDRESS       STATUS
e2e-test-satnam-master-ip us-central1 104.154.69.40 IN_USE
Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

...................................................................................................................................................................................

@zmerlynn
Copy link
Member

Let's livedebug on IRC.

@alex-mohr
Copy link
Contributor

@satnam6502 @zmerlynn Was there a verdict from IRC livedebug?

@satnam6502
Copy link
Contributor Author

Not enough whisky. Or we need to delete the e2e network and retry.

@liangchenye liangchenye mentioned this issue Feb 23, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zmerlynn zmerlynn

Labels
priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@j3ffml @zmerlynn @satnam6502 @bgrant0607 @alex-mohr