Skip to content

GCE: Services (LoadBalancer) with static ip causes panic in 1.7 #48848

Closed
@nicksardo

Description

/kind bug

What happened:
Services of type LoadBalancer (GCE) with IP addresses promoted to static (by user or by orphan) cause the service controller to panic. The constant panicking will cause a reboot loop of the pod.

How to reproduce it (as minimally and precisely as possible):

  1. Create a service on GCP
  2. Expose it
  3. In the GCP Console, promote the IP address of the LB to static and set the name to the loadBalancerName which is the same name as the forwarding rule.

Environment:

  • Kubernetes version:
    1.7.0 AND 1.7.1
  • Cloud provider or hardware configuration**:
    GCE

Symptoms

  1. kube-controller-manager crash loops on the master.

Mitigation
Option 1:
Build your own version of release-1.7 branch for the master. Only the kube-controller-manager image needs to be updated.

Option 2:
For all affected services, set the loadBalancerIP field to the known external IP address.

Steps:

  1. Gather list of reserved IPs

➜ gcloud compute addresses list
NAME REGION ADDRESS STATUS
a8b891421675f11e7bee242010a80000 us-central1 35.35.35.35 IN_USE

  1. Gather list of all Services of type LoadBalancer

➜ kubectl get svc --all-namespaces | grep "LoadBalancer"
default my-service LoadBalancer 10.0.99.201 35.35.35.35 80:32587/TCP,443:30446/TCP 58m

  1. For each service of type LoadBalancer with an IP that's reserved (in the list above), continue to next step.
  2. Set the service's loadBalancerIP field equal to the known IP.

Example
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2017-07-13T00:09:31Z
labels:
app: echo
name: my-service
namespace: default
resourceVersion: "46816"
selfLink: /api/v1/namespaces/default/services/my-service
uid: 8b891421-675f-11e7-bee2-42010a800002
spec:
clusterIP: 10.0.99.201
externalTrafficPolicy: Cluster
loadBalancerIP: 35.35.35.35 <------- Field set to status.loadBalancer.ingress.ip below.
ports:

  • name: port-1
    nodePort: 32587
    port: 80
    protocol: TCP
    targetPort: 80
  • name: port-2
    nodePort: 30446
    port: 443
    protocol: TCP
    targetPort: 443
    selector:
    app: echo
    sessionAffinity: ClientIP
    type: LoadBalancer
    status:
    loadBalancer:
    ingress:
    • ip: 35.35.35.35

Metadata

Assignees

Labels

area/platform/gcekind/bugCategorizes issue or PR as related to a bug.sig/networkCategorizes an issue or PR as relevant to SIG Network.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions