Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorrect IP address routed(?) to container #30531

Closed
nvnobelen opened this issue Aug 12, 2016 · 3 comments
Closed

Incorrect IP address routed(?) to container #30531

nvnobelen opened this issue Aug 12, 2016 · 3 comments
Labels
area/kubectl

Comments

@nvnobelen
Copy link

nvnobelen commented Aug 12, 2016 •
edited
Loading

This is on a fresh GCE Kubernetes 1.3.5 cluster created with the stock kube-up.sh script
In testing nginx logging (can be replicated with any other ip logging), the following can be observed:

The behaviour has also been observed on a 1.2.4 & 1.2.6 cluster (no other versions tested)

Kubernetes version: 1.2.4 & 1.2.6, 1.3.5

Environment:
1.2.6:

  • Google Cloud Engine
  • container-vm-v20160321
  • linux kernel 3.16.0-4-amd64

1.3.5:

  • Google Cloud Engine
  • container-v1-3-v20160604
  • linux kernel 4.4.4+

Testing the logging in nginx I observed the following under load (and also when there is low load):

174.6.196.75 58 0.059 [12/Aug/2016:19:51:19 +0000]  "GET / HTTP/1.1" 200 4 "-" "loadtest/1.4.4" "-" "-"
10.240.0.11 59 0.068 [12/Aug/2016:19:51:19 +0000]  "GET / HTTP/1.1" 200 4 "-" "loadtest/1.4.4" "-" "-"
10.240.0.11 60 0.062 [12/Aug/2016:19:51:20 +0000]  "GET / HTTP/1.1" 200 4 "-" "loadtest/1.4.4" "-" "-"
10.240.0.11 61 0.062 [12/Aug/2016:19:51:20 +0000]  "GET / HTTP/1.1" 200 4 "-" "loadtest/1.4.4" "-" "-"
10.240.0.11 62 0.062 [12/Aug/2016:19:51:20 +0000]  "GET / HTTP/1.1" 200 4 "-" "loadtest/1.4.4" "-" "-"
10.240.0.11 63 0.060 [12/Aug/2016:19:51:20 +0000]  "GET / HTTP/1.1" 200 4 "-" "loadtest/1.4.4" "-" "-"
10.240.0.11 64 0.075 [12/Aug/2016:19:51:21 +0000]  "GET / HTTP/1.1" 200 4 "-" "loadtest/1.4.4" "-" "-"
10.240.0.7 65 0.062 [12/Aug/2016:19:51:21 +0000]  "GET / HTTP/1.1" 200 4 "-" "loadtest/1.4.4" "-" "-"

In this the request is coming from the same source ip (174.6.196.75) but for some reason the address of the minion (10.240.0.11) is used in a certain percentage of cases, and the 10.240.0.7 which is an ip from another minion also shows up in the log.

The configuration of the containers is as follows:


---
apiVersion: v1
kind: Service
metadata:
  name: tracking-nginx-service
  labels:
    name: tracking-nginx-service
spec:
  type: LoadBalancer
  selector:
    app: tracking-nginx
  ports:
  - 
    name: "http"
    port: 80
    targetPort: "tracking-http"
    protocol: "TCP"

---
apiVersion: v1
kind: ReplicationController
metadata:
  name: tracking
spec:
  replicas: 2
  selector:
    app: tracking-nginx
  template:
    metadata:
      name: tracking-nginx
      labels:
        app: tracking-nginx
    spec:
      containers:
      - name: tracking-nginx
        resources:
          limits:
            cpu: 50m
            memory: 100Mi
        image: gcr.io/our_nginx_container
        imagePullPolicy: Always
        ports:
        -
          name: "tracking-http"
          containerPort: 80
      hostNetwork: true

hostNetwork is used to translate the container ip to the 174.6.xx.xx address. Removing hostNetwork results in the container ip instead of the 174.6 address, for the rest the behaviour stays the same.

Reproduce by:

  • Installing Kubernetes 1.2.6 stock with GCE container-vm-v20160321.
  • Run any ip logger to log ips, run data through Google loadbalancer (not known if this actually influences the behaviour).
@nvnobelen
Copy link
Author

This seems to be related to #3760

@nvnobelen
Copy link
Author

And 1.4-alpha2 still has this issue despite posted fixes

@nvnobelen nvnobelen mentioned this issue Aug 12, 2016
Closed
17 tasks
@nvnobelen
Copy link
Author

Issue is still in the development queue as it turns out: #30105

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/kubectl
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nvnobelen @k8s-github-robot