Unrelated ingresses targeting the same service/port: annotations for session stickyness not respected #12399
Description
What happened:
We have two ingresses on separate hostnames which target the same service and port. When we annotated the second ingress with
nginx.ingress.kubernetes.io/upstream-hash-by: $remote_addr
this was not added to the nginx configuration and session stickyness did not work.
What you expected to happen:
ingresses annotated with
nginx.ingress.kubernetes.io/upstream-hash-by: $remote_addr
are sticky to pods based on the remote client IP.
We were able to trace it down to the list provided by
kubectl ingress-nginx backends
which would show that the upstream hash
"upstreamHashByConfig": {
"upstream-hash-by-subset-size": 3
},
for both ingresses because it is in a section labeled by the namespace-service-port triplicate.
NGINX Ingress controller version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.12.0-beta.0
Build: 80154a3694b52736c88de408811ebd1888712520
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.5
-------------------------------------------------------------------------------
and
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.10.1
Build: 4fb5aac1dd3669daa3a14d9de3e3cdb371b4c518
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.3
-------------------------------------------------------------------------------
and
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.11.3
Build: 0106de65cfccb74405a6dfa7d9daffc6f0a6ef1a
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.5
-------------------------------------------------------------------------------
Kubernetes version v1.29.9 and v1.30.0
Environment:
- Cloud provider or hardware configuration: on-premise bare metal, reproduced in minikube
- OS (e.g. from /etc/os-release): flatcar (various versions 3975.2.2)
- Kernel (e.g.
uname -a): 6.6.54-flatcar
How to reproduce this issue:
Install minikube
Install the ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml
or v1.11
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/refs/heads/release-1.11/deploy/static/provider/baremetal/deploy.yaml
Create two ingresses
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo-server
labels:
app: echo-server
spec:
replicas: 5
selector:
matchLabels:
app: echo-server
template:
metadata:
labels:
app: echo-server
allow-from: public
spec:
containers:
- name: echo-server
image: ealen/echo-server:latest
env:
- name: PORT
value: "5678"
ports:
- containerPort: 5678
securityContext:
runAsNonRoot: true
runAsUser: 1000
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault #
---
apiVersion: v1
kind: Service
metadata:
name: echo-server
labels:
app: echo-server
spec:
selector:
app: echo-server
ports:
- protocol: TCP
port: 80
targetPort: 5678
type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: echo-server-ingress
annotations:
nginx.ingress.kubernetes.io/upstream-hash-by: $remote_addr
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: host1.minikube.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: echo-server
port:
number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: echo-server-ingress-public
annotations:
spec:
ingressClassName: nginx
rules:
- host: host2.minikube.local
http:
paths:
- backend:
service:
name: echo-server
port:
number: 80
path: /
pathType: Prefix
It is important that both ingresses in the above example are in the same namespace, and both point to the same service and same port, and that the first is alphabetically before the second.
The session stickyness annotation will not be applied and can be verified with
kubectl ingress-nginx backends -n ingress-nginx
this should output:
"name": "default-echo-server-test",
[...]
},
"upstreamHashByConfig": {
"upstream-hash-by-subset-size": 3
},
[...]
},
expected was output
"upstreamHashByConfig": {
"upstream-hash-by": "$remote_addr", # <---- this is missing above
"upstream-hash-by-subset-size": 3
},
Because these are two separate hostnames I would expect the ingress controller to handle them separately, and that annotations on the seemingly unrelated ingress. Note that the annotation should only be on one of the ingresses and the order matters. This is tested in our cluster on v1.29 with ingress-controller version v1.10and reproduceable in minikube with v1.12.0-beta
Metadata
Assignees
Labels
Type
Projects
Status
No status