Two k8s ingress with same secret resulting in blackhole 404 for port 443 #54757
Open
Description
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
I applied below two ingress yamls to cluster
k get ingress -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"istio"},"name":"mdnnddy-ingress-2","namespace":"default"},"spec":{"rules":[{"host":"altostrat.com","http":{"paths":[{"backend":{"service":{"name":"hello-service","port":{"number":60000}}},"path":"/greet-the-world","pathType":"Exact"},{"backend":{"service":{"name":"hello-service","port":{"number":60001}}},"path":"/greet-kubernetes","pathType":"Exact"}]}}],"tls":[{"hosts":["altostrat.com"],"secretName":"example-server-creds-1"}]}}
kubernetes.io/ingress.class: istio
creationTimestamp: "2025-01-20T04:33:37Z"
generation: 1
name: mdnnddy-ingress-2
namespace: default
resourceVersion: "70628013"
uid: 3e66aaee-cbc5-4425-90ac-5398451c229b
spec:
rules:
- host: altostrat.com
http:
paths:
- backend:
service:
name: hello-service
port:
number: 60000
path: /greet-the-world
pathType: Exact
- backend:
service:
name: hello-service
port:
number: 60001
path: /greet-kubernetes
pathType: Exact
tls:
- hosts:
- altostrat.com
secretName: example-server-creds-1
status:
loadBalancer:
ingress:
- ip: 34.46.104.172
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"istio"},"name":"myddd-ingress-2","namespace":"default"},"spec":{"rules":[{"host":"altostrat.com","http":{"paths":[{"backend":{"service":{"name":"hello-service","port":{"number":60000}}},"path":"/greet-the-worlda","pathType":"Exact"},{"backend":{"service":{"name":"hello-service","port":{"number":60001}}},"path":"/greet-kubernetesa","pathType":"Exact"}]}}],"tls":[{"hosts":["altostrat.com"],"secretName":"example-server-creds-1"}]}}
kubernetes.io/ingress.class: istio
creationTimestamp: "2025-01-20T04:33:25Z"
generation: 1
name: myddd-ingress-2
namespace: default
resourceVersion: "70627857"
uid: 58104301-9507-43d4-87af-5e01adbd204a
spec:
rules:
- host: altostrat.com
http:
paths:
- backend:
service:
name: hello-service
port:
number: 60000
path: /greet-the-worlda
pathType: Exact
- backend:
service:
name: hello-service
port:
number: 60001
path: /greet-kubernetesa
pathType: Exact
tls:
- hosts:
- altostrat.com
secretName: example-server-creds-1
status:
loadBalancer:
ingress:
- ip: 34.46.104.172
kind: List
metadata:
resourceVersion: ""
I can confirm that secret is present in istio-system namespace where ingress pods are deployed
k get secret -n istio-system
NAME TYPE DATA AGE
example-server-creds-1 kubernetes.io/tls 2 8m46s
istio-ca-secret istio.io/ca-root 5 31d
This is route configuration from ingress pod
istioctl proxy-config routes istio-ingressgateway-74dbbdd4d5-2tjfw -n istio-system -o json
[
{
"name": "http.80",
"virtualHosts": [
{
"name": "altostrat.com:80",
"domains": [
"altostrat.com"
],
"routes": [
{
"match": {
"path": "/greet-kubernetesa",
"caseSensitive": true
},
"route": {
"cluster": "outbound|60001||hello-service.default.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"hostSelectionRetryMaxAttempts": "5",
"retriableStatusCodes": [
503
]
},
"maxGrpcTimeout": "0s"
},
"metadata": {
"filterMetadata": {
"istio": {
"config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/altostrat-com-myddd-ingress-2-istio-autogenerated-k8s-ingress"
}
}
},
"decorator": {
"operation": "hello-service.default.svc.cluster.local:60001/greet-kubernetesa"
}
},
{
"match": {
"path": "/greet-the-worlda",
"caseSensitive": true
},
"route": {
"cluster": "outbound|60000||hello-service.default.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"hostSelectionRetryMaxAttempts": "5",
"retriableStatusCodes": [
503
]
},
"maxGrpcTimeout": "0s"
},
"metadata": {
"filterMetadata": {
"istio": {
"config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/altostrat-com-myddd-ingress-2-istio-autogenerated-k8s-ingress"
}
}
},
"decorator": {
"operation": "hello-service.default.svc.cluster.local:60000/greet-the-worlda"
}
},
{
"match": {
"path": "/greet-kubernetes",
"caseSensitive": true
},
"route": {
"cluster": "outbound|60001||hello-service.default.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"hostSelectionRetryMaxAttempts": "5",
"retriableStatusCodes": [
503
]
},
"maxGrpcTimeout": "0s"
},
"metadata": {
"filterMetadata": {
"istio": {
"config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/altostrat-com-myddd-ingress-2-istio-autogenerated-k8s-ingress"
}
}
},
"decorator": {
"operation": "hello-service.default.svc.cluster.local:60001/greet-kubernetes"
}
},
{
"match": {
"path": "/greet-the-world",
"caseSensitive": true
},
"route": {
"cluster": "outbound|60000||hello-service.default.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
}
}
],
"hostSelectionRetryMaxAttempts": "5",
"retriableStatusCodes": [
503
]
},
"maxGrpcTimeout": "0s"
},
"metadata": {
"filterMetadata": {
"istio": {
"config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/altostrat-com-myddd-ingress-2-istio-autogenerated-k8s-ingress"
}
}
},
"decorator": {
"operation": "hello-service.default.svc.cluster.local:60000/greet-the-world"
}
}
],
"includeRequestAttemptCount": true
}
],
"validateClusters": false,
"maxDirectResponseBodySizeBytes": 1048576,
"ignorePortInHostMatching": true
},
{
"name": "https.443.https-443-ingress-mdnnddy-ingress-2-default-0.mdnnddy-ingress-2-istio-autogenerated-k8s-ingress-default.istio-system",
"virtualHosts": [
{
"name": "blackhole:443",
"domains": [
"*"
]
}
],
"validateClusters": false,
"maxDirectResponseBodySizeBytes": 1048576,
"ignorePortInHostMatching": true
},
{
"virtualHosts": [
{
"name": "backend",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/healthz/ready"
},
"route": {
"cluster": "agent"
}
}
]
}
]
},
{
"virtualHosts": [
{
"name": "backend",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/stats/prometheus"
},
"route": {
"cluster": "prometheus_stats"
}
}
]
}
]
}
]
As we can see port 443 is giving blackhole.
Version
client version is 1.20.4
control plane version: 1.23.2
data plane version: 1.23.2
Additional Information
Istiod logs from 2 pods
ster.local: 100.898495ms since last change, 100.898408ms since last push, full=false"}
{"severity":"Info","timestamp":"2025-01-20T04:27:15.097645438Z","logger":"ads","message":"XDS: Incremental Pushing ConnectedEndpoints:2 Version:2025-01-20T04:26:50Z/1"}
{"severity":"Info","timestamp":"2025-01-20T04:33:21.789856841Z","logger":"ads","message":"Push debounce stable[17] 1 for config Secret/istio-system/example-server-creds: 100.280702ms since last change, 100.280621ms since last push, full=false"}
{"severity":"Info","timestamp":"2025-01-20T04:33:21.790043643Z","logger":"ads","message":"XDS: Incremental Pushing ConnectedEndpoints:2 Version:2025-01-20T04:26:50Z/1"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.111480673Z","logger":"ads","message":"Push debounce stable[18] 4 for config VirtualService/default/myddd-ingress-2-virtualservice and 1 more configs: 100.63319ms since last change, 109.968428ms since last push, full=true"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.112785978Z","logger":"ads","message":"XDS: Pushing Services:11 ConnectedEndpoints:2 Version:2025-01-20T04:33:26Z/2"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.113783308Z","logger":"delta","message":"CDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-fqpkt.istio-system resources:19 removed:0 size:22.0kB cached:15/18"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.11707303Z","logger":"delta","message":"LDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-fqpkt.istio-system resources:2 removed:0 size:5.3kB"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.117841539Z","logger":"delta","message":"CDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-2tjfw.istio-system resources:19 removed:0 size:22.0kB cached:15/18"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.118197599Z","logger":"delta","message":"LDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-2tjfw.istio-system resources:2 removed:0 size:5.3kB"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.139165353Z","logger":"delta","message":"RDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-2tjfw.istio-system resources:2 removed:0 size:2.4kB cached:0/0 filtered:0"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.141136086Z","logger":"delta","message":"RDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-fqpkt.istio-system resources:2 removed:0 size:2.4kB cached:0/0 filtered:0"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.144095428Z","logger":"delta","message":"SDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-2tjfw.istio-system resources:1 removed:0 size:2.9kB cached:0/1 filtered:0"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.14450919Z","logger":"delta","message":"SDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-fqpkt.istio-system resources:1 removed:0 size:2.9kB cached:1/1 filtered:0"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.718503783Z","logger":"ads","message":"Push debounce stable[19] 4 for config VirtualService/default/mdnnddy-ingress-2-virtualservice and 1 more configs: 100.625074ms since last change, 109.013479ms since last push, full=true"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.718980562Z","logger":"ads","message":"XDS: Pushing Services:11 ConnectedEndpoints:2 Version:2025-01-20T04:33:37Z/3"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.719188169Z","logger":"model","message":"skipping server on gateway istio-system/myddd-ingress-2-istio-autogenerated-k8s-ingress-default, duplicate host names: [altostrat.com]"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.719533448Z","logger":"delta","message":"CDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-2tjfw.istio-system resources:19 removed:0 size:22.0kB cached:18/18"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.720024141Z","logger":"delta","message":"LDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-2tjfw.istio-system resources:2 removed:0 size:5.3kB"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.72030535Z","message":"Gateway missing for route https.443.https-443-ingress-myddd-ingress-2-default-0.myddd-ingress-2-istio-autogenerated-k8s-ingress-default.istio-system. This is normal if gateway was recently deleted."}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.720421885Z","logger":"delta","message":"RDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-2tjfw.istio-system resources:2 removed:0 size:2.4kB cached:0/0"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.72063442Z","logger":"model","message":"skipping server on gateway istio-system/myddd-ingress-2-istio-autogenerated-k8s-ingress-default, duplicate host names: [altostrat.com]"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.720960651Z","logger":"delta","message":"CDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-fqpkt.istio-system resources:19 removed:0 size:22.0kB cached:18/18"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.721311569Z","logger":"delta","message":"LDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-fqpkt.istio-system resources:2 removed:0 size:5.3kB"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.72141396Z","message":"Gateway missing for route https.443.https-443-ingress-myddd-ingress-2-default-0.myddd-ingress-2-istio-autogenerated-k8s-ingress-default.istio-system. This is normal if gateway was recently deleted."}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.721639519Z","logger":"delta","message":"RDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-fqpkt.istio-system resources:2 removed:0 size:2.4kB cached:0/0"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.752566074Z","message":"constructed http route config for route https.443.https-443-ingress-mdnnddy-ingress-2-default-0.mdnnddy-ingress-2-istio-autogenerated-k8s-ingress-default.istio-system on port 443 with no vhosts; Setting up a default 404 vhost"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.752789586Z","logger":"delta","message":"RDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-fqpkt.istio-system resources:1 removed:0 size:158B cached:0/0 filtered:2"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.754028133Z","message":"constructed http route config for route https.443.https-443-ingress-mdnnddy-ingress-2-default-0.mdnnddy-ingress-2-istio-autogenerated-k8s-ingress-default.istio-system on port 443 with no vhosts; Setting up a default 404 vhost"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.75418476Z","logger":"delta","message":"RDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-2tjfw.istio-system resources:1 removed:0 size:158B cached:0/0 filtered:2"}
and
{"severity":"Info","timestamp":"2025-01-20T04:27:12.860350566Z","logger":"delta","message":"ADS: new delta connection for node:istio-ingressgateway-5b7d89679b-w2tvp.istio-system-2"}
{"severity":"Info","timestamp":"2025-01-20T04:27:12.863672915Z","logger":"delta","message":"CDS: PUSH request for node:istio-ingressgateway-5b7d89679b-w2tvp.istio-system resources:19 removed:0 size:22.0kB cached:15/18 filtered:0"}
{"severity":"Info","timestamp":"2025-01-20T04:27:12.867849641Z","logger":"delta","message":"EDS: PUSH request for node:istio-ingressgateway-5b7d89679b-w2tvp.istio-system resources:18 removed:0 size:6.7kB empty:0 cached:18/18 filtered:0"}
{"severity":"Info","timestamp":"2025-01-20T04:27:12.868113337Z","logger":"delta","message":"LDS: PUSH request for node:istio-ingressgateway-5b7d89679b-w2tvp.istio-system resources:0 removed:0 size:0B"}
{"severity":"Info","timestamp":"2025-01-20T04:27:12.868275256Z","logger":"delta","message":"PCDS: PUSH request for node:istio-ingressgateway-5b7d89679b-w2tvp.istio-system resources:1 removed:0 size:1.1kB"}
{"severity":"Info","timestamp":"2025-01-20T04:27:14.701085454Z","logger":"delta","message":"ADS: \"10.56.0.9:54776\" istio-ingressgateway-5b7d89679b-w2tvp.istio-system-2 terminated"}
{"severity":"Info","timestamp":"2025-01-20T04:27:15.097273799Z","logger":"ads","message":"Push debounce stable[13] 1 for config ServiceEntry/istio-system/istio-ingressgateway.istio-system.svc.cluster.local: 100.420738ms since last change, 100.420499ms since last push, full=false"}
{"severity":"Info","timestamp":"2025-01-20T04:27:15.097459613Z","logger":"ads","message":"XDS: Incremental Pushing ConnectedEndpoints:1 Version:2025-01-20T04:26:53Z/1"}
{"severity":"Info","timestamp":"2025-01-20T04:33:21.790155351Z","logger":"ads","message":"Push debounce stable[14] 1 for config Secret/istio-system/example-server-creds: 100.338537ms since last change, 100.338314ms since last push, full=false"}
{"severity":"Info","timestamp":"2025-01-20T04:33:21.790393115Z","logger":"ads","message":"XDS: Incremental Pushing ConnectedEndpoints:1 Version:2025-01-20T04:26:53Z/1"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.001808255Z","logger":"ingress status","message":"updating IPs ([{34.46.104.172 []}])","ingress":{"Namespace":"default","Name":"myddd-ingress-2"}}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.112688683Z","logger":"ads","message":"Push debounce stable[15] 4 for config VirtualService/default/myddd-ingress-2-virtualservice and 1 more configs: 100.520668ms since last change, 109.365133ms since last push, full=true"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.114048227Z","logger":"ads","message":"XDS: Pushing Services:11 ConnectedEndpoints:1 Version:2025-01-20T04:33:26Z/2"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.115131801Z","logger":"delta","message":"CDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-qmcfz.istio-system resources:19 removed:0 size:22.0kB cached:15/18"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.118902239Z","logger":"delta","message":"LDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-qmcfz.istio-system resources:2 removed:0 size:5.3kB"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.146323766Z","logger":"delta","message":"RDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-qmcfz.istio-system resources:2 removed:0 size:2.4kB cached:0/0 filtered:0"}
{"severity":"Info","timestamp":"2025-01-20T04:33:26.153228183Z","logger":"delta","message":"SDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-qmcfz.istio-system resources:1 removed:0 size:2.9kB cached:0/1 filtered:0"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.609606511Z","logger":"ingress status","message":"updating IPs ([{34.46.104.172 []}])","ingress":{"Namespace":"default","Name":"mdnnddy-ingress-2"}}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.718362617Z","logger":"ads","message":"Push debounce stable[16] 4 for config VirtualService/default/mdnnddy-ingress-2-virtualservice and 1 more configs: 100.799881ms since last change, 107.913251ms since last push, full=true"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.719060864Z","logger":"ads","message":"XDS: Pushing Services:11 ConnectedEndpoints:1 Version:2025-01-20T04:33:37Z/3"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.719408795Z","logger":"model","message":"skipping server on gateway istio-system/myddd-ingress-2-istio-autogenerated-k8s-ingress-default, duplicate host names: [altostrat.com]"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.720733259Z","logger":"delta","message":"CDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-qmcfz.istio-system resources:19 removed:0 size:22.0kB cached:18/18"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.721586227Z","logger":"delta","message":"LDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-qmcfz.istio-system resources:2 removed:0 size:5.3kB"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.722132281Z","message":"Gateway missing for route https.443.https-443-ingress-myddd-ingress-2-default-0.myddd-ingress-2-istio-autogenerated-k8s-ingress-default.istio-system. This is normal if gateway was recently deleted."}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.722426631Z","logger":"delta","message":"RDS: PUSH for node:istio-ingressgateway-74dbbdd4d5-qmcfz.istio-system resources:2 removed:0 size:2.4kB cached:0/0"}
{"severity":"Warning","timestamp":"2025-01-20T04:33:37.750601281Z","message":"constructed http route config for route https.443.https-443-ingress-mdnnddy-ingress-2-default-0.mdnnddy-ingress-2-istio-autogenerated-k8s-ingress-default.istio-system on port 443 with no vhosts; Setting up a default 404 vhost"}
{"severity":"Info","timestamp":"2025-01-20T04:33:37.750939147Z","logger":"delta","message":"RDS: PUSH request for node:istio-ingressgateway-74dbbdd4d5-qmcfz.istio-system resources:1 removed:0 size:158B cached:0/0 filtered:2"}