Skip to content

Pilot agent on a VM doesn't clean all iptables chains, so will exit if re-started #52835

Closed
@birkland

Description

@birkland

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
    This is not a question about how to use Istio

Bug Description

On a VM with 1.23.0, try the following:

  1. run istio-start.sh
  2. Stop Istio (e.g something like pkill pilot-agent; pkill envoy)
  3. run istio-start.sh clean
  4. run istio-start.sh to try to start Istio again. It will terminate prematurely and neither pilot agent nor envoy will be running:
2024-08-23T18:22:22.108620Z     info    Running command (without lock): iptables-nft-restore --noflush
2024-08-23T18:22:22.110672Z     error   Command error output: xtables other problem: line 5 failed: Chain already exists.
2024-08-23T18:22:22.110800Z     info    Running command (without lock): iptables-nft-save 
2024-08-23T18:22:22.111989Z     info    Command output: 
# Generated by iptables-nft-save v1.8.10 (nf_tables) on Fri Aug 23 18:22:22 2024
*raw
:PREROUTING ACCEPT [2923:1031893]
:OUTPUT ACCEPT [3089:416325]
:ISTIO_OUTPUT - [0:0]
COMMIT
# Completed on Fri Aug 23 18:22:22 2024
# Generated by iptables-nft-save v1.8.10 (nf_tables) on Fri Aug 23 18:22:22 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER_OUTPUT - [0:0]
:DOCKER_POSTROUTING - [0:0]
:ISTIO_OUTPUT - [0:0]
-A OUTPUT -d 127.0.0.11/32 -j DOCKER_OUTPUT
-A POSTROUTING -d 127.0.0.11/32 -j DOCKER_POSTROUTING
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:43113
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:47857
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p tcp -m tcp --sport 43113 -j SNAT --to-source :53
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 47857 -j SNAT --to-source :53
COMMIT
# Completed on Fri Aug 23 18:22:22 2024

2024-08-23T18:22:22.112019Z     error   exit status 1

You'll see a leftover empty ISTIO_OUTPUT chain in the raw table. Istio 1.22 and earlier don't create this chain at all. Istio 1.23 does, but doesn't attempt to remove it when doing istio-clean-iptables.

So it looks like the pilot-agent istio-clean-iptables cmd needs to account for that chain in 1.23, at least.

Version

1.23.0

Additional Information

root@671dd6898073:/# apt list --installed | grep istio

istio-sidecar/now 1.23.0 arm64 [installed,local]

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Pilot agent on a VM doesn't clean all iptables chains, so will exit if re-started · Issue #52835 · istio/istio