cloud-init is reseting DNS network config from dhcp after Fedora 40 -> 41 upgrade on Hetzner cloud #5961
Open
Description
Bug report
After upgrade from Fedora 40 to Fedora 41 (cloud-init-24.1.4-3.fc40.noarch
-> cloud-init-24.2-3.fc41.noarch
) on Hetzner cloud; the custom DNS configuration done by network manager is overridden during reboot by cloud-init. The log also show some exceptions, see below.
Before with Fedora 40 and cloud-init-24.1.4-3.fc40.noarch
the log just showed, without errors:
2024-11-12 18:50:05,852 - stages.py[DEBUG]: Allowed events: {<EventScope.NETWORK: 'network'>: {<EventType.BOOT_NEW_INSTANCE: 'boot-new-instance'>}}
2024-11-12 18:50:05,852 - stages.py[DEBUG]: Event Denied: scopes=['network'] EventType=boot-legacy
2024-11-12 18:50:05,852 - stages.py[DEBUG]: No network config applied. Neither a new instance nor datasource network update allowed
2024-11-12 18:50:05,852 - stages.py[DEBUG]: applying net config names for {'config': [{'mac_address': '96:00:02:9f:7f:6e', 'name': 'eth0', 'subnets': [{'ipv4': True, 'type': 'dhcp'}, {'address': '2a01:4f8:c013:ce8::1/64', 'dns_nameservers': ['2a01:4ff:ff00::add:2', '2a01:4ff:ff00::add:1'], 'gateway': 'fe80::1', 'ipv6': True, 'type': 'static'}], 'type': 'physical'}], 'version': 1}
And the custom DNS configuration done by nmcli wasn't changed. Now there are exceptions and the DNS config is reset.
Steps to reproduce the problem
I didn't do any changes to cloud-init. There's configuration provided by Hetzner:
$ cat /etc/cloud/cloud.cfg.d/90-hetznercloud.cfg
# The top level settings are used as module
# and system configuration.
# A set of users which may be applied and/or used by various modules
# when a 'default' entry is found it will reference the 'default_user'
# from the distro configuration specified below
users:
- default
# If this is set, 'root' will not be able to ssh in and they
# will get a message to login instead as the default $user
disable_root: false
mount_default_fields: [~, ~, 'auto', 'defaults,nofail', '0', '2']
resize_rootfs_tmp: /dev
ssh_pwauth: 0
# This will cause the set+update hostname module to not operate (if true)
preserve_hostname: false
# Example datasource config
# datasource:
# Ec2:
# metadata_urls: [ 'blah.com' ]
# timeout: 5 # (defaults to 50 seconds)
# max_wait: 10 # (defaults to 120 seconds)
# The modules that run in the 'init' stage
cloud_init_modules:
- migrator
- seed_random
- bootcmd
- write-files
- [ growpart, always ]
- [ resizefs, always ]
- disk_setup
- mounts
- set_hostname
- update_hostname
- [ update_etc_hosts, once-per-instance ]
- ca-certs
- rsyslog
- users-groups
- ssh
# The modules that run in the 'config' stage
cloud_config_modules:
- ssh-import-id
- locale
- set-passwords
- spacewalk
- yum-add-repo
- ntp
- timezone
- disable-ec2-metadata
- runcmd
# The modules that run in the 'final' stage
cloud_final_modules:
- package-update-upgrade-install
- puppet
- chef
- mcollective
- salt-minion
- rightscale_userdata
- scripts-vendor
- scripts-per-once
- scripts-per-boot
- scripts-per-instance
- scripts-user
- ssh-authkey-fingerprints
- keys-to-console
- phone-home
- final-message
- power-state-change
# System and/or distro specific settings
# (not accessible to handlers/transforms)
system_info:
# This will affect which distro class gets used
distro: fedora
# Default user name + that default users groups (if added/used)
default_user:
name: root
lock_passwd: false
shell: /bin/bash
# Other config here will be given to the distro class and/or path classes
paths:
cloud_dir: /var/lib/cloud/
templates_dir: /etc/cloud/templates/
ssh_svcname: sshd
$ cat /etc/cloud/cloud.cfg.d/92-hetznercloud-ds.cfg
# Hetzner cloud-init defaults
# This file was written during the Hetzner image generation process
datasource_list: [ Hetzner, None ]
Environment details
- Cloud-init version: cloud-init-24.2-3.fc41.noarch
- Operating System Distribution: Fedora 41
- Cloud provider, platform or installer type: Hetzner cloud
cloud-init logs
2025-01-10 11:05:23,189 - subp.py[DEBUG]: Running command ['ip', '--json', 'addr'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,200 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/hetzner/v1/metadata/instance-id' with {'url': 'http://169.254.169.254/hetzner/v1/metadata/instance-id', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 5.0, 'headers': {'User-Agent': 'Cloud-Init/24.2'}} configuration
2025-01-10 11:05:23,207 - distros[DEBUG]: DHCP client selected: dhcpcd
2025-01-10 11:05:23,208 - dhcp.py[DEBUG]: Performing a dhcp discovery on ens3
2025-01-10 11:05:23,208 - subp.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,215 - util.py[DEBUG]: Reading from /sys/class/net/ens3/type (quiet=False)
2025-01-10 11:05:23,215 - util.py[DEBUG]: Read 2 bytes from /sys/class/net/ens3/type
2025-01-10 11:05:23,215 - subp.py[DEBUG]: Running command ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,433 - subp.py[DEBUG]: ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3'] took 0.2s to run
2025-01-10 11:05:23,433 - dhcp.py[DEBUG]: dhcpcd exited with code: 1 stderr: 'dhcpcd-10.0.8 starting\n DUID 00:01:00:01:2f:12:a1:d3:96:00:02:9f:7f:6e\n ens3: IAID 02:9f:7f:6e\n dhcpcd_fork_cb: dhcpcd manager hungup' stdout: ''
2025-01-10 11:05:23,433 - DataSourceHetzner.py[ERROR]: Bailing, DHCP Exception:
2025-01-10 11:05:23,433 - handlers.py[DEBUG]: finish: init-local/search-Hetzner: FAIL: no local data found from DataSourceHetzner
2025-01-10 11:05:23,433 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceHetzner.DataSourceHetzner'> failed
2025-01-10 11:05:23,433 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceHetzner.DataSourceHetzner'> failed
Traceback (most recent call last):
File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 666, in dhcp_discovery
out, err = subp.subp(
~~~~~~~~~^
command,
^^^^^^^^
timeout=self.timeout,
^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/lib/python3.13/site-packages/cloudinit/subp.py", line 295, in subp
raise ProcessExecutionError(
stdout=out, stderr=err, exit_code=rc, cmd=args
)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3']
Exit code: 1
Reason: -
Stdout:
Stderr: dhcpcd-10.0.8 starting
DUID 00:01:00:01:2f:12:a1:d3:96:00:02:9f:7f:6e
ens3: IAID 02:9f:7f:6e
dhcpcd_fork_cb: dhcpcd manager hungup
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 1053, in find_source
if s.update_metadata_if_supported(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
[EventType.BOOT_NEW_INSTANCE]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
):
^
File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 929, in update_metadata_if_supported
result = self.get_data()
File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 485, in get_data
return_value = self._check_and_get_data()
File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 417, in _check_and_get_data
return self._get_data()
~~~~~~~~~~~~~~^^
File "/usr/lib/python3.13/site-packages/cloudinit/sources/DataSourceHetzner.py", line 60, in _get_data
with EphemeralDHCPv4(
...<17 lines>...
)
File "/usr/lib/python3.13/site-packages/cloudinit/net/ephemeral.py", line 306, in __enter__
return self.obtain_lease()
~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.13/site-packages/cloudinit/net/ephemeral.py", line 329, in obtain_lease
self.lease = maybe_perform_dhcp_discovery(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self.distro, self.iface, self.dhcp_log_func
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 105, in maybe_perform_dhcp_discovery
return distro.dhcp_client.dhcp_discovery(interface, dhcp_log_func, distro)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 733, in dhcp_discovery
raise NoDHCPLeaseError from error
cloudinit.net.dhcp.NoDHCPLeaseError