Skip to content

cloud-init is reseting DNS network config from dhcp after Fedora 40 -> 41 upgrade on Hetzner cloud #5961

Open
@pschiffe

Description

Bug report

After upgrade from Fedora 40 to Fedora 41 (cloud-init-24.1.4-3.fc40.noarch -> cloud-init-24.2-3.fc41.noarch) on Hetzner cloud; the custom DNS configuration done by network manager is overridden during reboot by cloud-init. The log also show some exceptions, see below.

Before with Fedora 40 and cloud-init-24.1.4-3.fc40.noarch the log just showed, without errors:

2024-11-12 18:50:05,852 - stages.py[DEBUG]: Allowed events: {<EventScope.NETWORK: 'network'>: {<EventType.BOOT_NEW_INSTANCE: 'boot-new-instance'>}}
2024-11-12 18:50:05,852 - stages.py[DEBUG]: Event Denied: scopes=['network'] EventType=boot-legacy
2024-11-12 18:50:05,852 - stages.py[DEBUG]: No network config applied. Neither a new instance nor datasource network update allowed
2024-11-12 18:50:05,852 - stages.py[DEBUG]: applying net config names for {'config': [{'mac_address': '96:00:02:9f:7f:6e', 'name': 'eth0', 'subnets': [{'ipv4': True, 'type': 'dhcp'}, {'address': '2a01:4f8:c013:ce8::1/64', 'dns_nameservers': ['2a01:4ff:ff00::add:2', '2a01:4ff:ff00::add:1'], 'gateway': 'fe80::1', 'ipv6': True, 'type': 'static'}], 'type': 'physical'}], 'version': 1}

And the custom DNS configuration done by nmcli wasn't changed. Now there are exceptions and the DNS config is reset.

Steps to reproduce the problem

I didn't do any changes to cloud-init. There's configuration provided by Hetzner:

$ cat /etc/cloud/cloud.cfg.d/90-hetznercloud.cfg 
# The top level settings are used as module
# and system configuration.

# A set of users which may be applied and/or used by various modules
# when a 'default' entry is found it will reference the 'default_user'
# from the distro configuration specified below
users:
   - default

# If this is set, 'root' will not be able to ssh in and they
# will get a message to login instead as the default $user
disable_root: false

mount_default_fields: [~, ~, 'auto', 'defaults,nofail', '0', '2']
resize_rootfs_tmp: /dev
ssh_pwauth:   0

# This will cause the set+update hostname module to not operate (if true)
preserve_hostname: false

# Example datasource config
# datasource:
#    Ec2:
#      metadata_urls: [ 'blah.com' ]
#      timeout: 5 # (defaults to 50 seconds)
#      max_wait: 10 # (defaults to 120 seconds)

# The modules that run in the 'init' stage
cloud_init_modules:
 - migrator
 - seed_random
 - bootcmd
 - write-files
 - [ growpart, always ]
 - [ resizefs, always ]
 - disk_setup
 - mounts
 - set_hostname
 - update_hostname
 - [ update_etc_hosts, once-per-instance ]
 - ca-certs
 - rsyslog
 - users-groups
 - ssh

# The modules that run in the 'config' stage
cloud_config_modules:
 - ssh-import-id
 - locale
 - set-passwords
 - spacewalk
 - yum-add-repo
 - ntp
 - timezone
 - disable-ec2-metadata
 - runcmd

# The modules that run in the 'final' stage
cloud_final_modules:
 - package-update-upgrade-install
 - puppet
 - chef
 - mcollective
 - salt-minion
 - rightscale_userdata
 - scripts-vendor
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - ssh-authkey-fingerprints
 - keys-to-console
 - phone-home
 - final-message
 - power-state-change

# System and/or distro specific settings
# (not accessible to handlers/transforms)
system_info:
   # This will affect which distro class gets used
   distro: fedora
   # Default user name + that default users groups (if added/used)
   default_user:
     name: root
     lock_passwd: false
     shell: /bin/bash
   # Other config here will be given to the distro class and/or path classes
   paths:
      cloud_dir: /var/lib/cloud/
      templates_dir: /etc/cloud/templates/
   ssh_svcname: sshd
$ cat /etc/cloud/cloud.cfg.d/92-hetznercloud-ds.cfg 
# Hetzner cloud-init defaults
# This file was written during the Hetzner image generation process

datasource_list: [ Hetzner, None ]

Environment details

  • Cloud-init version: cloud-init-24.2-3.fc41.noarch
  • Operating System Distribution: Fedora 41
  • Cloud provider, platform or installer type: Hetzner cloud

cloud-init logs

2025-01-10 11:05:23,189 - subp.py[DEBUG]: Running command ['ip', '--json', 'addr'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,200 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/hetzner/v1/metadata/instance-id' with {'url': 'http://169.254.169.254/hetzner/v1/metadata/instance-id', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 5.0, 'headers': {'User-Agent': 'Cloud-Init/24.2'}} configuration
2025-01-10 11:05:23,207 - distros[DEBUG]: DHCP client selected: dhcpcd
2025-01-10 11:05:23,208 - dhcp.py[DEBUG]: Performing a dhcp discovery on ens3
2025-01-10 11:05:23,208 - subp.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,215 - util.py[DEBUG]: Reading from /sys/class/net/ens3/type (quiet=False)
2025-01-10 11:05:23,215 - util.py[DEBUG]: Read 2 bytes from /sys/class/net/ens3/type
2025-01-10 11:05:23,215 - subp.py[DEBUG]: Running command ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,433 - subp.py[DEBUG]: ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3'] took 0.2s to run
2025-01-10 11:05:23,433 - dhcp.py[DEBUG]: dhcpcd exited with code: 1 stderr: 'dhcpcd-10.0.8 starting\n        DUID 00:01:00:01:2f:12:a1:d3:96:00:02:9f:7f:6e\n        ens3: IAID 02:9f:7f:6e\n        dhcpcd_fork_cb: dhcpcd manager hungup' stdout: ''
2025-01-10 11:05:23,433 - DataSourceHetzner.py[ERROR]: Bailing, DHCP Exception: 
2025-01-10 11:05:23,433 - handlers.py[DEBUG]: finish: init-local/search-Hetzner: FAIL: no local data found from DataSourceHetzner
2025-01-10 11:05:23,433 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceHetzner.DataSourceHetzner'> failed
2025-01-10 11:05:23,433 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceHetzner.DataSourceHetzner'> failed
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 666, in dhcp_discovery
    out, err = subp.subp(
               ~~~~~~~~~^
        command,
        ^^^^^^^^
        timeout=self.timeout,
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/cloudinit/subp.py", line 295, in subp
    raise ProcessExecutionError(
        stdout=out, stderr=err, exit_code=rc, cmd=args
    )
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3']
Exit code: 1
Reason: -
Stdout: 
Stderr: dhcpcd-10.0.8 starting
        DUID 00:01:00:01:2f:12:a1:d3:96:00:02:9f:7f:6e
        ens3: IAID 02:9f:7f:6e
        dhcpcd_fork_cb: dhcpcd manager hungup

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 1053, in find_source
    if s.update_metadata_if_supported(
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        [EventType.BOOT_NEW_INSTANCE]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 929, in update_metadata_if_supported
    result = self.get_data()
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 485, in get_data
    return_value = self._check_and_get_data()
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 417, in _check_and_get_data
    return self._get_data()
           ~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/DataSourceHetzner.py", line 60, in _get_data
    with EphemeralDHCPv4(
    ...<17 lines>...
        )
  File "/usr/lib/python3.13/site-packages/cloudinit/net/ephemeral.py", line 306, in __enter__
    return self.obtain_lease()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/cloudinit/net/ephemeral.py", line 329, in obtain_lease
    self.lease = maybe_perform_dhcp_discovery(
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.distro, self.iface, self.dhcp_log_func
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 105, in maybe_perform_dhcp_discovery
    return distro.dhcp_client.dhcp_discovery(interface, dhcp_log_func, distro)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 733, in dhcp_discovery
    raise NoDHCPLeaseError from error
cloudinit.net.dhcp.NoDHCPLeaseError

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working correctlynewAn issue that still needs triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions