image-refresh fedora-rawhide-boot done: https://github.com/cockpit-project/bots/commits/image-refresh-fedora-rawhide-boot-20230321-100623

Success. Log: https://cockpit-logs.us-east-1.linodeobjects.com/image-refresh-logs/fedora-rawhide-boot-20230321-100606.log

This will likely fail due to new pykickstart commands not being wired in in anaconda, which would be resolved in rhinstaller/anaconda#4561. No idea how fast this would get here, though...

lol a deadlock :D
ok, going to merge my part first

Nope it is a different issue:

===============================================================================
An unknown error has occurred
===============================================================================
anaconda 39.6 exception report
Traceback (most recent call first):
  File "/usr/lib64/python3.11/subprocess.py", line 1901, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
  File "/usr/lib64/python3.11/subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.11/site-packages/pyanaconda/core/util.py", line 151, in startProgram
    return subprocess.Popen(argv,
  File "/usr/lib64/python3.11/site-packages/pyanaconda/ui/webui/__init__.py", line 96, in run
    startProgram([
  File "/sbin/anaconda", line 572, in <module>
    anaconda.intf.run()
FileNotFoundError: [Errno 2] No such file or directory: '/usr/libexec/cockpit-ws'

@jkonecny12 @VladimirSlavik this fails exactly the same way that it failed on rhinstaller/anaconda#4624 where I see that it was merged with red tests.

Sorry, but the rest of our tests were red before merging that, so it was picking one of the two.

The "pick one" situation needs more discussion, I think.

Sorry, but the rest of our tests were red before merging that, so it was picking one of the two.

The "pick one" situation needs more discussion, I think.

if I get a bit better understanding on why the tests were failing in the first place, without the PR which broke the webui tests, I might be able to help improve the workflow. Is pykickstart updates the problem?

Sorry, but the rest of our tests were red before merging that, so it was picking one of the two.

The "pick one" situation needs more discussion, I think.

So the pykickstart PR broke the webui part in a way that you understand? @jkonecny12 @VladimirSlavik Because to me it's not clear still why webui broke (which deterministically did according to tests) with rhinstaller/anaconda#4624

It seems during boot we are not able to download updates image. Not sure what happens there but I'm starting to blame "Rawhide issues"...

if I get a bit better understanding on why the tests were failing in the first place, without the PR which broke the webui tests, I might be able to help improve the workflow. Is pykickstart updates the problem?

Yes. In a nutshell, anaconda infra uses nightly pykickstart builds, while cockpit tests take it from the official image where it needs to get from a finished tagged release, build, and propagation to mirrors.

This helps fast iteration, which in turn helps the actual import and version mismatches. Whatever kickstart merges, the next day we can merge the anaconda part and all works again.

So the pykickstart PR broke the webui part in a way that you understand?

Retroactively, yes, it's pretty clear. The changes I merged required a newer version of pykickstart, which was already tagged so available for our CI, but wasn't in the (older) image that cockpit tests used. Also, the pykickstart version that we need is two versions ahead of what was in the image, so the chance it propagated far enough to be picked up here is lower.

if I get a bit better understanding on why the tests were failing in the first place, without the PR which broke the webui tests, I might be able to help improve the workflow. Is pykickstart updates the problem?

Yes. In a nutshell, anaconda infra uses nightly pykickstart builds, while cockpit tests take it from the official image where it needs to get from a finished tagged release, build, and propagation to mirrors.

This helps fast iteration, which in turn helps the actual import and version mismatches. Whatever kickstart merges, the next day we can merge the anaconda part and all works again.

So the pykickstart PR broke the webui part in a way that you understand?

Retroactively, yes, it's pretty clear. The changes we merged required a newer version of pykickstart, which was already tagged so available for our CI, but wasn't in the (older) image that cockpit tests used. Also, the pykickstart version that we need is two versions ahead of what was in the image, so the chance it propagated far enough to be picked up here is lower.

Is this https://bodhi.fedoraproject.org/updates/FEDORA-2023-eecd218040 the pykickstart version that you want to have in this image in order for the tests to be fixed?

Yup, the 47 one.

Yup, the 47 one.

I just booted from the ISO from #4551 how can I check the pykickstart version?
There are not any pykickstart files installed in /usr/bin

@KKoukiou you can do that by reading /root/lorax-packages.log file (if they are not added by updates-image which pykickstart is not)

But I already verified that the new image is what we are looking for but for some reason it seems that Anaconda is not able to apply updates-image. Not sure why...

@KKoukiou normally grep kickstart /root/lorax-packages.log or if you doubt the actual files, then cat /usr/lib/python3.11/site-packages/pykickstart/__init__.py

@KKoukiou normally grep kickstart /root/lorax-packages.log or if you doubt the actual files, then cat /usr/lib/python3.11/site-packages/pykickstart/__init__.py

thanks! So i just checked and the pykickstart is the exact same version as you expect.

The issue was at the end again on my side. Fixed and made improvement to avoid that again in the future here: rhinstaller/anaconda@d7070a4

Testing what really happens now.

I'm working on more stable fix but we merged this rhinstaller/anaconda#4633 so the result should be green now.

Needs pixel tests updates because new fonts are shipped - to be done shortly by @jkonecny12

PR created: rhinstaller/anaconda#4634