Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Immucore unable to locate COS_STATE partition after installation of image larger than 5Gb on Kairos v3.1.3 #2950

Closed
robin-ny opened this issue Oct 17, 2024 · 3 comments
Closed

Immucore unable to locate COS_STATE partition after installation of image larger than 5Gb on Kairos v3.1.3 #2950

robin-ny opened this issue Oct 17, 2024 · 3 comments
Labels
bug Something isn't working triage Add this label to issues that should be triaged and prioretized in the next planning call unconfirmed

Comments

@robin-ny
Copy link

Kairos version:

PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
KAIROS_VERSION_ID="v3.1.3"
KAIROS_ARTIFACT="kairos-ubuntu-24.04-core-amd64-generic-v3.1.3"
KAIROS_VARIANT="core"
KAIROS_VERSION="v3.1.3"
KAIROS_PRETTY_NAME="kairos-core-ubuntu-24.04 v3.1.3"
KAIROS_MODEL="generic"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_FLAVOR_RELEASE="24.04"
KAIROS_TARGETARCH="amd64"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_FLAVOR="ubuntu"
KAIROS_FAMILY="ubuntu"
KAIROS_RELEASE="v3.1.3"
KAIROS_ID="kairos"
KAIROS_NAME="kairos-core-ubuntu-24.04"
KAIROS_ID_LIKE="kairos-core-ubuntu-24.04"
KAIROS_IMAGE_REPO="quay.io/kairos/ubuntu:24.04-core-amd64-generic-v3.1.3"
KAIROS_IMAGE_LABEL="24.04-core-amd64-generic-v3.1.3"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"

CPU architecture, OS, and Version:

Linux live 6.8.0-47-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug

I build two images based on the Kairos version shown above (quay.io/kairos/ubuntu:24.04-core-amd64-generic-v3.1.3), which contain additional software:

REPOSITORY                                         TAG                               IMAGE ID             CREATED         SIZE
internal.registry.net/kairos/ubuntu                24.04-core-amd64-small-v3.1.3       661bdd139b33   19 hours ago    3.8GB
internal.registry.net/kairos/ubuntu                24.04-core-amd64-large-v3.1.3       5d2ca0fa0e9e     19 hours ago    5.1GB

The image 24.04-core-amd64-small-v3.1.3 can be installed and booted without any issues.
The image 24.04-core-amd64-large-v3.1.3 can also be installed without any errors, however on boot the partition COS_STATE can't seem to be found by immucore, which leads to reboot cycling:
image

The install section in the cloud config looks as following:

#cloud-config
install:
  grub_options:
    extra_cmdline: "rd.immucore.debug"
  device: "auto"
  reboot: false
  poweroff: true
  auto: true
  source: oci:internal.registry.net/kairos/ubuntu:24.04-core-amd64-large-v3.1.3
  partitions:
    oem:
      size: 512
      fs: ext4
    recovery:
      size: 20000
      fs: ext4
    state:
      size: 40000
      fs: ext4
    persistent:
      size: 0
  system:
    size: 8000
  passive:
    size: 8000
  recovery-system:
    size: 8000

An Image with the same software installed but using Kairos version v3.1.2 (quay.io/kairos/ubuntu:24.04-core-amd64-generic-v3.1.2) boots normally.

REPOSITORY                                         TAG                               IMAGE ID             CREATED         SIZE
internal.registry.net/kairos/ubuntu                24.04-core-amd64-large-v3.1.2       f0a42a621130   24 hours ago    5.14GB

To Reproduce

  1. Build a Kairos Image >5GB based on quay.io/kairos/ubuntu:24.04-core-amd64-generic-v3.1.3
  2. Install the image
  3. Boot and see errors

Expected behavior

Images using Kairos v3.1.3 >5Gb should boot normally.

Additional context
Installation has been tested both on a physical machine and on a vm, with the same results.
@robin-ny robin-ny added bug Something isn't working triage Add this label to issues that should be triaged and prioretized in the next planning call unconfirmed labels Oct 17, 2024
@jimmykarily jimmykarily moved this to In Progress 🏃 in 🧙Issue tracking board Oct 21, 2024
@Itxaka
Copy link
Member

Itxaka commented Oct 22, 2024 •
edited
Loading

mmmh, looks like there is some size issues

Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2CEFF841-4C1D-5A14-8E46-ED3AEB621093

Device         Start               End           Sectors  Size Type
/dev/vda1       2048              4095              2048    1M BIOS boot
/dev/vda2       4096           1052671           1048576  512M Linux filesystem
/dev/vda3    1052672          42012671          40960000 19.5G Linux filesystem
/dev/vda4   42012672         123932671          81920000 39.1G Linux filesystem
/dev/vda5  123932672 36028797102847999 36028796978915328   16E Linux filesystem

I dont think my disk has 16 Exabytes, last time I checked at least :D

this seems to be a side effect of something breaking. With master I dont even get to install it fails to find COS_STATE

2024-10-22T09:01:30Z INF Partitioning device...
2024-10-22T09:01:30Z INF Creating partition table for partition type gpt
2024-10-22T09:01:30Z INF Created partition table for partition type gpt
2024-10-22T09:01:30Z DBG Running cmd: 'udevadm trigger'
2024-10-22T09:01:30Z DBG Running cmd: 'udevadm settle'
2024-10-22T09:01:30Z DBG Formatting partition: COS_OEM
2024-10-22T09:01:30Z DBG Running cmd: 'mkfs.ext4 -L COS_OEM /dev/vda2'
2024-10-22T09:01:30Z DBG Formatting partition: COS_RECOVERY
2024-10-22T09:01:30Z DBG Running cmd: 'mkfs.ext4 -L COS_RECOVERY /dev/vda3'
2024-10-22T09:01:30Z DBG Formatting partition: COS_STATE
2024-10-22T09:01:30Z ERR Failed finding partition COS_STATE by partition label: lstat /dev/disk/by-partlabel/state: no such file or directory
2024-10-22T09:01:30Z DBG Running cmd: 'mkfs.ext4 -L COS_STATE '
mke2fs 1.47.0 (5-Feb-2023)
The file  does not exist and no size was specified.

2024-10-22T09:01:30Z ERR Failed formatting partition: exit status 1
1 error occurred:
	* exit status 1

@Itxaka
Copy link
Member

Itxaka commented Oct 22, 2024

oh well, yes, If I go over the total disk, then it breaks, makes sense 🤦

Adjusting the sizes to a more decent thing seems to work here:


Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2CEFF841-4C1D-5A14-8E46-ED3AEB621093

Device        Start      End  Sectors  Size Type
/dev/vda1      2048     4095     2048    1M BIOS boot
/dev/vda2      4096  1052671  1048576  512M Linux filesystem
/dev/vda3   1052672 21532671 20480000  9.8G Linux filesystem
/dev/vda4  21532672 62492671 40960000 19.5G Linux filesystem
/dev/vda5  62492672 83884031 21391360 10.2G Linux filesystem

That booted without issues....

@robin-ny do you have the install logs? Maybe it failed somewhere during the install and the install did not fail for some reason?

The logs may be under /var/log/kairos

@robin-ny
Copy link
Author

@Itxaka I found the issue in the meantime. It had nothing to do with image size, sorry for the red herring 🤦
From 24.04-core-amd64-generic-v3.1.2 to 24.04-core-amd64-generic-v3.1.3 the ubuntu kernel got updated from 6.8.0-41 to 6.8.0-47

In my image I install nvidia drivers and I did not update the installation to reflect the kernel change.

...
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    nvidia-driver-535 \
    linux-modules-nvidia-535-6.8.0-41-generic\
...

After changing the package the system booted normally 🥳

@github-project-automation github-project-automation bot moved this from Todo 🖊 to Done ✅ in 🧙Issue tracking board Oct 22, 2024
@jimmykarily jimmykarily mentioned this issue Oct 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working triage Add this label to issues that should be triaged and prioretized in the next planning call unconfirmed
Projects
🧙Issue tracking board
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Itxaka @robin-ny