Skip to content

[BUG] mkfs.xfs: libxfs_device_zero write failed: Input/output error #8604

Open
@reysravanga

Description

Describe the bug

While mounting the PVC to pod longhorn failed to mount with timeout error.

To Reproduce

  1. Longhorn failed to mount the PVCto pod with following error: volume attachment is being deleted
  2. scaledown the pod replicas zero and detach the volume
  3. Volume turned in to faulty mode - to fix faulty volume salvage the volume
  4. scaled up the pod replicas to 1 - envounted below error :
 Warning  FailedMount             6m20s  kubelet                  MountVolume.MountDevice failed for volume "pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf" failed: type:("xfs") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf/globalmount") options:("defaults") errcode:(exit status 1) output:(error reading existing superblock: No data available
mkfs.xfs: pwrite failed: No data available
libxfs_bwrite: write failed on (unknown) bno 0x1116ffff00/0x100, err=61
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: No data available
libxfs_bwrite: write failed on (unknown) bno 0x0/0x100, err=61
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: No data available
libxfs_bwrite: write failed on xfs_sb bno 0x0/0x8, err=61
mkfs.xfs: Releasing dirty buffer to free list!
mkfs.xfs: libxfs_device_zero write failed: Input/output error
meta-data=/dev/longhorn/pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf isize=512    agcount=35, agsize=268435455 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=9175040000, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
)
  Warning  FailedMount  57s (x3 over 5m30s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[storage-cloned], unattached volumes=[kube-api-access-7p6b4 storage-cloned]: timed out waiting for the condition
  Warning  FailedMount  7s (x10 over 6m20s)  kubelet  MountVolume.MountDevice failed for volume "pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf" : rpc error: code = InvalidArgument desc = volume pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf hasn't been attached yet

Expected behavior

longhorn volume should be mounted without any error.

Environment

  • Longhorn version:
  • Impacted volume (PV):
  • Installation method (e.g. Rancher Catalog App/Helm/Kubectl): helm
  • Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: bearmetal k8s
    • Number of control plane nodes in the cluster: 1
    • Number of worker nodes in the cluster: 2
  • Node config
    • OS type and version: ubuntu 22.04
    • Kernel version: Linux node1 5.4.0-174-generic EXT4-fs (sdb): Remounting filesystem read-only #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
    • CPU per node: 48
    • Memory per node:
    • Disk type (e.g. SSD/NVMe/HDD): SSD
    • Network bandwidth between the nodes (Gbps):
  • Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
  • Number of Longhorn volumes in the cluster: 64

Additional context

Metadata

Assignees

No one assigned

    Labels

    area/resilienceSystem or volume resiliencekind/bugrequire/backportRequire backport. Only used when the specific versions to backport have not been definied.require/qa-review-coverageRequire QA to review coverage

    Type

    No type

    Projects

    • Status

      In Progress
    • Status

      New Issues

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions