[BUG] mkfs.xfs: libxfs_device_zero write failed: Input/output error #8604
Open
Description
Describe the bug
While mounting the PVC to pod longhorn failed to mount with timeout error.
To Reproduce
- Longhorn failed to mount the PVCto pod with following error: volume attachment is being deleted
- scaledown the pod replicas zero and detach the volume
- Volume turned in to faulty mode - to fix faulty volume salvage the volume
- scaled up the pod replicas to 1 - envounted below error :
Warning FailedMount 6m20s kubelet MountVolume.MountDevice failed for volume "pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf" failed: type:("xfs") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf/globalmount") options:("defaults") errcode:(exit status 1) output:(error reading existing superblock: No data available
mkfs.xfs: pwrite failed: No data available
libxfs_bwrite: write failed on (unknown) bno 0x1116ffff00/0x100, err=61
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: No data available
libxfs_bwrite: write failed on (unknown) bno 0x0/0x100, err=61
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: No data available
libxfs_bwrite: write failed on xfs_sb bno 0x0/0x8, err=61
mkfs.xfs: Releasing dirty buffer to free list!
mkfs.xfs: libxfs_device_zero write failed: Input/output error
meta-data=/dev/longhorn/pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf isize=512 agcount=35, agsize=268435455 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=0 bigtime=0 inobtcount=0
data = bsize=4096 blocks=9175040000, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
)
Warning FailedMount 57s (x3 over 5m30s) kubelet Unable to attach or mount volumes: unmounted volumes=[storage-cloned], unattached volumes=[kube-api-access-7p6b4 storage-cloned]: timed out waiting for the condition
Warning FailedMount 7s (x10 over 6m20s) kubelet MountVolume.MountDevice failed for volume "pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf" : rpc error: code = InvalidArgument desc = volume pvc-e9b4c5e1-1c5d-4428-bd0e-a02167136cbf hasn't been attached yet
Expected behavior
longhorn volume should be mounted without any error.
Environment
- Longhorn version:
- Impacted volume (PV):
- Installation method (e.g. Rancher Catalog App/Helm/Kubectl): helm
- Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: bearmetal k8s
- Number of control plane nodes in the cluster: 1
- Number of worker nodes in the cluster: 2
- Node config
- OS type and version: ubuntu 22.04
- Kernel version: Linux node1 5.4.0-174-generic EXT4-fs (sdb): Remounting filesystem read-only #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
- CPU per node: 48
- Memory per node:
- Disk type (e.g. SSD/NVMe/HDD): SSD
- Network bandwidth between the nodes (Gbps):
- Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
- Number of Longhorn volumes in the cluster: 64
Additional context
Metadata
Assignees
Labels
Type
Projects
Status
In Progress
Status
New Issues