kubelet cannot find device for dir /var/lib/kubelet in cached partitions map #38337
Description
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "btrfs kubelet"
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
Kubernetes version (use kubectl version): v1.4.6
Environment:
- Cloud provider or hardware configuration: bare metal
- OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)
- Kernel (e.g.
uname -a): Linux vilnus 3.10.0-327.36.3.el7.x86_64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux - Install tools: Ansible script from https://github.com/kubernetes/contrib.git with custom changes.
- Others: Cluster with 4 bare metal nodes, same OS on all nodes
What happened:
kubelet continuously reports following error messages:
Dec 06 01:27:26 vilnus kubelet[2447]: E1206 01:27:26.798004 2447 kubelet.go:2132] Failed to check if disk space is available on the root partition: failed to get fs info for "root": error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 37 in cached partitions map
As (I assume) a side effect heapster pod fails with following error messages:
2016-12-07T22:48:05.074258000Z E1207 22:48:05.073677 1 summary.go:114] error while getting metrics summary from Kubelet antego(192.168.81.104:10255): request failed - "500 Internal Server Error", response: "Internal Error: failed RootFsInfo: error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 36 in cached partitions map"
When kubelet starts there is no device with major:minor ID '0:37':
Dec 06 01:27:21 vilnus kubelet[2447]: I1206 01:27:21.304451 2447 fs.go:117] Filesystem partitions: map[/dev/sda4:{mountpoint:/ major:0 minor:35 fsType:btrfs blockSize:0} /dev/sda2:{mountp
oint:/boot major:8 minor:2 fsType:xfs blockSize:0}]
stat tool does not report device with 0:35 but 0:37 (i.e. 25h/37d):
[root@vilnus rubinstein]# stat /
File: ‘/’
Size: 152 Blocks: 0 IO Block: 4096 directory
Device: 25h/37d Inode: 256 Links: 1
Access: (0555/dr-xr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:root_t:s0
Access: 2016-12-07 23:49:42.368681876 +0100
Modify: 2016-12-06 01:27:50.993617031 +0100
Change: 2016-12-06 01:27:50.993617031 +0100
Birth: -
Here is also /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Nov 8 15:25:05 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=4706dd7f-81aa-4bb6-bcb6-e742aae08456 / btrfs subvol=root 0 0
UUID=72a8c6a4-b442-48ef-a6ad-a8da2146fab6 /boot xfs defaults 0 0
UUID=039B-CD7E /boot/efi vfat umask=0077,shortname=winnt 0 0
UUID=56f454ff-1d8e-4900-b034-1f777df92c30 swap swap defaults 0 0
UUID=4706dd7f-81aa-4bb6-bcb6-e742aae08456 /mnt/disk-4706dd7f-81aa-4bb6-bcb6-e742aae08456 btrfs subvol=/ 0 0
Same errors appears on all nodes and all nodes have / mounted to btrfs partition.
What you expected to happen:
No errors, kubernetes 1.2.0 installed from CentOS 7 package had no such errors.
How to reproduce it (as minimally and precisely as possible):
Kubernetes binaries were installed from here :
https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64
Run kubelet on btrfs root partition.
Anything else do we need to know:
I would like at least to have recommendations how to figure out what exactly issue is.