kubelet race condition: concurrent map writes #48045
Assignees
Labels
kind/bug
Categorizes issue or PR as related to a bug.
priority/important-soon
Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
sig/node
Categorizes an issue or PR as relevant to SIG Node.
sig/storage
Categorizes an issue or PR as relevant to SIG Storage.
Milestone
Comments
k8s-ci-robot
added
kind/bug
luxas
added
the
priority/important-soon
|
cc @bradfitz for Golang triaging |
|
Marking 1.7 until this is properly triaged. |
|
Yes, this may be a release blocker. |
dchen1107
added
sig/storage
This isn't a Go bug. This is the best-effort always-on race detector in the Go runtime. It has no false positives. If you get this crash, it's a real race that needs to be fixed. |
|
I have been working on a fix for the last couple of hours. The code is spaghetti at this point and I'm trying to clean it up along the way. |
|
/assign I'll be backup |
vdemeester
pushed a commit
to vdemeester/kubernetes
that referenced
this issue
Jun 28, 2017
Automatic merge from submit-queue (batch tested with PRs 48123, 48079) [Kubelet] Fix race condition in container manager **What this PR does / why we need it**: This fixes a race condition where the container manager capacity map was being updated without synchronization. It moves the storage capacity detection to kubelet initialization, which happens serially in one thread. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes kubernetes#48045 **Release note**: ```release-note Fixes kubelet race condition in container manager. ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
kubelet crashed:
fatal error: concurrent map writesWhat you expected to happen:
kubelet would function normally
How to reproduce it (as minimally and precisely as possible):
kubeadm init --kubernetes-version v1.7.0-rc.1wherekubeadm,kubectlandkubeletbinaries on host also are of versionv1.7.0-rc.1Anything else we need to know?:
This was on an arm64 machine, but I don't think it's arm64 related at a first glance.
After kubelet had failed a couple of times in a systemd crashloop, it eventually started up and worked fine.
Environment:
kubectl version): v1.7.0-rc.1uname -a):Linux pi5 4.9.13-bee42-v8 #1 SMP PREEMPT Fri Mar 3 16:42:37 UTC 2017 aarch64 GNU/Linuxcc @kubernetes/sig-node-bugs
The text was updated successfully, but these errors were encountered: