Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Projected volume plugin does not block on data fetch error #47020

Closed
liggitt opened this issue Jun 6, 2017 · 3 comments
Closed

Projected volume plugin does not block on data fetch error #47020

liggitt opened this issue Jun 6, 2017 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Milestone
v1.7

Comments

@liggitt
Copy link
Member

liggitt commented Jun 6, 2017 •
edited
Loading

When collecting data for a projected volume, if an error is encountered, the plugin logs, but then continues setting up the volume and returns as if it were successful:

E0604 10:31:19.324314    3580 projected.go:265] Couldn't get configMap e2e-tests-projected-mnbzg/projected-configmap-test-volume-f1dfa503-4910-11e7-8d9d-0242ac110002: User "system:node:e2e-34198-minion-group-qnk2" cannot get configmaps in the namespace "e2e-tests-proj│
E0604 10:31:19.324409    3580 projected.go:197] Error preparing data for projected volume projected-configmap-volume for pod e2e-tests-projected-mnbzg/pod-projected-configmaps-f1f51bf8-4910-11e7-8d9d-0242ac110002: User "system:node:e2e-34198-minion-group-qnk2" cannot │
I0604 10:31:19.324620    3580 atomic_writer.go:142] pod e2e-tests-projected-mnbzg/pod-projected-configmaps-f1f51bf8-4910-11e7-8d9d-0242ac110002 volume projected-configmap-volume: no update required for target directory /var/lib/kubelet/pods/f2278654-4910-11e7-9d2f-420│
I0604 10:31:19.324667    3580 operation_generator.go:472] MountVolume.SetUp succeeded for volume "projected-configmap-volume" (UniqueName: "kubernetes.io/projected/f2278654-4910-11e7-9d2f-42010a800002-projected-configmap-volume") pod "pod-projected-configmaps-f1f51bf8│

This is because the error returned from collectData() does not short-circuit the SetUp function:

	data, err := s.collectData()
	if err != nil {
		glog.Errorf("Error preparing data for projected volume %v for pod %v/%v: %s", s.volName, s.pod.Namespace, s.pod.Name, err.Error())
	}

This means that a pod can start before the configmaps/secrets intended to populate its projected volume exist.
@liggitt liggitt added kind/bug Categorizes issue or PR as related to a bug. sig/storage Categorizes an issue or PR as relevant to SIG Storage. labels Jun 6, 2017
This was referenced Jun 6, 2017
Merged
Merged
@liggitt
Copy link
Member Author

liggitt commented Jun 6, 2017

@kubernetes/sig-storage-bugs

@liggitt
Copy link
Member Author

liggitt commented Jun 6, 2017

cc @jpeeler @pmorie

@liggitt
Copy link
Member Author

liggitt commented Jun 6, 2017 •
edited
Loading

With the Node authorizer in place (#46796), it's a race between a node observing a newly bound pod, and the authorizer noticing the node now has access to secrets/configmaps for the pod... the other volume plugins just retry once or twice until they can get the data, but for projected volumes, if it fails to retrieve the object the very first time, it just continues happily setting up the pod without the requested data, and the tests fail.

@liggitt liggitt added this to the v1.7 milestone Jun 6, 2017
k8s-github-robot pushed a commit that referenced this issue Jun 7, 2017
Merge pull request #47021 from liggitt/projected-err
5c9b9ae 
Automatic merge from submit-queue (batch tested with PRs 46997, 47021)

Block on projected volume data fetch failure

Fixes #47020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Projects
None yet
Milestone
v1.7
Development

No branches or pull requests
1 participant
@liggitt