add enhanced volume and mount logging for block devices #24797

screeley44 · 2016-04-26T13:58:57Z

Adding better logging and debugging for block device volumes and the shared SafeFormatAndMount (aws, gce, flex, rbd, cinder, etc...)

k8s-bot · 2016-04-26T13:59:13Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-04-26T14:00:20Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-04-26T14:00:43Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

screeley44 · 2016-04-26T17:22:01Z

@kubernetes/sig-storage

saad-ali · 2016-04-26T22:04:43Z

pkg/util/mount/mount_linux.go

@@ -107,6 +107,7 @@ func doMount(source string, target string, fstype string, options []string) erro
 	command := exec.Command("mount", mountArgs...)
 	output, err := command.CombinedOutput()
 	if err != nil {
+		glog.Errorf("Mount failed: %w\nMounting arguments: %s %s %s %w\nOutput: %s\n", err, source, target, fstype, options, string(output))


Did you mean %v instead of %w?

@saad-ali - yes, that is what I meant and will be updating accordingly

k8s-bot · 2016-05-11T22:07:41Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

screeley44 · 2016-05-18T16:00:53Z

@saad-ali @pmorie - any more thoughts on this?

screeley44 · 2016-05-18T16:16:55Z

a snippet of some logging for aws ebs:

I0518 12:02:33.082433   16223 mount_linux.go:287] disk /dev/xvdba appears to be unformatted, attempting to format disk: ext4  with options: [-E lazy_itable_init=0,lazy_journal_init=0 -F /dev/xvdba]
I0518 12:02:34.512242   16223 mount_linux.go:292] disk successfully formatted (mkfs): ext4 - /dev/xvdba /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-d56f0667
I0518 12:02:34.535146   16223 aws_ebs.go:292] Successfully mounted /var/lib/kubelet/pods/ebc7822c-1d11-11e6-9195-06d8cc2ef6b3/volumes/kubernetes.io~aws-ebs/ebsvol

saad-ali · 2016-05-20T05:41:59Z

pkg/util/mount/mount_linux.go

@@ -267,6 +269,7 @@ func (mounter *SafeFormatAndMount) formatAndMount(source string, target string,
 	}

 	// Try to mount the disk
+	glog.Infof("attempting to mount disk: %s %s %s", fstype, source, target)


I think this should be V(4) since it is fairly common, and not alarming.

k8s-bot · 2016-06-15T00:35:31Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-github-robot · 2016-07-19T22:17:30Z

This PR hasn't been active in 60 days. Will be closed in 29 days.

You can add 'keep-open' label to prevent this from happening.

saad-ali · 2016-07-19T22:53:16Z

This is a useful PR. In particular logging attempting to format disk is a must-have

@screeley44 could you rebase and address feedback.

screeley44 · 2016-07-20T00:25:06Z

@saad-ali - ack

screeley44 · 2016-07-20T20:50:39Z

@saad-ali - rebased and addressed initial comments

screeley44 · 2016-07-20T20:51:38Z

@saad-ali - might need the "ok-to-test" label

saad-ali · 2016-07-20T21:20:00Z

@k8s-bot ok to test

saad-ali · 2016-07-21T02:42:53Z

pkg/volume/aws_ebs/aws_ebs.go

 	return nil
 }

 func makeGlobalPDPath(host volume.VolumeHost, volumeID string) string {
 	// Clean up the URI to be more fs-friendly
 	name := volumeID
 	name = strings.Replace(name, "://", "/", -1)
+	glog.V(4).Infof(path.Join(host.GetPluginDir(awsElasticBlockStorePluginName), "mounts", name))


This looks wrong

yeah, I think I was using that as debug and didn't remove it, don't think we need to log anything from here, I think I will remove

saad-ali · 2016-07-21T21:48:20Z

@k8s-bot test this issue: #22655

k8s-bot · 2016-07-21T22:22:13Z

GCE e2e build/test passed for commit 11d1289.

k8s-github-robot · 2016-07-21T23:35:52Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-bot · 2016-07-22T00:12:19Z

GCE e2e build/test passed for commit 11d1289.

k8s-github-robot · 2016-07-22T00:12:33Z

Automatic merge from submit-queue

…97-upstream-release-1.3 Automatic merge from submit-queue Automated cherry pick of #24797 upstream release 1.3 Cherry pick #24797 ("add enhanced volume and mount logging for block devices") to upstream release 1.3.

k8s-cherrypick-bot · 2016-07-27T01:30:37Z

Commit found in the "release-1.3" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

…ck-of-#24797-upstream-release-1.3 Automatic merge from submit-queue Automated cherry pick of kubernetes#24797 upstream release 1.3 Cherry pick kubernetes#24797 ("add enhanced volume and mount logging for block devices") to upstream release 1.3.

Automatic merge from submit-queue add enhanced volume and mount logging for block devices Fixes kubernetes#24568 Adding better logging and debugging for block device volumes and the shared SafeFormatAndMount (aws, gce, flex, rbd, cinder, etc...)

googlebot added the cla: yes label Apr 26, 2016

k8s-github-robot assigned saad-ali Apr 26, 2016

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-label-needed labels Apr 26, 2016

saad-ali reviewed Apr 26, 2016
View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 19, 2016

pmorie added sig/storage Categorizes an issue or PR as relevant to SIG Storage. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels May 19, 2016

k8s-github-robot removed the release-note-label-needed label May 19, 2016

saad-ali reviewed May 20, 2016
View reviewed changes

saad-ali added the keep-open label Jul 19, 2016

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 20, 2016

saad-ali reviewed Jul 21, 2016
View reviewed changes

Add volume and mount logging

11d1289

screeley44 changed the title ~~[WIP] add enhanced volume and mount logging for block devices~~ add enhanced volume and mount logging for block devices Jul 21, 2016

saad-ali added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 21, 2016

saad-ali added this to the v1.3 milestone Jul 21, 2016

saad-ali added the cherrypick-candidate label Jul 21, 2016

k8s-github-robot merged commit 4694a6d into kubernetes:master Jul 22, 2016

fabioy added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jul 24, 2016

saad-ali mentioned this pull request Jul 25, 2016

Automated cherry pick of #24797 upstream release 1.3 #29566

Merged

k8s-cherrypick-bot removed the cherrypick-candidate label Jul 27, 2016

This was referenced Aug 1, 2016

1.3 upstream picks openshift/origin#9790

Closed

3.3 Upstream patches for storage openshift/origin#10111

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add enhanced volume and mount logging for block devices #24797

add enhanced volume and mount logging for block devices #24797

screeley44 commented Apr 26, 2016 •

edited

Loading

k8s-bot commented Apr 26, 2016

k8s-bot commented Apr 26, 2016

k8s-bot commented Apr 26, 2016

screeley44 commented Apr 26, 2016

saad-ali Apr 26, 2016

screeley44 Apr 26, 2016

k8s-bot commented May 11, 2016

screeley44 commented May 18, 2016

screeley44 commented May 18, 2016

saad-ali May 20, 2016

k8s-bot commented Jun 15, 2016

k8s-github-robot commented Jul 19, 2016

saad-ali commented Jul 19, 2016

screeley44 commented Jul 20, 2016

screeley44 commented Jul 20, 2016

screeley44 commented Jul 20, 2016

saad-ali commented Jul 20, 2016

saad-ali Jul 21, 2016

screeley44 Jul 21, 2016

saad-ali commented Jul 21, 2016

k8s-bot commented Jul 21, 2016

k8s-github-robot commented Jul 21, 2016

k8s-bot commented Jul 22, 2016

k8s-github-robot commented Jul 22, 2016

k8s-cherrypick-bot commented Jul 27, 2016

add enhanced volume and mount logging for block devices #24797

add enhanced volume and mount logging for block devices #24797

Conversation

screeley44 commented Apr 26, 2016 • edited Loading

k8s-bot commented Apr 26, 2016

k8s-bot commented Apr 26, 2016

k8s-bot commented Apr 26, 2016

screeley44 commented Apr 26, 2016

saad-ali Apr 26, 2016

Choose a reason for hiding this comment

screeley44 Apr 26, 2016

Choose a reason for hiding this comment

k8s-bot commented May 11, 2016

screeley44 commented May 18, 2016

screeley44 commented May 18, 2016

saad-ali May 20, 2016

Choose a reason for hiding this comment

k8s-bot commented Jun 15, 2016

k8s-github-robot commented Jul 19, 2016

saad-ali commented Jul 19, 2016

screeley44 commented Jul 20, 2016

screeley44 commented Jul 20, 2016

screeley44 commented Jul 20, 2016

saad-ali commented Jul 20, 2016

saad-ali Jul 21, 2016

Choose a reason for hiding this comment

screeley44 Jul 21, 2016

Choose a reason for hiding this comment

saad-ali commented Jul 21, 2016

k8s-bot commented Jul 21, 2016

k8s-github-robot commented Jul 21, 2016

k8s-bot commented Jul 22, 2016

k8s-github-robot commented Jul 22, 2016

k8s-cherrypick-bot commented Jul 27, 2016

screeley44 commented Apr 26, 2016 •

edited

Loading