Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: deflake the blockfile testsuite #8696

Merged
merged 5 commits into from
Jun 26, 2023
Merged

Conversation

fuweid
Copy link
Member

@fuweid fuweid commented Jun 15, 2023

mount supports direct-io option for loop block device. In the original loop setup, the Direct flag doesn't work even if the syscall returns successfully. In the patch, I use ioctl to set the direct-io and return error if the old kernel doesn't support this.

It's a cleanup for the snapshot test. The detach umount might cause unexpected issue. It's hard to debug in the github action. The umount without detach might be slow but it's easy to figure out the flaky cases.

The patch is trying to deflake the blockfile snapshot test case. The view-type snapshot mounts the device with readonly. If the filesystem needs to recovery, the kernel will return the EROFS and the error in dmesg is like.

2023-05-25T05:31:07.6688425Z [Thu May 25 05:29:39 2023] EXT4-fs (loop10): INFO: recovery required on readonly filesystem
2023-05-25T05:31:07.6688980Z [Thu May 25 05:29:39 2023] EXT4-fs (loop10): write access unavailable, cannot proceed (try mounting with noload)

For this case, I introduce a viewHook helper for test. It's used to mount the snapshot with write permission, so that the kernel can handle recovery at first. Then the ro mount won't fail.

And using 8MiB is to reduce IO pressure. :)

Even if we use direct-io for loop, we still have the unsync error, like

 ~ /onlyme/file-13(mode: 777, uid: 0, gid: 0) -> /onlyme/file-13(mode: 755, uid: 0, gid: 0)

So I change the fs/fstest to force sync for new file. Hopefully, containerd/continuity#228 it's acceptable.


#!/usr/bin/env bash

set -euo pipefail

for i in {1..101}
do
   echo "$i times"
   sudo -E PATH=$PATH make root-test EXTRA_TESTFLAGS="-failfast"
done

I run this branch for test in azure Standard D4s v4. It's happy. :)

fuweid added 3 commits June 15, 2023 23:51
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
* Use direct-io mode to reduce IO.

* Add testViewHook helper to recovery the backing file since the ext4
  might need writable permission to handle recovery. If the backing file
  needs recovery and it's for View snapshot, the readonly mount will
  cause error.

* Use 8 MiB as capacity to reduce the IO.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
@k8s-ci-robot
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@fuweid fuweid marked this pull request as ready for review June 16, 2023 05:21
@kzys
Copy link
Member

kzys commented Jun 16, 2023

@fuweid Approved and merged continuity changes. You don't have to modify vendor/ anymore.

@fuweid fuweid force-pushed the deflaky-blockfile branch 2 times, most recently from f51291f to 2aed141 Compare June 17, 2023 00:31
Pin it with 1e0d26eb2381594984ee80989c9c229dbd930d9f

Signed-off-by: Wei Fu <fuweid89@gmail.com>
@fuweid fuweid force-pushed the deflaky-blockfile branch from 2aed141 to 59b0b39 Compare June 17, 2023 00:36
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants