cc @innobead @joshimoo

cc @longhorn/qa

For longhorn's current design, we quite rely on saving some files persistent on the host, so it's necessary to have some error handling and self-healing/recovering when the persistent data on the host has some problems like getting deleted or empty like in this case.

For QA, need to have more negative testing to cover this kind of external interrupt to test Longhorn resilience.

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

1. Create a pod using Longhorn volume
2. Write some data to the volume then get the md5sum
3. Delete the pod and wait for the volume detached
4. Randomly pick up replicas and manually delete or empty the volume meta file in this replica data path
5. Recreate the pod and wait for the volume attached
6. Check if the volume is Healthy after the volume attached
7. Check if data is not corrupted
8. Check if r/w to volume is ok in pod

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at
  fix(replica): volume metafile deleted or empty longhorn-engine#775

• Which areas/issues this PR might have potential impacts on?
  Area/engine

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)
  test(replica): volume metafile deleted or empty longhorn-tests#1214

Verified in longhorn master 207e74 with test steps
Result Pass

After create pod with longhorn volume then delete the pod let volume in detach state:

1. If volume.meta were removed or emptied in replica file path, replicas will recovery soon after pod recreated. Volume will attach to node and data still kept consistent if volume have replica with correct meta file
2. But if volume-head-001.img.meta were also removed/emptied in some replicas, after pod recreated, volume will keep in Degraded status, replicas which had file volume-head-001.img.meta modified, will keep in fail status. Because volume-head-001.img.meta is not this ticket's scope, maybe we can consider handle those situation in the future.