Describe the bug

To Reproduce
Steps to reproduce the behavior:

1. Launch a DR volume with 2 replicas
2. During the restoring, crash one replica
3. When Longhorn restarts/rebuilds the crashed replica but hasn't started the restoring for the replica, crash another restoring replica. (Race here)
4. After the rebuilding replica finishes restore, Longhorn cannot find another RW replica to verify the snapshot chain for this rebuilding replica. Then the DR volume gets stuck there.

Expected behavior
The restarted replica should become the healthy one even if there is no other RW replicas in the volume.

Log

Environment:

• Longhorn version: master-06/30/2021

Additional context

• https://github.com/longhorn/longhorn-manager/blob/27c93ca97bf6966c7a946ef223366f3ac6e70d62/controller/engine_controller.go#L952-L963
• https://github.com/longhorn/longhorn-engine/blob/7f42612eb488b2b829a18711d605323f7a77c50a/pkg/controller/rebuild.go#L64