Describe the bug (🐛 if you encounter this issue)

Replica auto-balance doesn't respect the node selector and causes a re-balance loop.

To Reproduce

Steps to reproduce the behavior:

1. Have 3 node cluster (node-1, node-2, node-3)
2. Tag node-1, node-2 (select-tag)
3. Tag all nodes as different zones (zone-1, zone-2, zone-3)
4. Create volume with node-selector (select-tag)
5. Enable replica auto-rebalance
6. Replica auto-rebalance attempts to schedule a new replica to zone-3, but couldn't due to the volume node selector.

Expected behavior

Rebalance should only check in zone-1 and zone-2 due to the node selector.

Log or Support bundle

None

Environment

• Longhorn version: day-1 issue

Additional context

https://github.com/longhorn/longhorn-manager/blob/cd31ec6e46ae76e56758382503d2ef5082f93ba1/controller/volume_controller.go#L2229