Description
When running my app (https://github.com/ArthurBrussee/brush), training proceeds steadily for a while, until the app crashes. The symptons seem hard to pin down, it happens fairly randomly. Just before the crash the Vulkan validation layer spits out a bunch of errors about semaphores. Most tellingly some semaphore value seems to be u64::MAX which Vulkan trips over.

This causes a device loss (possibly?) after which wgpu crashes because of #6378, I think.

I have not been able to reproduce this on Metal, not sure about Vulkan + Linux.

Extra materials

Log with validation errors
log.txt

Platform
wgpu (trunk or 23.0 or 23.1 repro), windows 11, Vulkan, 4070 on 566.36.