Phone Calls are automatically routed to Airpods Max even when I'm not wearing them

Being a complete doofus, I lost the "smart" cover for my Airpods PRo Max. Now, when I receive a call on my iPhone 14 Pro Max, the audio is automatically routed through my Airpods Max (if they are in BT range, obvs). This is really annoying because I then have to manually disconnect them for EVERY call I receive, if I don't want to use the APM for the call. Is there a way to stop this from happening? Like, If I was wearing the APMs that would be cool, but often they are in a different room. Has anyone got any suggestions?

iOS 17.5.1