Quote from the fish help (3.6.0): "Specifying -a or --all without -M or --mode erases all binds in all modes regardless of sequence." (emphasis mine)

I think the intention was to erase ! and $ from all modes? Instead its just erasing every other plugins keybindings