I don't know what others thoughts on this are, but scipy is only used in numerical_inverse as is a pretty large package to pull in if you aren't ever planning on using numerical_inverse.

On asdf, afaik you aren't ever going to want to use the asdf related functionality of gwcs unless you already have asdf installed? i.e. if you are opening or saving an asdf file with the asdf API which includes a gwcs component, so the asdf dependency for gwcs could be made into a test only requirement?

If there are no strong objections to this I will open a PR?