Even though my VS process is arm64:

Building libraries projects results in an x64 compiler process being used:

Digging deeper, it appears this is issue is specific to dotnet/runtime infrastructure which by default uses an OOB compiler that doesn't support arm64:

I should stress that this has major impact on build performance. For reference, building System.Text.Json.sln from the CLI using the arm64 sdk takes 32 seconds whereas building the same solution from VS takes 8 minutes.