Well, it turned out to be something very simple and quite frankly quite stupid. I only had a 'real' Apple keyboard and no 'Magic mouse'.
All the time I used a bluetooth mouse, which was fine as long I was not trying to downgrade.
But as soon as I started that proces, the non-Apple mouse did not work and the Apple keyboard stopped working at the last step.
So I did the disk utility checks first (no problems there luckily) and then borrowed an Apple mouse from someone nearby. And that did the trick! Not immediately, I first had to 'get them to know each other' (meaning: start the iMac with Catalina, its current version) before starting the downgrade.
When I reached the end and - with the Apple mouse - I clicked on 'install on Apple' it said (....): "you cannot install this version because there's a newer one installed". Quite frustrated and not hindered by any doubts I used the disk utiliy - while still in the downgrade proces - to erase 'Apple' and then it started to install High Sierra. I am typing this while it still has 7 minutes to go, but I am optimistic that I will have a downgraded iMac.
And for someone who reads this when trying the same thing: this iMac from 2012 does not 'see' our Wifi (we have three different routers, probably too new) so I use the hotspot on mytelephone. Because once you start the downgrade proces, it cannot use an ethernet cable. It can only work with wifi.