Will have a look when i am back home. But remember that zooming in the magnifier does not zoom an image. It zooms the UI which means it's a vector graphic. When taking a screenshot it produces a pixel based image. You will never get the same zoom quality when comparing vector to pixel images.
Taking a screenshot of a single control should be possible though. Have you tried that?

I know, I was thinking that the screenshot feature first zooms the vector wpf according to the dpi setting, and then takes that screenshot. What else is the dpi setting otherwise for? That's even what the original contributor claims where I found the feature described first: https://qmino.wordpress.com/2010/09/20/high-res-screenshots-of-wpf-apps/

Interestingly, the screenshot feature does vector zoom a bit, just not anywhere near the claimed dpi for a larger window. When "screenshotting" the radio button control only, the vector zoom is much better, but it introduces a new problem: The image gets distorted. See how the radio button is not round anymore?

Compare that with the tree preview:

My guess is: The screenshot feature tries to zoom but gets somehow limited by the actual screen size. If that's a WPF renderer limitation, the magnifier and preview seem not to be limited by it. So a solution could be to take the zoomed screenshot in multiple zoomed parts and stitch the raster image together. Similar as to how high-dpi browser screenshot tools do it (eg. Chrome -> Print full screen).

Ok. That looks wrong. Will have a look in about a week.

So a week was a bit optimistic.
The screenshots are actually really high dpi which means the image is just create with greater dimensions.
So if you choose 600 dpi and your control has a width of 800px the image will have 5000px.
I still had no time to have a look at the distorted issue you posted.

@djkrose i finally found some time to investigate this further and indeed the quality is way below what can be expected when increasing the DPI for the image.
The poor quality is caused by some conversion in the capture code.

Glad you could find it. Thanks for investigating!

Finally found a solution.
Using 4800 DPI (left before, right after)

Just have to do some finishing before i push the code changes.