Create a custom 404 page with absolute URLs hardcoded.

pip install sphinx-notfound-page

Add this extension in your conf.py file as:

extensions = [
 # ... other extensions here

 'notfound.extension',
]

Check out the full documentation at https://sphinx-notfound-page.readthedocs.io/

Strongly based on @ericholscher's solution from readthedocs/readthedocs.org#353