Tracking "Connection reset by peer" issue with Copernicus zipper API #30
Open
Description
In an attempt to bypass the continual issues with incorrect MD5 checksums #29, I deployed an updated version of the downloader with checksum validation disabled. But we immediately began to see high rates of errors for
[ERROR] ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/var/task/handler.py", line 74, in handler
download_file(
File "/var/task/handler.py", line 213, in download_file
Body=response.raw.read(),
File "/var/task/urllib3/response.py", line 593, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/var/lang/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/var/task/urllib3/response.py", line 461, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
I submitted the following issue to the Copernicus support system
We are seeing high rates of our downloads being interrupted with "Connection reset by peer" errors indicating TCP interruptions by the ESA Copernicus zipper API. Our requests are concurrency limited so they do not exceed the 15 connections we have been allotted.
ref https://github.com/NASA-IMPACT/hls_development/issues/148