No alert when MISP is timing out for correlation / enrichment #26
Open
Description
Launching a search query for a domain or IP on a MISP instance is resource intensive and often cause the query to timeout.
Currently, pDNSSOC-cli stops processing the alert when this happens.
Instead, it should proceed and create an alert in alerts/matches.json, just with the information available and an indication that MISP timed out (again).
CRITICAL:pymisp:Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.4.182 - Python 3.10', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'MISP-session-af16f=<SNIP>', 'Content-Length': '317', 'content-type': 'application/json'}
Request body:
{"returnFormat": "json", "type": "domain", "withAttachments": 0, "metadata": 0, "enforceWarninglist": 1, "to_ids": 1, "includeEventUuid": 0, "includeEventTags": 0, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0, "excludeDecayed": 0}
Response (if any):
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
Exception in thread Thread-3 (daemonized_retro):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/pdnssoccli/subcommands/daemonize.py", line 37, in daemonized_retro
ctx.invoke(correlate, **{'retro_lookup': True, 'files':[correlation_config['archive_dir']]})
File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pdnssoccli/subcommands/correlate.py", line 163, in correlate
attributes = misp.search(controller='attributes', type_attribute='domain', to_ids=1, pythonify=True, **args)
File "/usr/local/lib/python3.10/dist-packages/pymisp/api.py", line 2722, in search
normalized_response = self._check_json_response(response)
File "/usr/local/lib/python3.10/dist-packages/pymisp/api.py", line 3658, in _check_json_response
r = self._check_response(response, expect_json=True)
File "/usr/local/lib/python3.10/dist-packages/pymisp/api.py", line 3676, in _check_response
raise MISPServerError(f'Error code 500:\n{response.text}')
pymisp.exceptions.MISPServerError: Error code 500:
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>