Syntax errors crash the process even with --reload #2244
Closed
Description
Running gunicorn --reload doesn't work as intended. Syntax errors cause it to crash instead of emitting the error and watching for further updates.
[2020-01-21 21:01:06 +0100] [19910] [INFO] Starting gunicorn 20.0.4
[2020-01-21 21:01:06 +0100] [19910] [INFO] Listening at: http://127.0.0.1:8000 (19910)
[2020-01-21 21:01:06 +0100] [19910] [INFO] Using worker: sync
[2020-01-21 21:01:06 +0100] [19913] [INFO] Booting worker with pid: 19913
[2020-01-21 21:01:10 +0100] [19913] [INFO] Worker reloading: /home/liquid/tmp/gunicorn/app.py modified
[2020-01-21 21:01:10 +0100] [19913] [INFO] Worker exiting (pid: 19913)
[2020-01-21 21:01:10 +0100] [19931] [INFO] Booting worker with pid: 19931
[2020-01-21 21:01:10 +0100] [19931] [ERROR] invalid syntax (app.py, line 4)
Traceback (most recent call last):
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
return self.load_wsgiapp()
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
mod = importlib.import_module(module)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 724, in exec_module
File "<frozen importlib._bootstrap_external>", line 860, in get_code
File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/liquid/tmp/gunicorn/app.py", line 4
def index():
^
SyntaxError: invalid syntax
[2020-01-21 21:01:10 +0100] [19931] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
return self.load_wsgiapp()
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
mod = importlib.import_module(module)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 724, in exec_module
File "<frozen importlib._bootstrap_external>", line 860, in get_code
File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/liquid/tmp/gunicorn/app.py", line 4
def index():
^
SyntaxError: invalid syntax
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
self.load_wsgi()
File "/home/liquid/tmp/gunicorn/venv/lib/python3.7/site-packages/gunicorn/workers/base.py", line 157, in load_wsgi
self.reloader.add_extra_file(exc_val.filename)
AttributeError: 'NoneType' object has no attribute 'add_extra_file'
[2020-01-21 21:01:10 +0100] [19931] [INFO] Worker exiting (pid: 19931)
[2020-01-21 21:01:10 +0100] [19910] [INFO] Shutting down: Master
[2020-01-21 21:01:10 +0100] [19910] [INFO] Reason: Worker failed to boot.
In my understanding when a file gets updated a new worker is initialized and the old one is killed. When the application contains a syntax error the worker crashes because the reloader which is being used while handling a syntax error isn't initialized when load_wsgi runs.
This is where the error occurs due to self.reloader being None:
https://github.com/benoitc/gunicorn/blob/master/gunicorn/workers/base.py#L158
I'm not quite sure what this line does. I haven't looked into the reloader logic yet but wouldn't it track that file already?
This is where the ordering problem occurs:
https://github.com/benoitc/gunicorn/blob/master/gunicorn/workers/base.py#L120-L135
Moving the call to load_wsgi after the reloader setup fixes the issue. I'm not sure if this won't cause other ordering issues though.
Should I submit a pull request with that change?