Fix run_with_server.py to run on non-Linux python platforms. #3003
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
For some reason,
run_with_server.py
barfs on macOS with Python 3.9 installed from Homebrew. If I run it, I get:and no saved results.
Looking at the code, it seems that on macOS, something is different about how
global
variables are inherited by the forked processes in themultuprocessing.Pool
. Onmaster
,initialize(parsed_args)
is only called once in the primary process, but it seems that all the workers fully inherit all the global variables. On macOS with Python 3.9, the global variables aren't present when the worker runs, and it barfs with the error printed above.This PR makes a few slight modifications:
multiprocessing.Pool
- each worker performs its own initialization call (onmaster
, I think it's simply a mistake in invocation doinginitializer=initialize(parsed_args)
asinitialize
returns nothing).multiprocessing.Value
into the master process, and passes in copies to the workers.With these changes,
run_with_server.py
works for me on macOS. Can someone on Linux please give this a shot and confirm it's still working there? Given the changes I had to make, I'm really not sure I understand how this was ever working on Linux - it seems to be some pretty fundamental difference in howmultiprocessing.Pool
is implemented.