Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve parallel default selection on macOS #5383

Merged
merged 12 commits into from
Mar 4, 2021

Conversation

mtreinish
Copy link
Member

@mtreinish mtreinish commented Nov 11, 2020

Summary

In #5324 we changed the default for running with parallel map on macOS
to default to false unconditionally, but that was likely too strict.
While there are a myriad of issues related to Python multiproccessing
and macOS it doesn't fail in every macOS environment. Additionally the
performance benefits from running in parallel is signtificant. This
commit attempts to make the default change a bit more nuanced. Instead
of defaulting to false on all macOS systems it only does that for Python
3.8 and macOS. This is because in python 3.8 the use of spawn instead of
fork means in the best cases if it works completely as expected it will
be slower (see #5323) but also means there are unforseen consequences
(see #5376) because it's spawning a new interpreter instead of forking.
But, often it just doesn't work. For other python versions there are
still potential issues on newer versions of macOS (>=10.14) but in general
it still provides a speedup if it works.

Details and comments

Fixes #5373

In Qiskit#5324 we changed the default for running with parallel map on macOS
to default to false unconditionally, but that was likely too strict.
While there are a myriad of issues related to Python multiproccessing
and macOS it doesn't fail in every macOS environment. Additionally the
performance benefits from running in parallel is signtificant. This
commit attempts to make the default change a bit more nuanced. Instead
of defaulting to false on all macOS systems it only does that for Python
3.8 and macOS. This is because in python 3.8 the use of spawn instead of
fork means in the best cases if it works completely as expected it will
be slower (see $5323) but also means there are unforseen consequences
(see Qiskit#5376) because it's spawning a new interpreter instead of forking.
But, often it just doesn't work. For other python versions there are
still potential issues on newer versions of macOS (>=10.14) so to catch
potential issues there a small parallel benchmark is run on import to
ensure that it both works and is faster than serial. If both are true
then we default it parallelism to True, otherwise it's False.

Fixes Qiskit#5373
@mtreinish mtreinish added this to the 0.17 milestone Nov 11, 2020
@mtreinish mtreinish requested a review from a team as a code owner November 11, 2020 17:26
@mtreinish mtreinish changed the title Improve parallel default selection on macOS [WIP] Improve parallel default selection on macOS Nov 11, 2020
@mtreinish
Copy link
Member Author

Heh, ok the first draft of the benchmark is hanging the on import in macOS. So this definitely needs work still.

The use of the parallel benchmark for macOS to determine whether
parallel execution is actually faster or not was hanging on import. I
expect this was due to it being executed on import and not having
sufficient context awareness in the subprocesses to handle that. To
avoid the hang this just makes the default for macOS parallel processing
to be True if Python < 3.8.
@mtreinish mtreinish changed the title [WIP] Improve parallel default selection on macOS Improve parallel default selection on macOS Jan 12, 2021
@mtreinish mtreinish marked this pull request as ready for review January 12, 2021 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve default selection for parallel macOS
2 participants