add support for spawn multiprocessing mode #638
Conversation
Codecov ReportBase: 99.58% // Head: 99.58% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## master #638 +/- ##
=======================================
Coverage 99.58% 99.58%
=======================================
Files 87 87
Lines 5972 5992 +20
=======================================
+ Hits 5947 5967 +20
Misses 25 25
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
|
Successfully tested parallel training of NN ensemble with this PR branch on a MacBook with an M1 processor (arm architecture) by @holehan, according to this report. Thanks a lot! |
osma
force-pushed
the
issue637-multiprocessing-spawn
branch
from
3f507d9 to
36c79c0
Compare
|
Rebased on current |
|
Kudos, SonarCloud Quality Gate passed!
|
|
Tested parallel processing in this PR branch using Windows 10 (Intel Core i5-8250U processor). Succesfully trained an nn-ensemble model, ran into no problems related to PR. |
This (draft) PR attempts to add support for the
spawnmultiprocessing mode. This is the only mode supported on Windows, and the default mode on Mac OS. Theforkmode, supported on Linux and some other *nix systems, is more efficient because it allows loading models only once and then reusing them in forked child processes, with most of the memory shared between the processes. Butspawnmode is still better than not being able to perform multiprocessing at all.This PR needs more testing, ideally on both Mac OS and Windows. Currently there is just a single unit test that exercises parallel evaluation using the
spawnmode.Fixes #637