Skip to content

Extending Auto-sklearn can be incompatible with parallel execution #1290

Closed
@mfeurer

Description

When running Auto-sklearn extended with a classifier/regressor etc with n_jobs>1 it can happen that the classifier/regressor Auto-sklearn was extended with is not available in the subprocess. Therefore, all function evaluations will crash.

This can be circumvented by calling add_classifier etc in the subprocesses executing the machine learning runs. Sometimes, this is done automatically when for example the call to add_classifier is done on the script's highest level and is not guarded by __name__ == "__main__". However, it would not work in a notebook for example.

Potential solution:

  1. Move the _addons attribute every component type has to a global addons registry.
  2. Pass all registered addons to the workers
  3. When doing a function evaluations, check whether addons need to be registered on the worker and do so

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions