Discussion about this moved to #686

To elaborate a bit more, If you are using a locally written python module, and using it inside an electron, cloudpickle would not pickle them. One has to explicitly tell cloud pickle to pickle that module. We also need a same interface where we pass on these local non-pip-installable modules to the Deps that can be pickled by value by cloudpickle.

Ah yes, thanks for elaborating @santoshkumarradha, I will add an issue to the Import Deps epic to cover the UX proposal and implementation.

@cjao , great qs,

0. I would agree with you and this is more on the line of the following

deps_object_1.apply()
deps_object_2.apply()

def new_func(*args, **kwargs):	
	task(*args, **kwargs)

deps_object_3.apply()

Rather than the former. It is meant to have a unified set of call back/call forward methods before and after python function execution ( not inside of it) good catch.

Are the different classes of deps assumed to be independent? What would happen if a pip package requires some system package to be installed using shell commands?
Where should the PipDeps be installed? In a venv? Or a condaenv?

Great questions, although we see pip and conda in the same variable there, what we need is to differentiate pip with pipenv and conda with conda env and through error while defining ImportDeps to make sure only one is passed.

Should the CondaDeps automatically activate the condaenv? If so, would that overlap with the executor's execute_in_conda_env method?

Again great Q, essentially the execute_in_conda_env was a lean patch to support multiple conda env which should be depreciated once this is release. The goal with Deps is to run the given function in the required environment by installing the unavailable packages (or creating the env if it is not available in conda) in the given list.

Hope this helps

Thanks for the clarifications. Could you elaborate on the order in which the dependencies are to be processed? Conda first, and then Pip? What happens if users define their own Deps subclasses? And how do these dependencies interact with the File deps? Which ones are to be handled first?

We must take care to not compromise the clarity of the program logic whenever we perform magic behind the scenes (in this case, translating dependency declarations into Python statements). Unless the deps are truly independent -- any order which they are processed yields the same results -- it will be hard to reason about the code unless there is a deterministic processing order.

For now, we will process the different classes of deps in a canonical order.

I agree, the order I would choose for default ( after which we can have a backlog feature addition to create UX for modifying this behaviour) is

Env->Bash->Module->Conda->Pip->Call_before
*function*
Call_after

typical cluster modules contain conda as well as python, so having module load conda and then doing stuff of instaling conda env or switching conda env would be useful. Same for pip where module load python is often used to invoke cluster's default python.

Sounds good, I see @AlejandroEsquivel had already written down this order in the MVP planning issue.

again, good q. Ideally lets' design it in a way that we dont include Deps object in the executer but rather just the list of commands/functions to be executed. We want to keep the transferring to executer as lean as possible. Having said that designing objects to be picklable is extremely important for future proofing and transporting (especially since we are dealing with remote execution)

tldr - lets make it picklable but design deps in a way that we dont need to ship the object.

The bulk of a Deps object's footprint would be its associated data (e.g. a list of pip packages). If the Deps object can in fact be serialized, it should be easy to ship the object and call apply() in the remote environment.

I don't see much point in introducing a Deps object to store the list of bash commands or pip dependencies only to deconstruct the object right before execution.

I don't see much point in introducing a Deps object to store the list of bash commands or pip dependencies only to deconstruct the object right before execution.

I guess the reasoning to send it via non pickled sense is that we might want to restrict pickle to only objects that definitely need to be pickled (only functions) to the executer while others are done via REST and such. This might maybe let us be more clever on how we handle larger units ( by breaking the message etc..) instead of being tied to one unit blob. Also, since pickle is not cross compatable, it might be worth while to keep them separate and just execute them which reduces one head ache off us taking care of making sure of pickle compatibility.

The reason to have Deps is not just for our convenience to store, but rather for user to have more structure while defining. Passing requirements as second image in sankalp's proposal although we allow it, ideally is not how we want user to interact as one can very much make mistakes and not even have type hints on what all is allowed. Hence passing more structured data gives the user the ease to exactly know what all they can pass and how to pass them.

I see, thanks for clarifying.

One way to avoid pickling any class or function is to have Deps.apply() simply emit the sequence of bash commands and leave it to the executor to figure out how to apply them. functools.partial doesn't seem to avoid pickling; it just fills in some of the parameters of a function.

One constraint here is that we really don't want to keep SSH connections open for very long; thus either call_before and call_after must be extremely lightweight, or we pack the hooks into the same task as the main task; but this would prevent environment setup (e.g. conda activate) before the main task is deserialized.

Can you elaborate on why we would do this rather than ship all the deps execution statement along with the function to happen before and after ? for instance, one can easily define install and module load statements in the SLURM job while in case of SLURM and similarly for others. Down side is that we cant get back communication of whether the said Deps state have been finished or not, but thats an okay state to be in for MVP and then add it on as feature.

I think we're talking about the same thing. module load or conda activate are the sort of commands that would go in call_before.

I was thinking initially that putting deps (especially complicated deps involving many shell commands) in their own electrons might be easier for error handling -- failure of any dep would be handled like failure of any other electron, and one can easily see exactly what step failed.