don't save Modules in hparams #915
Conversation
| .squeeze(0) | ||
| .numpy() |
There was a problem hiding this comment.
_ScaleTransformMixin unsqueezes and converts to a torch tensor. If I don't include .numpy() here, I get this warning: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). scale = torch.cat([torch.ones(pad), torch.tensor(scale, dtype=torch.float)])
|
woof- on second thought, i don't think my idea was a good one. I don't think it's that much of a problem to double-save like 32 floating point numbers |
|
Okay, sounds like we will go with #898 for saving the scalers. Questions: I reorganized some of |
|
I moved the tests to #955 I decided to not move the changes to |
Description
Related to #898, this PR address the fact that we double save
nn.Modulesin both thehparamsand in thestate_dict. This PR follows David's idea here.The main downside to this PR is a lot more code is required to implement this versus continuing to double save. The main upside is that model files will be a little smaller.
Other notes
metricsinMPNNis a list ofModulesso the whole list is pickled and unpickled to save and load. This is different than all the other Modules likecriterionwhich are rebuilt each time a model is loaded.The default
task_weightsforcriterionis an array, so I don't need to check if there aretask_weightsin thestate_dict. For the other cases, the default isNone/Identityso in those cases, I need to check if their are corresponding parameters in thestate_dictand if so build aModulewith zeros.While working on this PR, I found that for multicomponent message passing, if a single block is shared between two components, the weights for that block appear twice in the
state_dict. I don't know if the weights just have two key values, or if they are truly saved twice in the model file. Not sure if this is something we should try to change.