Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH : set train / valid of criterion. #621

Merged
merged 1 commit into from
May 29, 2020

Conversation

YannDubs
Copy link
Contributor

@YannDubs YannDubs commented Apr 20, 2020

closes #565

Usually criterions in pytorch inherit from nn.Module, meaning that they should have a self.training option that says whether it is during train or validation. yet skorch does not update it. This is useful for cases where there are multiple losses and the weights of each loss change during training (but not testing). E.g. beta-vae with annealing, Gans ....

@BenjaminBossan

@BenjaminBossan
Copy link
Collaborator

Thanks for taking this one.

Do you think there is a good way of testing this feature, so as to prevent regressions in the future?

@BenjaminBossan
Copy link
Collaborator

ping @YannDubs

@YannDubs
Copy link
Contributor Author

Sorry I didn't really understand what you asked for. Are you asking for a unittest that checks whether or not the criterion is correctly on train or valid ? Or are you asking for a concrete usecases that could also be tested ?

I see 2 use cases for this (I actually used this for both):
1/ For annealing and changing of hyperparameters. Typically in vea you want to anneal beta (the lagrange multiplier), but it doesn't really make sense to annealy beta during validation
2/ More and more people run optimization procedures / have networks in the loss. For example the discriminator of GAN is often used as a loss for the generator (and not really as a module in itseld, as we discard it afterwards).

The second use case is more tricky as it requires to make other changes to skorch. For example I had to change the checkpointing procedures so as to also checkpoint the criterion. That's something I wanted to open as an issue to see what was you thought about this ...

@ottonemo ottonemo mentioned this pull request May 13, 2020
2 tasks
@ottonemo
Copy link
Member

I think what Benjamin meant was going for was unit testing this feature :)

The second use case is more tricky as it requires to make other changes to skorch. For example I had to change the checkpointing procedures so as to also checkpoint the criterion. That's something I wanted to open as an issue to see what was you thought about this ...

That's a good point. With the addition from this PR the criterion is left out from saving and with the changes in #597 it is easier to add custom modules to the neural net instances which are then also exempt from checkpointing. I feel that we need a checkpointing approach that scales better with 'auxiliary' objects that arise in more complex scenarios because I don't see adding yet another f_-parameter to the checkpoint callback as a good idea.

Let's discuss our options in #635.

@BenjaminBossan
Copy link
Collaborator

I think what Benjamin meant was going for was unit testing this feature :)

Yes, but ideally it's close to a real use case so that we have more confidence that the right thing happens.

@BenjaminBossan
Copy link
Collaborator

I have a suggestion for two unit tests:

    def test_criterion_training_set_correctly(self, net_cls, module_cls, data):
        # check that criterion's training attribute is set correctly

        X, y = data[0][:50], data[1][:50]  # don't need all the data
        side_effect = []

        class MyCriterion(nn.NLLLoss):
            """Criterion that records its training attribute"""
            def forward(self, *args, **kwargs):
                side_effect.append(self.training)
                return super().forward(*args, **kwargs)

        net = net_cls(module_cls, criterion=MyCriterion, max_epochs=1)
        net.fit(X, y)

        # called once with training=True for train step, once with
        # training=False for validation step
        assert side_effect == [True, False]

        net.partial_fit(X, y)
        # same logic as before
        assert side_effect == [True, False, True, False]

    def test_criterion_is_not_a_torch_module(self, net_cls, module_cls, data):
        X, y = data[0][:50], data[1][:50]  # don't need all the data

        def my_criterion():
            return torch.nn.functional.nll_loss

        net = net_cls(module_cls, criterion=my_criterion, max_epochs=1)
        net.fit(X, y)  # does not raise

They could be added to the end of TestNeuralNet in test_net.py. They are a bit artificial but should do the job. The first one fails with the provided fix, the second one is just a safeguard for potential regressions.

What do you think @YannDubs?

@YannDubs
Copy link
Contributor Author

YannDubs commented May 22, 2020

I'm putting this on hold until Neurips deadline (10 June ). I will then send you a long proposal I have that will need this PR (and other things including #635 ). Happy to write tests and example notebooks (I'm thinking VAE and GAN) for that if you like the proposal. This will also close #482 . I prefer waiting util then as this PR is key and I will want to test many (real) usecases.
In short I think VAE's should be trained as NeuralNet(module=Encoder,criterion=Decoder) and GAN as NeuralNet(module=Generator,criterion=Discriminator). I'll be more precise then

@BenjaminBossan
Copy link
Collaborator

In short I think VAE's should be trained as NeuralNet(module=Encoder,criterion=Decoder) and GAN as NeuralNet(module=Generator,criterion=Discriminator). I'll be more precise then

That sounds very intriguing, I haven't thought of that possibility. Looking forward to your proposal.

As to this PR, I think we can also just merge it for now and add the tests later (I think the ones I pasted above should be enough for now). If that makes your work easier, please say so and I'll merge.

@YannDubs
Copy link
Contributor Author

Sounds good to me ! thanks

@BenjaminBossan BenjaminBossan merged commit a621d0f into skorch-dev:master May 29, 2020
BenjaminBossan pushed a commit that referenced this pull request May 29, 2020
As discussed in #621.

Also: Add entry to CHANGES.md
@BenjaminBossan
Copy link
Collaborator

I will then send you a long proposal I have that will need this PR (and other things including #635 )

Hi @YannDubs this PR has been merged and there is a PR to address #635 (namely #652). Are you making progress with your proposal?

@BenjaminBossan
Copy link
Collaborator

ping @YannDubs any news?

BenjaminBossan added a commit that referenced this pull request Aug 30, 2020
This release of skorch contains a few minor improvements and some nice additions. As always, we fixed a few bugs and improved the documentation. Our [learning rate scheduler](https://skorch.readthedocs.io/en/latest/callbacks.html#skorch.callbacks.LRScheduler) now optionally logs learning rate changes to the history; moreover, it now allows the user to choose whether an update step should be made after each batch or each epoch.

If you always longed for a metric that would just use whatever is defined by your criterion, look no further than [`loss_scoring`](https://skorch.readthedocs.io/en/latest/scoring.html#skorch.scoring.loss_scoring). Also, skorch now allows you to easily change the kind of nonlinearity to apply to the module's output when `predict` and `predict_proba` are called, by passing the `predict_nonlinearity` argument.

Besides these changes, we improved the customization potential of skorch. First of all, the `criterion` is now set to `train` or `valid`, depending on the phase -- this is useful if the criterion should act differently during training and validation. Next we made it easier to add custom modules, optimizers, and criteria to your neural net; this should facilitate implementing architectures like GANs. Consult the [docs](https://skorch.readthedocs.io/en/latest/user/neuralnet.html#subclassing-neuralnet) for more on this. Conveniently, [`net.save_params`](https://skorch.readthedocs.io/en/latest/net.html#skorch.net.NeuralNet.save_params) can now persist arbitrary attributes, including those custom modules.
As always, these improvements wouldn't have been possible without the community. Please keep asking questions, raising issues, and proposing new features. We are especially grateful to those community members, old and new, who contributed via PRs:

```
Aaron Berk
guybuk
kqf
Michał Słapek
Scott Sievert
Yann Dubois
Zhao Meng
```

Here is the full list of all changes:

### Added

- Added the `event_name` argument for `LRScheduler` for optional recording of LR changes inside `net.history`. NOTE: Supported only in Pytorch>=1.4
- Make it easier to add custom modules or optimizers to a neural net class by automatically registering them where necessary and by making them available to set_params
- Added the `step_every` argument for `LRScheduler` to set whether the scheduler step should be taken on every epoch or on every batch.
- Added the `scoring` module with `loss_scoring` function, which computes the net's loss (using `get_loss`) on provided input data.
- Added a parameter `predict_nonlinearity` to `NeuralNet` which allows users to control the nonlinearity to be applied to the module output when calling `predict` and `predict_proba` (#637, #661)
- Added the possibility to save the criterion with `save_params` and with checkpoint callbacks
- Added the possibility to save custom modules with `save_params` and with checkpoint callbacks

### Changed

- Removed support for schedulers with a `batch_step()` method in `LRScheduler`.
- Raise `FutureWarning` in `CVSplit` when `random_state` is not used. Will raise an exception in a future (#620)
- The behavior of method `net.get_params` changed to make it more consistent with sklearn: it will no longer return "learned" attributes like `module_`; therefore, functions like `sklearn.base.clone`, when called with a fitted net, will no longer return a fitted net but instead an uninitialized net; if you want a copy of a fitted net, use `copy.deepcopy` instead;`net.get_params` is used under the hood by many sklearn functions and classes, such as `GridSearchCV`, whose behavior may thus be affected by the change. (#521, #527)
- Raise `FutureWarning` when using `CyclicLR` scheduler, because the default behavior has changed from taking a step every batch to taking a step every epoch. (#626)
- Set train/validation on criterion if it's a PyTorch module (#621)
- Don't pass `y=None` to `NeuralNet.train_split` to enable the direct use of split functions without positional `y` in their signatures. This is useful when working with unsupervised data (#605).
- `to_numpy` is now able to unpack dicts and lists/tuples (#657, #658)
- When using `CrossEntropyLoss`, softmax is now automatically applied to the output when calling `predict` or `predict_proba`

### Fixed

- Fixed a bug where `CyclicLR` scheduler would update during both training and validation rather than just during training.
- Fixed a bug introduced by moving the `optimizer.zero_grad()` call outside of the train step function, making it incompatible with LBFGS and other optimizers that call the train step several times per batch (#636)
- Fixed pickling of the `ProgressBar` callback (#656)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request] Calling .eval on criterion
3 participants