XLNet bias fix on resize embeddings (cf #1124) #1162
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thomwolf
reviewed
Sep 1, 2019
| @@ -327,6 +327,14 @@ def _tie_or_clone_weights(self, first_module, second_module): | |||
| else: | |||
| first_module.weight = second_module.weight | |||
|
|
|||
| if hasattr(first_module, 'bias') and first_module.bias is not None: | |||
| first_module.bias.data = torch.nn.functional.pad( | |||
There was a problem hiding this comment.
It's a nice and concise way to do it but I'm worried about two things here:
torch.nn.functional.padis not present before pytorch 1.2.0 (and I think we should aim to keep compatibility with +1.0.1 if possible for now).- when we reduce the size of the embeddings (which is supported right now), this will break, I believe.
There was a problem hiding this comment.
I believe torch.nn.functional.pad was actually introduced in torch way back, and is available in the documentation of version 1.0.0. I've run this code with torch 1.0.0 installed successfully.
The pad function actually accepts negative indices, and will then remove the overflowing elements. In this scenario, it removes the last elements, similar to the resize_token_embeddings method.
Here's an example, running on torch 1.0.0:
from pytorch_transformers import XLNetTokenizer, XLNetLMHeadModel
model: XLNetLMHeadModel = XLNetLMHeadModel.from_pretrained("xlnet-base-cased")
tok = XLNetTokenizer.from_pretrained("xlnet-base-cased")
print(model.lm_loss.bias.shape, model.lm_loss.weight.shape)
# torch.Size([32000]) torch.Size([32000, 768])
tok.add_tokens(["token"])
model.resize_token_embeddings(len(tok))
print(model.lm_loss.bias.shape, model.lm_loss.weight.shape)
# torch.Size([32001]) torch.Size([32001, 768])
model.resize_token_embeddings(len(tok) - 100)
print(model.lm_loss.bias.shape, model.lm_loss.weight.shape)
# torch.Size([31901]) torch.Size([31901, 768])There was a problem hiding this comment.
Damned, my doc search capabilities look quite rusty! Ok all good then :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed an issue where the linear layer bias wouldn't be resized along the weight resize when there was an embedding matrix resize with XLNet (cf #1124).
This fix works for any model that needs to tie its weights between an embedding layer & a linear layer if . that linear layer has a bias.