Update vit.py #274
Update vit.py #274
Conversation
In the official code of Vision Transformer, the final LayerNorm is conducted after `self.transformer(x)` but before taking the mean or class token from `x`. Therefore, it is revised here to be consistent with the official version.
|
@LuYuchenOrRobert oh yes, this would affect mean pooling (but not cls token) i should change all the ViTs (whichever ones are using global mean pooling) to have this 'correct' order |
|
@LuYuchenOrRobert you want to see if 3e5d1be looks right to you? i'll finish converting the rest of the vision transformers using global mean pooling later this week |
|
@lucidrains Sorry for my late response. It looks right now. Thank you for your contribution! |
|
Hi, @lucidrains. If you look at the forward function of the Transformer class in vit-pytorch/vit_pytorch/vit.py Line 122 in 96f66d2 Because Attention and Feedforward class already have their LayerNorm. I also think the call to |
In the official code of Vision Transformer, the final LayerNorm is conducted after
self.transformer(x)but before taking the mean or class token fromx. It is revised here to be consistent with the official version as chaging the position of LayerNorm does affect the calculation results (not sure about whether affects the model performance).