Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Unification] Move all encoders/transformers to models folders #239

Closed
wants to merge 6 commits into from
Closed

[Unification] Move all encoders/transformers to models folders #239

wants to merge 6 commits into from

Conversation

RdoubleA
Copy link
Contributor

@RdoubleA RdoubleA commented Aug 3, 2022 •
edited
Loading

Stack from ghstack (oldest at bottom):

Differential Revision: D38506859

Summary

Move all model specific encoders and transformers from modules/encoders/ and modules/layers/transformer.py to their corresponding model specific folders, following the naming convention models/<model_name>/<modality>_encoder.py. The encoders folder will remain for general use encoders.

Test plan

pytest -vv

This was referenced Aug 3, 2022
Closed
Closed
Closed
Closed
Closed
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 3, 2022
This was referenced Aug 4, 2022
Closed
Closed
Closed
@codecov-commenter
Copy link

codecov-commenter commented Aug 4, 2022 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (gh/RdoubleA/28/base@5484836). Click here to learn what that means.
The diff coverage is n/a.

@@                  Coverage Diff                   @@
##             gh/RdoubleA/28/base     #239   +/-   ##
======================================================
  Coverage                       ?   92.12%           
======================================================
  Files                          ?       50           
  Lines                          ?     3059           
  Branches                       ?        0           
======================================================
  Hits                           ?     2818           
  Misses                         ?      241           
  Partials                       ?        0

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

This was referenced Aug 5, 2022
Closed
Closed
sophiazhi
sophiazhi suggested changes Aug 5, 2022
View reviewed changes
Copy link
Contributor

@sophiazhi sophiazhi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should be a change in the CLIP import statement in videoclip.

also maybe test/models/albef/test_albef.py should be renamed to test/models/albef/test_model.py (and similar change for clip and flava) since the model is now at model.py?

ebsmothers
ebsmothers reviewed Aug 5, 2022
View reviewed changes
Copy link
Contributor

@ebsmothers ebsmothers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. One question: why are the architecture tests getting moved around in this PR?

test/models/clip/test_clip.py Outdated
from torchmultimodal.modules.encoders.clip_resnet_encoder import ResNetForCLIP
from torchmultimodal.modules.encoders.clip_text_encoder import CLIPTextEncoder
from torchmultimodal.models.clip.model import CLIP
from torchmultimodal.models.clip.resnet_encoder import ResNetForCLIP
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering, do we wanna just call this image_encoder to be consistent with what we have for other models? (I know Ankita has a PR coming in for CLIP ViT encoder, so maybe we wanna keep things as you have them to keep these in separate files?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can place both in an image_encoder.py file for CLIP, what do you think? cc @ankitade

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds good to me. Ultimately will leave it up to Ankita as she is working more closely with the CLIP code these days

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes image_encoder.py is fine

@RdoubleA
Copy link
Contributor Author

RdoubleA commented Aug 5, 2022

Looks good overall. One question: why are the architecture tests getting moved around in this PR?

those tests should've been moved already in an old PR but they weren't for some reason, so I just went ahead and did it in this PR

ebsmothers
ebsmothers approved these changes Aug 6, 2022
View reviewed changes
Copy link
Contributor

@ebsmothers ebsmothers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me pending the decision on the CLIP image encoders

@RdoubleA
Copy link
Contributor Author

RdoubleA commented Aug 8, 2022

@RdoubleA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@RdoubleA RdoubleA mentioned this pull request Aug 9, 2022
Closed
@RdoubleA
Update on "[Unification] Move all encoders/transformers to models fol…
532068c 
…ders"


Differential Revision: [D38506859](https://our.internmc.facebook.com/intern/diff/D38506859)

## Summary
Move all model specific encoders and transformers from `modules/encoders/` and `modules/layers/transformer.py` to their corresponding model specific folders, following the naming convention `models/<model_name>/<modality>_encoder.py`. The encoders folder will remain for general use encoders.

## Test plan
`pytest -vv`

[ghstack-poisoned]
@RdoubleA
Copy link
Contributor Author

RdoubleA commented Aug 9, 2022

@RdoubleA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@RdoubleA
Update on "[Unification] Move all encoders/transformers to models fol…
56897d8 
…ders"


Differential Revision: [D38506859](https://our.internmc.facebook.com/intern/diff/D38506859)

## Summary
Move all model specific encoders and transformers from `modules/encoders/` and `modules/layers/transformer.py` to their corresponding model specific folders, following the naming convention `models/<model_name>/<modality>_encoder.py`. The encoders folder will remain for general use encoders.

## Test plan
`pytest -vv`

[ghstack-poisoned]
@RdoubleA
Copy link
Contributor Author

@RdoubleA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@RdoubleA
Update on "[Unification] Move all encoders/transformers to models fol…
5afb2bb 
…ders"


Differential Revision: [D38506859](https://our.internmc.facebook.com/intern/diff/D38506859)

## Summary
Move all model specific encoders and transformers from `modules/encoders/` and `modules/layers/transformer.py` to their corresponding model specific folders, following the naming convention `models/<model_name>/<modality>_encoder.py`. The encoders folder will remain for general use encoders.

## Test plan
`pytest -vv`

[ghstack-poisoned]
@RdoubleA
Copy link
Contributor Author

@RdoubleA has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot facebook-github-bot deleted the gh/RdoubleA/28/head branch August 15, 2022 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sophiazhi sophiazhi sophiazhi requested changes

@ebsmothers ebsmothers ebsmothers approved these changes

@ankitade ankitade Awaiting requested review from ankitade

@langong347 langong347 Awaiting requested review from langong347

@katrina433 katrina433 Awaiting requested review from katrina433

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@RdoubleA @codecov-commenter @ankitade @sophiazhi @ebsmothers @facebook-github-bot