[FLAVA] Add notebook that remaps checkpoint #188
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
RdoubleA
added a commit
that referenced
this pull request
Jul 21, 2022
…ith generalized class" ## Summary Replaces the `FLAVASelfAttention` class with the generalized `SelfAttention`. Multihead splitting and merging was moved upto `FLAVAAttention`. Since this change modifies layers, the checkpoint was loaded, remapped, and saved to a new location. The link has been updated. The remapping was done in #188. ## Test plan `pytest -vv` and `pytest test/models/flava/test_flava_checkpoint.py -vv` ``` ======================================================================================== test session starts ======================================================================================== platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /fsx/users/rafiayub/conda/envs/torchmm/bin/python cachedir: .pytest_cache rootdir: /data/home/rafiayub/torchmultimodal, configfile: pyproject.toml plugins: cov-3.0.0, mock-3.8.2 collected 224 items examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_fixed_weight_and_bias PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_arg_lengths PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_output_dims PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_multiple_layer PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_scripting PASSED [ 2%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_single_layer PASSED [ 2%] examples/cnn_lstm/test/test_cnn_lstm.py::TestCNNLSTMModule::test_forward PASSED [ 3%] examples/cnn_lstm/test/test_lstm_encoder.py::TestLSTMEncoder::test_lstm_encoder PASSED [ 3%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_contrastive_alignment_loss PASSED [ 4%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_construct_positive_map PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[12-expected0] PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[5-expected1] PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_invalid_inputs PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_valid_inputs PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_s3d_base PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_basicconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_sepconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3b-192-256] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3c-256-480] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4b-480-512] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4c-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4d-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4e-512-528] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4f-528-832] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5b-832-832] PASSED [ 11%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5c-832-1024] PASSED [ 11%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_attention_mask PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_invalid_channels PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestProjection::test_forward PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_pretrained_trainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_pretrained_untrainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_untrained_trainable PASSED [ 15%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_untrained_untrainable PASSED [ 15%] test/architectures/test_late_fusion.py::TestLateFusion::test_forward PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_missing_key_in_modalities PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_script PASSED [ 16%] test/architectures/test_two_tower.py::TestTwoTower::test_shared_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower_scripting PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings_momentum PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_copy_params_momentum_models PASSED [ 21%] test/models/test_albef.py::test_dequeue_and_enqueue PASSED [ 21%] test/models/test_albef.py::test_momentum_update PASSED [ 22%] test/models/test_albef.py::test_similarity PASSED [ 22%] test/models/test_albef.py::test_neg_embeddings PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_forward PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_resnet_forward PASSED [ 24%] test/models/test_clip.py::TestCLIP::test_clip_vit_forward PASSED [ 24%] test/models/test_mdetr.py::TestMDETR::test_transformer_encoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_transformer_decoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_full_mdetr_model PASSED [ 25%] test/models/test_omnivore.py::test_omnivore_swin_t_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_s_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_b_forward PASSED [ 27%] test/models/test_omnivore.py::test_omnivore_forward_wrong_input_type PASSED [ 27%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_hidden_dim_assertion PASSED [ 28%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_forward PASSED [ 28%] test/models/test_video_vqvae.py::TestVideoEncoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoDecoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_encode PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_decode PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_tokenize PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_forward PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_encode PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_decode PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_tokenize PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_forward PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_classification PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_pretraining PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image_text PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image_and_text PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_text PASSED [ 36%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_text PASSED [ 36%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_classification PASSED [ 37%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_embedding PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_image_encoder PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_embedding PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer PASSED [ 39%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer_attn_mask PASSED [ 39%] test/modules/encoders/test_albef_multimodal_encoder.py::test_multimodal_encoder PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_image_hidden_size PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_text_hidden_size PASSED [ 41%] test/modules/encoders/test_albef_multimodal_encoder.py::test_not_matching_input_batch_size PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_text_encoder PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_invalid_input_length PASSED [ 42%] test/modules/encoders/test_albef_text_encoder.py::test_not_matching_attention_mask_shape PASSED [ 42%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_vision_transformer PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_input_length PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_channel_dim PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_height PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_width PASSED [ 45%] test/modules/encoders/test_clip_resnet_encoder.py::TestCLIPModule::test_resnet PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_initialization PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_clip_parameters PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_attention_mask PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward_over_context_length PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_hash PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_invalid_pooling PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_max PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_mean PASSED [ 50%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_sum PASSED [ 50%] test/modules/encoders/test_mdetr_image_encoder.py::TestMDETRImageEncoder::test_resnet_101_forward PASSED [ 50%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_modified_transformer PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_embeddings PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_encoder PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_forward PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_invalid_partitioning PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_scripting PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_transformer_pooling PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_encoder PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_scripting PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_patch_merging_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d_zero_shift PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_max_pooling PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_mean_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_sum_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_scripting PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_input_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_no_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_scripted_model PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_apply_attention PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_auto_mapping PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_invalid_pooling PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_max PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_mean PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_median PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_min PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_modality_normalize PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_sum PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_transformer PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_get_deepset_transformer PASSED [ 64%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript PASSED [ 64%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_use_cache PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_causal_use_cache PASSED [ 66%] test/modules/layers/test_attention.py::test_scaled_dot_product_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_full_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_axial_attention PASSED [ 67%] test/modules/layers/test_attention.py::test_split_multihead PASSED [ 67%] test/modules/layers/test_attention.py::test_merge_multihead PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_forward PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_channel_dim PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_codebook_restart PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_ema_update_embedding PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_and_preprocess PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_smaller_encoded PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_postprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess_channel_dim_assertion PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_quantized_output PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_assert PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_output PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_assert PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_output PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconvtranspose3d_forward PASSED [ 75%] test/modules/layers/test_mlp.py::TestMLP::test_activation_and_normalization PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_dropout_default PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_no_dropout PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_no_hidden_layers PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_pass_hidden_dims PASSED [ 78%] test/modules/layers/test_mlp.py::TestMLP::test_torchscript PASSED [ 78%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_sets_embedding PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_bad_embedding_dim PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_broadcast PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward PASSED [ 80%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward_invalid_input PASSED [ 80%] test/modules/layers/test_transformer.py::TestTransformer::test_flava_encoder_forward PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_missing_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_target PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_without_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_sim_targets PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss_invalid_input_hidden_size PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_labels PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_missing_momentum_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss PASSED [ 87%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_with_distillation PASSED [ 87%] test/modules/losses/test_commitment.py::TestCommitment::test_loss_value PASSED [ 87%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_local_loss PASSED [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_multi_gpu_loss SKIPPED (Not enough GPUs to run the test: required 2) [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_single_gpu_loss SKIPPED (Not enough GPUs to run the test: required 1) [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_invalid PASSED [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_max PASSED [ 90%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_min PASSED [ 90%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_soft_token_prediction_loss PASSED [ 91%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_box_losses PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_single_transform PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_single_transform PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_call PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_wrong_channels PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_sample_frames PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_resize_hw PASSED [ 95%] test/transforms/test_video_transform.py::TestVideoTransform::test_normalize PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_different_lengths PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_same_lengths PASSED [ 96%] test/utils/test_common.py::test_shift_dim PASSED [ 96%] test/utils/test_common.py::TestTensorSlice::test_default PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_size_minus_one PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_uneven_begin_size PASSED [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_begin XFAIL (Invalid begin) [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_size XFAIL (Invalid size) [ 99%] test/utils/test_common.py::TestToTupleTuple::test_int PASSED [ 99%] test/utils/test_common.py::TestToTupleTuple::test_tuple PASSED [100%] ========================================================================================= warnings summary ========================================================================================== test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining /data/home/rafiayub/torchmultimodal/test/models/flava/test_flava_checkpoint.py:81: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). else torch.tensor(dict_actual[key]) test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:246: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:240: UserWarning: 'norm' was found in ScriptModule constants, but it is a non-constant submodule. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:169: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.code_avg.grad, msg_has_grad test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:171: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.embedding.grad, msg_has_grad test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward /data/home/rafiayub/torchmultimodal/torchmultimodal/modules/layers/conv.py:49: UserWarning: Padding was specified but will not be used in favor of same padding, use Conv3d directly for custom padding warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================= 220 passed, 2 skipped, 2 xfailed, 7 warnings in 81.22s (0:01:21) ================================================================== ``` [ghstack-poisoned]
RdoubleA
added a commit
that referenced
this pull request
Jul 21, 2022
…class" ## Summary Replaces the `FLAVASelfAttention` class with the generalized `SelfAttention`. Multihead splitting and merging was moved upto `FLAVAAttention`. Since this change modifies layers, the checkpoint was loaded, remapped, and saved to a new location. The link has been updated. The remapping was done in #188. ## Test plan `pytest -vv` and `pytest test/models/flava/test_flava_checkpoint.py -vv` ``` ======================================================================================== test session starts ======================================================================================== platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /fsx/users/rafiayub/conda/envs/torchmm/bin/python cachedir: .pytest_cache rootdir: /data/home/rafiayub/torchmultimodal, configfile: pyproject.toml plugins: cov-3.0.0, mock-3.8.2 collected 224 items examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_fixed_weight_and_bias PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_arg_lengths PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_output_dims PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_multiple_layer PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_scripting PASSED [ 2%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_single_layer PASSED [ 2%] examples/cnn_lstm/test/test_cnn_lstm.py::TestCNNLSTMModule::test_forward PASSED [ 3%] examples/cnn_lstm/test/test_lstm_encoder.py::TestLSTMEncoder::test_lstm_encoder PASSED [ 3%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_contrastive_alignment_loss PASSED [ 4%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_construct_positive_map PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[12-expected0] PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[5-expected1] PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_invalid_inputs PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_valid_inputs PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_s3d_base PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_basicconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_sepconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3b-192-256] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3c-256-480] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4b-480-512] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4c-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4d-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4e-512-528] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4f-528-832] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5b-832-832] PASSED [ 11%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5c-832-1024] PASSED [ 11%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_attention_mask PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_invalid_channels PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestProjection::test_forward PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_pretrained_trainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_pretrained_untrainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_untrained_trainable PASSED [ 15%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_untrained_untrainable PASSED [ 15%] test/architectures/test_late_fusion.py::TestLateFusion::test_forward PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_missing_key_in_modalities PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_script PASSED [ 16%] test/architectures/test_two_tower.py::TestTwoTower::test_shared_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower_scripting PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings_momentum PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_copy_params_momentum_models PASSED [ 21%] test/models/test_albef.py::test_dequeue_and_enqueue PASSED [ 21%] test/models/test_albef.py::test_momentum_update PASSED [ 22%] test/models/test_albef.py::test_similarity PASSED [ 22%] test/models/test_albef.py::test_neg_embeddings PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_forward PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_resnet_forward PASSED [ 24%] test/models/test_clip.py::TestCLIP::test_clip_vit_forward PASSED [ 24%] test/models/test_mdetr.py::TestMDETR::test_transformer_encoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_transformer_decoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_full_mdetr_model PASSED [ 25%] test/models/test_omnivore.py::test_omnivore_swin_t_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_s_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_b_forward PASSED [ 27%] test/models/test_omnivore.py::test_omnivore_forward_wrong_input_type PASSED [ 27%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_hidden_dim_assertion PASSED [ 28%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_forward PASSED [ 28%] test/models/test_video_vqvae.py::TestVideoEncoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoDecoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_encode PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_decode PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_tokenize PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_forward PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_encode PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_decode PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_tokenize PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_forward PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_classification PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_pretraining PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image_text PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image_and_text PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_text PASSED [ 36%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_text PASSED [ 36%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_classification PASSED [ 37%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_embedding PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_image_encoder PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_embedding PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer PASSED [ 39%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer_attn_mask PASSED [ 39%] test/modules/encoders/test_albef_multimodal_encoder.py::test_multimodal_encoder PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_image_hidden_size PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_text_hidden_size PASSED [ 41%] test/modules/encoders/test_albef_multimodal_encoder.py::test_not_matching_input_batch_size PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_text_encoder PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_invalid_input_length PASSED [ 42%] test/modules/encoders/test_albef_text_encoder.py::test_not_matching_attention_mask_shape PASSED [ 42%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_vision_transformer PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_input_length PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_channel_dim PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_height PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_width PASSED [ 45%] test/modules/encoders/test_clip_resnet_encoder.py::TestCLIPModule::test_resnet PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_initialization PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_clip_parameters PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_attention_mask PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward_over_context_length PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_hash PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_invalid_pooling PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_max PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_mean PASSED [ 50%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_sum PASSED [ 50%] test/modules/encoders/test_mdetr_image_encoder.py::TestMDETRImageEncoder::test_resnet_101_forward PASSED [ 50%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_modified_transformer PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_embeddings PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_encoder PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_forward PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_invalid_partitioning PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_scripting PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_transformer_pooling PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_encoder PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_scripting PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_patch_merging_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d_zero_shift PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_max_pooling PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_mean_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_sum_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_scripting PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_input_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_no_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_scripted_model PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_apply_attention PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_auto_mapping PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_invalid_pooling PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_max PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_mean PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_median PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_min PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_modality_normalize PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_sum PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_transformer PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_get_deepset_transformer PASSED [ 64%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript PASSED [ 64%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_use_cache PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_causal_use_cache PASSED [ 66%] test/modules/layers/test_attention.py::test_scaled_dot_product_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_full_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_axial_attention PASSED [ 67%] test/modules/layers/test_attention.py::test_split_multihead PASSED [ 67%] test/modules/layers/test_attention.py::test_merge_multihead PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_forward PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_channel_dim PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_codebook_restart PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_ema_update_embedding PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_and_preprocess PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_smaller_encoded PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_postprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess_channel_dim_assertion PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_quantized_output PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_assert PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_output PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_assert PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_output PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconvtranspose3d_forward PASSED [ 75%] test/modules/layers/test_mlp.py::TestMLP::test_activation_and_normalization PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_dropout_default PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_no_dropout PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_no_hidden_layers PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_pass_hidden_dims PASSED [ 78%] test/modules/layers/test_mlp.py::TestMLP::test_torchscript PASSED [ 78%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_sets_embedding PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_bad_embedding_dim PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_broadcast PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward PASSED [ 80%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward_invalid_input PASSED [ 80%] test/modules/layers/test_transformer.py::TestTransformer::test_flava_encoder_forward PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_missing_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_target PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_without_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_sim_targets PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss_invalid_input_hidden_size PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_labels PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_missing_momentum_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss PASSED [ 87%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_with_distillation PASSED [ 87%] test/modules/losses/test_commitment.py::TestCommitment::test_loss_value PASSED [ 87%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_local_loss PASSED [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_multi_gpu_loss SKIPPED (Not enough GPUs to run the test: required 2) [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_single_gpu_loss SKIPPED (Not enough GPUs to run the test: required 1) [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_invalid PASSED [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_max PASSED [ 90%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_min PASSED [ 90%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_soft_token_prediction_loss PASSED [ 91%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_box_losses PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_single_transform PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_single_transform PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_call PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_wrong_channels PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_sample_frames PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_resize_hw PASSED [ 95%] test/transforms/test_video_transform.py::TestVideoTransform::test_normalize PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_different_lengths PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_same_lengths PASSED [ 96%] test/utils/test_common.py::test_shift_dim PASSED [ 96%] test/utils/test_common.py::TestTensorSlice::test_default PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_size_minus_one PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_uneven_begin_size PASSED [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_begin XFAIL (Invalid begin) [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_size XFAIL (Invalid size) [ 99%] test/utils/test_common.py::TestToTupleTuple::test_int PASSED [ 99%] test/utils/test_common.py::TestToTupleTuple::test_tuple PASSED [100%] ========================================================================================= warnings summary ========================================================================================== test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining /data/home/rafiayub/torchmultimodal/test/models/flava/test_flava_checkpoint.py:81: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). else torch.tensor(dict_actual[key]) test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:246: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:240: UserWarning: 'norm' was found in ScriptModule constants, but it is a non-constant submodule. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:169: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.code_avg.grad, msg_has_grad test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:171: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.embedding.grad, msg_has_grad test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward /data/home/rafiayub/torchmultimodal/torchmultimodal/modules/layers/conv.py:49: UserWarning: Padding was specified but will not be used in favor of same padding, use Conv3d directly for custom padding warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================= 220 passed, 2 skipped, 2 xfailed, 7 warnings in 81.22s (0:01:21) ================================================================== ``` [ghstack-poisoned]
RdoubleA
added a commit
that referenced
this pull request
Jul 21, 2022
…ith generalized class" ## Summary Replaces the `FLAVASelfAttention` class with the generalized `SelfAttention`. Multihead splitting and merging was moved upto `FLAVAAttention`. Since this change modifies layers, the checkpoint was loaded, remapped, and saved to a new location. The link has been updated. The remapping was done in #188. ## Test plan `pytest -vv` and `pytest test/models/flava/test_flava_checkpoint.py -vv` ``` ======================================================================================== test session starts ======================================================================================== platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /fsx/users/rafiayub/conda/envs/torchmm/bin/python cachedir: .pytest_cache rootdir: /data/home/rafiayub/torchmultimodal, configfile: pyproject.toml plugins: cov-3.0.0, mock-3.8.2 collected 224 items examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_fixed_weight_and_bias PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_arg_lengths PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_output_dims PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_multiple_layer PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_scripting PASSED [ 2%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_single_layer PASSED [ 2%] examples/cnn_lstm/test/test_cnn_lstm.py::TestCNNLSTMModule::test_forward PASSED [ 3%] examples/cnn_lstm/test/test_lstm_encoder.py::TestLSTMEncoder::test_lstm_encoder PASSED [ 3%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_contrastive_alignment_loss PASSED [ 4%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_construct_positive_map PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[12-expected0] PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[5-expected1] PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_invalid_inputs PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_valid_inputs PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_s3d_base PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_basicconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_sepconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3b-192-256] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3c-256-480] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4b-480-512] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4c-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4d-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4e-512-528] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4f-528-832] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5b-832-832] PASSED [ 11%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5c-832-1024] PASSED [ 11%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_attention_mask PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_invalid_channels PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestProjection::test_forward PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_pretrained_trainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_pretrained_untrainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_untrained_trainable PASSED [ 15%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_untrained_untrainable PASSED [ 15%] test/architectures/test_late_fusion.py::TestLateFusion::test_forward PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_missing_key_in_modalities PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_script PASSED [ 16%] test/architectures/test_two_tower.py::TestTwoTower::test_shared_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower_scripting PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings_momentum PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_copy_params_momentum_models PASSED [ 21%] test/models/test_albef.py::test_dequeue_and_enqueue PASSED [ 21%] test/models/test_albef.py::test_momentum_update PASSED [ 22%] test/models/test_albef.py::test_similarity PASSED [ 22%] test/models/test_albef.py::test_neg_embeddings PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_forward PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_resnet_forward PASSED [ 24%] test/models/test_clip.py::TestCLIP::test_clip_vit_forward PASSED [ 24%] test/models/test_mdetr.py::TestMDETR::test_transformer_encoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_transformer_decoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_full_mdetr_model PASSED [ 25%] test/models/test_omnivore.py::test_omnivore_swin_t_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_s_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_b_forward PASSED [ 27%] test/models/test_omnivore.py::test_omnivore_forward_wrong_input_type PASSED [ 27%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_hidden_dim_assertion PASSED [ 28%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_forward PASSED [ 28%] test/models/test_video_vqvae.py::TestVideoEncoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoDecoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_encode PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_decode PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_tokenize PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_forward PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_encode PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_decode PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_tokenize PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_forward PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_classification PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_pretraining PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image_text PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image_and_text PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_text PASSED [ 36%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_text PASSED [ 36%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_classification PASSED [ 37%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_embedding PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_image_encoder PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_embedding PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer PASSED [ 39%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer_attn_mask PASSED [ 39%] test/modules/encoders/test_albef_multimodal_encoder.py::test_multimodal_encoder PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_image_hidden_size PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_text_hidden_size PASSED [ 41%] test/modules/encoders/test_albef_multimodal_encoder.py::test_not_matching_input_batch_size PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_text_encoder PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_invalid_input_length PASSED [ 42%] test/modules/encoders/test_albef_text_encoder.py::test_not_matching_attention_mask_shape PASSED [ 42%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_vision_transformer PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_input_length PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_channel_dim PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_height PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_width PASSED [ 45%] test/modules/encoders/test_clip_resnet_encoder.py::TestCLIPModule::test_resnet PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_initialization PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_clip_parameters PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_attention_mask PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward_over_context_length PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_hash PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_invalid_pooling PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_max PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_mean PASSED [ 50%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_sum PASSED [ 50%] test/modules/encoders/test_mdetr_image_encoder.py::TestMDETRImageEncoder::test_resnet_101_forward PASSED [ 50%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_modified_transformer PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_embeddings PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_encoder PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_forward PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_invalid_partitioning PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_scripting PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_transformer_pooling PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_encoder PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_scripting PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_patch_merging_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d_zero_shift PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_max_pooling PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_mean_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_sum_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_scripting PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_input_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_no_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_scripted_model PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_apply_attention PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_auto_mapping PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_invalid_pooling PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_max PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_mean PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_median PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_min PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_modality_normalize PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_sum PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_transformer PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_get_deepset_transformer PASSED [ 64%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript PASSED [ 64%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_use_cache PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_causal_use_cache PASSED [ 66%] test/modules/layers/test_attention.py::test_scaled_dot_product_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_full_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_axial_attention PASSED [ 67%] test/modules/layers/test_attention.py::test_split_multihead PASSED [ 67%] test/modules/layers/test_attention.py::test_merge_multihead PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_forward PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_channel_dim PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_codebook_restart PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_ema_update_embedding PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_and_preprocess PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_smaller_encoded PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_postprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess_channel_dim_assertion PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_quantized_output PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_assert PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_output PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_assert PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_output PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconvtranspose3d_forward PASSED [ 75%] test/modules/layers/test_mlp.py::TestMLP::test_activation_and_normalization PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_dropout_default PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_no_dropout PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_no_hidden_layers PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_pass_hidden_dims PASSED [ 78%] test/modules/layers/test_mlp.py::TestMLP::test_torchscript PASSED [ 78%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_sets_embedding PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_bad_embedding_dim PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_broadcast PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward PASSED [ 80%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward_invalid_input PASSED [ 80%] test/modules/layers/test_transformer.py::TestTransformer::test_flava_encoder_forward PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_missing_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_target PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_without_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_sim_targets PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss_invalid_input_hidden_size PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_labels PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_missing_momentum_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss PASSED [ 87%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_with_distillation PASSED [ 87%] test/modules/losses/test_commitment.py::TestCommitment::test_loss_value PASSED [ 87%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_local_loss PASSED [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_multi_gpu_loss SKIPPED (Not enough GPUs to run the test: required 2) [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_single_gpu_loss SKIPPED (Not enough GPUs to run the test: required 1) [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_invalid PASSED [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_max PASSED [ 90%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_min PASSED [ 90%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_soft_token_prediction_loss PASSED [ 91%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_box_losses PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_single_transform PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_single_transform PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_call PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_wrong_channels PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_sample_frames PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_resize_hw PASSED [ 95%] test/transforms/test_video_transform.py::TestVideoTransform::test_normalize PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_different_lengths PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_same_lengths PASSED [ 96%] test/utils/test_common.py::test_shift_dim PASSED [ 96%] test/utils/test_common.py::TestTensorSlice::test_default PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_size_minus_one PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_uneven_begin_size PASSED [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_begin XFAIL (Invalid begin) [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_size XFAIL (Invalid size) [ 99%] test/utils/test_common.py::TestToTupleTuple::test_int PASSED [ 99%] test/utils/test_common.py::TestToTupleTuple::test_tuple PASSED [100%] ========================================================================================= warnings summary ========================================================================================== test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining /data/home/rafiayub/torchmultimodal/test/models/flava/test_flava_checkpoint.py:81: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). else torch.tensor(dict_actual[key]) test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:246: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:240: UserWarning: 'norm' was found in ScriptModule constants, but it is a non-constant submodule. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:169: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.code_avg.grad, msg_has_grad test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:171: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.embedding.grad, msg_has_grad test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward /data/home/rafiayub/torchmultimodal/torchmultimodal/modules/layers/conv.py:49: UserWarning: Padding was specified but will not be used in favor of same padding, use Conv3d directly for custom padding warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================= 220 passed, 2 skipped, 2 xfailed, 7 warnings in 81.22s (0:01:21) ================================================================== ``` Differential Revision: [D38056496](https://our.internmc.facebook.com/intern/diff/D38056496) [ghstack-poisoned]
RdoubleA
added a commit
that referenced
this pull request
Jul 21, 2022
…class" ## Summary Replaces the `FLAVASelfAttention` class with the generalized `SelfAttention`. Multihead splitting and merging was moved upto `FLAVAAttention`. Since this change modifies layers, the checkpoint was loaded, remapped, and saved to a new location. The link has been updated. The remapping was done in #188. ## Test plan `pytest -vv` and `pytest test/models/flava/test_flava_checkpoint.py -vv` ``` ======================================================================================== test session starts ======================================================================================== platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /fsx/users/rafiayub/conda/envs/torchmm/bin/python cachedir: .pytest_cache rootdir: /data/home/rafiayub/torchmultimodal, configfile: pyproject.toml plugins: cov-3.0.0, mock-3.8.2 collected 224 items examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_fixed_weight_and_bias PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_arg_lengths PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_output_dims PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_multiple_layer PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_scripting PASSED [ 2%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_single_layer PASSED [ 2%] examples/cnn_lstm/test/test_cnn_lstm.py::TestCNNLSTMModule::test_forward PASSED [ 3%] examples/cnn_lstm/test/test_lstm_encoder.py::TestLSTMEncoder::test_lstm_encoder PASSED [ 3%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_contrastive_alignment_loss PASSED [ 4%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_construct_positive_map PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[12-expected0] PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[5-expected1] PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_invalid_inputs PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_valid_inputs PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_s3d_base PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_basicconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_sepconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3b-192-256] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3c-256-480] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4b-480-512] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4c-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4d-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4e-512-528] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4f-528-832] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5b-832-832] PASSED [ 11%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5c-832-1024] PASSED [ 11%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_attention_mask PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_invalid_channels PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestProjection::test_forward PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_pretrained_trainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_pretrained_untrainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_untrained_trainable PASSED [ 15%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_untrained_untrainable PASSED [ 15%] test/architectures/test_late_fusion.py::TestLateFusion::test_forward PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_missing_key_in_modalities PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_script PASSED [ 16%] test/architectures/test_two_tower.py::TestTwoTower::test_shared_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower_scripting PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings_momentum PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_copy_params_momentum_models PASSED [ 21%] test/models/test_albef.py::test_dequeue_and_enqueue PASSED [ 21%] test/models/test_albef.py::test_momentum_update PASSED [ 22%] test/models/test_albef.py::test_similarity PASSED [ 22%] test/models/test_albef.py::test_neg_embeddings PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_forward PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_resnet_forward PASSED [ 24%] test/models/test_clip.py::TestCLIP::test_clip_vit_forward PASSED [ 24%] test/models/test_mdetr.py::TestMDETR::test_transformer_encoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_transformer_decoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_full_mdetr_model PASSED [ 25%] test/models/test_omnivore.py::test_omnivore_swin_t_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_s_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_b_forward PASSED [ 27%] test/models/test_omnivore.py::test_omnivore_forward_wrong_input_type PASSED [ 27%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_hidden_dim_assertion PASSED [ 28%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_forward PASSED [ 28%] test/models/test_video_vqvae.py::TestVideoEncoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoDecoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_encode PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_decode PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_tokenize PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_forward PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_encode PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_decode PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_tokenize PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_forward PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_classification PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_pretraining PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image_text PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image_and_text PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_text PASSED [ 36%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_text PASSED [ 36%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_classification PASSED [ 37%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_embedding PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_image_encoder PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_embedding PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer PASSED [ 39%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer_attn_mask PASSED [ 39%] test/modules/encoders/test_albef_multimodal_encoder.py::test_multimodal_encoder PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_image_hidden_size PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_text_hidden_size PASSED [ 41%] test/modules/encoders/test_albef_multimodal_encoder.py::test_not_matching_input_batch_size PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_text_encoder PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_invalid_input_length PASSED [ 42%] test/modules/encoders/test_albef_text_encoder.py::test_not_matching_attention_mask_shape PASSED [ 42%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_vision_transformer PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_input_length PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_channel_dim PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_height PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_width PASSED [ 45%] test/modules/encoders/test_clip_resnet_encoder.py::TestCLIPModule::test_resnet PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_initialization PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_clip_parameters PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_attention_mask PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward_over_context_length PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_hash PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_invalid_pooling PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_max PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_mean PASSED [ 50%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_sum PASSED [ 50%] test/modules/encoders/test_mdetr_image_encoder.py::TestMDETRImageEncoder::test_resnet_101_forward PASSED [ 50%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_modified_transformer PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_embeddings PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_encoder PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_forward PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_invalid_partitioning PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_scripting PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_transformer_pooling PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_encoder PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_scripting PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_patch_merging_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d_zero_shift PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_max_pooling PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_mean_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_sum_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_scripting PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_input_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_no_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_scripted_model PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_apply_attention PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_auto_mapping PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_invalid_pooling PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_max PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_mean PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_median PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_min PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_modality_normalize PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_sum PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_transformer PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_get_deepset_transformer PASSED [ 64%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript PASSED [ 64%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_use_cache PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_causal_use_cache PASSED [ 66%] test/modules/layers/test_attention.py::test_scaled_dot_product_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_full_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_axial_attention PASSED [ 67%] test/modules/layers/test_attention.py::test_split_multihead PASSED [ 67%] test/modules/layers/test_attention.py::test_merge_multihead PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_forward PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_channel_dim PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_codebook_restart PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_ema_update_embedding PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_and_preprocess PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_smaller_encoded PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_postprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess_channel_dim_assertion PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_quantized_output PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_assert PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_output PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_assert PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_output PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconvtranspose3d_forward PASSED [ 75%] test/modules/layers/test_mlp.py::TestMLP::test_activation_and_normalization PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_dropout_default PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_no_dropout PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_no_hidden_layers PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_pass_hidden_dims PASSED [ 78%] test/modules/layers/test_mlp.py::TestMLP::test_torchscript PASSED [ 78%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_sets_embedding PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_bad_embedding_dim PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_broadcast PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward PASSED [ 80%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward_invalid_input PASSED [ 80%] test/modules/layers/test_transformer.py::TestTransformer::test_flava_encoder_forward PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_missing_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_target PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_without_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_sim_targets PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss_invalid_input_hidden_size PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_labels PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_missing_momentum_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss PASSED [ 87%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_with_distillation PASSED [ 87%] test/modules/losses/test_commitment.py::TestCommitment::test_loss_value PASSED [ 87%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_local_loss PASSED [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_multi_gpu_loss SKIPPED (Not enough GPUs to run the test: required 2) [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_single_gpu_loss SKIPPED (Not enough GPUs to run the test: required 1) [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_invalid PASSED [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_max PASSED [ 90%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_min PASSED [ 90%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_soft_token_prediction_loss PASSED [ 91%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_box_losses PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_single_transform PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_single_transform PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_call PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_wrong_channels PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_sample_frames PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_resize_hw PASSED [ 95%] test/transforms/test_video_transform.py::TestVideoTransform::test_normalize PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_different_lengths PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_same_lengths PASSED [ 96%] test/utils/test_common.py::test_shift_dim PASSED [ 96%] test/utils/test_common.py::TestTensorSlice::test_default PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_size_minus_one PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_uneven_begin_size PASSED [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_begin XFAIL (Invalid begin) [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_size XFAIL (Invalid size) [ 99%] test/utils/test_common.py::TestToTupleTuple::test_int PASSED [ 99%] test/utils/test_common.py::TestToTupleTuple::test_tuple PASSED [100%] ========================================================================================= warnings summary ========================================================================================== test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining /data/home/rafiayub/torchmultimodal/test/models/flava/test_flava_checkpoint.py:81: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). else torch.tensor(dict_actual[key]) test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:246: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:240: UserWarning: 'norm' was found in ScriptModule constants, but it is a non-constant submodule. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:169: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.code_avg.grad, msg_has_grad test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:171: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.embedding.grad, msg_has_grad test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward /data/home/rafiayub/torchmultimodal/torchmultimodal/modules/layers/conv.py:49: UserWarning: Padding was specified but will not be used in favor of same padding, use Conv3d directly for custom padding warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================= 220 passed, 2 skipped, 2 xfailed, 7 warnings in 81.22s (0:01:21) ================================================================== ``` Differential Revision: [D38056496](https://our.internmc.facebook.com/intern/diff/D38056496) [ghstack-poisoned]
ankitade
approved these changes
Jul 25, 2022
|
@RdoubleA has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
facebook-github-bot
pushed a commit
that referenced
this pull request
Jul 25, 2022
Summary: Pull Request resolved: #168 ## Summary Replaces the `FLAVASelfAttention` class with the generalized `SelfAttention`. Multihead splitting and merging was moved upto `FLAVAAttention`. Since this change modifies layers, the checkpoint was loaded, remapped, and saved to a new location. The link has been updated. The remapping was done in #188. ## Test plan `pytest -vv` and `pytest test/models/flava/test_flava_checkpoint.py -vv` ``` ======================================================================================== test session starts ======================================================================================== platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /fsx/users/rafiayub/conda/envs/torchmm/bin/python cachedir: .pytest_cache rootdir: /data/home/rafiayub/torchmultimodal, configfile: pyproject.toml plugins: cov-3.0.0, mock-3.8.2 collected 224 items examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_fixed_weight_and_bias PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_arg_lengths PASSED [ 0%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_invalid_output_dims PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_multiple_layer PASSED [ 1%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_scripting PASSED [ 2%] examples/cnn_lstm/test/test_cnn_encoder.py::TestCNNEncoder::test_single_layer PASSED [ 2%] examples/cnn_lstm/test/test_cnn_lstm.py::TestCNNLSTMModule::test_forward PASSED [ 3%] examples/cnn_lstm/test/test_lstm_encoder.py::TestLSTMEncoder::test_lstm_encoder PASSED [ 3%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_contrastive_alignment_loss PASSED [ 4%] examples/mdetr/test/test_contrastive_alignment_loss.py::TestContrastiveAlignmentLoss::test_construct_positive_map PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[12-expected0] PASSED [ 4%] examples/mdetr/test/test_matcher.py::TestMatcher::test_matcher[5-expected1] PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_invalid_inputs PASSED [ 5%] examples/mdetr/test/test_postprocessors.py::TestFlickrPostProcessor::test_valid_inputs PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_s3d_base PASSED [ 6%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_basicconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_sepconv3d PASSED [ 7%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3b-192-256] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed3c-256-480] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4b-480-512] PASSED [ 8%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4c-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4d-512-512] PASSED [ 9%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4e-512-528] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed4f-528-832] PASSED [ 10%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5b-832-832] PASSED [ 11%] examples/mugen/test/retrieval/test_s3d.py::TestS3D::test_mixed[Mixed5c-832-1024] PASSED [ 11%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestTextEncoder::test_attention_mask PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_forward PASSED [ 12%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoEncoder::test_invalid_channels PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestProjection::test_forward PASSED [ 13%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_pretrained_trainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_pretrained_untrainable PASSED [ 14%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_forward_untrained_trainable PASSED [ 15%] examples/mugen/test/retrieval/test_video_clip.py::TestVideoCLIPBuilder::test_untrained_untrainable PASSED [ 15%] test/architectures/test_late_fusion.py::TestLateFusion::test_forward PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_missing_key_in_modalities PASSED [ 16%] test/architectures/test_late_fusion.py::TestLateFusion::test_script PASSED [ 16%] test/architectures/test_two_tower.py::TestTwoTower::test_shared_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower PASSED [ 17%] test/architectures/test_two_tower.py::TestTwoTower::test_two_tower_scripting PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings PASSED [ 18%] test/models/test_albef.py::test_albef_image_embeddings_momentum PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings PASSED [ 19%] test/models/test_albef.py::test_albef_text_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings PASSED [ 20%] test/models/test_albef.py::test_albef_multimodal_embeddings_momentum PASSED [ 20%] test/models/test_albef.py::test_copy_params_momentum_models PASSED [ 21%] test/models/test_albef.py::test_dequeue_and_enqueue PASSED [ 21%] test/models/test_albef.py::test_momentum_update PASSED [ 22%] test/models/test_albef.py::test_similarity PASSED [ 22%] test/models/test_albef.py::test_neg_embeddings PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_forward PASSED [ 23%] test/models/test_clip.py::TestCLIP::test_clip_resnet_forward PASSED [ 24%] test/models/test_clip.py::TestCLIP::test_clip_vit_forward PASSED [ 24%] test/models/test_mdetr.py::TestMDETR::test_transformer_encoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_transformer_decoder PASSED [ 25%] test/models/test_mdetr.py::TestMDETR::test_full_mdetr_model PASSED [ 25%] test/models/test_omnivore.py::test_omnivore_swin_t_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_s_forward PASSED [ 26%] test/models/test_omnivore.py::test_omnivore_swin_b_forward PASSED [ 27%] test/models/test_omnivore.py::test_omnivore_forward_wrong_input_type PASSED [ 27%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_hidden_dim_assertion PASSED [ 28%] test/models/test_video_vqvae.py::TestAttentionResidualBlock::test_forward PASSED [ 28%] test/models/test_video_vqvae.py::TestVideoEncoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoDecoder::test_forward PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_encode PASSED [ 29%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_decode PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_tokenize PASSED [ 30%] test/models/test_video_vqvae.py::TestVideoVQVAE::test_forward PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_encode PASSED [ 31%] test/models/test_vqvae.py::TestVQVAE::test_decode PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_tokenize PASSED [ 32%] test/models/test_vqvae.py::TestVQVAE::test_forward PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_classification PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVA::test_forward_pretraining PASSED [ 33%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_image_text PASSED [ 34%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_image_and_text PASSED [ 35%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_masked_text PASSED [ 36%] test/models/flava/test_flava.py::TestFLAVAModel::test_forward_text PASSED [ 36%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_classification PASSED [ 37%] test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_embedding PASSED [ 37%] test/models/flava/test_flava_image_encoder.py::TestFlavaImageEncoder::test_image_encoder PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_embedding PASSED [ 38%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer PASSED [ 39%] test/models/flava/test_flava_text_encoder.py::TestFlavaTextEncoder::test_text_transformer_attn_mask PASSED [ 39%] test/modules/encoders/test_albef_multimodal_encoder.py::test_multimodal_encoder PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_image_hidden_size PASSED [ 40%] test/modules/encoders/test_albef_multimodal_encoder.py::test_invalid_text_hidden_size PASSED [ 41%] test/modules/encoders/test_albef_multimodal_encoder.py::test_not_matching_input_batch_size PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_text_encoder PASSED [ 41%] test/modules/encoders/test_albef_text_encoder.py::test_invalid_input_length PASSED [ 42%] test/modules/encoders/test_albef_text_encoder.py::test_not_matching_attention_mask_shape PASSED [ 42%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_vision_transformer PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_input_length PASSED [ 43%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_channel_dim PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_height PASSED [ 44%] test/modules/encoders/test_albef_vision_encoder.py::TestALBEFVisionEncoder::test_invalid_image_width PASSED [ 45%] test/modules/encoders/test_clip_resnet_encoder.py::TestCLIPModule::test_resnet PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_initialization PASSED [ 45%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_clip_parameters PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_attention_mask PASSED [ 46%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_forward_over_context_length PASSED [ 47%] test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_hash PASSED [ 48%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_invalid_pooling PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_max PASSED [ 49%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_mean PASSED [ 50%] test/modules/encoders/test_embedding_encoder.py::TestEmbeddingEncoder::test_embedding_encoder_sum PASSED [ 50%] test/modules/encoders/test_mdetr_image_encoder.py::TestMDETRImageEncoder::test_resnet_101_forward PASSED [ 50%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_modified_transformer PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_embeddings PASSED [ 51%] test/modules/encoders/test_mdetr_text_encoder.py::TestMDETRTextEncoder::test_mdetr_text_encoder PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_forward PASSED [ 52%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_invalid_partitioning PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_scripting PASSED [ 53%] test/modules/encoders/test_mil_encoder.py::TestMILEncoder::test_transformer_pooling PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_encoder PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3d::test_swin_transformer_3d_scripting PASSED [ 54%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_patch_merging_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d PASSED [ 55%] test/modules/encoders/test_swin_transformer_3d_encoder.py::TestSwinTransformer3dComponents::test_shifted_window_attention_3d_zero_shift PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_max_pooling PASSED [ 56%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_mean_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_forward_sum_pooling PASSED [ 57%] test/modules/encoders/test_weighted_embedding_encoder.py::TestEmbeddingEncoder::test_scripting PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_input_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_no_projection_dim PASSED [ 58%] test/modules/fusions/test_attention_fusion.py::TestAttentionFusionModule::test_scripted_model PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_apply_attention PASSED [ 59%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_auto_mapping PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_invalid_pooling PASSED [ 60%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_max PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_mean PASSED [ 61%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_median PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_min PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_modality_normalize PASSED [ 62%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_sum PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_deepset_transformer PASSED [ 63%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_get_deepset_transformer PASSED [ 64%] test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript PASSED [ 64%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_use_cache PASSED [ 65%] test/modules/layers/test_attention.py::TestMultiheadAttention::test_multi_head_attention_causal_use_cache PASSED [ 66%] test/modules/layers/test_attention.py::test_scaled_dot_product_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_full_attention PASSED [ 66%] test/modules/layers/test_attention.py::test_axial_attention PASSED [ 67%] test/modules/layers/test_attention.py::test_split_multihead PASSED [ 67%] test/modules/layers/test_attention.py::test_merge_multihead PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_forward PASSED [ 68%] test/modules/layers/test_attention.py::TestAxialBlock::test_axial_block_channel_dim PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_codebook_restart PASSED [ 69%] test/modules/layers/test_codebook.py::TestCodebook::test_ema_update_embedding PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_and_preprocess PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_init_embedding_smaller_encoded PASSED [ 70%] test/modules/layers/test_codebook.py::TestCodebook::test_postprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess PASSED [ 71%] test/modules/layers/test_codebook.py::TestCodebook::test_preprocess_channel_dim_assertion PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_quantized_output PASSED [ 72%] test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_assert PASSED [ 73%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_same_padding_output PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_assert PASSED [ 74%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_calculate_transpose_padding_output PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward PASSED [ 75%] test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconvtranspose3d_forward PASSED [ 75%] test/modules/layers/test_mlp.py::TestMLP::test_activation_and_normalization PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_dropout_default PASSED [ 76%] test/modules/layers/test_mlp.py::TestMLP::test_no_dropout PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_no_hidden_layers PASSED [ 77%] test/modules/layers/test_mlp.py::TestMLP::test_pass_hidden_dims PASSED [ 78%] test/modules/layers/test_mlp.py::TestMLP::test_torchscript PASSED [ 78%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_sets_embedding PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_init_bad_embedding_dim PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_broadcast PASSED [ 79%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward PASSED [ 80%] test/modules/layers/test_position_embedding.py::TestBroadcastedPositionEmbedding::test_forward_invalid_input PASSED [ 80%] test/modules/layers/test_transformer.py::TestTransformer::test_flava_encoder_forward PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim PASSED [ 81%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_missing_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_m PASSED [ 82%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_invalid_sim_target PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_without_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_distillation PASSED [ 83%] test/modules/losses/test_albef.py::TestImageTextContrastiveLoss::test_itc_loss_with_sim_targets PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss_invalid_input_hidden_size PASSED [ 84%] test/modules/losses/test_albef.py::TestImageTextMatchingLoss::test_itm_loss PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_labels PASSED [ 85%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_invalid_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_missing_momentum_embeddings PASSED [ 86%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss PASSED [ 87%] test/modules/losses/test_albef.py::TestMaskedLanguageModelingLoss::test_mlm_loss_with_distillation PASSED [ 87%] test/modules/losses/test_commitment.py::TestCommitment::test_loss_value PASSED [ 87%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_local_loss PASSED [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_multi_gpu_loss SKIPPED (Not enough GPUs to run the test: required 2) [ 88%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_single_gpu_loss SKIPPED (Not enough GPUs to run the test: required 1) [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_invalid PASSED [ 89%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_max PASSED [ 90%] test/modules/losses/test_contrastive_loss_with_temperature.py::TestContrastiveLossWithTemperature::test_temperature_clamp_min PASSED [ 90%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_soft_token_prediction_loss PASSED [ 91%] test/modules/losses/test_mdetr_losses.py::TestMDETRLosses::test_box_losses PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_single_transform PASSED [ 91%] test/transforms/test_bert_text_transform.py::TestBertTextTransform::test_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_multi_transform PASSED [ 92%] test/transforms/test_clip_transform.py::TestCLIPTransform::test_clip_single_transform PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_call PASSED [ 93%] test/transforms/test_video_transform.py::TestVideoTransform::test_wrong_channels PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_sample_frames PASSED [ 94%] test/transforms/test_video_transform.py::TestVideoTransform::test_resize_hw PASSED [ 95%] test/transforms/test_video_transform.py::TestVideoTransform::test_normalize PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_different_lengths PASSED [ 95%] test/utils/test_assertion.py::TestAssertEqualLengths::test_same_lengths PASSED [ 96%] test/utils/test_common.py::test_shift_dim PASSED [ 96%] test/utils/test_common.py::TestTensorSlice::test_default PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_size_minus_one PASSED [ 97%] test/utils/test_common.py::TestTensorSlice::test_uneven_begin_size PASSED [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_begin XFAIL (Invalid begin) [ 98%] test/utils/test_common.py::TestTensorSlice::test_invalid_size XFAIL (Invalid size) [ 99%] test/utils/test_common.py::TestToTupleTuple::test_int PASSED [ 99%] test/utils/test_common.py::TestToTupleTuple::test_tuple PASSED [100%] ========================================================================================= warnings summary ========================================================================================== test/models/flava/test_flava_checkpoint.py::TestFLAVACheckpoint::test_flava_model_for_pretraining /data/home/rafiayub/torchmultimodal/test/models/flava/test_flava_checkpoint.py:81: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). else torch.tensor(dict_actual[key]) test/modules/encoders/test_clip_text_encoder.py::TestCLIPTextEncoder::test_scripting test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:246: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/fusions/test_deepset_fusion.py::TestDeepSetFusionModule::test_torchscript /fsx/users/rafiayub/conda/envs/torchmm/lib/python3.9/site-packages/torch/jit/_recursive.py:240: UserWarning: 'norm' was found in ScriptModule constants, but it is a non-constant submodule. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:169: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.code_avg.grad, msg_has_grad test/modules/layers/test_codebook.py::TestCodebook::test_register_buffer_tensors /data/home/rafiayub/torchmultimodal/test/modules/layers/test_codebook.py:171: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/conda/conda-bld/pytorch_1656918644659/work/build/aten/src/ATen/core/TensorBody.h:478.) assert not self.vq.embedding.grad, msg_has_grad test/modules/layers/test_conv.py::TestSamePadConv3d::test_samepadconv3d_forward /data/home/rafiayub/torchmultimodal/torchmultimodal/modules/layers/conv.py:49: UserWarning: Padding was specified but will not be used in favor of same padding, use Conv3d directly for custom padding warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================= 220 passed, 2 skipped, 2 xfailed, 7 warnings in 81.22s (0:01:21) ================================================================== ``` Test Plan: Imported from OSS Reviewed By: langong347 Differential Revision: D38056496 Pulled By: RdoubleA fbshipit-source-id: 6f1be7d56469e72f856c73ee46bfab1eb05d4989
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
As more components of FLAVA are unified/generalized with the other models, the checkpoint will fall out of sync and will no longer be able to load. Here, I upload a notebook that lets you quickly write a mapping function to remap the state_dict of the old checkpoint to the new architecture and executes it. I've added it to examples since I will likely run this notebook more times as unification proceeds.
Colab link: https://colab.research.google.com/drive/1j9Hl6uroFx2Ud642jfhQjFEFlo9uKrOi?usp=sharing
Test plan:
None, this does not overwrite the existing checkpoint. The updated FLAVA checkpoint link is reflected in #168