https://github.com/navervision/KELIP

LiT: they released some models last week https://github.com/google-research/vision_transformer#lit-models

Audio: besides AudioCLIP there's also wav2clip with a different approach: https://github.com/descriptinc/lyrebird-wav2clip

https://github.com/allenai/reclip

https://github.com/facebookresearch/Detic

https://github.com/sallymmx/ActionCLIP

https://github.com/ChenRocks/UNITER

https://github.com/raoyongming/DenseCLIP

https://github.com/ttlmh/Bridge-Prompt

https://github.com/sonoisa/clip-japanese

• https://mmf.sh/docs/notes/projects
  • older stuff, e.g. ViLBET
• https://github.com/facebookresearch/multimodal
  • Newer stuff, e.g. FLAVA
  • https://github.com/facebookresearch/multimodal/tree/main/examples/flava
  • helloooooooo.... https://github.com/facebookresearch/multimodal/blob/668eabf273369eaf83a49016853425357f475c56/torchmultimodal/models/flava.py#L53
  • https://huggingface.co/aps/flava_full_pretrained_encoders_torchmm/tree/main

SLIP demo install using dapm and loading weights using old strategy from DD: alembics/disco-diffusion@c509aa1

!wget https://dl.fbaipublicfiles.com/slip/slip_base_100ep.pt
!pip install napm

import napm
url = 'https://github.com/facebookresearch/SLIP'
napm.pseudoinstall_git_repo(url, add_install_dir_to_path=True)

import torch
import napm
import SLIP
from SLIP.models import SLIP_VITB16, SLIP, SLIP_VITL16

sd = torch.load('slip_base_100ep.pt', map_location=torch.device('cpu') )
real_sd = {}
for k, v in sd['state_dict'].items():
  new_key = '.'.join(k.split('.')[1:]) # strips "module" prefix. sure, why not.
  #print(k, new_key) 
  real_sd[new_key] = v
del sd

SLIPB16model = SLIP_VITB16(ssl_mlp_dim=4096, ssl_emb_dim=256)
SLIPB16model.load_state_dict(real_sd)

CLOOB demo using dapm

!pip install git+https://github.com/openai/CLIP
import napm

url = "https://github.com/crowsonkb/cloob-training"
napm.pseudoinstall_git_repo(url, package_name='cloob')

import cloob
from cloob.cloob_training import model_pt, pretrained

config = pretrained.get_config('cloob_laion_400m_vit_b_16_16_epochs')
model = model_pt.get_pt_model(config)
checkpoint = pretrained.download_checkpoint(config)
model.load_state_dict(model_pt.get_pt_params(config, checkpoint), )
#model.eval().requires_grad_(False).to('cuda')

CoCa https://arxiv.org/abs/2205.01917

OTTER https://github.com/facebookresearch/OTTER

ruDOLPH

• https://colab.research.google.com/drive/1gmTDA13u709OXiAeXWGm7sPixRhEJCga?usp=sharing
• https://github.com/ai-forever/ru-dolph

https://socraticmodels.github.io/

https://github.com/yxuansu/MAGIC

https://github.com/rinnakk/japanese-clip (not the same as this)

This is a very large and seemingly very good CLIP in Chinese that @Dango233 has shown me: https://wukong-dataset.github.io/wukong-dataset/benchmark.html

One problem though: It's pre-trained weights are on Mindspore (Huawei's PyTorch) so someone would need to convert that...

https://github.com/mindspore-ai/models/tree/master/research/mm/wukong

maybe just the fine tuned model?

https://github.com/j-min/clip-caption-reward

A new (better, it seems) Multilingual CLIP https://github.com/FreddeFrallan/Multilingual-CLIP

@apolinario indeed and now it's packaged properly on pypi as multilingual-clip

it's also available for easy testing at https://rom1504.github.io/clip-retrieval/?useMclip=true&query=%E9%BB%84%E8%89%B2%E3%81%84%E7%8C%AB&back=https%3A%2F%2Fknn5.laion.ai&index=laion5B

@rom1504 @apolinario the m-clip release gave me a thought: maybe we could host mmc on pypi with essentially none of the other perceptors installed at all. Simple instructions for "finalizing" the mmc install could live in the README (as well as one-liners for specific perceptors PRN), and we could add a warning on import too. maybe we could ship an update script or a CLI command.

My thinking here is if we ship the core tooling as a bare library, then anyone could attach the mocking utilities upstream to quickly make new perceptors drop-in-able if they aren't already, which conversely would make them trivial to add to mmc (since they'd already be hooked into a conformant API one way or another).

Actually, it might be cleaner and simpler to isolate a simple mocking wrapper and package that for pypi?

I'm mostly just thinking out-loud now. Thoughts?

I like the idea and spirit and I feel eventually if MMC gets way too many perceptors making some optional make a lot of sense. Now starting with all optional, I'm not sure - regardless I think your idea holds - just not sure if we ship empty or with some basics (OpenAI + OpenCLIP for e.g.) and let users further install from then on

(New perceptor: https://github.com/microsoft/UniCL)

https://github.com/goel-shashank/CyCLIP

https://github.com/facebookresearch/omnivore

https://github.com/microsoft/GLIP

https://github.com/microsoft/RegionCLIP

https://github.com/Lednik7/CLIP-ONNX/tree/main/clip_onnx

https://github.com/OFA-Sys/OFA

Turkish CLIP https://github.com/yusufani/TrCLIP

https://github.com/salesforce/LAVIS

EVA-CLIP - https://github.com/baaivision/EVA/blob/master/clip/README.md

basically already api compliant