I think it could be valuable

I'm wondering where would be a good place to support all clip models

do you think a all-clip package could be good ? I can build it from https://github.com/rom1504/clip-retrieval/blob/main/clip_retrieval/clip_inference/mapper.py and https://github.com/rom1504/clip-retrieval/blob/main/clip_retrieval/clip_inference/load_clip.py

I was pointed to https://github.com/dmarx/Multi-Modal-Comparators ; looking into it

opened ai-forever/ru-clip#20 and rinnakk/japanese-clip#2 as it was pointed out to me that translating class names is not trivial

can we use http://compling.hss.ntu.edu.sg/omw/ to automatically get class names for imagenet in all languages

for other datasets, maybe https://www.wikidata.org/wiki/Wikidata:Main_Page could work

opened dmarx/Multi-Modal-Comparators#26

will wait a little

can we use http://compling.hss.ntu.edu.sg/omw/ to automatically get class names for imagenet in all languages

That looks very good, I suspected that we need to change the data structure of classnames/templates from plain fixed dicts in the code to something a bit more advanced due to these kind of use cases. So, maybe we could think about the data structure. We would have multiple languages, multiple datasets. For each (language, dataset) pair, multiple sets of templates (e.g., LiT experimented with different sets of templates and showed their effect on zero-shot accuracy) and also possibly multiple sets of classnames. So for each evaluation run we need to select the dataset, the language, which version of the templates to use, which version of the classnames to use, and we should have a default/recommended value for both templates and classnames to use. We will also need to populate this database, as we add more datasets and languages, sometimes in a programmatic way, e.g. from the multilingual wordnet link you sent. I don't know if a relational database is needed e.g. with SQLite, possibly too much, perhaps just a JSON file or a CSV would suffice. Any thoughts on this @rom1504 ?

I think a json file is probably best for small data like this where we can have a clear schema and we want things to be easy to explore/check

moar models! dmarx/Multi-Modal-Comparators#2

https://huggingface.co/clip-italian/clip-italian

https://github.com/OFA-Sys/Chinese-CLIP

@rom1504 @mehdidc Hi, thank you for considering this. For Japanese CLIP, I sent a PR to support our models. I am happy to be reviewed. Thanks again.
#50

can we directly use "open_clip" models from huggingface hub? https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224

import open_clip

model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
tokenizer = open_clip.get_tokenizer('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')

@usuyama Yes, that would be cool, it works fine already, e.g.
clip_benchmark eval --dataset=vtab/pcam --model="hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224" --pretrained=""

@mehdidc Great!

Tried and seeing this error - any thoughts?

notebook: https://colab.research.google.com/drive/1UmHOIOGNtOTZeOcc47jL-cZep2iEFx39?usp=sharing

@usuyama Here you would also need to io install VTAB's repo: pip install task_adaptation==0.1

@usuyama As an alternative, you can use the WDS version is much faster to download/prepare: clip_benchmark eval --dataset=wds/vtab/pcam --dataset_root "https://huggingface.co/datasets/clip-benchmark/wds_vtab-pcam/tree/main" --model="hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224" --pretrained=""