Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multimodal Embedding #7866

Closed
4 of 5 tasks
taowang1993 opened this issue Sep 1, 2024 · 3 comments
Closed
4 of 5 tasks

Multimodal Embedding #7866

taowang1993 opened this issue Sep 1, 2024 · 3 comments
Assignees
@Yawen-1010
Labels
💪 enhancement New feature or request 👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database.

Comments

@taowang1993
Copy link
Contributor

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Currently, Dify supports only text embedding.

But I need to show users images from my documents such as diagrams and graphs.

This feature is very useful in education, medicine, legal and finance domains.

Major vector DBs already support multimodal RAG.

https://weaviate.io/blog/multimodal-models

https://milvus.io/docs/multimodal_rag_with_milvus.md

https://jina.ai/news/jina-clip-v1-a-truly-multimodal-embeddings-model-for-text-and-image/

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
@dosubot dosubot bot added 👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database. 💪 enhancement New feature or request labels Sep 1, 2024
@dosubot dosubot bot mentioned this issue Sep 5, 2024
Closed
5 tasks
@friedinando
Copy link

+1

@Yawen-1010 Yawen-1010 self-assigned this Oct 22, 2024
@monotykamary
Copy link

https://docs.voyageai.com/docs/multimodal-embeddings
I think this is the near future for multimodal RAG, especially since OCR for Open WebUI and now Claude's Visual PDFs are getting heavier use-cases.

@dosubot DosuBot
Copy link

dosubot bot commented Dec 9, 2024

Hi, @taowang1993. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

  • You requested the addition of multimodal embedding support in Dify, emphasizing its importance in various fields.
  • @friedinando expressed support for this feature.
  • @monotykamary shared documentation on multimodal embeddings and highlighted the growing relevance of multimodal RAG.

Next Steps

  • Could you confirm if this issue is still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open.
  • If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot dosubot bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Dec 9, 2024
@dosubot dosubot bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 24, 2024
@dosubot dosubot bot removed the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Yawen-1010 Yawen-1010

Labels
💪 enhancement New feature or request 👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@monotykamary @friedinando @taowang1993 @Yawen-1010