Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api: enable passing images in raw mode #8209

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

rylativity
Copy link

@rylativity rylativity commented Dec 22, 2024

I belive this should be considered a bug fix, as it resolves unexpected behavior and clarifies how to use undocumented capabilities - Images were not being passed to multimodal models when images were provided and raw was set to true as described in this issue.

This change introduces an optional boolean parameter prepend_images_to_raw_prompt, which if set to true will add the image placeholder fields necessary to pass the images through to the multimodal model. This parameter only has effect if raw is also set to true and images are provided. The prepend_images_to_raw_prompt parameter can be omitted, and the default behavior remains entirely unchanged to avoid breaking existing users.

It also updates the api.md documentation to describe the new functionality (including an example request/response), and also provides and example of manually passing an image placeholder in the text of a raw prompt directly, which offers more control over where the image is inserted in the raw prompt.

Ryan Stewart added 4 commits December 21, 2024 23:22
… prompts when set to true; associated documentation update in docs/api.md
api.md to provide examples of how to pass images to model alongside raw
prompt; also adds missing response to existing example
ation to api.md on how the image placeholder can be used
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant