Replies: 6 comments
-
Thanks for the report @njedema. I'm able to reproduce on my end; will keep you updated on a solution. |
Beta Was this translation helpful? Give feedback.
-
After thinking about this a bit more, this will actually be an issue with all Drivers. It just so happens that Llama has a relatively small context window. Do we want to truncate prompt input to models? It might hide an underlying issue with the input data being too large. |
Beta Was this translation helpful? Give feedback.
-
If we do then we need to log a warning. I am on the fence about this leaning towards not truncating and explicitly failing. IMO, should be on the user to truncate. Another option is adding an optional |
Beta Was this translation helpful? Give feedback.
-
Migrated to discussion to discuss whether Prompt Drivers should have input truncation. |
Beta Was this translation helpful? Give feedback.
-
I think a |
Beta Was this translation helpful? Give feedback.
-
Describe the bug
BedrockLlamaPromptModelDriver() does not enforce truncation for any PromptTasks. If the task prompt is longer than 2048 tokens when rendered, the following ValidationError is raised until the code exits.
To Reproduce
Steps to reproduce the behavior:
The follow snippet is sufficient to reproduce the error:
First, verify that you are able to invoke Llama2 on Bedrock; ensure that you change the PROFILE_NAME
Now run the code with a really long prompt. GETing content of a really long Wiki page will do. Again, be sure to change PROFILE_NAME.
Expected behavior
I expect GripTape to truncate the response to the maximum number of tokens allowed by Bedrock Llama2. Griptape does this for other Bedrock prompt drivers, such as BedrockClaudePromptModelDriver()
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Beta Was this translation helpful? Give feedback.
All reactions