Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow set the type of K/V cache separately #8332

Open
ag2s20150909 opened this issue Jan 7, 2025 · 0 comments
Open

Allow set the type of K/V cache separately #8332

ag2s20150909 opened this issue Jan 7, 2025 · 0 comments
Labels
feature request New feature or request

Comments

@ag2s20150909
Copy link

ag2s20150909 commented Jan 7, 2025

Allow set the type of K/V cache separately

On Qwen2-7B,
when K/V cache both q4_0 produces weird results.
when k is q4_0 and v is q8_0 produces weird results.
when k is q8_0 and v is q4_0 produces normal results.

@ag2s20150909 ag2s20150909 added the feature request New feature or request label Jan 7, 2025
@ag2s20150909 ag2s20150909 changed the title Allows to set the type of K/V cache separately Allow set the type of K/V cache separately Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant