You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On Qwen2-7B,
when K/V cache both q4_0 produces weird results.
when k is q4_0 and v is q8_0 produces weird results.
when k is q8_0 and v is q4_0 produces normal results.
The text was updated successfully, but these errors were encountered:
Allow set the type of K/V cache separately
On Qwen2-7B,
when K/V cache both
q4_0
produces weird results.when k is
q4_0
and v isq8_0
produces weird results.when k is
q8_0
and v isq4_0
produces normal results.The text was updated successfully, but these errors were encountered: