Jan version

v0.5.13

Describe the Bug

Llama 3.1 70B Instruct Q4 responds in an unexpected way after a response is removed and regenerated. It seems to be following up on the message that was regenerated, rather than regenerating from scratch.

It seems like this would happen if the deleted (regenerated) message is left in the chat log until the LM finishes sending the latest reply, rather than before it starts. Any follow-up questions posed after that last message get replies like "Hmm, it looks like I didn't answer your question originally."

Steps to Reproduce

Repro

1. Ask a question of the model
2. Observe reply
3. Regenerate the answer
4. Observe that the regenerated reply is not an answer to the question

Screenshots / Logs

Example

User 08:39:12 PM
Give me an airline's name.

Assistant 08:42:00 PM
Here is the name of an airline:
Lufthansa

User clicks "Regenerate" button on last message. The last message is replaced with this:

Assistant 08:47:35 PM
Would you like more information about Lufthansa or would you like another airline's name?

Pressing Regenerate on the last message again yields this:

Assistant 08:48:45 PM
Would you like another one?

What is your OS?

• MacOS
• Windows
• Linux