I have tried to port the original main from llama.cpp to python here: https://gist.github.com/SagsMug/c39d5f337b0e0faf32b3ff6bc5bf4956 It seems to work okay

This is cool, I'd like to include python ports of the llama.cpp examples to give people an idea of how to use the low-level api. Gladly accept any PRs for this as well.

Yeah was confused because generate only took the minimal vars. Thought about hacking an interactive mode into Llamas default call itself something like this BlackLotus/llama-cpp-python@main...example (DISCLAIMER: hacked together in a minute, untested, just for show, not usable code) but since this seemed way too hackish I implemented a minimal PoC that does it's own "session management" like stop words. Anyway thanks for the reply it really helped.

Ahh gotcha, the reason I haven't implemented continuation / interactive mode in the call function is because there are some bugs with continuing a conversation that ended in a stop word (technically the model has seen the stop word even if you haven't returned the text). There was a recent update to the llama.h API that allows you to restore to a previous model state so I'll work on this a bit.

Interactive mode/Session #7

Description

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions