I think control_vector and lora related stuff should be re-grouped into a module, maybe called adapters (if someone has a better naming, feel free to comment). That's because they work kinda the same way, by "adding things" on top of the original cgraph.

I think this is a good place to merge this change. The project builds faster now and hopefully the code is organized a bit better. Will continue refactoring in follow-up PRs and any suggestions and recommendations are welcome. I've left some TODOs around the code and will try to address those next. After that will be looking for ways to separate the KV cache from the llama_context and enable support for multiple KV cache implementations.

After this PR, common_tokenize fails to tokenize any text, exits without console errors on llama_tokenize. Windows 10, static cpu build. Model and ctx are loaded at that moment (main-based application).

What change caused this? I don't see changes to llama_tokenize_internal function, for example. Are models required to be reconverted after this PR?

There should be no functional changes. Try to clean your build folder.

There should be no functional changes.

There's definitely a change, I see llama_n_ctx returning random numbers instead of the actual context size. At the same time, the context size is reported correctly on loading:

llama_new_context_with_model: n_ctx_per_seq (16384) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 16384, offload = 0, type_k = 'f16', type_v = 'f16', n_layer = 56, can_shift = 1

but later:

load: llama_n_ctx = 458134640

There is a slight change in common_init_result, both model and ctx are now unique pointer. I'm not sure if it's the root problem. (i.e. you need to modify your code to use model.get() instead of just model)

Please note that we don't provide stable interface for common library, so undocumented breaking changes are expected.

you need to modify your code to use model.get() instead of just model

That's already done according to the changes in main this PR introduces. The problem is that everything before llama_n_ctx seems to work fine, but it looks like ctx either doesn't load properly, or is unloaded (which didn't happen before and probably shouldn't happen).

undocumented breaking changes

That's exactly what I'm trying to find.

Be careful that with unique ptr, the ctx maybe deallocated when it goes out of scope. You should make sure that common_init_result always valid in the scope, otherwise .release() the ptr from unique ptr and use it as raw ptr.

ctx maybe deallocated when it goes out of scope

Yes, looks like that's what happens here. I see that llama_free(ctx) and llama_free_model(model) are no longer used, so deallocation and unloading happen automatically. It there a way to lock model and ctx manually now?

Although, it's a bit weird that this wasn't needed previously.

llama_free is no longer needed because common_init_result is in scope of main(), and deleter will be called automatically when we exit from main() (go out of scope)

Another way is: (but remember to free it manually afterwards)

common_init_result llama_init = common_init_from_params(params);
llama_model * model = llama_init.model.release();
llama_context * ctx = llama_init.context.release();

llama_free(ctx);
llama_free_model(model);

llama_model * model = llama_init.model.release();
llama_context * ctx = llama_init.context.release();

Yes, thank you, that's exactly what I needed!

So, release() allows to match previous behavior (for example, reloading works exactly as it used to), while .get() relies on common_init_result (I tested it, and keeping common_init_result as a part of the object helps, but regen doesn't work - looks like it should be cleared/reset manually too).

In cpp, you have notion of "ownership". For example, std::string owns a char * buffer under the hood that you can access it using str.data(), and buffer is freed once std::string goes out of scope.

Same for unique_ptr, llama_model * model is owned by unique_ptr and get() let you access it. You can also release() it from being owned.

unique_ptr cannot be reassigned, you can only reset() it or std::move it. Edit: it can be reassigned, which will be equivalent to reset()

Same for unique_ptr, llama_model * model is owned by unique_ptr and get() let you access it. You can also release() it from being owned.

I understand that, but the problem is that release() isn't used anywhere for model and ctx, so there are no examples to learn that from. Plus, in practical sense, functionality has changed in the examples (they are examples to learn from!), because it used to rely on free functions, while now they rely on simply going out of scope.