I found and fixed a bug that caused more VRAM than necessary being used, it was very noticeable in llama 3 models. The memory debug option makes it much easier to debug memory issues with the backend.

@0cc4m Thank you for the update! Unfortunately, it didn't fix the RAM problem on my side.

I tested with Hermes-2-Pro-Llama-3-Instruct-Merged-DPO-Q6_K.gguf, but it still uses way too much RAM.

Relevant part of logs with GGML_VULKAN_MEMORY_DEBUG enabled: vk_report2.txt

It's blazing fast now compared to Clblast (especially pp, very noticeable), but the model barely fits on my 16GB system.

Additionally, here's a report where the model simply stopped working - no gpu activity, with 8 gpu layers: vk_report_halted.txt

@MaggotHATE Thank you, that report shows that your high RAM use is not Vulkan host memory, since it only shows 80MB used. According to the log it should only be around 4.5GB RAM used.

When I record the total RAM used of CPU, CUDA and Vulkan with your model and ngl 9 on my system, I get 6.8GB for CPU, 7GB for CUDA and 5.4GB for Vulkan. ngl 1000 leads to 0.8GB for CUDA and 1.2GB for Vulkan. I'm not sure how to reproduce your issue.

What I did was run the new llama-cli with build/bin/llama-cli -f ~/llama.cpp/input.txt -c 2048 --ignore-eos -m models/Hermes-2-Pro-Llama-3-8B-Q6_K.gguf -ngl 9, wait until it is generating and then check the process memory usage with smem -c "pid user command rss". Can you check what that reports for you?

The CPU buffer appearing two times in llm_load_tensors is unexpected. I investigated why this happens, and it seems that ggml_backend_vk_host_buffer_type returns different values at different times during llm_load_tensors. I think this is because vk_instance.contexts[0].initialized is only set after the get_device_memory function is called, but there are calls to ggml_backend_vk_host_buffer_type before that happens, so it returns the cpu buffer type.

smem -c "pid user command rss"

Sorry, I didn't include more info: I use Windows 10, so I'm not sure it's relevant for me. I have just checked llama-cli with your parameters, but the RAM issue is the same. I don't know how to report process memory usage is a detailed way in Windows.

Since I have Vulkan SDK installation, maybe its Configurator would be of any help?

The CPU buffer appearing two times in llm_load_tensors is unexpected.

Now that I remember, InfinityKuno-2x7B.Q3_K_S.gguf doesn't give me this RAM problem. So I checked, and the report shows one instance of llm_load_tensors: CPU buffer size vk_report_MOE.txt

Thank you, that's a good hint. I'll look into that. I'm reworking the backend anyways to solve #7575.

It seems that this change broke Vulkan support for some GPUs (#8092)

I can confirm that this PR breaks Vulkan for me as mentioned in #8092 (using a Nvidia RTX 2060 (laptop), and running the model https://huggingface.co/TheBloke/airoboros-mistral2.2-7B-GGUF/blob/main/airoboros-mistral2.2-7b.Q4_K_S.gguf)