Efficient inference using llama-mtmd-cli for high resolution images with reduced GPU VRAM usage

[deepshnv]

Proposed Optimizations

i) Modify load_tensors method in clip.cpp to assign/allocate vision model weights to the CPU and stream it to the device only at runtime. This provides memory savings == the total weight of the vision tensors and do not make the image encoding terribly slow as is seen with --no-mmproj-offload which keeps the computation also on the CPU

ii) During multi-modal inference, defer the LLM model and context initialization from mtmd-cli, and rather invoke it JIT from mtmd-helper after encoding(pre-encoding) the IMAGE chunk is done and CLIP models' smemory resources are freed. This preserves downstream decode order but frees vision VRAM earlier.
Thus peak VRAM reduces: SUM(clip encode, text decode) -> MAX(clip encode, text decode)

iii) Currently if we want to run VQA inference for very large images with Flash-Attention and GPU support on, e.g. command-line:- llama-mtmd-cli.exe -m Qwen2.5-VL-7B-Instruct-q8_0.gguf --mmproj Qwen2.5-VL-7B-Instruct-mmproj-bf16.gguf -p <PROMPT> --image <4K_IMAGE> -ngl 99 -c 12000 --image-max-tokens 8192 -n 100, I encounter runtime assertion failure: cpy.cu:359: GGML_ASSERT(ggml_nbytes(src0) <= INT_MAX) failed. Basically we start exceeding 2GB ggml_cpy limit.
The proposed solution here is to implement tiled flash-attention to avoid 2GB/INT_MAX ggml_cuda_cpy for inputs with large FA masks. Tile the Q tensor into smaller chunks, processes each with flash attention, concatenate results and apply output projection.

My Implementation

I have implemented each of these proposed code changes and you can find my changes here: https://github.com/deepshnv/llama.cpp/tree/deepshnv/mtmd_vram_opti
@ggerganov @slaren, I would request you to please go through these implementations. I will leave it to you and the other llama.cpp devs to decide if and how much of my code to use.
The draft PR can be found here: #17802

Results showing reduced VRAM usage while maintaining the performance metrics

All these numbers were gotten on the same PC with a RTX 5080 GPU
Command-line used: llama-mtmd-cli.exe -m Qwen2.5-VL-7B-Instruct-q8_0.gguf --mmproj Qwen2.5-VL-7B-Instruct-mmproj-bf16.gguf -p <PROMPT> --image <IMAGE> // rest of the cmdline flags

AI usage: The entire implementation is mostly my own code changes with minimal help from AI(cursor) limited to code structuring, navigation and commenting. Even then I did my due diligence and manual review before accepting any change.

[pwilkin]

I think at least @ngxson as the godfather of MTMD should also take a look.

[ngxson]

The proposed change is CUDA-specific and thus it isn't beneficial to other backends. I don't have much experiences working with CUDA backend so honestly I'm not the best person to review this.

But ideally I would prefer to make this as transparent to end-users as possible. Not everyone use CUDA. libmtmd also designed to work on all sort of embedded devices, like on ARM smartphone, so I prefer keeping the code simple and as backend-agnostic as possible.

Also, these kind of optimizations can be done easier when we move libmtmd into libllama in the future

[deepshnv]

I have tried to keep my changes as backend-agnostic as possible. Can you please point me to the CUDA-specific parts so that I can try and fix the implementation to make them backend-agnostic .
Also the implemented changes are guarded via the "--clip-reduced-vram" cmdline flag, except for the INT_MAX ggml_cuda_cpy limit part but a flag can be added for that as well, so even for non-CUDA backends it would not affect the baseline execution.

[ngxson]

A true backend-agnostic feature should have no changes to the public API. The array of API mtmd_invoke_llm_init_if_needed, mtmd_preencode_image has no effects for non-CUDA backend. They all rely on the fact that there must be some kind of JIT in the backend.

Now about the DX aspect, these APIs can be extremely confused to downstream developers who use llama.cpp/mtmd in their produce. Many of them aren't familiar with backend work and they will simply remove these code if it is too complicated to understand.

The main library llama.h also never exposes anything backend-specific functionalities for the same reason.

For features like this, I expect implementation which will make the least or ideally no changes either the public API mtmd.h or the user interface arg.cpp

---

Beside, I think we should keep mtmd system easy to understand because it was very buggy in the past, where each contributor with put their own thing without having an overview on the global architecture of the library.

I would prefer keeping these kinds of improvements as suggestions, an re-visit them when we finally move mtmd into libllama - we can even have them sharing the same GGML sched which can greatly improve performance and memory usage at the same time.

[deepshnv]

Hi ngxson, please look at the updated PR, i've made a new commit

I have tried to paraphrase your concerns and have addressed them below:

1. Public API methods in mtmd.h: Of the optimizations mentioned in the OP, only (ii) required changes to public API. I have addressed in my recent commit by moving the methods mtmd_invoke_llm_init_if_needed, mtmd_preencode_image etc. from mtmd.h to the mtmd-helper files. Also now only mtmd_set_llm_init_callback and mtmd_set_llm_context remain exported as MTMD_API as they are needed in mtmd-cli.cpp, rest other JIT functions are now internal. Does this address your primary concern?
2. changes to args.cpp: These are simple changes to reduce VRAM demand (and are vendor agnostic). I took inspiration from existing llama.cpp features/flags like --override-tensor, --no-mmproj-offload flags from arg.cpp, and usage of offload_kqv, n_gpu_layers in llama.h to control VRAM usage. These flags are very useful for systems with any GPU and are part of what makes llama-cpp so valuable.
   My changes reduce VRAM demand for high-res image inferencing. They allow users to run powerful native-res VLMs (such as Qwen2*VL) on high-res images alongside other concurrent applications such as video games. This will be useful for all systems with GPUs (vendor agnostic).
   Please let me know if you find my updated PR acceptable (or if there are any changes you'd like to see before approving). Thanks.

[ngxson]

Firstly, this change is not valid because you're delegating all the accesses to internal mtmd_context to mtmd-helper. The helper must NOT have any access to the internal data of the context - if you're doing this, you're not understanding the structure of mtmd.

For the rest of the implementation, IMO it's still far too complicated compared to the (human) resources that we have in the project to maintain it. It's not worth the efforts to maintain all of the duplicated code inside big blocks like if (ctx_clip.clip_reduced_vram), common_init_from_existing_model, build_flash_attn_tiled just to improve single-backend performance - I even doubt that other maintainers would be happy to take this risk.

At this point, the top priority of mtmd is support more models and modalities (that includes audio and image generation in the future). So, merging this much code right now will make it quite complicated for maintainers who will the code base of mtmd. In the far future, this will also make it tricker to refactor the code or even bring mtmd into the main libllama.

So no, sorry I cannot accept adding this much risk to the code base.

---

Just to note that recently, we added flash attention into mtmd which improved performance significantly while doesn't change much of the code (since most of the implementation is already confined in ggml) - improvements that make the least change to the downstream code like this are always more preferable and easier to maintain.