prefix caching in llama.cpp

[vectorport]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and TensorRT-LLM for FLUX. The interesting difference for me has been long context.

Trying to avoid rebuilding the whole thing twice.

[embershed99]

Spent some time on this today and wanted to compare notes.

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. Turns out 'just one more test' was a lie.

[mangotinker73]

worth checking the driver first.

[vrambandit]

Interesting, I hadn't tried that. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[kernelrunner]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[embershed99]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetagrid15]

Ran a quick check on my box. On dual 3090 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.

[northgrid33]

Appreciate the detail. That matches what I saw on A6000.

[bytesupply76]

Ran a quick check on my box. On mini PC with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did.

[northcraft90]

Interesting, I hadn't tried that. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.