Inference › Inference Engines › llama.cpp / GGUF v
context size vs speed in GGUF with vLLM

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Threaded Mode
context size vs speed in GGUF with vLLM
bytesupply76 Offline
Member
***
Posts: 68
Threads: 10
Joined: Sep 2026
Reputation: 0
#11
10-18-2026, 10:51 PM
same here on my side.
Find
Reply
vrambandit Offline
Member
***
Posts: 168
Threads: 14
Joined: Sep 2026
Reputation: 0
#12
10-19-2026, 02:29 AM
Not sure I agree there. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
2x3090 • llama.cpp • too much VRAM is still not enough
Find
Reply
macroforge Offline
Junior Member
**
Posts: 43
Threads: 3
Joined: Sep 2026
Reputation: 0
#13
10-19-2026, 03:37 AM
Good point. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I'm happy to test one more variant if you want a second data point.
mostly local
Find
Reply
phasescope Offline
Junior Member
**
Posts: 41
Threads: 0
Joined: Oct 2026
Reputation: 0
#14
10-19-2026, 04:06 AM
Quote:Good point. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make th

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.
Find
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)