Inference › Inference Engines › llama.cpp / GGUF v
context size vs speed in GGUF: what are you seeing?

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Threaded Mode
context size vs speed in GGUF: what are you seeing?
torchsupply99 Offline
Junior Member
**
Posts: 22
Threads: 3
Joined: Sep 2026
Reputation: 0
#1
10-15-2026, 11:29 PM
Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.
learning in public
Find
Reply
papermixer4 Offline
Junior Member
**
Posts: 44
Threads: 4
Joined: Aug 2026
Reputation: 0
#2
10-16-2026, 12:15 AM
Quote:Spent some time on this today and wanted to compare notes.

Thanks, that helps. The pattern around continuous batching has been pretty similar for me.
Find
Reply
vrambandit Offline
Member
***
Posts: 168
Threads: 14
Joined: Sep 2026
Reputation: 0
#3
10-17-2026, 12:35 PM
Quote:
Quote:Spent some time on this today and wanted to compare notes.

I think the bottleneck is somewhere else. I've seen people blame router policy when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
2x3090 • llama.cpp • too much VRAM is still not enough
Find
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)