do small edge boxes make sense with llama.cpp

[kernelghost]

Did a bit of testing and thought I'd post the result here.

After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

[sanepeak48]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed. At this point the real bottleneck is my patience.

[cobaltworks25]

That matches what I was seeing. The pattern around prefill/decode split has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tensorkite]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.

[amberpatch25]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.

Thanks, that helps. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[visionforge]

I think the bottleneck is somewhere else. I've seen people blame VRAM headroom when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[tensorfield88]

I think the bottleneck is somewhere else. I've seen people blame VRAM headroom when the real issue was much more mundane. Worth testing the simpler ex

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. At this point the real bottleneck is my patience.

[flintriver50]

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[audiorunner74]

A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[tensorport]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there.