+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: batching changes after update for Llama (/showthread.php?tid=816)
Pages:
1
2
Re: batching changes after update for Llama - edgecoast - 10-18-2026
Maybe, but I had the opposite result. I've seen people blame router policy when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
I'm happy to test one more variant if you want a second data point.
Re: batching changes after update for Llama - phaseport36 - 10-18-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps. I keep promising myself this is the last rebuild.
Re: batching changes after update for Llama - kernelmike - 10-19-2026
subscribed, curious where this lands.
Re: batching changes after update for Llama - llm_wren - 10-19-2026
Maybe, but I had the opposite result. I've seen people blame prefix caching when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.