+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: vLLM / SGLang (https://www.datasculpting.ai/inference/forumdisplay.php?fid=36)
+--- Thread: prefill/decode separation in practice with Ollama (/showthread.php?tid=250)
Pages:
1
2
prefill/decode separation in practice with Ollama - oxidemixer - 09-12-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around tokens/sec.
Curious if anyone else landed in the same place.
This is mostly for local work rather than public serving.
Re: prefill/decode separation in practice with Ollama - batchsize - 09-13-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: prefill/decode separation in practice with Ollama - deltapoint17 - 09-13-2026
I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: prefill/decode separation in practice with Ollama - batchsize - 09-13-2026
Quote:I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Good point. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: prefill/decode separation in practice with Ollama - batchcraft - 09-13-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: prefill/decode separation in practice with Ollama - batchsize - 09-16-2026
I tested this tonight. On mini PC with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: prefill/decode separation in practice with Ollama - thetashed24 - 09-16-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.
Re: prefill/decode separation in practice with Ollama - batchsize - 09-18-2026
Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: prefill/decode separation in practice with Ollama - nanogarden - 09-18-2026
Quote:Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of add
Maybe, but I had the opposite result. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere.
Re: prefill/decode separation in practice with Ollama - batchcraft - 09-18-2026
Quote:[quote]Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.