prefill/decode separation in practice with Ollama

Thread Rating:

0 Vote(s) - 0 Average
1
2
3
4
5

Linear Mode

prefill/decode separation in practice with Ollama

batchsize Offline

Member

Posts: 137
Threads: 16
Joined: Sep 2026
Reputation: 0

09-16-2026, 01:22 PM

I tested this tonight. On mini PC with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

throughput matters, but tail latency matters too

Find

« Next Oldest | Next Newest »

Messages In This Thread

prefill/decode separation in practice with Ollama - by oxidemixer - 09-12-2026, 11:49 PM

Re: prefill/decode separation in practice with Ollama - by batchsize - 09-13-2026, 02:03 AM

Re: prefill/decode separation in practice with Ollama - by deltapoint17 - 09-13-2026, 02:47 AM

Re: prefill/decode separation in practice with Ollama - by batchsize - 09-13-2026, 11:31 AM

Re: prefill/decode separation in practice with Ollama - by batchcraft - 09-13-2026, 12:36 PM

Re: prefill/decode separation in practice with Ollama - by promptpoint - 09-19-2026, 11:18 PM

Re: prefill/decode separation in practice with Ollama - by batchsize - 09-16-2026, 01:22 PM

Re: prefill/decode separation in practice with Ollama - by turboarch95 - 09-20-2026, 01:43 AM

Re: prefill/decode separation in practice with Ollama - by thetashed24 - 09-16-2026, 05:48 PM

Re: prefill/decode separation in practice with Ollama - by nanogarden - 09-18-2026, 06:33 AM

Re: prefill/decode separation in practice with Ollama - by batchsize - 09-18-2026, 06:07 AM

Re: prefill/decode separation in practice with Ollama - by batchcraft - 09-18-2026, 10:36 AM

Re: prefill/decode separation in practice with Ollama - by batchcraft - 09-20-2026, 09:59 AM

Re: prefill/decode separation in practice with Ollama - by nanogarden - 09-20-2026, 12:37 PM

View a Printable Version

Forum Jump:

Users browsing this thread: 1 Guest(s)

Login
Username:
Password:	Lost Password?
	Remember me

Sign in with X.com