09-20-2026, 04:46 PM
Maybe I'm missing something obvious.
My current setup is llama.cpp with GB200, and I keep running into prefill/decode split.
Not looking for the perfect answer, mostly what worked in real use.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
My current setup is llama.cpp with GB200, and I keep running into prefill/decode split.
Not looking for the perfect answer, mostly what worked in real use.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
throughput matters, but tail latency matters too