prefill/decode separation in practice with Ollama

[nanogarden]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of add

Maybe, but I had the opposite result. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere.