prefill/decode separation in practice with Ollama

[batchcraft]

[quote]Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.