Ollama for power users: anyone testing this?

[vectorember]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and vLLM for Gemma. The interesting difference for me has been prefill/decode split.

Interested in tradeoffs more than fan loyalty.

[tensorfield88]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[pixelspark]

What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[vectorhouse71]

What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[dockridge]

worth checking the driver first.

[batchscope72]

Need a little more detail here. Is this local-only or are you routing anything externally?

[orbitbench6]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[edgeriver]

same here on my side.

[fusedhall]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.