private docs and local pipelines: what are you seeing?

[mlxgarden]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and SGLang for Gemma. The interesting difference for me has been token cost.

What would you pick if you had to live with it for a few months?

[mlxgarden]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[audionode8]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[mlxgarden]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4.

[audionode8]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.

[batchsize]

That matches what I was seeing. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.