+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Security, Privacy & Safety (https://www.datasculpting.ai/inference/forumdisplay.php?fid=23)
+--- Forum: Local Privacy (https://www.datasculpting.ai/inference/forumdisplay.php?fid=76)
+--- Thread: private docs and local pipelines: what are you seeing? (/showthread.php?tid=569)
private docs and local pipelines: what are you seeing? - mlxgarden - 09-28-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between Ollama and SGLang for Gemma. The interesting difference for me has been token cost.
What would you pick if you had to live with it for a few months?
Re: private docs and local pipelines: what are you seeing? - mlxgarden - 09-28-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: private docs and local pipelines: what are you seeing? - audionode8 - 09-30-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: private docs and local pipelines: what are you seeing? - mlxgarden - 09-30-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4.
Re: private docs and local pipelines: what are you seeing? - audionode8 - 09-30-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.
Re: private docs and local pipelines: what are you seeing? - batchsize - 09-30-2026
That matches what I was seeing. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.