+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: little utility you keep installed (/showthread.php?tid=570)
Pages:
1
2
little utility you keep installed - mlxgarden - 09-29-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: llama.cpp on Threadripper box, keeping small local stacks in mind from the start.
Feel free to tear it apart if you see something dumb.
Re: little utility you keep installed - vramstream - 09-29-2026
I hit almost the same issue last night.
Re: little utility you keep installed - fastquill74 - 09-29-2026
Quote:I hit almost the same issue last night.
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: little utility you keep installed - rasterbench - 10-01-2026
Quote:Quote:I hit almost the same issue last night.
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: little utility you keep installed - deltagrid - 10-01-2026
Quote:Quote:Quote:I hit almost the same issue last night.
Tried to reproduce it just now. On 5090 with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: little utility you keep installed - cachebench - 10-01-2026
What worked for me was pretty simple. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: little utility you keep installed - mintstation99 - 10-01-2026
That matches what I was seeing. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
I'm happy to test one more variant if you want a second data point.
Re: little utility you keep installed - llm_wren - 10-01-2026
I hit almost the same issue last night.
Re: little utility you keep installed - audioriver - 10-01-2026
That matches what I was seeing. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: little utility you keep installed - vramstream - 10-02-2026
Good point. I don't think your conclusion is far off.