desktop runtimes for non-devs

[edgeriver]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, 5090, with a lot of attention on tool calling.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

[edgeriver]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[amberhall]

I'd start with the simpler stack.

[latentscope35]

I hit almost the same issue last night.

[prompttinker44]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[deltagrid]

Did a small comparison this afternoon. On mini PC with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality.

[dockridge]

Can you fill in a bit more context? Is this local-only or are you routing anything externally? Turns out 'just one more test' was a lie.

[modbench]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[torchtrack24]

following this because I'm in the same boat. At this point the real bottleneck is my patience.

[mellowsupply]

That matches what I was seeing. The pattern around embeddings has been pretty similar for me. The logs were somehow calmer than I was.