desktop apps around local AI

[llmpeak]

Has anyone else run into this?

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling token cost.

Would appreciate practical advice rather than theory.

[rasterbench]

I think the bottleneck is somewhere else. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

[llmpeak]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking.

[kernelgrid]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[mlxgarden]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[deltagrid]

That matches what I was seeing. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[audioriver]

worth checking the driver first.

[vramstream]

What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[audioriver]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[vramstream]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.