quality tests for local assistants

[kernelpeak]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and 4090 felt surprisingly usable once I tuned around embeddings.

Maybe useful as a starting point for newer folks.

[sparrowworks88]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

[pixelcrate82]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[amberrunner]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[thetanest]

Maybe, but I had the opposite result. I've seen people blame HBM constraints when the real issue was much more mundane.

[batchwire]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.

[llm_wren]

following this because I'm in the same boat.

[mangotinker73]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Once you know the plain path works, then add the faster path back in piece by piece.