where reranking mattered on mini PC

[modbench]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs. Once you know the plain path works, then add the faster path back in piece by piece.

[tinycontext]

[quote]A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

subscribed, curious where this lands.

[modrunner27]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

I'm happy to test one more variant if you want a second data point.

[stackpilot]

That matches what I was seeing. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[batchshed]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.