non-AI thing you built recently for Qwen

[sparrowshed]

Did a bit of testing and thought I'd post the result here.

I tried this with Qwen using PyTorch on mini PC. The headline result was better than I expected, but quant drift still ended up being the limiting factor.

Happy to post more details if anyone wants them.

[modbench]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

If you post your exact settings, people here can probably narrow it down pretty fast.

[visionyard23]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Wort

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[cudacrate]

What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece. Turns out 'just one more test' was a lie.

[mlxgarden]

What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. O

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[benchplease]

[quote]What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem surv

I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. The logs were somehow calmer than I was.

[macropoint]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.