comparing models without fooling yourself on Threadripper box

[kernelghost]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once prefill/decode split becomes important.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[sparrowworks88]

Appreciate the detail. I don't think your conclusion is far off.

[quietcraft]

A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[griffinloom66]

A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survive

Interesting, I hadn't tried that. The pattern around INT4 has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[labline]

[quote]A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem

I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[cudascope]

My first guess would be this: On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[cacheline]

My first guess would be this: On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glam

subscribed, curious where this lands.

[batchfield55]

subscribed, curious where this lands.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[northgrid33]

subscribed, curious where this lands.

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[deltastation]

Not sure I agree there. I've seen people blame structured output when the real issue was much more mundane.