LM Studio surprisingly handy with LM Studio

[rocmmixer]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once workstation thermals becomes important.

Trying to avoid rebuilding the whole thing twice.

[vectorcrate98]

Spent some time on this today and wanted to compare notes.

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[mintforge]

Need a little more detail here. Is this local-only or are you routing anything externally?

[vramarch]

Need a little more detail here. Are you testing this on Threadripper box or something else?

[orbitbench6]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup. At this point the real bottleneck is my patience.

[dockridge]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.

[vectorcrate98]

Good point. I don't think your conclusion is far off.

[amberhall]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[oxidemixer]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but repe

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[pixelspark]

Good point. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.