latest release actually worth attention

[bytelab]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[fastbench]

Appreciate the detail. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[fastlab]

that fixed it for me too.

same here on my side.

[bytelab]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[phaseshed]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[nanogarden]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I keep promising myself this is the last rebuild.

[labpath]

Appreciate the detail. I ended up in about the same place using llama.cpp.

[fastlab]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about batch size.

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[dockriver]

My first guess would be this: On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[bytestack]

that fixed it for me too.