small agent harness that stayed useful with LM Studio

[amberplane]

Wanted to compare notes with people doing similar things.

I'm mostly working with Qwen in llama.cpp on 5090. The part I'm trying to understand is NVFP4.

Would appreciate practical advice rather than theory.

[southden]

Wanted to compare notes with people doing similar things.

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[cudacrate]

Interesting, I hadn't tried that. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[southden]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[northgrid33]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[llm_wren]

Good point. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[northgrid33]

that fixed it for me too.

[bytegarden73]

Thanks, that helps. The pattern around small local stacks has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[southden]

Good point. The pattern around reranking has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[ggufvault]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?