best small quant that still feels smart

[deltagrid]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: TensorRT-LLM on A6000, keeping structured output in mind from the start.

Maybe useful as a starting point for newer folks.

[promptfield]

What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[relaynest82]

Need a little more detail here. Is this local-only or are you routing anything externally?

[nanobrook]

that fixed it for me too.

[fastbench]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you know the plain path works, then add the faster path back in piece by piece.

[paperrunner]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you know

subscribed, curious where this lands.

[cudaline]

subscribed, curious where this lands.

[vrambandit]

subscribed, curious where this lands.

That matches what I was seeing. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cudaline]

Need a little more detail here. What model, what runtime, and roughly what context size?

[mangopath4]

worth checking the driver first.