Inference
setup notes worth sharing: what are you seeing? - Printable Version

+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: setup notes worth sharing: what are you seeing? (/showthread.php?tid=553)

setup notes worth sharing: what are you seeing? - mangoquill - 09-29-2026

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

Re: setup notes worth sharing: what are you seeing? - batchshed - 09-29-2026

I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

Re: setup notes worth sharing: what are you seeing? - nanobrook - 09-29-2026

worth checking the driver first. Turns out 'just one more test' was a lie.

Re: setup notes worth sharing: what are you seeing? - llmpeak - 09-30-2026

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

Re: setup notes worth sharing: what are you seeing? - relayforge - 09-30-2026

Thanks, that helps. I don't think your conclusion is far off.

Re: setup notes worth sharing: what are you seeing? - mangoquill - 10-03-2026

Ran a quick check on my box. On mini PC with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.

Re: setup notes worth sharing: what are you seeing? - mlxgarden - 10-03-2026

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

Re: setup notes worth sharing: what are you seeing? - rocmhouse90 - 10-04-2026

Quote:My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

If you post your exact settings, people here can probably narrow it down pretty fast.

Re: setup notes worth sharing: what are you seeing? - audioriver - 10-04-2026

I tested this tonight. On 7900 XTX with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.