+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Proprietary / Frontier (https://www.datasculpting.ai/inference/forumdisplay.php?fid=48)
+--- Thread: 70B still worth the pain (/showthread.php?tid=845)
Pages:
1
2
Re: 70B still worth the pain - routeline23 - 10-08-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: 70B still worth the pain - edgekite - 10-08-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.
Re: 70B still worth the pain - plaingarden18 - 10-08-2026
My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: 70B still worth the pain - deltastation - 10-11-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.