+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Proprietary / Frontier (https://www.datasculpting.ai/inference/forumdisplay.php?fid=48)
+--- Thread: 8B vs 14B sweet spot (/showthread.php?tid=498)
Pages:
1
2
8B vs 14B sweet spot - deepseeked - 09-26-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and GB200.
Simple and boring is a valid answer. At this point the real bottleneck is my patience.
Re: 8B vs 14B sweet spot - circuitkite - 09-27-2026
I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: 8B vs 14B sweet spot - localtinker - 09-27-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: 8B vs 14B sweet spot - rasterbench - 09-27-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding.
Re: 8B vs 14B sweet spot - sanescope - 09-28-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding.
same here on my side.
Re: 8B vs 14B sweet spot - benchplease - 09-28-2026
Good point. That matches what I saw on dual 3090.
Re: 8B vs 14B sweet spot - kernelghost - 09-28-2026
Quote:Good point. That matches what I saw on dual 3090.
Interesting, I hadn't tried that. That matches what I saw on 4090. The biggest change for me was simplifying the stack instead of adding one more optimization layer. Turns out 'just one more test' was a lie.
Re: 8B vs 14B sweet spot - batchwire - 09-29-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.
Re: 8B vs 14B sweet spot - thetagrid15 - 10-02-2026
Can you fill in a bit more context? Are you testing this on mini PC or something else?
Re: 8B vs 14B sweet spot - fusedhall56 - 10-05-2026
worth checking the driver first.