+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Benchmarks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=65)
+--- Thread: comparing models without fooling yourself for Phi (/showthread.php?tid=397)
Pages:
1
2
comparing models without fooling yourself for Phi - cacheline - 09-22-2026
Did a bit of testing and thought I'd post the result here.
I tried this with FLUX using MLX on A6000. The headline result was better than I expected, but tail latency still ended up being the limiting factor.
Curious if anyone else landed in the same place.
Re: comparing models without fooling yourself for Phi - cudascope - 09-22-2026
A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: comparing models without fooling yourself for Phi - mangopath4 - 09-22-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Once you know the plain path works, then add the faster path back in piece by piece.
Re: comparing models without fooling yourself for Phi - batchfield55 - 09-25-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Once yo
I think the bottleneck is somewhere else. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: comparing models without fooling yourself for Phi - nanobrook - 09-26-2026
Thanks, that helps. The pattern around MoE serving has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: comparing models without fooling yourself for Phi - mangoquill - 09-29-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once you know the plain path works, then add the faster path back in piece by piece.
Re: comparing models without fooling yourself for Phi - cacheline - 09-29-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once yo
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: comparing models without fooling yourself for Phi - circuitworks90 - 10-02-2026
I'd start with the simpler stack.
Re: comparing models without fooling yourself for Phi - cudapeak - 10-05-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.
Re: comparing models without fooling yourself for Phi - griffinloom66 - 10-06-2026
My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.