comparing models without fooling yourself for Phi

[cacheline]

Did a bit of testing and thought I'd post the result here.

I tried this with FLUX using MLX on A6000. The headline result was better than I expected, but tail latency still ended up being the limiting factor.

Curious if anyone else landed in the same place.

[cudascope]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[mangopath4]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Once you know the plain path works, then add the faster path back in piece by piece.

[batchfield55]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Once yo

I think the bottleneck is somewhere else. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[nanobrook]

Thanks, that helps. The pattern around MoE serving has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mangoquill]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once you know the plain path works, then add the faster path back in piece by piece.

[cacheline]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once yo

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[circuitworks90]

I'd start with the simpler stack.

[cudapeak]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[griffinloom66]

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.