why did throughput drop after update

[cacheline]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and mini PC.

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[routecraft]

I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[cacheline]

That matches what I was seeing. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[mangofield33]

following this because I'm in the same boat.

[cacheline]

following this because I'm in the same boat.

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[thetaplane]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[turboarch95]

Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[thetagrid15]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[batchsize]

same here on my side.

[promptpoint]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?