4090 vs 5090 for 70B quants

[batchwire]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between PyTorch and MLX for Kimi. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.

[cudapoint]

Spent some time on this today and wanted to compare notes.

Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[rocmriley]

subscribed, curious where this lands.

[signaltrack44]

That matches what I was seeing. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[turbogrid11]

That matches what I was seeing. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[fastscope]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[turbogrid11]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[southspark]

following this because I'm in the same boat.

[rocmriley]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[batchshed]

Good point. I ended up in about the same place using SGLang.