working config for 32k context for Llama

[mlxgarden]

Did a bit of testing and thought I'd post the result here.

After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around prefill/decode split.

Curious if anyone else landed in the same place.

[rocmriley]

What exact setup are you running? Is this local-only or are you routing anything externally?

[mangoquill]

What exact setup are you running? Is this local-only or are you routing anything externally?

Appreciate the detail. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmhouse90]

What exact setup are you running? Is this local-only or are you routing anything externally?

Appreciate the detail. The pattern around batch size has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[vectorport]

I hit almost the same issue last night. At this point the real bottleneck is my patience.

[rocmhouse90]

I hit almost the same issue last night. At this point the real bottleneck is my patience.

I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[audioriver]

I hit almost the same issue last night.

[rocmcrate26]

I hit almost the same issue last night.

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[rocmhouse90]

worth checking the driver first.

If you post your exact settings, people here can probably narrow it down pretty fast.

[rasterbench]

Appreciate the detail. That matches what I saw on 7900 XTX.