long context that stayed useful with MLX

[promptfield]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between SGLang and MLX for Llama. The interesting difference for me has been embeddings.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.

[mellowsupply]

What exact setup are you running? What model, what runtime, and roughly what context size?

[turboscope5]

What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[fastfield4]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[bytesupply76]

Thanks, that helps. The pattern around INT4 has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[papermixer4]

That matches what I was seeing. The pattern around tail latency has been pretty similar for me. I keep promising myself this is the last rebuild.

[sanescope]

Thanks, that helps. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[southspark]

Ran a quick check on my box. On M3 Max with PyTorch, I saw basically the same behavior.