recipes for a good local stack

[vectorport]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

[vectorport]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[mlxgarden]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once

That matches what I was seeing. The pattern around KV cache pressure has been pretty similar for me.

[rocmriley]

Maybe, but I had the opposite result. I've seen people blame INT4 when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[quietstack]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[relayforge]

Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

[rocmriley]

Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

Appreciate the detail. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[mlxgarden]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[thetaplane]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[modgarden]

Tried to reproduce it just now. On B200 with LM Studio, I saw basically the same behavior. I expected the model choice to matter more than it did.