latency vs throughput for small deployments with PyTorch

[amberrunner]

Curious what people here are seeing.

My current setup is TensorRT-LLM with M3 Max, and I keep running into KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[vramport]

I hit almost the same issue last night.

[rocmpath]

same here on my side.

[tensorfield88]

same here on my side.

worth checking the driver first.

[papermixer4]

same here on my side.

I'd start with the simpler stack.

[localpatch]

same here on my side.

[kernelmike]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[tinycontext]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was l

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[kernelmike]

[quote]Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that move

Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[signalcraft14]

[quote][quote]Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing th

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. At this point the real bottleneck is my patience.