frankenmerges anyone kept using

[llmarch51]

Did a bit of testing and thought I'd post the result here.

I tried this with Gemma using TensorRT-LLM on 7900 XTX. The headline result was better than I expected, but HBM constraints still ended up being the limiting factor.

Curious if anyone else landed in the same place.

[lablab]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[marbleyard32]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[batchcraft]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[minthall]

Thanks, that helps. I don't think your conclusion is far off.

[relayforge]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[routearch]

following this because I'm in the same boat.

[vectorpoint]

Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

[northquill]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[thetashed24]

Did a small comparison this afternoon. On 4090 with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.