+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: MCP / APIs / Tool Calling (https://www.datasculpting.ai/inference/forumdisplay.php?fid=46)
+--- Thread: frankenmerges anyone kept using (/showthread.php?tid=591)
Pages:
1
2
frankenmerges anyone kept using - llmarch51 - 09-29-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Gemma using TensorRT-LLM on 7900 XTX. The headline result was better than I expected, but HBM constraints still ended up being the limiting factor.
Curious if anyone else landed in the same place.
Re: frankenmerges anyone kept using - lablab - 09-30-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: frankenmerges anyone kept using - marbleyard32 - 10-01-2026
Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: frankenmerges anyone kept using - batchcraft - 10-01-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: frankenmerges anyone kept using - minthall - 10-02-2026
Thanks, that helps. I don't think your conclusion is far off.
Re: frankenmerges anyone kept using - relayforge - 10-04-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Re: frankenmerges anyone kept using - routearch - 10-04-2026
following this because I'm in the same boat.
Re: frankenmerges anyone kept using - vectorpoint - 10-05-2026
Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere.
Re: frankenmerges anyone kept using - northquill - 10-06-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: frankenmerges anyone kept using - thetashed24 - 10-06-2026
Did a small comparison this afternoon. On 4090 with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.