+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Prompting & Context Engineering (https://www.datasculpting.ai/inference/forumdisplay.php?fid=15)
+--- Forum: Prompts / System Prompts (https://www.datasculpting.ai/inference/forumdisplay.php?fid=62)
+--- Thread: prompt length getting silly (/showthread.php?tid=509)
Pages:
1
2
prompt length getting silly - mlxgarden - 09-26-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of LM Studio and 5090 felt surprisingly usable once I tuned around tool calling.
Maybe useful as a starting point for newer folks.
Re: prompt length getting silly - tensorworks - 09-27-2026
Interesting, I hadn't tried that. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: prompt length getting silly - batchember - 09-28-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Re: prompt length getting silly - paperport64 - 09-28-2026
I think the bottleneck is somewhere else. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: prompt length getting silly - mlxgarden - 09-28-2026
A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: prompt length getting silly - cacheline - 09-29-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: prompt length getting silly - mintstation99 - 09-30-2026
Interesting, I hadn't tried that. The pattern around prefill/decode split has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: prompt length getting silly - mlxgarden - 09-30-2026
My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: prompt length getting silly - quiettoken - 10-01-2026
same here on my side.
Re: prompt length getting silly - batchember - 10-01-2026
I think the bottleneck is somewhere else. I've seen people blame prefill/decode split when the real issue was much more mundane.