+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: Off Topic (https://www.datasculpting.ai/inference/forumdisplay.php?fid=27)
+--- Thread: desk setup thread for Qwen (/showthread.php?tid=516)
Pages:
1
2
desk setup thread for Qwen - paperport64 - 09-26-2026
Has anyone else run into this?
I'm mostly working with Mixtral in PyTorch on Threadripper box. The part I'm trying to understand is token cost.
Not looking for the perfect answer, mostly what worked in real use.
Re: desk setup thread for Qwen - fastfield4 - 09-26-2026
Thanks, that helps. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: desk setup thread for Qwen - quietsupply11 - 09-27-2026
worth checking the driver first.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: desk setup thread for Qwen - rasterpoint - 09-27-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: desk setup thread for Qwen - bytelab - 09-27-2026
Appreciate the detail. The pattern around tail latency has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: desk setup thread for Qwen - macrostack69 - 09-28-2026
Thanks, that helps. The pattern around reranking has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: desk setup thread for Qwen - deepseeked - 09-29-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Re: desk setup thread for Qwen - dockworks23 - 09-29-2026
Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
A couple questions before I guess too hard. What model, what runtime, and roughly what context size? Turns out 'just one more test' was a lie.
Re: desk setup thread for Qwen - embersmith - 09-29-2026
What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: desk setup thread for Qwen - vramscope - 09-29-2026
I hit almost the same issue last night.