desk setup thread for Qwen

[paperport64]

Has anyone else run into this?

I'm mostly working with Mixtral in PyTorch on Threadripper box. The part I'm trying to understand is token cost.

Not looking for the perfect answer, mostly what worked in real use.

[fastfield4]

Thanks, that helps. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[quietsupply11]

worth checking the driver first.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[rasterpoint]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[bytelab]

Appreciate the detail. The pattern around tail latency has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[macrostack69]

Thanks, that helps. The pattern around reranking has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[deepseeked]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[dockworks23]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

A couple questions before I guess too hard. What model, what runtime, and roughly what context size? Turns out 'just one more test' was a lie.

[embersmith]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[vramscope]

I hit almost the same issue last night.