how much orchestration is too much: what are you seeing?

[cudaline]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of TensorRT-LLM and dual 3090 felt surprisingly usable once I tuned around embeddings.

Feel free to tear it apart if you see something dumb.

[bytegarden73]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[llm_wren]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[southden]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path w

subscribed, curious where this lands.

[llm_wren]

[quote]My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain

Interesting, I hadn't tried that. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vectorport]

[quote][quote]My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know th

I hit almost the same issue last night.

[northgrid33]

I'd start with the simpler stack.

[labline]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[ggufvault]

My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[llm_wren]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Once you know the plain path works, then add the faster path back in piece by piece.