workflow automation with local models on dual 3090

[bytegarden73]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once tail latency becomes important.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.

[labline]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Once you know the plain path works, then add the faster path back in piece by piece.

[llm_wren]

Thanks, that helps. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.