router in front of vLLM

[phasestream53]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around Ollama and dual 3090.

I'm especially interested in mistakes worth avoiding.

[cacheline]

Posting this in case it saves someone else a little time.

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt? At this point the real bottleneck is my patience.

[oxidemixer]

that fixed it for me too.