what music do you benchmark to

[bytehall]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, llama.cpp, dual 3090, with a lot of attention on continuous batching.

Feel free to tear it apart if you see something dumb.

This is mostly for local work rather than public serving.

[phasestream53]

I hit almost the same issue last night.

[batchpeak]

That matches what I was seeing. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.