what music do you benchmark to

[bytehall]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, llama.cpp, dual 3090, with a lot of attention on continuous batching.

Feel free to tear it apart if you see something dumb.

This is mostly for local work rather than public serving.