latency vs throughput for small deployments on 7900 XTX

[rasterpoint]

Thanks, that helps. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

If you post your exact settings, people here can probably narrow it down pretty fast.