structured output failures with vLLM
|
structured output failures with vLLM
|
10-19-2026, 02:30 AM
Quote:Thanks, that helps. The pattern around quant drift has been pretty similar for me. I got better results once I stopped chasing one metric and just loo Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
throughput matters, but tail latency matters too
|
|
« Next Oldest | Next Newest »
|
| Messages In This Thread |
|
structured output failures with vLLM - by southtinker - 10-17-2026, 06:42 PM
Re: structured output failures with vLLM - by batchpeak - 10-17-2026, 10:53 PM
Re: structured output failures with vLLM - by sparrowhall - 10-18-2026, 04:51 PM
Re: structured output failures with vLLM - by cudapoint - 10-18-2026, 05:08 PM
Re: structured output failures with vLLM - by embershed99 - 10-18-2026, 09:09 PM
Re: structured output failures with vLLM - by mangovault67 - 10-19-2026, 01:29 AM
Re: structured output failures with vLLM - by macrotinker - 10-19-2026, 01:59 AM
Re: structured output failures with vLLM - by turboarch95 - 10-19-2026, 02:29 AM
Re: structured output failures with vLLM - by batchsize - 10-19-2026, 02:30 AM
Re: structured output failures with vLLM - by vramghost - 10-19-2026, 04:02 AM
Re: structured output failures with vLLM - by vramport - 10-19-2026, 04:23 AM
Re: structured output failures with vLLM - by quantsupply - 10-19-2026, 05:25 AM
|
