structured output failures

[benchplease]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

[marbleyard32]

Posting this in case it saves someone else a little time.

Can you fill in a bit more context? Are you testing this on A6000 or something else?

If you post your exact settings, people here can probably narrow it down pretty fast.

[torchshed]

Maybe, but I had the opposite result. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

[griffinloom66]

following this because I'm in the same boat.

[northcraft90]

Thanks, that helps. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[orbitgarden]

Not sure I agree there. I've seen people blame INT4 when the real issue was much more mundane.

[byteyard]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

[fastfield4]

worth checking the driver first.

[vramstream]

Good point. I ended up in about the same place using TensorRT-LLM.

[rocmpath]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.