+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Prompting & Context Engineering (https://www.datasculpting.ai/inference/forumdisplay.php?fid=15)
+--- Forum: Structured Output (https://www.datasculpting.ai/inference/forumdisplay.php?fid=63)
+--- Thread: structured output failures (/showthread.php?tid=1208)
Pages:
1
2
structured output failures - benchplease - 10-14-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
Simple and boring is a valid answer.
Re: structured output failures - marbleyard32 - 10-15-2026
Quote:Posting this in case it saves someone else a little time.
Can you fill in a bit more context? Are you testing this on A6000 or something else?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: structured output failures - torchshed - 10-15-2026
Maybe, but I had the opposite result. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Re: structured output failures - griffinloom66 - 10-15-2026
following this because I'm in the same boat.
Re: structured output failures - northcraft90 - 10-15-2026
Thanks, that helps. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: structured output failures - orbitgarden - 10-18-2026
Not sure I agree there. I've seen people blame INT4 when the real issue was much more mundane.
Re: structured output failures - byteyard - 10-18-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.
Re: structured output failures - fastfield4 - 10-18-2026
worth checking the driver first.
Re: structured output failures - vramstream - 10-18-2026
Good point. I ended up in about the same place using TensorRT-LLM.
Re: structured output failures - rocmpath - 10-18-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.