+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Prompting & Context Engineering (https://www.datasculpting.ai/inference/forumdisplay.php?fid=15)
+--- Forum: Structured Output (https://www.datasculpting.ai/inference/forumdisplay.php?fid=63)
+--- Thread: structured output failures (/showthread.php?tid=47)
Pages:
1
2
structured output failures - cacheline - 08-28-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Mistral, llama.cpp, 4090, with a lot of attention on reranking.
Feel free to tear it apart if you see something dumb.
This is mostly for local work rather than public serving.
Re: structured output failures - torchshed66 - 08-28-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: structured output failures - dockworks23 - 08-28-2026
subscribed, curious where this lands.
Re: structured output failures - vramforge - 08-28-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: structured output failures - nanopatch - 08-30-2026
Thanks, that helps. That matches what I saw on dual 3090.
Re: structured output failures - vectorshed43 - 08-30-2026
That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: structured output failures - rocmriley - 08-30-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: structured output failures - dockkite69 - 08-30-2026
following this because I'm in the same boat.
Re: structured output failures - mellowwire98 - 08-31-2026
Quote:following this because I'm in the same boat.
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.
Re: structured output failures - mintstation99 - 09-03-2026
My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.