structured output failures

[cacheline]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, llama.cpp, 4090, with a lot of attention on reranking.

Feel free to tear it apart if you see something dumb.

This is mostly for local work rather than public serving.

[torchshed66]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[dockworks23]

subscribed, curious where this lands.

[vramforge]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[nanopatch]

Thanks, that helps. That matches what I saw on dual 3090.

[vectorshed43]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[rocmriley]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[dockkite69]

following this because I'm in the same boat.

[mellowwire98]

following this because I'm in the same boat.

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.

[mintstation99]

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.