when RAG was overkill for Gemma

[griffinloom66]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and B200.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[tensorfield88]

That matches what I was seeing. I don't think your conclusion is far off.

[rocmcrate26]

What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[oxiderunner]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[macrostack69]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[sysstation69]

What worked for me was pretty simple. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[mintforge]

I think the bottleneck is somewhere else. I get why that seems right, but on GB200 the bottleneck felt elsewhere.

[cudapeak]

I don't think that's the whole story. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere.

[sparrowcraft]

that fixed it for me too.

[batchfield55]

Appreciate the detail. The pattern around speculative decoding has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.