Search / Retrieval / Reranking discussion for Gemma

[tensorfield88]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

[cobaltlab]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding.

[deltagrid]

I'd start with the simpler stack.

[llmarch51]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.

[localcraft17]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.

[relaysupply90]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vectorcrate98]

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[circuitwire58]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[deepseeked]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

[turbosupply83]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.