hybrid retrieval results: early impressions

[macropilot]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of TensorRT-LLM and mini PC felt surprisingly usable once I tuned around VRAM headroom.

Feel free to tear it apart if you see something dumb.

[kernelghost]

same here on my side.

[orbitgarden]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.

[tinycontext]

What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[phaseshed]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[routecraft]

following this because I'm in the same boat.

[tensorriver]

Thanks, that helps. I don't think your conclusion is far off.

[batchshed]

that fixed it for me too.

[marbleyard32]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals. Once you know the plain path works, then add the faster path back in piece by piece.

[cacheplane98]

subscribed, curious where this lands.