sqlite + embeddings surprisingly ok: early impressions

[sanescope]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and mini PC.

I'm especially interested in mistakes worth avoiding.

[torchsupply99]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[audiopath66]

What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[kernelgrid]

What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. N

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[relayforge]

[quote]What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem surv

Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me.

[edgeriver]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.