+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Model Merging (https://www.datasculpting.ai/inference/forumdisplay.php?fid=57)
+--- Thread: sqlite + embeddings surprisingly ok for Mixtral (/showthread.php?tid=1417)
Pages:
1
2
sqlite + embeddings surprisingly ok for Mixtral - sparrowworks88 - 10-18-2026
I'm trying to sanity-check my setup.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.
Would appreciate practical advice rather than theory.
Re: sqlite + embeddings surprisingly ok for Mixtral - ggufwire22 - 10-18-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: sqlite + embeddings surprisingly ok for Mixtral - macropoint - 10-18-2026
subscribed, curious where this lands.
Re: sqlite + embeddings surprisingly ok for Mixtral - relaysupply90 - 10-19-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: sqlite + embeddings surprisingly ok for Mixtral - oxidestream - 10-19-2026
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: sqlite + embeddings surprisingly ok for Mixtral - dockkite69 - 10-19-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Not glamorous advice, but repeatable baselines save a lot of time.
Re: sqlite + embeddings surprisingly ok for Mixtral - llm_wren - 10-19-2026
That matches what I was seeing. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: sqlite + embeddings surprisingly ok for Mixtral - torchdesk - 10-19-2026
Quote:same here on my side.
What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: sqlite + embeddings surprisingly ok for Mixtral - marbleyard32 - 10-19-2026
Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: sqlite + embeddings surprisingly ok for Mixtral - circuitkite - 10-19-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?