sqlite + embeddings surprisingly ok for Mixtral

[sparrowworks88]

I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.

Would appreciate practical advice rather than theory.

[ggufwire22]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[macropoint]

subscribed, curious where this lands.

[relaysupply90]

Need a little more detail here. Is this local-only or are you routing anything externally?

[oxidestream]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[dockkite69]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Not glamorous advice, but repeatable baselines save a lot of time.

[llm_wren]

That matches what I was seeing. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[torchdesk]

same here on my side.

What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[marbleyard32]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[circuitkite]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?