AI pin style devices but useful? on 4090

[sparrowghost]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

Would rather learn from your scars than collect my own all at once.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[mangopath4]

same here on my side.

[lablab]

Did a small comparison this afternoon. On 4090 with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.

[oxidestream]

Interesting, I hadn't tried that. I ended up in about the same place using Ollama.

[audiorunner74]

Interesting, I hadn't tried that. I ended up in about the same place using Ollama.

What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[rasterbench]

Appreciate the detail. The pattern around embeddings has been pretty similar for me.

[routeline23]

Good point. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rasterbench]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[routeline23]

I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[rasterbench]

I tested this tonight. On dual 3090 with llama.cpp, I saw basically the same behavior.