on-device inference worth the effort

[marbleyard32]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like tokens/sec decides everything downstream.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[vramarch]

I tested this tonight. On dual 3090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[batchpeak]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[plainline]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[routeline23]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lo

A couple questions before I guess too hard. Are you testing this on 4090 or something else?

I'm happy to test one more variant if you want a second data point.

[sparrowghost]

What exact setup are you running? Is this local-only or are you routing anything externally?

[turbosupply83]

What exact setup are you running? Is this local-only or are you routing anything externally?

Did a small comparison this afternoon. On 5090 with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[routeline23]

What exact setup are you running? Is this local-only or are you routing anything externally?

I hit almost the same issue last night.

If you post your exact settings, people here can probably narrow it down pretty fast.

[sparrowghost]

What exact setup are you running? Is this local-only or are you routing anything externally?

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamorous advice, but repeatable baselines save a lot of time.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[plainscope50]

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.