https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:27 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1470 Sun, 18 Oct 2026 21:21:00 +0000 deltastation]]> https://www.datasculpting.ai/inference/showthread.php?tid=1470 https://www.datasculpting.ai/inference/showthread.php?tid=1427 Sat, 17 Oct 2026 19:11:55 +0000 cudascope]]> https://www.datasculpting.ai/inference/showthread.php?tid=1427
I tried this with Phi using LM Studio on Threadripper box. The headline result was better than I expected, but consumer GPUs still ended up being the limiting factor.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I tried this with Phi using LM Studio on Threadripper box. The headline result was better than I expected, but consumer GPUs still ended up being the limiting factor.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1406 Sat, 17 Oct 2026 10:12:35 +0000 audioforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1406
I've gone back and forth between Ollama and Ollama for DeepSeek. The interesting difference for me has been tokens/sec.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I've gone back and forth between Ollama and Ollama for DeepSeek. The interesting difference for me has been tokens/sec.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1264 Thu, 15 Oct 2026 18:31:39 +0000 kernelarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1264
I'm mostly working with DeepSeek in vLLM on GB200. The part I'm trying to understand is tokens/sec.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with DeepSeek in vLLM on GB200. The part I'm trying to understand is tokens/sec.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1295 Thu, 15 Oct 2026 16:17:31 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=1295
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around router policy.

Nothing scientific here, just practical notes from a real setup.]]>
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around router policy.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1235 Wed, 14 Oct 2026 19:53:23 +0000 cacheline]]> https://www.datasculpting.ai/inference/showthread.php?tid=1235
Short version: Ollama behaved well for me, but only after I stopped ignoring quant drift.

Curious if anyone else landed in the same place.]]>
Short version: Ollama behaved well for me, but only after I stopped ignoring quant drift.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1010 Sun, 11 Oct 2026 05:17:39 +0000 kernelarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1010
The more I work with local setups, the more it seems like INT4 decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like INT4 decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=986 Sat, 10 Oct 2026 08:57:20 +0000 kernelarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=986
Maybe useful as a starting point: vLLM on M3 Max, keeping INT4 in mind from the start.

Feel free to tear it apart if you see something dumb.]]>
Maybe useful as a starting point: vLLM on M3 Max, keeping INT4 in mind from the start.

Feel free to tear it apart if you see something dumb.]]> https://www.datasculpting.ai/inference/showthread.php?tid=919 Thu, 08 Oct 2026 21:53:13 +0000 kernelarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=919
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling HBM constraints.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling HBM constraints.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=669 Fri, 02 Oct 2026 23:36:46 +0000 quiethouse]]> https://www.datasculpting.ai/inference/showthread.php?tid=669
The more I work with local setups, the more it seems like NVFP4 decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like NVFP4 decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=676 Fri, 02 Oct 2026 15:51:01 +0000 promptspark58]]> https://www.datasculpting.ai/inference/showthread.php?tid=676
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=647 Fri, 02 Oct 2026 02:38:11 +0000 kernelghost]]> https://www.datasculpting.ai/inference/showthread.php?tid=647
Two paths both seem viable, but they break down differently once prefill/decode split becomes important.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Two paths both seem viable, but they break down differently once prefill/decode split becomes important.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=657 Thu, 01 Oct 2026 13:35:35 +0000 bytelab]]> https://www.datasculpting.ai/inference/showthread.php?tid=657
I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]>
I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=629 Thu, 01 Oct 2026 01:57:46 +0000 mellowwire98]]> https://www.datasculpting.ai/inference/showthread.php?tid=629
I tried this with Llama using PyTorch on M3 Max. The headline result was better than I expected, but consumer GPUs still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.]]>
I tried this with Llama using PyTorch on M3 Max. The headline result was better than I expected, but consumer GPUs still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=613 Wed, 30 Sep 2026 23:39:56 +0000 sparrowworks88]]> https://www.datasculpting.ai/inference/showthread.php?tid=613
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with quant drift.

Wondering if this is just where the whole space is heading now.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with quant drift.

Wondering if this is just where the whole space is heading now.]]>