https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:48 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1259 Fri, 16 Oct 2026 04:00:59 +0000 fusedtinker]]> https://www.datasculpting.ai/inference/showthread.php?tid=1259
This isn't revolutionary, but the combination of llama.cpp and mini PC felt surprisingly usable once I tuned around consumer GPUs.

If someone wants the exact settings I can clean them up and post them.

This is mostly for local work rather than public serving.]]>
This isn't revolutionary, but the combination of llama.cpp and mini PC felt surprisingly usable once I tuned around consumer GPUs.

If someone wants the exact settings I can clean them up and post them.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1276 Thu, 15 Oct 2026 21:58:39 +0000 turbosupply83]]> https://www.datasculpting.ai/inference/showthread.php?tid=1276
One thing I keep noticing is that people underestimate consumer GPUs until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
One thing I keep noticing is that people underestimate consumer GPUs until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1146 Tue, 13 Oct 2026 20:48:35 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1146
The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Curious whether that lines up with what other people are seeing.]]>
The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1073 Sun, 11 Oct 2026 23:31:01 +0000 macroforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1073
My current setup is PyTorch with B200, and I keep running into workstation thermals.

Would appreciate practical advice rather than theory.

The part that surprised me was how much the boring details mattered more than the model choice. At this point the real bottleneck is my patience.]]>
My current setup is PyTorch with B200, and I keep running into workstation thermals.

Would appreciate practical advice rather than theory.

The part that surprised me was how much the boring details mattered more than the model choice. At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1003 Sun, 11 Oct 2026 00:05:17 +0000 turbosupply83]]> https://www.datasculpting.ai/inference/showthread.php?tid=1003
Two paths both seem viable, but they break down differently once quant drift becomes important.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Two paths both seem viable, but they break down differently once quant drift becomes important.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=942 Sat, 10 Oct 2026 04:03:22 +0000 cobalttinker]]> https://www.datasculpting.ai/inference/showthread.php?tid=942
The more I work with local setups, the more it seems like HBM constraints decides everything downstream.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
The more I work with local setups, the more it seems like HBM constraints decides everything downstream.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=894 Fri, 09 Oct 2026 03:49:12 +0000 flintriver50]]> https://www.datasculpting.ai/inference/showthread.php?tid=894
Short version: Ollama behaved well for me, but only after I stopped ignoring small local stacks.

Happy to post more details if anyone wants them.]]>
Short version: Ollama behaved well for me, but only after I stopped ignoring small local stacks.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=864 Wed, 07 Oct 2026 22:35:20 +0000 dockkite69]]> https://www.datasculpting.ai/inference/showthread.php?tid=864
My current setup is Ollama with Threadripper box, and I keep running into structured output.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
My current setup is Ollama with Threadripper box, and I keep running into structured output.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=886 Wed, 07 Oct 2026 11:44:08 +0000 latentcraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=886
One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.]]>
One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=718 Sat, 03 Oct 2026 20:11:24 +0000 amberhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=718
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling token cost.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling token cost.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=672 Fri, 02 Oct 2026 15:03:55 +0000 localcraft17]]> https://www.datasculpting.ai/inference/showthread.php?tid=672
I'm mostly working with Qwen in Ollama on 4090. The part I'm trying to understand is KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with Qwen in Ollama on 4090. The part I'm trying to understand is KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=634 Thu, 01 Oct 2026 17:51:55 +0000 quiettoken]]> https://www.datasculpting.ai/inference/showthread.php?tid=634
I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]>
I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=444 Thu, 24 Sep 2026 00:35:57 +0000 marbletrack]]> https://www.datasculpting.ai/inference/showthread.php?tid=444
I've gone back and forth between Ollama and TensorRT-LLM for Qwen. The interesting difference for me has been token cost.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
I've gone back and forth between Ollama and TensorRT-LLM for Qwen. The interesting difference for me has been token cost.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=451 Wed, 23 Sep 2026 15:11:41 +0000 batchfield55]]> https://www.datasculpting.ai/inference/showthread.php?tid=451
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=377 Sun, 20 Sep 2026 19:37:47 +0000 batchpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=377
Two paths both seem viable, but they break down differently once batch size becomes important.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.]]>
Two paths both seem viable, but they break down differently once batch size becomes important.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.]]>