https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:30 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1481 Sun, 18 Oct 2026 22:13:00 +0000 paperrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=1481 https://www.datasculpting.ai/inference/showthread.php?tid=1373 Sat, 17 Oct 2026 01:47:52 +0000 sanevault]]> https://www.datasculpting.ai/inference/showthread.php?tid=1373
The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.]]>
The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1158 Tue, 13 Oct 2026 22:03:12 +0000 audiopath66]]> https://www.datasculpting.ai/inference/showthread.php?tid=1158
The more I work with local setups, the more it seems like token cost decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]>
The more I work with local setups, the more it seems like token cost decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1120 Mon, 12 Oct 2026 11:09:06 +0000 forgemixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=1120
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it. The logs were somehow calmer than I was.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it. The logs were somehow calmer than I was.]]> https://www.datasculpting.ai/inference/showthread.php?tid=772 Mon, 05 Oct 2026 21:12:53 +0000 rasterbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=772
Short version: llama.cpp behaved well for me, but only after I stopped ignoring workstation thermals.

Curious if anyone else landed in the same place.]]>
Short version: llama.cpp behaved well for me, but only after I stopped ignoring workstation thermals.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=748 Mon, 05 Oct 2026 05:46:29 +0000 oxidemixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=748
The more I work with local setups, the more it seems like batch size decides everything downstream.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
The more I work with local setups, the more it seems like batch size decides everything downstream.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=731 Sun, 04 Oct 2026 10:07:06 +0000 mintquill]]> https://www.datasculpting.ai/inference/showthread.php?tid=731
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=667 Sat, 03 Oct 2026 00:43:43 +0000 bytelab]]> https://www.datasculpting.ai/inference/showthread.php?tid=667
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with router policy.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with router policy.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=670 Fri, 02 Oct 2026 19:42:05 +0000 thetashed24]]> https://www.datasculpting.ai/inference/showthread.php?tid=670
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Curious if anyone else landed in the same place.]]>
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=587 Wed, 30 Sep 2026 05:44:10 +0000 dockbrook21]]> https://www.datasculpting.ai/inference/showthread.php?tid=587
I've gone back and forth between SGLang and PyTorch for Qwen. The interesting difference for me has been batch size.

Interested in tradeoffs more than fan loyalty.]]>
I've gone back and forth between SGLang and PyTorch for Qwen. The interesting difference for me has been batch size.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=519 Sat, 26 Sep 2026 18:40:30 +0000 fastquill74]]> https://www.datasculpting.ai/inference/showthread.php?tid=519
Maybe useful as a starting point: Ollama on 4090, keeping tail latency in mind from the start.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
Maybe useful as a starting point: Ollama on 4090, keeping tail latency in mind from the start.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=355 Fri, 18 Sep 2026 23:45:36 +0000 torchdesk]]> https://www.datasculpting.ai/inference/showthread.php?tid=355
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with reranking.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with reranking.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=311 Thu, 17 Sep 2026 02:11:05 +0000 mangopeak72]]> https://www.datasculpting.ai/inference/showthread.php?tid=311
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=302 Tue, 15 Sep 2026 22:04:16 +0000 turboscope]]> https://www.datasculpting.ai/inference/showthread.php?tid=302
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=278 Mon, 14 Sep 2026 16:26:24 +0000 northhall88]]> https://www.datasculpting.ai/inference/showthread.php?tid=278
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding.]]>