https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:40:32 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1445 Mon, 19 Oct 2026 03:27:55 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=1445
Maybe useful as a starting point: LM Studio on B200, keeping structured output in mind from the start.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
Maybe useful as a starting point: LM Studio on B200, keeping structured output in mind from the start.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1455 Sun, 18 Oct 2026 21:23:00 +0000 labline]]> https://www.datasculpting.ai/inference/showthread.php?tid=1455 Quote:subscribed, curious where this lands.
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?]]> Quote:subscribed, curious where this lands.
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1478 Sun, 18 Oct 2026 17:28:05 +0000 circuitkite]]> https://www.datasculpting.ai/inference/showthread.php?tid=1478
I tried this with Phi using SGLang on 4090. The headline result was better than I expected, but workstation thermals still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]>
I tried this with Phi using SGLang on 4090. The headline result was better than I expected, but workstation thermals still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1465 Sun, 18 Oct 2026 16:52:44 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1465
The more I work with local setups, the more it seems like structured output decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like structured output decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1497 Sun, 18 Oct 2026 16:15:19 +0000 oxidegrove40]]> https://www.datasculpting.ai/inference/showthread.php?tid=1497
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1432 Sun, 18 Oct 2026 01:01:16 +0000 mellowsupply]]> https://www.datasculpting.ai/inference/showthread.php?tid=1432
My current setup is PyTorch with M3 Max, and I keep running into router policy.

Not looking for the perfect answer, mostly what worked in real use.]]>
My current setup is PyTorch with M3 Max, and I keep running into router policy.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1422 Sat, 17 Oct 2026 23:24:27 +0000 batchsize]]> https://www.datasculpting.ai/inference/showthread.php?tid=1422
One thing I keep noticing is that people underestimate tool calling until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]>
One thing I keep noticing is that people underestimate tool calling until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1227 Thu, 15 Oct 2026 04:52:29 +0000 deltagrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=1227
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tool calling.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tool calling.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1250 Wed, 14 Oct 2026 21:36:04 +0000 deltapoint17]]> https://www.datasculpting.ai/inference/showthread.php?tid=1250
I've gone back and forth between vLLM and llama.cpp for Llama. The interesting difference for me has been token cost.

Trying to avoid rebuilding the whole thing twice.]]>
I've gone back and forth between vLLM and llama.cpp for Llama. The interesting difference for me has been token cost.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1202 Wed, 14 Oct 2026 16:12:38 +0000 cozykite]]> https://www.datasculpting.ai/inference/showthread.php?tid=1202
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about VRAM headroom.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about VRAM headroom.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1107 Mon, 12 Oct 2026 18:36:41 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1107
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty. The logs were somehow calmer than I was.]]>
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty. The logs were somehow calmer than I was.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1078 Mon, 12 Oct 2026 05:31:48 +0000 pixelspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=1078
This isn't revolutionary, but the combination of SGLang and A6000 felt surprisingly usable once I tuned around quant drift.

Maybe useful as a starting point for newer folks.]]>
This isn't revolutionary, but the combination of SGLang and A6000 felt surprisingly usable once I tuned around quant drift.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1040 Mon, 12 Oct 2026 05:12:37 +0000 signalcraft14]]> https://www.datasculpting.ai/inference/showthread.php?tid=1040
I'm mostly working with Mixtral in MLX on B200. The part I'm trying to understand is tokens/sec.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm mostly working with Mixtral in MLX on B200. The part I'm trying to understand is tokens/sec.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1058 Mon, 12 Oct 2026 03:23:56 +0000 mintquill]]> https://www.datasculpting.ai/inference/showthread.php?tid=1058
Maybe useful as a starting point: vLLM on 4090, keeping MoE serving in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]>
Maybe useful as a starting point: vLLM on 4090, keeping MoE serving in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1051 Sun, 11 Oct 2026 15:43:03 +0000 routecrate]]> https://www.datasculpting.ai/inference/showthread.php?tid=1051
This isn't revolutionary, but the combination of SGLang and 4090 felt surprisingly usable once I tuned around tool calling.

If someone wants the exact settings I can clean them up and post them.]]>
This isn't revolutionary, but the combination of SGLang and 4090 felt surprisingly usable once I tuned around tool calling.

If someone wants the exact settings I can clean them up and post them.]]>