https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:40:32 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1479 Sun, 18 Oct 2026 06:38:48 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=1479
Two paths both seem viable, but they break down differently once MoE serving becomes important.

What would you pick if you had to live with it for a few months?]]>
Two paths both seem viable, but they break down differently once MoE serving becomes important.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1434 Sun, 18 Oct 2026 04:06:49 +0000 fusedhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=1434
Two paths both seem viable, but they break down differently once router policy becomes important.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Two paths both seem viable, but they break down differently once router policy becomes important.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1403 Sat, 17 Oct 2026 21:07:22 +0000 dockridge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1403
Short version: SGLang behaved well for me, but only after I stopped ignoring tool calling.

Happy to post more details if anyone wants them.]]>
Short version: SGLang behaved well for me, but only after I stopped ignoring tool calling.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1360 Sat, 17 Oct 2026 01:33:54 +0000 rocmpath]]> https://www.datasculpting.ai/inference/showthread.php?tid=1360
Maybe useful as a starting point: LM Studio on B200, keeping continuous batching in mind from the start.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Maybe useful as a starting point: LM Studio on B200, keeping continuous batching in mind from the start.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1289 Thu, 15 Oct 2026 22:09:47 +0000 vectorember]]> https://www.datasculpting.ai/inference/showthread.php?tid=1289
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1284 Thu, 15 Oct 2026 21:15:37 +0000 oxidemixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=1284
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around INT4.

Happy to post more details if anyone wants them.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around INT4.

Happy to post more details if anyone wants them.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1105 Mon, 12 Oct 2026 18:15:32 +0000 vectorcrate98]]> https://www.datasculpting.ai/inference/showthread.php?tid=1105
Just sharing a setup that ended up working reasonably well: Kimi, vLLM, dual 3090, with a lot of attention on NVFP4.

Feel free to tear it apart if you see something dumb.]]>
Just sharing a setup that ended up working reasonably well: Kimi, vLLM, dual 3090, with a lot of attention on NVFP4.

Feel free to tear it apart if you see something dumb.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1100 Mon, 12 Oct 2026 18:06:58 +0000 fusedhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=1100
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tool calling.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it. At this point the real bottleneck is my patience.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tool calling.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it. At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1115 Mon, 12 Oct 2026 07:45:14 +0000 rocmmixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=1115
Two paths both seem viable, but they break down differently once workstation thermals becomes important.

Trying to avoid rebuilding the whole thing twice.]]>
Two paths both seem viable, but they break down differently once workstation thermals becomes important.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1080 Mon, 12 Oct 2026 04:16:30 +0000 vectorember]]> https://www.datasculpting.ai/inference/showthread.php?tid=1080
This isn't revolutionary, but the combination of PyTorch and mini PC felt surprisingly usable once I tuned around long context.

Maybe useful as a starting point for newer folks.]]>
This isn't revolutionary, but the combination of PyTorch and mini PC felt surprisingly usable once I tuned around long context.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1001 Sun, 11 Oct 2026 04:19:32 +0000 fusedhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=1001
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=992 Sat, 10 Oct 2026 21:40:42 +0000 mellowwire98]]> https://www.datasculpting.ai/inference/showthread.php?tid=992
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around structured output.

Happy to post more details if anyone wants them.]]>
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around structured output.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=981 Fri, 09 Oct 2026 23:13:56 +0000 thetanest]]> https://www.datasculpting.ai/inference/showthread.php?tid=981
My current setup is PyTorch with 5090, and I keep running into long context.

Would appreciate practical advice rather than theory.]]>
My current setup is PyTorch with 5090, and I keep running into long context.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=907 Thu, 08 Oct 2026 09:30:15 +0000 fusedhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=907
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=865 Thu, 08 Oct 2026 04:14:40 +0000 dockridge]]> https://www.datasculpting.ai/inference/showthread.php?tid=865
This isn't revolutionary, but the combination of MLX and 7900 XTX felt surprisingly usable once I tuned around long context.

Feel free to tear it apart if you see something dumb.]]>
This isn't revolutionary, but the combination of MLX and 7900 XTX felt surprisingly usable once I tuned around long context.

Feel free to tear it apart if you see something dumb.]]>