https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:49 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1500 Mon, 19 Oct 2026 00:58:22 +0000 vramport]]> https://www.datasculpting.ai/inference/showthread.php?tid=1500
I'm mostly working with DeepSeek in llama.cpp on B200. The part I'm trying to understand is NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm mostly working with DeepSeek in llama.cpp on B200. The part I'm trying to understand is NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1331 Fri, 16 Oct 2026 18:37:36 +0000 rocmriley]]> https://www.datasculpting.ai/inference/showthread.php?tid=1331
Short version: SGLang behaved well for me, but only after I stopped ignoring long context.

Curious if anyone else landed in the same place.]]>
Short version: SGLang behaved well for me, but only after I stopped ignoring long context.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1209 Wed, 14 Oct 2026 22:13:59 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1209
This isn't revolutionary, but the combination of Ollama and 7900 XTX felt surprisingly usable once I tuned around tool calling.

If someone wants the exact settings I can clean them up and post them.]]>
This isn't revolutionary, but the combination of Ollama and 7900 XTX felt surprisingly usable once I tuned around tool calling.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1148 Wed, 14 Oct 2026 02:19:49 +0000 deltagrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=1148
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]>
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1123 Tue, 13 Oct 2026 05:33:34 +0000 phasetrack]]> https://www.datasculpting.ai/inference/showthread.php?tid=1123
Maybe useful as a starting point: PyTorch on GB200, keeping HBM constraints in mind from the start.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. At this point the real bottleneck is my patience.]]>
Maybe useful as a starting point: PyTorch on GB200, keeping HBM constraints in mind from the start.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1012 Sat, 10 Oct 2026 20:08:07 +0000 fusedriver]]> https://www.datasculpting.ai/inference/showthread.php?tid=1012
I tried this with Qwen using Ollama on 7900 XTX. The headline result was better than I expected, but quant drift still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.]]>
I tried this with Qwen using Ollama on 7900 XTX. The headline result was better than I expected, but quant drift still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=709 Sat, 03 Oct 2026 22:49:35 +0000 vramghost]]> https://www.datasculpting.ai/inference/showthread.php?tid=709
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Maybe obvious to some of you, but it clicked for me this week.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Maybe obvious to some of you, but it clicked for me this week.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=581 Tue, 29 Sep 2026 22:15:32 +0000 cozyarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=581
I'm mostly working with Mixtral in SGLang on GB200. The part I'm trying to understand is small local stacks.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm mostly working with Mixtral in SGLang on GB200. The part I'm trying to understand is small local stacks.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=261 Sun, 13 Sep 2026 19:13:29 +0000 cudapoint]]> https://www.datasculpting.ai/inference/showthread.php?tid=261
Maybe useful as a starting point: llama.cpp on Threadripper box, keeping MoE serving in mind from the start.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Maybe useful as a starting point: llama.cpp on Threadripper box, keeping MoE serving in mind from the start.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=217 Thu, 10 Sep 2026 06:21:00 +0000 syshouse]]> https://www.datasculpting.ai/inference/showthread.php?tid=217
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=181 Mon, 07 Sep 2026 23:29:00 +0000 turboarch95]]> https://www.datasculpting.ai/inference/showthread.php?tid=181
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Wondering if this is just where the whole space is heading now.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=154 Mon, 07 Sep 2026 02:58:09 +0000 cozygrid12]]> https://www.datasculpting.ai/inference/showthread.php?tid=154
I'm mostly working with Mixtral in TensorRT-LLM on GB200. The part I'm trying to understand is batch size.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm mostly working with Mixtral in TensorRT-LLM on GB200. The part I'm trying to understand is batch size.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=8 Sun, 23 Aug 2026 01:58:11 +0000 cacheworks]]> https://www.datasculpting.ai/inference/showthread.php?tid=8
I tried this with Mistral using MLX on M3 Max. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I tried this with Mistral using MLX on M3 Max. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>