https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:40:32 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1437 Sat, 17 Oct 2026 07:22:44 +0000 vramscope]]> https://www.datasculpting.ai/inference/showthread.php?tid=1437
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around Ollama and 7900 XTX.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around Ollama and 7900 XTX.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1321 Fri, 16 Oct 2026 08:22:32 +0000 ggufwire22]]> https://www.datasculpting.ai/inference/showthread.php?tid=1321
I tried this with FLUX using vLLM on GB200. The headline result was better than I expected, but router policy still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]>
I tried this with FLUX using vLLM on GB200. The headline result was better than I expected, but router policy still ended up being the limiting factor.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1219 Thu, 15 Oct 2026 05:23:21 +0000 ggufwire22]]> https://www.datasculpting.ai/inference/showthread.php?tid=1219
I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1189 Wed, 14 Oct 2026 02:48:44 +0000 quantnest]]> https://www.datasculpting.ai/inference/showthread.php?tid=1189
One thing I keep noticing is that people underestimate workstation thermals until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]>
One thing I keep noticing is that people underestimate workstation thermals until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1055 Sun, 11 Oct 2026 19:55:08 +0000 edgecoast]]> https://www.datasculpting.ai/inference/showthread.php?tid=1055
Two paths both seem viable, but they break down differently once VRAM headroom becomes important.

What would you pick if you had to live with it for a few months?

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
Two paths both seem viable, but they break down differently once VRAM headroom becomes important.

What would you pick if you had to live with it for a few months?

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1013 Sun, 11 Oct 2026 01:32:38 +0000 vectorshed43]]> https://www.datasculpting.ai/inference/showthread.php?tid=1013
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and 7900 XTX.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and 7900 XTX.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=824 Wed, 07 Oct 2026 00:07:34 +0000 torchdesk]]> https://www.datasculpting.ai/inference/showthread.php?tid=824
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=830 Tue, 06 Oct 2026 20:58:38 +0000 vectorshed43]]> https://www.datasculpting.ai/inference/showthread.php?tid=830
I tried this with Phi using PyTorch on 5090. The headline result was better than I expected, but batch size still ended up being the limiting factor.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I tried this with Phi using PyTorch on 5090. The headline result was better than I expected, but batch size still ended up being the limiting factor.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=787 Tue, 06 Oct 2026 03:16:23 +0000 vramscope]]> https://www.datasculpting.ai/inference/showthread.php?tid=787
Maybe useful as a starting point: LM Studio on 4090, keeping workstation thermals in mind from the start.

Maybe useful as a starting point for newer folks.]]>
Maybe useful as a starting point: LM Studio on 4090, keeping workstation thermals in mind from the start.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=738 Mon, 05 Oct 2026 00:40:14 +0000 rasterhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=738
I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=759 Sun, 04 Oct 2026 15:52:02 +0000 ggufwire22]]> https://www.datasculpting.ai/inference/showthread.php?tid=759
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with tool calling.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with tool calling.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=685 Sat, 03 Oct 2026 00:40:32 +0000 thetanest]]> https://www.datasculpting.ai/inference/showthread.php?tid=685
This isn't revolutionary, but the combination of TensorRT-LLM and Threadripper box felt surprisingly usable once I tuned around consumer GPUs.

If someone wants the exact settings I can clean them up and post them.]]>
This isn't revolutionary, but the combination of TensorRT-LLM and Threadripper box felt surprisingly usable once I tuned around consumer GPUs.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=600 Wed, 30 Sep 2026 03:48:12 +0000 southtinker]]> https://www.datasculpting.ai/inference/showthread.php?tid=600
My current setup is TensorRT-LLM with 7900 XTX, and I keep running into NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
My current setup is TensorRT-LLM with 7900 XTX, and I keep running into NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=588 Wed, 30 Sep 2026 02:50:11 +0000 torchdesk]]> https://www.datasculpting.ai/inference/showthread.php?tid=588
Maybe useful as a starting point: PyTorch on Threadripper box, keeping quant drift in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]>
Maybe useful as a starting point: PyTorch on Threadripper box, keeping quant drift in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=562 Mon, 28 Sep 2026 19:54:18 +0000 ggufwire22]]> https://www.datasculpting.ai/inference/showthread.php?tid=562
Short version: Ollama behaved well for me, but only after I stopped ignoring structured output.

Nothing scientific here, just practical notes from a real setup.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Short version: Ollama behaved well for me, but only after I stopped ignoring structured output.

Nothing scientific here, just practical notes from a real setup.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>