https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:48 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1485 Sun, 18 Oct 2026 18:38:00 +0000 dockkite69]]> https://www.datasculpting.ai/inference/showthread.php?tid=1485 https://www.datasculpting.ai/inference/showthread.php?tid=1469 Sun, 18 Oct 2026 18:00:00 +0000 ggufwire22]]> https://www.datasculpting.ai/inference/showthread.php?tid=1469 https://www.datasculpting.ai/inference/showthread.php?tid=1491 Sun, 18 Oct 2026 17:08:00 +0000 batchwire]]> https://www.datasculpting.ai/inference/showthread.php?tid=1491 https://www.datasculpting.ai/inference/showthread.php?tid=1460 Sun, 18 Oct 2026 17:06:00 +0000 embersmith]]> https://www.datasculpting.ai/inference/showthread.php?tid=1460 https://www.datasculpting.ai/inference/showthread.php?tid=1393 Sat, 17 Oct 2026 22:35:56 +0000 turbogrid11]]> https://www.datasculpting.ai/inference/showthread.php?tid=1393
My current setup is llama.cpp with 4090, and I keep running into workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.]]>
My current setup is llama.cpp with 4090, and I keep running into workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1379 Sat, 17 Oct 2026 22:16:38 +0000 flintriver50]]> https://www.datasculpting.ai/inference/showthread.php?tid=1379
This isn't revolutionary, but the combination of llama.cpp and 7900 XTX felt surprisingly usable once I tuned around token cost.

Maybe useful as a starting point for newer folks.]]>
This isn't revolutionary, but the combination of llama.cpp and 7900 XTX felt surprisingly usable once I tuned around token cost.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1355 Sat, 17 Oct 2026 05:33:49 +0000 paperrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=1355
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1325 Sat, 17 Oct 2026 00:47:33 +0000 signaltrack44]]> https://www.datasculpting.ai/inference/showthread.php?tid=1325
I tried this with Mistral using TensorRT-LLM on 5090. The headline result was better than I expected, but HBM constraints still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I tried this with Mistral using TensorRT-LLM on 5090. The headline result was better than I expected, but HBM constraints still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1351 Fri, 16 Oct 2026 18:30:35 +0000 rocmriley]]> https://www.datasculpting.ai/inference/showthread.php?tid=1351
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and dual 3090.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and dual 3090.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1309 Fri, 16 Oct 2026 02:20:42 +0000 batchwire]]> https://www.datasculpting.ai/inference/showthread.php?tid=1309
I've gone back and forth between PyTorch and MLX for Kimi. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.]]>
I've gone back and forth between PyTorch and MLX for Kimi. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1303 Thu, 15 Oct 2026 18:12:38 +0000 sanevault]]> https://www.datasculpting.ai/inference/showthread.php?tid=1303
I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1275 Thu, 15 Oct 2026 14:45:22 +0000 phasenode]]> https://www.datasculpting.ai/inference/showthread.php?tid=1275
Just sharing a setup that ended up working reasonably well: DeepSeek, llama.cpp, A6000, with a lot of attention on consumer GPUs.

If someone wants the exact settings I can clean them up and post them.]]>
Just sharing a setup that ended up working reasonably well: DeepSeek, llama.cpp, A6000, with a lot of attention on consumer GPUs.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1210 Wed, 14 Oct 2026 11:10:46 +0000 vrambandit]]> https://www.datasculpting.ai/inference/showthread.php?tid=1210
One thing I keep noticing is that people underestimate HBM constraints until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]>
One thing I keep noticing is that people underestimate HBM constraints until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1180 Tue, 13 Oct 2026 19:44:08 +0000 torchshed]]> https://www.datasculpting.ai/inference/showthread.php?tid=1180
Two paths both seem viable, but they break down differently once token cost becomes important.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Two paths both seem viable, but they break down differently once token cost becomes important.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1168 Tue, 13 Oct 2026 19:34:24 +0000 torchshed]]> https://www.datasculpting.ai/inference/showthread.php?tid=1168
Just sharing a setup that ended up working reasonably well: Qwen, vLLM, 5090, with a lot of attention on prefix caching.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs. I keep promising myself this is the last rebuild.]]>
Just sharing a setup that ended up working reasonably well: Qwen, vLLM, 5090, with a lot of attention on prefix caching.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs. I keep promising myself this is the last rebuild.]]>