https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:28 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1457 Sun, 18 Oct 2026 17:59:00 +0000 labnest]]> https://www.datasculpting.ai/inference/showthread.php?tid=1457
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.]]>
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1383 Sat, 17 Oct 2026 11:10:27 +0000 latentgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1383
After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around HBM constraints.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around HBM constraints.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1353 Sat, 17 Oct 2026 02:20:36 +0000 rocmrunner40]]> https://www.datasculpting.ai/inference/showthread.php?tid=1353
I'm mostly working with DeepSeek in llama.cpp on Threadripper box. The part I'm trying to understand is batch size.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with DeepSeek in llama.cpp on Threadripper box. The part I'm trying to understand is batch size.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1347 Fri, 16 Oct 2026 07:42:15 +0000 byteyard]]> https://www.datasculpting.ai/inference/showthread.php?tid=1347
One thing I keep noticing is that people underestimate NVFP4 until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]>
One thing I keep noticing is that people underestimate NVFP4 until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1262 Fri, 16 Oct 2026 05:43:05 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1262
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring HBM constraints.

Curious if anyone else landed in the same place.]]>
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring HBM constraints.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1279 Thu, 15 Oct 2026 16:18:56 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=1279
I'm mostly working with Phi in Ollama on M3 Max. The part I'm trying to understand is prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with Phi in Ollama on M3 Max. The part I'm trying to understand is prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1253 Wed, 14 Oct 2026 21:11:00 +0000 plainember52]]> https://www.datasculpting.ai/inference/showthread.php?tid=1253
I tried this with DeepSeek using MLX on A6000. The headline result was better than I expected, but router policy still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>
I tried this with DeepSeek using MLX on A6000. The headline result was better than I expected, but router policy still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1201 Wed, 14 Oct 2026 13:36:33 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1201
My current setup is PyTorch with dual 3090, and I keep running into HBM constraints.

If you've tested something similar, what did you keep and what did you stop bothering with? At this point the real bottleneck is my patience.]]>
My current setup is PyTorch with dual 3090, and I keep running into HBM constraints.

If you've tested something similar, what did you keep and what did you stop bothering with? At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1157 Wed, 14 Oct 2026 00:01:44 +0000 lablab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1157
Short version: MLX behaved well for me, but only after I stopped ignoring router policy.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Short version: MLX behaved well for me, but only after I stopped ignoring router policy.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1098 Tue, 13 Oct 2026 03:10:28 +0000 thetaarch43]]> https://www.datasculpting.ai/inference/showthread.php?tid=1098
Just sharing a setup that ended up working reasonably well: Gemma, vLLM, mini PC, with a lot of attention on consumer GPUs.

Feel free to tear it apart if you see something dumb.]]>
Just sharing a setup that ended up working reasonably well: Gemma, vLLM, mini PC, with a lot of attention on consumer GPUs.

Feel free to tear it apart if you see something dumb.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1129 Mon, 12 Oct 2026 22:03:41 +0000 macrolab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1129
I've gone back and forth between TensorRT-LLM and TensorRT-LLM for Llama. The interesting difference for me has been reranking.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I've gone back and forth between TensorRT-LLM and TensorRT-LLM for Llama. The interesting difference for me has been reranking.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1108 Mon, 12 Oct 2026 22:02:24 +0000 sanescope]]> https://www.datasculpting.ai/inference/showthread.php?tid=1108
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1088 Mon, 12 Oct 2026 09:29:40 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1088
I'm mostly working with DeepSeek in PyTorch on dual 3090. The part I'm trying to understand is consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm mostly working with DeepSeek in PyTorch on dual 3090. The part I'm trying to understand is consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1022 Sat, 10 Oct 2026 23:20:38 +0000 visionyard23]]> https://www.datasculpting.ai/inference/showthread.php?tid=1022
I tried this with DeepSeek using TensorRT-LLM on 4090. The headline result was better than I expected, but NVFP4 still ended up being the limiting factor.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
I tried this with DeepSeek using TensorRT-LLM on 4090. The headline result was better than I expected, but NVFP4 still ended up being the limiting factor.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=825 Tue, 06 Oct 2026 15:53:54 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=825
I'm mostly working with Phi in llama.cpp on GB200. The part I'm trying to understand is NVFP4.

Would appreciate practical advice rather than theory.]]>
I'm mostly working with Phi in llama.cpp on GB200. The part I'm trying to understand is NVFP4.

Would appreciate practical advice rather than theory.]]>