https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:30 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1386 Sat, 17 Oct 2026 22:20:54 +0000 cudaworks]]> https://www.datasculpting.ai/inference/showthread.php?tid=1386
Two paths both seem viable, but they break down differently once speculative decoding becomes important.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Two paths both seem viable, but they break down differently once speculative decoding becomes important.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1317 Sat, 17 Oct 2026 03:29:22 +0000 cudapeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=1317
After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Curious if anyone else landed in the same place.]]>
After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1298 Thu, 15 Oct 2026 15:08:24 +0000 deltapoint17]]> https://www.datasculpting.ai/inference/showthread.php?tid=1298
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and GB200.

I'm especially interested in mistakes worth avoiding.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and GB200.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1247 Wed, 14 Oct 2026 22:23:56 +0000 bytegarden73]]> https://www.datasculpting.ai/inference/showthread.php?tid=1247
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and 7900 XTX.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around SGLang and 7900 XTX.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1233 Wed, 14 Oct 2026 13:44:37 +0000 cozyarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1233
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and 7900 XTX.

I'm especially interested in mistakes worth avoiding.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and 7900 XTX.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1127 Mon, 12 Oct 2026 23:39:16 +0000 homelabhenry]]> https://www.datasculpting.ai/inference/showthread.php?tid=1127
One thing I keep noticing is that people underestimate batch size until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
One thing I keep noticing is that people underestimate batch size until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1054 Sun, 11 Oct 2026 14:49:36 +0000 relaysupply90]]> https://www.datasculpting.ai/inference/showthread.php?tid=1054
My current setup is Ollama with B200, and I keep running into batch size.

Would appreciate practical advice rather than theory.]]>
My current setup is Ollama with B200, and I keep running into batch size.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=999 Sat, 10 Oct 2026 16:58:57 +0000 pixelspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=999
Just sharing a setup that ended up working reasonably well: Phi, Ollama, Threadripper box, with a lot of attention on quant drift.

Feel free to tear it apart if you see something dumb.]]>
Just sharing a setup that ended up working reasonably well: Phi, Ollama, Threadripper box, with a lot of attention on quant drift.

Feel free to tear it apart if you see something dumb.]]> https://www.datasculpting.ai/inference/showthread.php?tid=954 Fri, 09 Oct 2026 16:30:01 +0000 macrotinker]]> https://www.datasculpting.ai/inference/showthread.php?tid=954
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and mini PC.

Would rather learn from your scars than collect my own all at once.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and mini PC.

Would rather learn from your scars than collect my own all at once.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=906 Fri, 09 Oct 2026 04:45:19 +0000 plainline]]> https://www.datasculpting.ai/inference/showthread.php?tid=906
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=929 Fri, 09 Oct 2026 03:17:00 +0000 northgarden37]]> https://www.datasculpting.ai/inference/showthread.php?tid=929
My current setup is PyTorch with GB200, and I keep running into VRAM headroom.

Not looking for the perfect answer, mostly what worked in real use.

This is mostly for local work rather than public serving.]]>
My current setup is PyTorch with GB200, and I keep running into VRAM headroom.

Not looking for the perfect answer, mostly what worked in real use.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=856 Wed, 07 Oct 2026 19:12:57 +0000 dockbrook21]]> https://www.datasculpting.ai/inference/showthread.php?tid=856
This isn't revolutionary, but the combination of TensorRT-LLM and Threadripper box felt surprisingly usable once I tuned around embeddings.

Maybe useful as a starting point for newer folks.]]>
This isn't revolutionary, but the combination of TensorRT-LLM and Threadripper box felt surprisingly usable once I tuned around embeddings.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=833 Wed, 07 Oct 2026 05:27:45 +0000 sparrowgrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=833
I tried this with Gemma using TensorRT-LLM on 7900 XTX. The headline result was better than I expected, but small local stacks still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I tried this with Gemma using TensorRT-LLM on 7900 XTX. The headline result was better than I expected, but small local stacks still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=814 Tue, 06 Oct 2026 06:03:23 +0000 modgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=814
Just sharing a setup that ended up working reasonably well: Llama, MLX, M3 Max, with a lot of attention on small local stacks.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: Llama, MLX, M3 Max, with a lot of attention on small local stacks.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=790 Tue, 06 Oct 2026 03:50:28 +0000 cacheline]]> https://www.datasculpting.ai/inference/showthread.php?tid=790
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>