https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:39 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1388 Sun, 18 Oct 2026 00:32:22 +0000 dockember]]> https://www.datasculpting.ai/inference/showthread.php?tid=1388
I'm mostly working with FLUX in SGLang on A6000. The part I'm trying to understand is KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with? The logs were somehow calmer than I was.]]>
I'm mostly working with FLUX in SGLang on A6000. The part I'm trying to understand is KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with? The logs were somehow calmer than I was.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1313 Fri, 16 Oct 2026 01:12:49 +0000 vramforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1313
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about NVFP4.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about NVFP4.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1278 Thu, 15 Oct 2026 09:28:26 +0000 edgequill]]> https://www.datasculpting.ai/inference/showthread.php?tid=1278
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1234 Wed, 14 Oct 2026 18:47:07 +0000 kernelmike]]> https://www.datasculpting.ai/inference/showthread.php?tid=1234
I've gone back and forth between SGLang and Ollama for Gemma. The interesting difference for me has been workstation thermals.

Trying to avoid rebuilding the whole thing twice.]]>
I've gone back and forth between SGLang and Ollama for Gemma. The interesting difference for me has been workstation thermals.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1167 Tue, 13 Oct 2026 15:56:03 +0000 stackarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1167
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and M3 Max.

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and M3 Max.

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1097 Tue, 13 Oct 2026 00:54:39 +0000 turboscope]]> https://www.datasculpting.ai/inference/showthread.php?tid=1097
I'm mostly working with FLUX in Ollama on Threadripper box. The part I'm trying to understand is long context.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm mostly working with FLUX in Ollama on Threadripper box. The part I'm trying to understand is long context.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1083 Sun, 11 Oct 2026 15:49:27 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1083
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1018 Sat, 10 Oct 2026 12:01:09 +0000 cudapeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=1018
Two paths both seem viable, but they break down differently once embeddings becomes important.

What would you pick if you had to live with it for a few months?]]>
Two paths both seem viable, but they break down differently once embeddings becomes important.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=935 Thu, 08 Oct 2026 17:36:59 +0000 cobaltspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=935
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around structured output.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around structured output.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=770 Tue, 06 Oct 2026 05:03:21 +0000 kernelrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=770
One thing I keep noticing is that people underestimate consumer GPUs until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]>
One thing I keep noticing is that people underestimate consumer GPUs until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=741 Sun, 04 Oct 2026 12:58:10 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=741
Short version: llama.cpp behaved well for me, but only after I stopped ignoring quant drift.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Short version: llama.cpp behaved well for me, but only after I stopped ignoring quant drift.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=711 Sun, 04 Oct 2026 02:39:36 +0000 cozywire]]> https://www.datasculpting.ai/inference/showthread.php?tid=711
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 7900 XTX.

I'm especially interested in mistakes worth avoiding.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 7900 XTX.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=716 Sat, 03 Oct 2026 11:54:45 +0000 mangopeak72]]> https://www.datasculpting.ai/inference/showthread.php?tid=716
I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=558 Mon, 28 Sep 2026 09:02:43 +0000 cachebench]]> https://www.datasculpting.ai/inference/showthread.php?tid=558
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.]]>
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=477 Fri, 25 Sep 2026 12:24:39 +0000 signalcraft14]]> https://www.datasculpting.ai/inference/showthread.php?tid=477
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>