https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:39 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1322 Sat, 17 Oct 2026 03:11:39 +0000 promptfield]]> https://www.datasculpting.ai/inference/showthread.php?tid=1322
The more I work with local setups, the more it seems like continuous batching decides everything downstream.

Curious whether that lines up with what other people are seeing.]]>
The more I work with local setups, the more it seems like continuous batching decides everything downstream.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1363 Fri, 16 Oct 2026 20:01:33 +0000 quantshed]]> https://www.datasculpting.ai/inference/showthread.php?tid=1363
My current setup is TensorRT-LLM with A6000, and I keep running into prefix caching.

Not looking for the perfect answer, mostly what worked in real use. Turns out 'just one more test' was a lie.]]>
My current setup is TensorRT-LLM with A6000, and I keep running into prefix caching.

Not looking for the perfect answer, mostly what worked in real use. Turns out 'just one more test' was a lie.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1311 Fri, 16 Oct 2026 05:53:08 +0000 flintriver50]]> https://www.datasculpting.ai/inference/showthread.php?tid=1311
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about token cost.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about token cost.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1283 Thu, 15 Oct 2026 19:08:28 +0000 rasterbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1283
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]>
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1087 Tue, 13 Oct 2026 01:32:06 +0000 pixelcrate82]]> https://www.datasculpting.ai/inference/showthread.php?tid=1087
Just sharing a setup that ended up working reasonably well: Gemma, Ollama, 7900 XTX, with a lot of attention on prefix caching.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Just sharing a setup that ended up working reasonably well: Gemma, Ollama, 7900 XTX, with a lot of attention on prefix caching.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=604 Thu, 01 Oct 2026 03:21:18 +0000 batchwire]]> https://www.datasculpting.ai/inference/showthread.php?tid=604
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with? At this point the real bottleneck is my patience.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with? At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=624 Wed, 30 Sep 2026 22:34:21 +0000 rocmcrate26]]> https://www.datasculpting.ai/inference/showthread.php?tid=624
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. I keep promising myself this is the last rebuild.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. I keep promising myself this is the last rebuild.]]> https://www.datasculpting.ai/inference/showthread.php?tid=290 Wed, 16 Sep 2026 00:56:36 +0000 llmpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=290
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around embeddings.

Happy to post more details if anyone wants them.]]>
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around embeddings.

Happy to post more details if anyone wants them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=287 Tue, 15 Sep 2026 00:12:26 +0000 minthall]]> https://www.datasculpting.ai/inference/showthread.php?tid=287
My current setup is PyTorch with 7900 XTX, and I keep running into consumer GPUs.

Would appreciate practical advice rather than theory.]]>
My current setup is PyTorch with 7900 XTX, and I keep running into consumer GPUs.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=259 Sun, 13 Sep 2026 21:05:55 +0000 vectorhouse71]]> https://www.datasculpting.ai/inference/showthread.php?tid=259
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=159 Sun, 06 Sep 2026 23:05:51 +0000 llm_wren]]> https://www.datasculpting.ai/inference/showthread.php?tid=159
Just sharing a setup that ended up working reasonably well: FLUX, TensorRT-LLM, A6000, with a lot of attention on router policy.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Just sharing a setup that ended up working reasonably well: FLUX, TensorRT-LLM, A6000, with a lot of attention on router policy.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=72 Mon, 31 Aug 2026 01:57:59 +0000 phasenode]]> https://www.datasculpting.ai/inference/showthread.php?tid=72
The more I work with local setups, the more it seems like small local stacks decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like small local stacks decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=50 Fri, 28 Aug 2026 03:46:34 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=50
I tried this with Mistral using LM Studio on GB200. The headline result was better than I expected, but VRAM headroom still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>
I tried this with Mistral using LM Studio on GB200. The headline result was better than I expected, but VRAM headroom still ended up being the limiting factor.

Happy to post more details if anyone wants them.]]>