https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:47 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1453 Sun, 18 Oct 2026 22:41:00 +0000 vrambandit]]> https://www.datasculpting.ai/inference/showthread.php?tid=1453 https://www.datasculpting.ai/inference/showthread.php?tid=1487 Sun, 18 Oct 2026 15:30:57 +0000 visionyard23]]> https://www.datasculpting.ai/inference/showthread.php?tid=1487
I've gone back and forth between Ollama and TensorRT-LLM for Mistral. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.]]>
I've gone back and forth between Ollama and TensorRT-LLM for Mistral. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1450 Sun, 18 Oct 2026 13:20:58 +0000 bytesupply76]]> https://www.datasculpting.ai/inference/showthread.php?tid=1450
Maybe useful as a starting point: Ollama on GB200, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Maybe useful as a starting point: Ollama on GB200, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1458 Sun, 18 Oct 2026 06:32:03 +0000 vrambandit]]> https://www.datasculpting.ai/inference/showthread.php?tid=1458
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with NVFP4.

Maybe obvious to some of you, but it clicked for me this week.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with NVFP4.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1391 Sat, 17 Oct 2026 14:04:23 +0000 vectorport]]> https://www.datasculpting.ai/inference/showthread.php?tid=1391
I've gone back and forth between vLLM and TensorRT-LLM for FLUX. The interesting difference for me has been long context.

Trying to avoid rebuilding the whole thing twice.]]>
I've gone back and forth between vLLM and TensorRT-LLM for FLUX. The interesting difference for me has been long context.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1424 Sat, 17 Oct 2026 10:08:54 +0000 bytesupply76]]> https://www.datasculpting.ai/inference/showthread.php?tid=1424
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Curious whether that lines up with what other people are seeing.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Curious whether that lines up with what other people are seeing.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1374 Fri, 16 Oct 2026 22:05:28 +0000 phasetrack]]> https://www.datasculpting.ai/inference/showthread.php?tid=1374
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with continuous batching.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with continuous batching.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1293 Fri, 16 Oct 2026 04:46:03 +0000 vectorport]]> https://www.datasculpting.ai/inference/showthread.php?tid=1293
I'm mostly working with FLUX in PyTorch on A6000. The part I'm trying to understand is HBM constraints.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm mostly working with FLUX in PyTorch on A6000. The part I'm trying to understand is HBM constraints.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1280 Thu, 15 Oct 2026 23:29:28 +0000 torchsupply99]]> https://www.datasculpting.ai/inference/showthread.php?tid=1280
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]>
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1257 Thu, 15 Oct 2026 17:10:21 +0000 bytesupply76]]> https://www.datasculpting.ai/inference/showthread.php?tid=1257
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around prefix caching.

Curious if anyone else landed in the same place.]]>
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around prefix caching.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1240 Wed, 14 Oct 2026 22:53:38 +0000 northcraft90]]> https://www.datasculpting.ai/inference/showthread.php?tid=1240
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with consumer GPUs.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with consumer GPUs.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1252 Wed, 14 Oct 2026 19:37:24 +0000 mellowwire98]]> https://www.datasculpting.ai/inference/showthread.php?tid=1252
The more I work with local setups, the more it seems like workstation thermals decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like workstation thermals decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1106 Tue, 13 Oct 2026 00:19:02 +0000 vrambandit]]> https://www.datasculpting.ai/inference/showthread.php?tid=1106
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1102 Mon, 12 Oct 2026 06:53:08 +0000 latentcraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=1102
Just sharing a setup that ended up working reasonably well: Mistral, vLLM, A6000, with a lot of attention on tail latency.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
Just sharing a setup that ended up working reasonably well: Mistral, vLLM, A6000, with a lot of attention on tail latency.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1045 Mon, 12 Oct 2026 01:53:41 +0000 embershed99]]> https://www.datasculpting.ai/inference/showthread.php?tid=1045
This isn't revolutionary, but the combination of PyTorch and B200 felt surprisingly usable once I tuned around consumer GPUs.

Feel free to tear it apart if you see something dumb.]]>
This isn't revolutionary, but the combination of PyTorch and B200 felt surprisingly usable once I tuned around consumer GPUs.

Feel free to tear it apart if you see something dumb.]]>