https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:30 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1468 Sun, 18 Oct 2026 18:55:00 +0000 sparrowbrook]]> https://www.datasculpting.ai/inference/showthread.php?tid=1468 https://www.datasculpting.ai/inference/showthread.php?tid=1456 Sun, 18 Oct 2026 17:44:00 +0000 cozywire]]> https://www.datasculpting.ai/inference/showthread.php?tid=1456 https://www.datasculpting.ai/inference/showthread.php?tid=1459 Sun, 18 Oct 2026 17:35:00 +0000 fusedhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=1459 https://www.datasculpting.ai/inference/showthread.php?tid=1461 Sun, 18 Oct 2026 17:05:00 +0000 fastbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=1461 https://www.datasculpting.ai/inference/showthread.php?tid=1330 Fri, 16 Oct 2026 18:00:27 +0000 batchsize]]> https://www.datasculpting.ai/inference/showthread.php?tid=1330
I tried this with Gemma using MLX on Threadripper box. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I tried this with Gemma using MLX on Threadripper box. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1314 Thu, 15 Oct 2026 15:35:35 +0000 dockworks23]]> https://www.datasculpting.ai/inference/showthread.php?tid=1314
Just sharing a setup that ended up working reasonably well: FLUX, TensorRT-LLM, mini PC, with a lot of attention on tail latency.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: FLUX, TensorRT-LLM, mini PC, with a lot of attention on tail latency.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1215 Thu, 15 Oct 2026 01:13:09 +0000 southspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=1215
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

Simple and boring is a valid answer.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1200 Wed, 14 Oct 2026 20:49:01 +0000 amberhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=1200
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefix caching.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefix caching.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1171 Wed, 14 Oct 2026 05:13:41 +0000 routeforge64]]> https://www.datasculpting.ai/inference/showthread.php?tid=1171
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling INT4.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling INT4.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1149 Wed, 14 Oct 2026 04:39:33 +0000 thetapath]]> https://www.datasculpting.ai/inference/showthread.php?tid=1149
I'm mostly working with FLUX in Ollama on B200. The part I'm trying to understand is KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with FLUX in Ollama on B200. The part I'm trying to understand is KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1187 Tue, 13 Oct 2026 16:59:19 +0000 cudahouse]]> https://www.datasculpting.ai/inference/showthread.php?tid=1187
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with consumer GPUs.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with consumer GPUs.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1135 Mon, 12 Oct 2026 23:11:15 +0000 vramarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1135
This isn't revolutionary, but the combination of TensorRT-LLM and 7900 XTX felt surprisingly usable once I tuned around embeddings.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
This isn't revolutionary, but the combination of TensorRT-LLM and 7900 XTX felt surprisingly usable once I tuned around embeddings.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1099 Mon, 12 Oct 2026 17:14:48 +0000 oxideworks59]]> https://www.datasculpting.ai/inference/showthread.php?tid=1099
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1116 Mon, 12 Oct 2026 11:10:05 +0000 lablab]]> https://www.datasculpting.ai/inference/showthread.php?tid=1116
Maybe useful as a starting point: vLLM on GB200, keeping KV cache pressure in mind from the start.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
Maybe useful as a starting point: vLLM on GB200, keeping KV cache pressure in mind from the start.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1060 Mon, 12 Oct 2026 04:48:26 +0000 batchscope72]]> https://www.datasculpting.ai/inference/showthread.php?tid=1060
The more I work with local setups, the more it seems like token cost decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like token cost decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>