https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:40 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1495 Mon, 19 Oct 2026 01:59:08 +0000 vrampilot]]> https://www.datasculpting.ai/inference/showthread.php?tid=1495
My current setup is llama.cpp with 7900 XTX, and I keep running into speculative decoding.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.]]>
My current setup is llama.cpp with 7900 XTX, and I keep running into speculative decoding.

Would appreciate practical advice rather than theory.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1447 Sun, 18 Oct 2026 19:18:32 +0000 cudacrate]]> https://www.datasculpting.ai/inference/showthread.php?tid=1447
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1440 Sun, 18 Oct 2026 19:13:00 +0000 embershed99]]> https://www.datasculpting.ai/inference/showthread.php?tid=1440 https://www.datasculpting.ai/inference/showthread.php?tid=1480 Sun, 18 Oct 2026 18:14:00 +0000 phaseport36]]> https://www.datasculpting.ai/inference/showthread.php?tid=1480
I'm happy to test one more variant if you want a second data point.]]>
I'm happy to test one more variant if you want a second data point.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1443 Sun, 18 Oct 2026 17:05:39 +0000 dockridge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1443
I've gone back and forth between SGLang and llama.cpp for Llama. The interesting difference for me has been speculative decoding.

What would you pick if you had to live with it for a few months?]]>
I've gone back and forth between SGLang and llama.cpp for Llama. The interesting difference for me has been speculative decoding.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=1439 Sun, 18 Oct 2026 15:37:52 +0000 kernelarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1439
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

I'm especially interested in mistakes worth avoiding.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1499 Sun, 18 Oct 2026 09:33:55 +0000 mangowire]]> https://www.datasculpting.ai/inference/showthread.php?tid=1499
Maybe useful as a starting point: TensorRT-LLM on mini PC, keeping consumer GPUs in mind from the start.

Maybe useful as a starting point for newer folks.]]>
Maybe useful as a starting point: TensorRT-LLM on mini PC, keeping consumer GPUs in mind from the start.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1421 Sat, 17 Oct 2026 14:23:01 +0000 tinycontext]]> https://www.datasculpting.ai/inference/showthread.php?tid=1421
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1308 Fri, 16 Oct 2026 04:03:23 +0000 rasterpoint]]> https://www.datasculpting.ai/inference/showthread.php?tid=1308
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1216 Wed, 14 Oct 2026 22:57:44 +0000 llm_wren]]> https://www.datasculpting.ai/inference/showthread.php?tid=1216
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring structured output.

Curious if anyone else landed in the same place.]]>
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring structured output.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1206 Wed, 14 Oct 2026 22:00:20 +0000 bytesupply76]]> https://www.datasculpting.ai/inference/showthread.php?tid=1206
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around small local stacks.

Nothing scientific here, just practical notes from a real setup. At this point the real bottleneck is my patience.]]>
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around small local stacks.

Nothing scientific here, just practical notes from a real setup. At this point the real bottleneck is my patience.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1243 Wed, 14 Oct 2026 09:36:25 +0000 batchsize]]> https://www.datasculpting.ai/inference/showthread.php?tid=1243
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1151 Wed, 14 Oct 2026 03:18:19 +0000 quietarch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1151
Short version: MLX behaved well for me, but only after I stopped ignoring quant drift.

Curious if anyone else landed in the same place.]]>
Short version: MLX behaved well for me, but only after I stopped ignoring quant drift.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1121 Mon, 12 Oct 2026 12:35:05 +0000 homelabhenry]]> https://www.datasculpting.ai/inference/showthread.php?tid=1121
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and 4090.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and 4090.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1046 Mon, 12 Oct 2026 03:21:30 +0000 llm_wren]]> https://www.datasculpting.ai/inference/showthread.php?tid=1046
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around tokens/sec.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs. I keep promising myself this is the last rebuild.]]>
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around tokens/sec.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs. I keep promising myself this is the last rebuild.]]>