https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:25:23 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1454 Sun, 18 Oct 2026 18:44:00 +0000 fusedriver]]> https://www.datasculpting.ai/inference/showthread.php?tid=1454 Quote:Appreciate the detail. I don't think your conclusion is far off.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.]]> Quote:Appreciate the detail. I don't think your conclusion is far off.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1411 Sat, 17 Oct 2026 17:22:00 +0000 marbleyard32]]> https://www.datasculpting.ai/inference/showthread.php?tid=1411
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and A6000.

Simple and boring is a valid answer.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and A6000.

Simple and boring is a valid answer.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1372 Sat, 17 Oct 2026 02:12:15 +0000 plainscope50]]> https://www.datasculpting.ai/inference/showthread.php?tid=1372
This isn't revolutionary, but the combination of llama.cpp and dual 3090 felt surprisingly usable once I tuned around prefix caching.

If someone wants the exact settings I can clean them up and post them.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
This isn't revolutionary, but the combination of llama.cpp and dual 3090 felt surprisingly usable once I tuned around prefix caching.

If someone wants the exact settings I can clean them up and post them.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1350 Fri, 16 Oct 2026 18:36:40 +0000 plainline]]> https://www.datasculpting.ai/inference/showthread.php?tid=1350
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about quant drift.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about quant drift.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1268 Thu, 15 Oct 2026 17:06:08 +0000 fusedriver]]> https://www.datasculpting.ai/inference/showthread.php?tid=1268
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1230 Thu, 15 Oct 2026 01:08:13 +0000 cozysmith]]> https://www.datasculpting.ai/inference/showthread.php?tid=1230
Maybe useful as a starting point: MLX on dual 3090, keeping tail latency in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]>
Maybe useful as a starting point: MLX on dual 3090, keeping tail latency in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1075 Mon, 12 Oct 2026 03:17:13 +0000 amberrunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=1075
One thing I keep noticing is that people underestimate consumer GPUs until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
One thing I keep noticing is that people underestimate consumer GPUs until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=879 Wed, 07 Oct 2026 22:26:41 +0000 cozysmith]]> https://www.datasculpting.ai/inference/showthread.php?tid=879
The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]>
The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=846 Tue, 06 Oct 2026 17:19:44 +0000 batchpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=846
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.

I'm especially interested in mistakes worth avoiding.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=778 Tue, 06 Oct 2026 00:38:07 +0000 marbleyard32]]> https://www.datasculpting.ai/inference/showthread.php?tid=778
I've gone back and forth between TensorRT-LLM and llama.cpp for Mistral. The interesting difference for me has been quant drift.

Interested in tradeoffs more than fan loyalty.]]>
I've gone back and forth between TensorRT-LLM and llama.cpp for Mistral. The interesting difference for me has been quant drift.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=703 Sat, 03 Oct 2026 06:28:30 +0000 fastquill74]]> https://www.datasculpting.ai/inference/showthread.php?tid=703
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=601 Thu, 01 Oct 2026 05:38:50 +0000 fastquill74]]> https://www.datasculpting.ai/inference/showthread.php?tid=601
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about quant drift.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about quant drift.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=594 Wed, 30 Sep 2026 04:53:24 +0000 fusedriver]]> https://www.datasculpting.ai/inference/showthread.php?tid=594
I tried this with Llama using PyTorch on mini PC. The headline result was better than I expected, but consumer GPUs still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.]]>
I tried this with Llama using PyTorch on mini PC. The headline result was better than I expected, but consumer GPUs still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=521 Sat, 26 Sep 2026 20:54:26 +0000 promptspark58]]> https://www.datasculpting.ai/inference/showthread.php?tid=521
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and 4090.

Simple and boring is a valid answer.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and 4090.

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=223 Fri, 11 Sep 2026 05:34:35 +0000 cozysmith]]> https://www.datasculpting.ai/inference/showthread.php?tid=223
Short version: llama.cpp behaved well for me, but only after I stopped ignoring KV cache pressure.

Happy to post more details if anyone wants them.]]>
Short version: llama.cpp behaved well for me, but only after I stopped ignoring KV cache pressure.

Happy to post more details if anyone wants them.]]>