https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 12:54:39 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1416 Sat, 17 Oct 2026 23:16:43 +0000 rocmpath]]> https://www.datasculpting.ai/inference/showthread.php?tid=1416
Maybe useful as a starting point: vLLM on 5090, keeping tool calling in mind from the start.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Maybe useful as a starting point: vLLM on 5090, keeping tool calling in mind from the start.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1263 Fri, 16 Oct 2026 02:53:35 +0000 turbosupply83]]> https://www.datasculpting.ai/inference/showthread.php?tid=1263
I'm mostly working with FLUX in SGLang on B200. The part I'm trying to understand is tail latency.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm mostly working with FLUX in SGLang on B200. The part I'm trying to understand is tail latency.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1068 Sun, 11 Oct 2026 22:13:47 +0000 orbitline]]> https://www.datasculpting.ai/inference/showthread.php?tid=1068
My current setup is LM Studio with dual 3090, and I keep running into prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
My current setup is LM Studio with dual 3090, and I keep running into prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=959 Sat, 10 Oct 2026 05:07:08 +0000 flintriver50]]> https://www.datasculpting.ai/inference/showthread.php?tid=959
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=975 Fri, 09 Oct 2026 22:00:30 +0000 lablab]]> https://www.datasculpting.ai/inference/showthread.php?tid=975
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=903 Thu, 08 Oct 2026 15:22:04 +0000 sparrowghost]]> https://www.datasculpting.ai/inference/showthread.php?tid=903
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

Would rather learn from your scars than collect my own all at once.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

Would rather learn from your scars than collect my own all at once.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=869 Wed, 07 Oct 2026 19:19:42 +0000 routeline23]]> https://www.datasculpting.ai/inference/showthread.php?tid=869
My current setup is TensorRT-LLM with 5090, and I keep running into continuous batching.

If you've tested something similar, what did you keep and what did you stop bothering with?

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
My current setup is TensorRT-LLM with 5090, and I keep running into continuous batching.

If you've tested something similar, what did you keep and what did you stop bothering with?

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=781 Mon, 05 Oct 2026 20:52:30 +0000 mintquill]]> https://www.datasculpting.ai/inference/showthread.php?tid=781
After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around small local stacks.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.]]>
After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around small local stacks.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=384 Sun, 20 Sep 2026 22:10:35 +0000 mangopath4]]> https://www.datasculpting.ai/inference/showthread.php?tid=384
One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=312 Wed, 16 Sep 2026 16:17:12 +0000 rocmpath]]> https://www.datasculpting.ai/inference/showthread.php?tid=312
The more I work with local setups, the more it seems like small local stacks decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
The more I work with local setups, the more it seems like small local stacks decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=178 Tue, 08 Sep 2026 05:40:03 +0000 audiorunner74]]> https://www.datasculpting.ai/inference/showthread.php?tid=178
One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=179 Tue, 08 Sep 2026 05:16:54 +0000 turboscope]]> https://www.datasculpting.ai/inference/showthread.php?tid=179
I've gone back and forth between llama.cpp and LM Studio for Llama. The interesting difference for me has been consumer GPUs.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I've gone back and forth between llama.cpp and LM Studio for Llama. The interesting difference for me has been consumer GPUs.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=55 Fri, 28 Aug 2026 12:25:00 +0000 flintriver50]]> https://www.datasculpting.ai/inference/showthread.php?tid=55
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.

Curious whether that lines up with what other people are seeing.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.

Curious whether that lines up with what other people are seeing.]]>