https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:33:49 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1449 Sun, 18 Oct 2026 21:09:00 +0000 vectorcrate98]]> https://www.datasculpting.ai/inference/showthread.php?tid=1449 https://www.datasculpting.ai/inference/showthread.php?tid=1475 Sun, 18 Oct 2026 18:37:00 +0000 edgeriver]]> https://www.datasculpting.ai/inference/showthread.php?tid=1475 https://www.datasculpting.ai/inference/showthread.php?tid=1400 Sat, 17 Oct 2026 23:11:08 +0000 nanogarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1400
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around consumer GPUs.

Curious if anyone else landed in the same place.]]>
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around consumer GPUs.

Curious if anyone else landed in the same place.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1341 Fri, 16 Oct 2026 14:36:09 +0000 vectorember]]> https://www.datasculpting.ai/inference/showthread.php?tid=1341
Maybe useful as a starting point: llama.cpp on GB200, keeping small local stacks in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]>
Maybe useful as a starting point: llama.cpp on GB200, keeping small local stacks in mind from the start.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1296 Thu, 15 Oct 2026 22:16:22 +0000 torchtrack24]]> https://www.datasculpting.ai/inference/showthread.php?tid=1296
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1286 Thu, 15 Oct 2026 13:43:29 +0000 audioforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1286
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and 4090.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and 4090.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1223 Wed, 14 Oct 2026 19:12:49 +0000 quantghost]]> https://www.datasculpting.ai/inference/showthread.php?tid=1223
One thing I keep noticing is that people underestimate MoE serving until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
One thing I keep noticing is that people underestimate MoE serving until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1160 Tue, 13 Oct 2026 09:29:35 +0000 vramghost]]> https://www.datasculpting.ai/inference/showthread.php?tid=1160
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.

Nothing scientific here, just practical notes from a real setup.]]>
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=899 Thu, 08 Oct 2026 20:46:09 +0000 routeline23]]> https://www.datasculpting.ai/inference/showthread.php?tid=899
Maybe useful as a starting point: Ollama on 7900 XTX, keeping reranking in mind from the start.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
Maybe useful as a starting point: Ollama on 7900 XTX, keeping reranking in mind from the start.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=857 Thu, 08 Oct 2026 05:10:33 +0000 fusedtinker]]> https://www.datasculpting.ai/inference/showthread.php?tid=857
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Wondering if this is just where the whole space is heading now.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Wondering if this is just where the whole space is heading now.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=858 Wed, 07 Oct 2026 19:02:13 +0000 emberpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=858
I'm mostly working with Llama in SGLang on dual 3090. The part I'm trying to understand is small local stacks.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm mostly working with Llama in SGLang on dual 3090. The part I'm trying to understand is small local stacks.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=868 Wed, 07 Oct 2026 18:22:06 +0000 fastbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=868
I've gone back and forth between SGLang and vLLM for DeepSeek. The interesting difference for me has been reranking.

Interested in tradeoffs more than fan loyalty.]]>
I've gone back and forth between SGLang and vLLM for DeepSeek. The interesting difference for me has been reranking.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=792 Tue, 06 Oct 2026 01:39:28 +0000 batchfield55]]> https://www.datasculpting.ai/inference/showthread.php?tid=792
The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=758 Sun, 04 Oct 2026 21:33:17 +0000 kernelpeak]]> https://www.datasculpting.ai/inference/showthread.php?tid=758
This isn't revolutionary, but the combination of Ollama and 4090 felt surprisingly usable once I tuned around embeddings.

Maybe useful as a starting point for newer folks.]]>
This isn't revolutionary, but the combination of Ollama and 4090 felt surprisingly usable once I tuned around embeddings.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=750 Sun, 04 Oct 2026 19:44:43 +0000 vectorcrate98]]> https://www.datasculpting.ai/inference/showthread.php?tid=750
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefix caching.

Would appreciate practical advice rather than theory.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefix caching.

Would appreciate practical advice rather than theory.]]>