https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:36:11 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1407 Sun, 18 Oct 2026 03:47:52 +0000 fusedhall56]]> https://www.datasculpting.ai/inference/showthread.php?tid=1407
I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]>
I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1435 Sun, 18 Oct 2026 02:34:24 +0000 thetafield]]> https://www.datasculpting.ai/inference/showthread.php?tid=1435
Two paths both seem viable, but they break down differently once long context becomes important.

Trying to avoid rebuilding the whole thing twice.]]>
Two paths both seem viable, but they break down differently once long context becomes important.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1377 Sat, 17 Oct 2026 18:48:50 +0000 mintstation99]]> https://www.datasculpting.ai/inference/showthread.php?tid=1377
I tried this with Phi using MLX on 4090. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.]]>
I tried this with Phi using MLX on 4090. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1291 Thu, 15 Oct 2026 11:21:38 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1291
Short version: MLX behaved well for me, but only after I stopped ignoring prefix caching.

Nothing scientific here, just practical notes from a real setup.]]>
Short version: MLX behaved well for me, but only after I stopped ignoring prefix caching.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1118 Mon, 12 Oct 2026 15:12:50 +0000 fusedhall56]]> https://www.datasculpting.ai/inference/showthread.php?tid=1118
I've gone back and forth between llama.cpp and vLLM for Gemma. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I've gone back and forth between llama.cpp and vLLM for Gemma. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1066 Sun, 11 Oct 2026 23:58:37 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1066
After a few runs on A6000, I ended up with decent performance, but the setup only really made sense once I adjusted around structured output.

Nothing scientific here, just practical notes from a real setup.]]>
After a few runs on A6000, I ended up with decent performance, but the setup only really made sense once I adjusted around structured output.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1020 Sat, 10 Oct 2026 13:23:56 +0000 localpatch]]> https://www.datasculpting.ai/inference/showthread.php?tid=1020
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=978 Fri, 09 Oct 2026 18:33:11 +0000 batchember]]> https://www.datasculpting.ai/inference/showthread.php?tid=978
The more I work with local setups, the more it seems like batch size decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]>
The more I work with local setups, the more it seems like batch size decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=933 Thu, 08 Oct 2026 19:42:11 +0000 mintstation99]]> https://www.datasculpting.ai/inference/showthread.php?tid=933
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling VRAM headroom.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling VRAM headroom.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=867 Wed, 07 Oct 2026 19:41:53 +0000 turboarch95]]> https://www.datasculpting.ai/inference/showthread.php?tid=867
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=890 Wed, 07 Oct 2026 16:18:55 +0000 quiettoken]]> https://www.datasculpting.ai/inference/showthread.php?tid=890
One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.]]>
One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=861 Wed, 07 Oct 2026 10:20:43 +0000 localpatch]]> https://www.datasculpting.ai/inference/showthread.php?tid=861
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling structured output.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling structured output.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=783 Mon, 05 Oct 2026 15:56:57 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=783
The more I work with local setups, the more it seems like long context decides everything downstream.

Wondering if this is just where the whole space is heading now.]]>
The more I work with local setups, the more it seems like long context decides everything downstream.

Wondering if this is just where the whole space is heading now.]]> https://www.datasculpting.ai/inference/showthread.php?tid=805 Mon, 05 Oct 2026 06:00:09 +0000 thetafield]]> https://www.datasculpting.ai/inference/showthread.php?tid=805
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

I'm especially interested in mistakes worth avoiding.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

I'm especially interested in mistakes worth avoiding.]]> https://www.datasculpting.ai/inference/showthread.php?tid=527 Mon, 28 Sep 2026 04:23:10 +0000 rasterhall]]> https://www.datasculpting.ai/inference/showthread.php?tid=527
The more I work with local setups, the more it seems like KV cache pressure decides everything downstream.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
The more I work with local setups, the more it seems like KV cache pressure decides everything downstream.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>