https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:24 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1428 Sat, 17 Oct 2026 06:16:04 +0000 promptpoint]]> https://www.datasculpting.ai/inference/showthread.php?tid=1428
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefix caching.

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.]]>
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefix caching.

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1269 Thu, 15 Oct 2026 21:16:46 +0000 routeline23]]> https://www.datasculpting.ai/inference/showthread.php?tid=1269
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling INT4.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling INT4.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1141 Tue, 13 Oct 2026 23:49:58 +0000 batchember]]> https://www.datasculpting.ai/inference/showthread.php?tid=1141
Just sharing a setup that ended up working reasonably well: FLUX, vLLM, M3 Max, with a lot of attention on quant drift.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: FLUX, vLLM, M3 Max, with a lot of attention on quant drift.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1154 Tue, 13 Oct 2026 16:28:03 +0000 emberquill]]> https://www.datasculpting.ai/inference/showthread.php?tid=1154
I'm mostly working with Kimi in Ollama on B200. The part I'm trying to understand is structured output.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with Kimi in Ollama on B200. The part I'm trying to understand is structured output.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1053 Mon, 12 Oct 2026 04:28:15 +0000 audioforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=1053
Just sharing a setup that ended up working reasonably well: Mixtral, SGLang, B200, with a lot of attention on VRAM headroom.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Just sharing a setup that ended up working reasonably well: Mixtral, SGLang, B200, with a lot of attention on VRAM headroom.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1026 Sun, 11 Oct 2026 04:14:38 +0000 routeline23]]> https://www.datasculpting.ai/inference/showthread.php?tid=1026
I'm mostly working with Mistral in Ollama on 5090. The part I'm trying to understand is HBM constraints.

If you've tested something similar, what did you keep and what did you stop bothering with?

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]>
I'm mostly working with Mistral in Ollama on 5090. The part I'm trying to understand is HBM constraints.

If you've tested something similar, what did you keep and what did you stop bothering with?

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.]]> https://www.datasculpting.ai/inference/showthread.php?tid=963 Fri, 09 Oct 2026 18:25:50 +0000 plainline]]> https://www.datasculpting.ai/inference/showthread.php?tid=963
I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]>
I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=893 Thu, 08 Oct 2026 17:54:04 +0000 routeline23]]> https://www.datasculpting.ai/inference/showthread.php?tid=893
Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, B200, with a lot of attention on tokens/sec.

If someone wants the exact settings I can clean them up and post them.]]>
Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, B200, with a lot of attention on tokens/sec.

If someone wants the exact settings I can clean them up and post them.]]> https://www.datasculpting.ai/inference/showthread.php?tid=808 Wed, 07 Oct 2026 04:37:41 +0000 orbitgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=808
Two paths both seem viable, but they break down differently once tool calling becomes important.

What would you pick if you had to live with it for a few months?]]>
Two paths both seem viable, but they break down differently once tool calling becomes important.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=771 Mon, 05 Oct 2026 12:30:12 +0000 orbitgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=771
I'm mostly working with Gemma in MLX on mini PC. The part I'm trying to understand is quant drift.

Would appreciate practical advice rather than theory.]]>
I'm mostly working with Gemma in MLX on mini PC. The part I'm trying to understand is quant drift.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=732 Mon, 05 Oct 2026 03:02:54 +0000 cacheworks]]> https://www.datasculpting.ai/inference/showthread.php?tid=732
Short version: vLLM behaved well for me, but only after I stopped ignoring NVFP4.

Nothing scientific here, just practical notes from a real setup.]]>
Short version: vLLM behaved well for me, but only after I stopped ignoring NVFP4.

Nothing scientific here, just practical notes from a real setup.]]> https://www.datasculpting.ai/inference/showthread.php?tid=665 Sat, 03 Oct 2026 00:19:33 +0000 phasenode]]> https://www.datasculpting.ai/inference/showthread.php?tid=665
I've gone back and forth between PyTorch and Ollama for FLUX. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.]]>
I've gone back and forth between PyTorch and Ollama for FLUX. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=661 Fri, 02 Oct 2026 00:12:58 +0000 routeline23]]> https://www.datasculpting.ai/inference/showthread.php?tid=661
The more I work with local setups, the more it seems like KV cache pressure decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
The more I work with local setups, the more it seems like KV cache pressure decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=637 Thu, 01 Oct 2026 23:29:50 +0000 routecraft]]> https://www.datasculpting.ai/inference/showthread.php?tid=637
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=506 Sun, 27 Sep 2026 05:34:21 +0000 forgemixer]]> https://www.datasculpting.ai/inference/showthread.php?tid=506
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around Ollama and 7900 XTX.

Would rather learn from your scars than collect my own all at once.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around Ollama and 7900 XTX.

Would rather learn from your scars than collect my own all at once.]]>