https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:25:22 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1362 Sat, 17 Oct 2026 01:40:20 +0000 quiettoken]]> https://www.datasculpting.ai/inference/showthread.php?tid=1362
Two paths both seem viable, but they break down differently once tool calling becomes important.

Interested in tradeoffs more than fan loyalty.]]>
Two paths both seem viable, but they break down differently once tool calling becomes important.

Interested in tradeoffs more than fan loyalty.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1346 Fri, 16 Oct 2026 18:34:53 +0000 vramstream]]> https://www.datasculpting.ai/inference/showthread.php?tid=1346
Just sharing a setup that ended up working reasonably well: Mixtral, PyTorch, 4090, with a lot of attention on long context.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: Mixtral, PyTorch, 4090, with a lot of attention on long context.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1326 Fri, 16 Oct 2026 16:00:57 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=1326
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]>
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1297 Thu, 15 Oct 2026 22:12:56 +0000 rocmcrate26]]> https://www.datasculpting.ai/inference/showthread.php?tid=1297
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around prefill/decode split.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.]]>
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around prefill/decode split.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1251 Wed, 14 Oct 2026 22:03:51 +0000 ggufwire22]]> https://www.datasculpting.ai/inference/showthread.php?tid=1251
I tried this with Mistral using LM Studio on GB200. The headline result was better than I expected, but token cost still ended up being the limiting factor.

Curious if anyone else landed in the same place.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
I tried this with Mistral using LM Studio on GB200. The headline result was better than I expected, but token cost still ended up being the limiting factor.

Curious if anyone else landed in the same place.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]> https://www.datasculpting.ai/inference/showthread.php?tid=598 Tue, 29 Sep 2026 13:47:38 +0000 rasterbench]]> https://www.datasculpting.ai/inference/showthread.php?tid=598
Two paths both seem viable, but they break down differently once workstation thermals becomes important.

Trying to avoid rebuilding the whole thing twice.]]>
Two paths both seem viable, but they break down differently once workstation thermals becomes important.

Trying to avoid rebuilding the whole thing twice.]]> https://www.datasculpting.ai/inference/showthread.php?tid=569 Mon, 28 Sep 2026 19:17:29 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=569
I've gone back and forth between Ollama and SGLang for Gemma. The interesting difference for me has been token cost.

What would you pick if you had to live with it for a few months?]]>
I've gone back and forth between Ollama and SGLang for Gemma. The interesting difference for me has been token cost.

What would you pick if you had to live with it for a few months?]]> https://www.datasculpting.ai/inference/showthread.php?tid=491 Fri, 25 Sep 2026 18:17:16 +0000 mangoquill]]> https://www.datasculpting.ai/inference/showthread.php?tid=491
Just sharing a setup that ended up working reasonably well: Llama, Ollama, 4090, with a lot of attention on quant drift.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: Llama, Ollama, 4090, with a lot of attention on quant drift.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=480 Fri, 25 Sep 2026 16:16:16 +0000 modgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=480
I'm mostly working with Llama in Ollama on 7900 XTX. The part I'm trying to understand is workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm mostly working with Llama in Ollama on 7900 XTX. The part I'm trying to understand is workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=230 Fri, 11 Sep 2026 20:14:30 +0000 batchwire]]> https://www.datasculpting.ai/inference/showthread.php?tid=230
I'm mostly working with Kimi in LM Studio on M3 Max. The part I'm trying to understand is workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm mostly working with Kimi in LM Studio on M3 Max. The part I'm trying to understand is workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=85 Mon, 31 Aug 2026 19:45:17 +0000 kernelgrid]]> https://www.datasculpting.ai/inference/showthread.php?tid=85
I'm mostly working with Kimi in llama.cpp on A6000. The part I'm trying to understand is INT4.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm mostly working with Kimi in llama.cpp on A6000. The part I'm trying to understand is INT4.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=38 Wed, 26 Aug 2026 09:39:08 +0000 mlxgarden]]> https://www.datasculpting.ai/inference/showthread.php?tid=38
My current setup is SGLang with 7900 XTX, and I keep running into batch size.

Would appreciate practical advice rather than theory. The logs were somehow calmer than I was.]]>
My current setup is SGLang with 7900 XTX, and I keep running into batch size.

Would appreciate practical advice rather than theory. The logs were somehow calmer than I was.]]> https://www.datasculpting.ai/inference/showthread.php?tid=22 Tue, 25 Aug 2026 03:08:57 +0000 turboarch95]]> https://www.datasculpting.ai/inference/showthread.php?tid=22
I tried this with Qwen using TensorRT-LLM on 5090. The headline result was better than I expected, but reranking still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.]]>
I tried this with Qwen using TensorRT-LLM on 5090. The headline result was better than I expected, but reranking still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.]]>