https://www.datasculpting.ai/inference/ Mon, 26 Oct 2026 11:30:29 +0000 MyBB https://www.datasculpting.ai/inference/showthread.php?tid=1358 Sat, 17 Oct 2026 02:46:31 +0000 southspark]]> https://www.datasculpting.ai/inference/showthread.php?tid=1358
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.

Maybe obvious to some of you, but it clicked for me this week.]]>
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1214 Wed, 14 Oct 2026 14:34:40 +0000 oxiderunner]]> https://www.datasculpting.ai/inference/showthread.php?tid=1214
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1176 Wed, 14 Oct 2026 01:14:20 +0000 northgarden37]]> https://www.datasculpting.ai/inference/showthread.php?tid=1176
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling batch size.

Not looking for the perfect answer, mostly what worked in real use.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling batch size.

Not looking for the perfect answer, mostly what worked in real use.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1178 Tue, 13 Oct 2026 21:35:27 +0000 mangofield33]]> https://www.datasculpting.ai/inference/showthread.php?tid=1178
After a few runs on A6000, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
After a few runs on A6000, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1094 Mon, 12 Oct 2026 20:43:23 +0000 thetashed24]]> https://www.datasculpting.ai/inference/showthread.php?tid=1094
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tokens/sec.

Would appreciate practical advice rather than theory.]]>
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tokens/sec.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=1036 Mon, 12 Oct 2026 03:52:26 +0000 thetaarch43]]> https://www.datasculpting.ai/inference/showthread.php?tid=1036
My current setup is TensorRT-LLM with 5090, and I keep running into KV cache pressure.

Would appreciate practical advice rather than theory.]]>
My current setup is TensorRT-LLM with 5090, and I keep running into KV cache pressure.

Would appreciate practical advice rather than theory.]]> https://www.datasculpting.ai/inference/showthread.php?tid=932 Thu, 08 Oct 2026 13:47:31 +0000 kernelghost]]> https://www.datasculpting.ai/inference/showthread.php?tid=932
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.]]>
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=692 Sat, 03 Oct 2026 04:43:38 +0000 relayforge]]> https://www.datasculpting.ai/inference/showthread.php?tid=692
Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, B200, with a lot of attention on continuous batching.

Maybe useful as a starting point for newer folks.]]>
Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, B200, with a lot of attention on continuous batching.

Maybe useful as a starting point for newer folks.]]> https://www.datasculpting.ai/inference/showthread.php?tid=589 Tue, 29 Sep 2026 22:27:25 +0000 byteyard]]> https://www.datasculpting.ai/inference/showthread.php?tid=589
The more I work with local setups, the more it seems like speculative decoding decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]>
The more I work with local setups, the more it seems like speculative decoding decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.]]> https://www.datasculpting.ai/inference/showthread.php?tid=590 Tue, 29 Sep 2026 15:11:07 +0000 turboscope]]> https://www.datasculpting.ai/inference/showthread.php?tid=590
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]>
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.]]> https://www.datasculpting.ai/inference/showthread.php?tid=517 Sun, 27 Sep 2026 03:25:37 +0000 cobaltworks25]]> https://www.datasculpting.ai/inference/showthread.php?tid=517
I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]>
I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.]]> https://www.datasculpting.ai/inference/showthread.php?tid=407 Mon, 21 Sep 2026 21:47:32 +0000 benchplease]]> https://www.datasculpting.ai/inference/showthread.php?tid=407
I'm mostly working with FLUX in Ollama on A6000. The part I'm trying to understand is prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?]]>
I'm mostly working with FLUX in Ollama on A6000. The part I'm trying to understand is prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?]]> https://www.datasculpting.ai/inference/showthread.php?tid=345 Sat, 19 Sep 2026 01:34:35 +0000 cacheplane98]]> https://www.datasculpting.ai/inference/showthread.php?tid=345
My current setup is vLLM with mini PC, and I keep running into small local stacks.

Not looking for the perfect answer, mostly what worked in real use. The logs were somehow calmer than I was.]]>
My current setup is vLLM with mini PC, and I keep running into small local stacks.

Not looking for the perfect answer, mostly what worked in real use. The logs were somehow calmer than I was.]]> https://www.datasculpting.ai/inference/showthread.php?tid=129 Fri, 04 Sep 2026 22:30:47 +0000 batchsize]]> https://www.datasculpting.ai/inference/showthread.php?tid=129
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]>
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.]]> https://www.datasculpting.ai/inference/showthread.php?tid=135 Fri, 04 Sep 2026 18:48:32 +0000 sanepeak48]]> https://www.datasculpting.ai/inference/showthread.php?tid=135
Just sharing a setup that ended up working reasonably well: Phi, MLX, 7900 XTX, with a lot of attention on INT4.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>
Just sharing a setup that ended up working reasonably well: Phi, MLX, 7900 XTX, with a lot of attention on INT4.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.]]>