+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: General (https://www.datasculpting.ai/inference/forumdisplay.php?fid=25)
+--- Thread: new here, what's your stack (/showthread.php?tid=91)
Pages:
1
2
new here, what's your stack - quiettoken - 09-01-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.
Curious whether that lines up with what other people are seeing.
This is mostly for local work rather than public serving.
Re: new here, what's your stack - rocmrunner40 - 09-03-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
Need a little more detail here. What model, what runtime, and roughly what context size?
I'm happy to test one more variant if you want a second data point.
Re: new here, what's your stack - vectorshed43 - 09-03-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: new here, what's your stack - quiettoken - 09-03-2026
Not sure I agree there. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: new here, what's your stack - tensorkite - 09-03-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.
Re: new here, what's your stack - rocmcrate26 - 09-04-2026
following this because I'm in the same boat.
Re: new here, what's your stack - quietarch - 09-04-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.
Re: new here, what's your stack - rocmrunner40 - 09-04-2026
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: new here, what's your stack - vectorshed43 - 09-04-2026
subscribed, curious where this lands.
Re: new here, what's your stack - ggufwire22 - 09-04-2026
Good point. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.