Posts: 184
Threads: 23
Joined: Sep 2026
Reputation:
0
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like structured output decides everything downstream.
Wondering if this is just where the whole space is heading now.
Mostly CUDA, sometimes wrong, usually measured
Posts: 39
Threads: 4
Joined: Sep 2026
Reputation:
0
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Posts: 32
Threads: 1
Joined: Sep 2026
Reputation:
0
Appreciate the detail. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Posts: 35
Threads: 3
Joined: Sep 2026
Reputation:
0
I tested this tonight. On Threadripper box with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed. At this point the real bottleneck is my patience.
Posts: 40
Threads: 9
Joined: Sep 2026
Reputation:
0
Not sure I agree there. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Posts: 63
Threads: 13
Joined: Sep 2026
Reputation:
0
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Posts: 83
Threads: 6
Joined: Sep 2026
Reputation:
0
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
