Posts: 35
Threads: 4
Joined: Oct 2026
Reputation:
0
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: DeepSeek, vLLM, A6000, with a lot of attention on reranking.
Maybe useful as a starting point for newer folks.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
trying not to rebuild again
Posts: 121
Threads: 15
Joined: Sep 2026
Reputation:
0
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
image models by night, local pipelines by day
Posts: 42
Threads: 1
Joined: Oct 2026
Reputation:
0
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching.
Posts: 31
Threads: 2
Joined: Oct 2026
Reputation:
0
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Posts: 42
Threads: 1
Joined: Oct 2026
Reputation:
0
I don't think that's the whole story. I've seen people blame structured output when the real issue was much more mundane. The logs were somehow calmer than I was.
Posts: 121
Threads: 15
Joined: Sep 2026
Reputation:
0
Quote:I don't think that's the whole story. I've seen people blame structured output when the real issue was much more mundane. The logs were somehow calmer
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
image models by night, local pipelines by day
Posts: 28
Threads: 5
Joined: Sep 2026
Reputation:
0
that fixed it for me too.
Posts: 46
Threads: 6
Joined: Sep 2026
Reputation:
0
that fixed it for me too.
Posts: 28
Threads: 5
Joined: Sep 2026
Reputation:
0
Interesting, I hadn't tried that. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
