70B still worth the pain

[rocmhouse90]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, SGLang, B200, with a lot of attention on prefix caching.

Feel free to tear it apart if you see something dumb.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[cudascope]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[mangotinker73]

Thanks, that helps. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[mangoquill]

Thanks, that helps. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[quiethouse]

I think the bottleneck is somewhere else. I've seen people blame small local stacks when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[latentpeak22]

I think the bottleneck is somewhere else. I've seen people blame small local stacks when the real issue was much more mundane. Not saying you're wrong

I'd start with the simpler stack.

[bytehall]

[quote]I think the bottleneck is somewhere else. I've seen people blame small local stacks when the real issue was much more mundane. Not saying you'r

That matches what I was seeing. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[turboyard94]

Did a small comparison this afternoon. On GB200 with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[batchscope72]

I hit almost the same issue last night.

[cudaline]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.