framework overhead in real deployments

[stackpilot]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with embeddings.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[benchplease]

This keeps coming up in conversation so I figured I'd start a thread.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding.

[deltastation]

that fixed it for me too.

[quantsupply]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling.

[labline]

I hit almost the same issue last night.

[cudascope]

I hit almost the same issue last night.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. Not glamorous advice, but repeatable baselines save a lot of time.

[rocmpath]

Appreciate the detail. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[fusedriver]

Thanks, that helps. That matches what I saw on M3 Max.

[cudaworks]

Thanks, that helps. That matches what I saw on M3 Max.

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?