best sandbox for trying risky tools: anyone testing this?

[cacheworks]

Did a bit of testing and thought I'd post the result here.

I tried this with Mistral using MLX on M3 Max. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.