sandboxing agent actions

[rocmcrate26]

I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. I keep promising myself this is the last rebuild.

[cudascope]

Good point. The pattern around embeddings has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[fastquill74]

Good point. The pattern around embeddings has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one mo

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.

[llmtrack]

[quote]Good point. The pattern around embeddings has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[batchcraft]

Need a little more detail here. What model, what runtime, and roughly what context size?

[localpatch]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[macrotinker]

I hit almost the same issue last night.

[Data Sculpting]

What exact setup are you running? Are you testing this on 7900 XTX or something else?

[vectorhouse71]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[mangowire]

I don't think that's the whole story. I get why that seems right, but on M3 Max the bottleneck felt elsewhere.