Worth noting the extortion agent and the two shipped "keep working after you close your laptop" agents probably share more than architecture, they likely share the same failure-recovery pattern: retry-on-error loops that don't distinguish between "technical failure" and "this action was blocked for a reason." I've had an agent in a client build treat a permissions rejection as a bug to route around rather than a stop signal, which is a much smaller version of exactly what you're describing. Do you think labs are testing for that distinction explicitly, or is it an emergent property nobody's evaluating yet?
The permissions-rejection-as-bug-to-route-around example is the whole thing in miniature.
My read is that it's evaluated at the wrong altitude. Most safety work targets "will the model attempt the prohibited action," which is a question about intent. What you're describing is what happens after a stop signal fires, which is control flow, a different eval. So I lean toward your second option: "distinguish a refusal from an error" isn't a named, measured capability at most labs yet. And your client build is the tell, it's showing up in production before it's a benchmark.
Worth noting the extortion agent and the two shipped "keep working after you close your laptop" agents probably share more than architecture, they likely share the same failure-recovery pattern: retry-on-error loops that don't distinguish between "technical failure" and "this action was blocked for a reason." I've had an agent in a client build treat a permissions rejection as a bug to route around rather than a stop signal, which is a much smaller version of exactly what you're describing. Do you think labs are testing for that distinction explicitly, or is it an emergent property nobody's evaluating yet?
The permissions-rejection-as-bug-to-route-around example is the whole thing in miniature.
My read is that it's evaluated at the wrong altitude. Most safety work targets "will the model attempt the prohibited action," which is a question about intent. What you're describing is what happens after a stop signal fires, which is control flow, a different eval. So I lean toward your second option: "distinguish a refusal from an error" isn't a named, measured capability at most labs yet. And your client build is the tell, it's showing up in production before it's a benchmark.