How to tell whether an AI initiative is working

Most AI initiatives get measured by the thing that's easiest to count: usage. Logins, queries, documents processed, seats activated. The dashboard goes up and to the right, everyone relaxes, and nobody notices that none of it is the point. Usage tells you people are touching the tool. It says nothing about whether the business is better off.

The discipline is to tie each initiative, before it launches, to the operating metric it was meant to move — cycle time, error rate, deals screened per analyst, cost per file, whatever the real outcome is — and then to decide, up front, what result would justify continuing. Not the hoped-for result. The threshold. The number below which you'd stop.

That second half is the one people avoid, and it's the tell. If you can't name the outcome that would make you kill an initiative, you're not measuring it — you're sponsoring it. A metric with no kill criterion is a mascot: it exists to make the project feel accountable while protecting it from ever being judged.

Naming the kill criterion up front does two things. It forces honesty about what “working” means before anyone's ego is attached to the answer. And it makes stopping a normal, pre-agreed outcome instead of an admission of failure — which is the only way organizations ever actually stop anything. The initiatives that quietly eat budget for two years are the ones that were never given a condition under which they'd end.

So the test is short. What operating metric was this supposed to move? By how much, by when? And what result would tell us to stop? If the initiative can't answer all three, the problem isn't the AI. It's that no one decided in advance what success or failure looked like — and a project that can't fail can't really succeed either.

Measure the outcome, not the activity. And write down the number that would make you walk away, while walking away is still cheap.