Originally published on LinkedIn on March 28, 2026.

There’s a conversation happening in AI about how to measure whether an AI agent actually helped someone. New frameworks, scoring systems, startups — Microsoft just introduced a Multimodal Agent Score (Dynamics 365 Blog, Feb 2026) to holistically evaluate AI agents across understanding, reasoning, and response quality.

It’s a good conversation. But I think it’s a much older question wearing a new outfit.

Every product team has wrestled with the same thing: did what we built actually work for the person using it? And the answer has always come in layers.

The Measurement Stack

Level 1 is solved. Level 2 is where most teams live. Level 3 is where the eval industry is printing money — LangSmith, Braintrust, Arize. But all of that still tells you if the agent did the right thing. Not whether the person felt served.

Why AI agents break the old model

In traditional products, Level 2 was a good enough proxy for Level 4. Clicks, conversions, repeat visits — reasonable signals because the interaction was structured. A funnel. Buttons. Predictable paths.

AI agents blow that up. Conversations are unstructured, variable, and don’t map to funnels. A 15-turn chat that looks successful might have left someone confused. A 2-turn conversation that looks like abandonment might have given them exactly what they needed. There’s no “like” button in a chat.

McKinsey’s “Trust in the Age of Agents” research (2025) found that 80% of organizations have already encountered risky behavior from AI agents. Their framing: “Agency isn’t a feature; it’s a transfer of decision rights.” When you transfer decision rights, you need to know if that’s working for the person — not just the system.

Not every product needs Level 4

Facebook’s newsfeed? Level 2 is enough. Low-stakes, high-volume. Trillion-dollar business on likes and time-spent. A resolution KPI for whether each post made you happy would be overkill.

And there’s academic evidence for why this is fine. Keiningham et al. published in the Journal of Marketing (2007) showing that changes in Net Promoter Score have almost no relationship with how customers allocate spending. A broader review by Dawes in the International Journal of Market Research (2024) confirmed: NPS is not a reliably superior predictor of growth compared with other satisfaction metrics. Behavioral proxies often tell you more than asking people how they feel.

For high-volume, low-stakes products — Level 2 isn’t a consolation prize. It’s the right answer.

Where Level 4 is genuinely needed

It comes down to stakes and trust. Harvard Business Review Analytic Services surveyed 603 business leaders (published Dec 2025 via Fortune) and found only 6% of companies fully trust AI agents to handle core business processes. 43% limit them to routine tasks only.

Financial advisory. An AI copilot drafts a client proposal. The advisor accepts it. But did they send it as-is, or silently rewrite it? A 2025 World Economic Forum / Capgemini report found 93% of financial advisors want final say over AI outputs. The question isn’t “was the proposal accurate” — it’s “did the tool build or erode trust?”

Healthcare. An AI suggests a treatment path. The doctor follows it. But were they confident, or busy and planning to second-guess later? A 2024 study in Frontiers in Digital Health found that even when clinical AI is accurate, adoption stalls without perceived trustworthiness. Accuracy doesn’t drive adoption. Confidence does.

Enterprise renewals. The contract went through, terms accurate, stakeholders notified. But does the customer feel like a partner or a line item? That feeling — not the contract accuracy — determines next year’s renewal.

Legal. An AI flags contract risks. The lawyer reads the output. Do they trust it enough to stop there? If they’re redoing the review, the AI didn’t resolve anything. It added a step.

The ice cream shop. No dashboard. No evals. The owner watches the kid’s face. Did the kid smile? Are they tugging their parent’s arm saying “can we come back tomorrow?” That’s the purest Smile Signal — unmeasurable, and yet the most powerful retention signal there is.

How would you actually measure Level 4?

Nobody has cracked this. But I think it’s a composite — three signals, weighted by domain:

Conversational signals (real-time, weakest). Corrections, rephrasing, sentiment shifts. Useful but limited. The biggest blind spot: silent failure — the person who gets a bad answer and just quietly leaves.

Downstream behavior (lagging, strongest). Did the user come back? Did the advisor send the proposal without rewriting? Did the doctor follow the suggestion on the next patient too? Closest to ground truth, but requires instrumenting beyond the conversation.

Contextual micro-feedback (intermittent, calibrating). Not NPS. Matt Dixon, Nick Toman, and Karen Freeman made the case in “Stop Trying to Delight Your Customers” (HBR, July 2010) and later in The Effortless Experience — delight has negligible impact on loyalty; reducing effort matters far more. Gartner’s CES research puts a number on it: high-effort experiences make customers 96% more likely to become disloyal. Low-friction, in-the-moment feedback — a thumbs up, a “this wasn’t helpful” button — is better than any post-interaction survey

My take

For low-stakes, high-volume products — don’t over-engineer this. Level 2 works. It’s always worked.

For high-stakes, trust-dependent domains — finance, healthcare, legal, enterprise — Level 4 isn’t optional. It’s where competitive advantage lives.

The companies that figure out how to blend these signals — and know which ones to weight for their domain — will build the Smile Signal of agentic products. Except hopefully one that actually correlates with what it claims to measure.

Sources

• Dixon, Toman, DeLisi — “Stop Trying to Delight Your Customers,” Harvard Business Review, July 2010

• Keiningham, Cooil, Andreassen, Aksoy — “A Longitudinal Examination of Net Promoter and Firm Revenue Growth,” Journal of Marketing, 2007

• Dawes — “The Net Promoter Score: What Should Managers Know?” International Journal of Market Research, 2024

• McKinsey — “Trust in the Age of Agents,” 2025

• Harvard Business Review Analytic Services — AI Agent Trust Survey, Dec 2025 (via Fortune)

• World Economic Forum / Capgemini — AI in Wealth Management, 2025

• Frontiers in Digital Health — Trust in Clinical AI Systems, 2024

• Gartner — Customer Effort Score Research; Agentic AI Predictions, 2025

• Microsoft Dynamics 365 Blog — “Multimodal Agent Score,” February 2026

Where have you seen the gap between “the system worked” and “the person felt served”

?

For me it was my college WhatsApp group. Beyond the usual socialization, alumni meets, sports and politics — a friend posted asking for help with AI. She volunteers for a healthcare NGO in West Bengal and her team was getting inundated with over 25,000 Bengali patient feedback calls per month.

My friends suggested ElevenLabs, Twilio, voice agents — all good suggestions, but the sad reality was this wasn’t a technical user group, and an NGO can’t afford any of that.

I got intrigued and asked to be connected.

The NGO was trying to push Bengali voice recordings into Gemini for translation, but it was causing more grief than helping. When I actually spoke to them I realized translation was just one problem, not the only problem. The other issue was actionable clinical intelligence. One exacerbated the other.

Nobody knew which clinics were underperforming. Nobody knew that chronic condition patients — diabetics, hypertensives, people on maintenance meds — are the ones who actually need follow-up calls, not everyone. The team was calling all patients equally and couldn’t understand why it wasn’t working.

So I built something. Pro bono, using Claude Code.

Before picking a transcription engine I ran a benchmark — four engines, same 50 real calls, rural West Bengal Bengali recorded on basic phones. Scored across five criteria: completeness, coherence, proper noun accuracy, medical content accuracy, and handling of noise and dropped calls.

Same audio. Very different results.

I expected Whisper V3 to win. It didn’t. Sarvam AI’s Saaras V3 was the clear winner — and the nuance was that it actually understood rural dialects, not just standard Bengali. That gap matters when clinical data is in the mix. A misheard medication name or a mangled outcome is a wrong decision downstream

The pipeline transcribes each call, extracts structured fields — recovery status, medication dropout, access barriers, whether the patient sought treatment elsewhere, whether they're on a maintenance condition — and surfaces everything by clinic in a dashboard. Patterns across thousands of calls, not summaries read one at a time.

Expected outcomes: 13pp reduction in medication dropout, 85% improvement in clinical escalation time, 2-3x increase in access barrier identification.

Open source, self-hostable, bring your own API. Any health NGO can deploy this on nonprofit cloud credits.

Still in progress but working.