“What’s the parental leave policy?” Here are two answers to the same question.
The first isn’t unfriendly. It’s warm, even. But it doesn’t know who’s asking, so it hands a France-specific clause to someone who might work in Ohio, then routes them to HR for the specifics it should have handled itself. The second reads cooler on the surface. It’s also the one that actually helps: it leads with the answer, uses what the system can reasonably know, and names the one real uncertainty instead of guessing at it.
Why warmth fails
When a message feels off, the instinct is to reach for tone: friendlier, softer, more empathetic. But the parental-leave “before” above was already warm, and it failed anyway. Its problems were never tonal. It didn’t use what it knew about the person asking, and instead of carrying them forward it pushed them off to HR. No amount of friendlier phrasing fixes that.
Anthropic’s 2025 research on 80,000 of their users across 159 countries found that unreliability was the single most common concern people raised about AI, named more often than job loss or privacy. Confidence without competence doesn’t just fail to build trust. It actively taxes it, because every answer now carries a fact-check cost.
That sits on a longer line of mind-perception research separating two things people judge about any agent: its agency, whether it can act, follow through, and get things right, and its experience, whether it can feel (Gray, Gray & Wegner, 2007). Agentic systems get trusted on agency. Warmth targets experience, the axis people are least willing to grant a machine, which is exactly why performed warmth reads as hollow at the moment a system is being asked to act.
What trust is made of
The move is the same on every surface: replace what the system performs with what it demonstrates.
| Dimension | Performed (breaks trust) | Demonstrated (builds trust) |
|---|---|---|
| Tone | “I understand this must be frustrating…” | Direct answer, no cushioning |
| Context | Asking users to re-explain role or history | Using what’s known, without announcing it |
| Memory | Referencing memory as a warmth gesture | Citing what’s remembered and where it came from |
| Control | Assuming permission to act | Full-state confirmation before anything consequential |
| Response structure | Structuring each response from scratch, differently every time | A defined response type that prescribes how it’s built |
I authored all five as behavioral requirements for ServiceNow’s Otto. Two of them, tone and context, have shipped and been measured. That proof, and the concrete rewrite behind it, lives in the case study on making AI behavior a design decision. This piece is about the argument underneath all five.
Don’t perform helpfulness. Provide it.
The clearest example is the simplest one: a system prompt. The one I inherited read:
Adjectives aren’t instructions. They don’t say what to lead with, how much detail to give, or how to handle uncertainty, and “concise” and “sympathetic” often pull in different directions. Left unresolved, that ambiguity shows up as inconsistent output: different structure, length, and tone from one response to the next.
The fix wasn’t picking a nicer tone. It was specifying priority (which trait wins when they conflict), format, and fallback behavior, so the response became a deliberate choice instead of a guess: lead with the answer, don’t open with performative warmth, say uncertainty briefly, close with a next move. The full rewrite, and what it measurably changed, is here.
Knowing someone without overreaching
Research confirms people want AI to know them, but are sensitive to overreach. Using context isn’t the risk. Using it without explaining it is.
“You usually file expenses on Fridays, so I’ve started this week’s draft. Want to review it?”
“I noticed you’re free at 3pm and you’ve been slow to reply lately. Should I clear your afternoon?”
I think about it as three trust boundaries. Context the user gave, or obviously expects the AI to use, is explicit and consensual: safe to act on. Context the AI infers is usable, but must be disclosed, or inference starts to feel like mind-reading. And some context is simply off-limits regardless of whether the system can technically reach it: a hidden profile, cross-app surveillance, comparing someone to their peers.
The failure mode: confidence without basis
“The bot got it wrong” isn’t specific enough to act on. “The bot answered a query it had no basis to answer” is a behavior a designer can specify against, and a materially different problem from a system that simply stopped too early. Naming failures this precisely is what let me argue for abstention (saying “I don’t know” on purpose) as a specified requirement instead of a nice-to-have. Answering an unanswerable question isn’t a UX defect. It’s the system producing confident content with no basis for it, which is the failure mode that actually harms someone acting on the answer.
You can’t write your way into being trusted
Saying “you can rely on me” doesn’t create reliability. Being reliable does. As these products get more agentic, the instinct to make them warmer and more personable can work against them: people extend trust to an agent based on whether it acts and follows through, not on whether it feels like a person, and performed warmth in that gap tends to read as hollow.
So the goal isn’t a warmer AI. It’s an AI that’s honest about what it is and calibrates warmth to what the task actually needs. I wrote more about that question of sounding human in a companion piece, and about the concrete standards this argument produced at ServiceNow in the case study on making AI behavior measurable.
Further reading
- “What 80,000 People Want from AI,” across 159 countries. Anthropic, 2025.
- Gray, H. M., Gray, K., & Wegner, D. M. “Dimensions of Mind Perception.” Science, 2007.
- Research on perceived agency and trust in human-machine interaction. Scientific Reports, 2024.