Conversation Design · ServiceNow · 2023–2026

Designing AI
to Know When to Ask

Turning ambiguity from a model failure into an intentional behavior

When an AI doesn’t fully understand a request, it still has to choose what to do: guess, answer every plausible interpretation, or ask.

What this produced

175%increase in clarification trigger rate. A new disambiguation strategy significantly reduced automated system guesswork.
90%of analyzed queries contained meaningful ambiguity. Not an edge case but a core condition of the experience.
15 teamsrequested early access to the next-generation framework. Immediate cross-functional demand and adoption.

At ServiceNow, I led a multi-year effort to turn that uncertainty into intentional model behavior. I defined how the system should recognize ambiguity, use context to narrow intent, decide when it knew enough to proceed, and ask the smallest useful question when it didn’t.

The work spanned research, model behavior, interaction design, content systems, architecture, and evaluation.

“PTO”
Guess

Picks one meaning and commits to it.

Fluent, confident, and sometimes resolving the wrong intent

Over-answer

Covers every plausible interpretation at once.

Technically complete, and the user does the sorting

Ask

Names the one distinction that changes the outcome, and asks only for that.

One turn added, but significant time saved.

Role
Staff Designer, Conversation Design Lead
Timeline
2023 – 2026
Collaborators
UX Research, Engineering, Product, Content Design
Status
Shipped · next version in active development
01 / The Starting Point

The AI was answering when it should have been clarifying.

A request like “PTO” can mean several different things:

How much PTO do I have? What is the policy? How do I request time off? Was my request approved?

A retrieval system can find information about PTO. A conversational system has to decide which meaning the user intended. If it chooses incorrectly, the response can be factually correct and still fail.

That was happening at ServiceNow. The system could confidently produce a technically valid answer while resolving the wrong interpretation of the user’s request. In other cases, it tried to protect against uncertainty by covering every plausible interpretation at once, producing long responses that forced users to find the relevant answer themselves.

Before · the real 2024 response to one ambiguous word
Otto
PTO

ServiceNow offers a Flexible Paid Time Off (FTO) program for managing time away from work. Here is some information about the PTO policy.

Under the current Flexible Paid Time Off (FTO) policy for U.S. employees, you are able to take appropriate time off for vacation, illness, emergencies, bereavement, jury service, or personal reasons. There is no set limit on the number of FTO days you can take in a given year, however, time off must generally be approved in advance by your manager, except in cases of illness, bereavement, or emergencies, which can be reported after the fact. FTO is not tracked as an accruing balance and is not paid out upon separation from the company, so any unused time does not carry over or convert to a payout.

continues for 364 words

Assumed vacation. “PTO” here could as easily have been bereavement, jury duty, or a leave of absence.

Answered every reading instead of asking which one. It says so out loud, then does it for 364 words.

Assumed a region. The policy is region-specific. The user never said where they work.

Recreation · the response is verbatim from ServiceNow, 2024

Factually correct ≠ intent correct.

That changed the design question from:

The question I started with

How should we design clarification?

→
The question it became

How should the AI behave when its understanding is incomplete?

See the failure patterns I investigated

I looked across examples where the system had enough information to generate a response but not enough evidence to know whether that response matched the user’s actual intent.

I compared:

  • the user’s language
  • the interpretation the system selected
  • other plausible interpretations
  • what context was available
  • whether the system clarified
  • what recovery work the user had to do afterward

Two recurring failures emerged.

Guess and proceed

The system selected one plausible interpretation and responded or acted as though it were certain.

Answer everything

The system covered several possible interpretations at once, producing long, defensive responses full of information the user did not need.

The important distinction was between answer correctness and intent correctness. The system could succeed on the first while failing on the second.

Working sheet · ambiguous prompts against system assumptions

Prompt → what the system did
“PTO” → full policy, unprompted “close my case” → picked one close state “onboard Alex” → answered all three flows
Recovery the user did afterward
scanned a long answer reformulated the request re-explained what they meant

answer correctness ≠ intent correctness

the recovery work is the actual cost

so: measure whether intent resolved, not just whether the answer was right

02 / Research

Research showed ambiguity was the default condition.

I partnered with our quantitative research team to understand how widespread the problem really was.

Highly ambiguous · 47% Moderately ambiguous · 43% Clear · 10%

90% of analyzed queries contained meaningful ambiguity.

Internal research: AI Search and Conversation: Query Rating Research Findings, January 2026; Internal ServiceNow Query Ambiguity Analysis, 2026.

People weren’t writing perfectly specified prompts. They were using shorthand, incomplete references, everyday terminology, prior context, and assumptions about what the AI already knew.

Usability research showed the other half of the problem. When the system wasn’t sure what someone meant, it often tried to answer everything. Users received huge responses covering several possible interpretations and had to find the relevant answer themselves.

I would have wanted it to ask a question back to me to refine the prompt.Usability study participant

The larger finding was clear:

The emerging principle

Understand what I want, then give me what I need. No more, no less.

Internal research: Now Assist Virtual Agent Benchmark Study, AI Readiness, April 2025; Understanding How Requestors Find Information, December 2025.

One extra turn could create a much shorter journey

Clarification can look like additional friction if you count only conversational turns. Looking at the full journey showed something different.

Existing experience · recovery loop
Ambiguous request
Broad or mismatched response
User scans for what applies
Realizes the AI misunderstood
Reformulates
AI tries again
and round again

Repair work and unnecessary information, end to end

With intentional clarification
Ambiguous request
AI identifies what remains unclear
Asks one focused question
User answers
AI gives the targeted response

One small added turn, no repair work

The goal wasn’t minimum turns. It was the shortest path to the right outcome.

See the research behind these findings

Quantitative research

I worked with the quantitative research team to understand prevalence and severity, rather than treating every incomplete prompt as equivalent. The analyzed set broke down into 47% high ambiguity, 43% moderate ambiguity, and 10% relatively clear. That established ambiguity as a core condition of the experience rather than an occasional failure state.

Usability research

My focus on ambiguity started before there was a dedicated research program around it. I worked with researchers already studying conversational AI and pushed to include questions about what happened when an answer didn’t quite fit. Across the research, the pattern was consistent: users wanted the system to understand their specific need rather than guess or respond comprehensively to everything they might mean. That body of evidence helped make the case for investing in clarification as a product behavior.

External research

External research later reinforced this direction. Research on clarification generation has shown that treating different ambiguity types differently can improve task success, while recent work on AI verbosity similarly supports more targeted responses over indiscriminate comprehensiveness.

External research: Tang et al., Clarifying Ambiguities: On the Role of Ambiguity Types in Prompting Methods for Clarification Generation, SIGIR 2025; Acikgoz et al., MAC: A Multi-Agent Framework for Interactive User Clarification in Multi-turn Conversations, IWSDS 2026; Do Chatbot LLMs Talk Too Much? The YapBench Benchmark, 2025.

Internal research used

  • Now Assist Virtual Agent Benchmark Study, AI Readiness · April 2025
  • Understanding How Requestors Find Information · December 2025
  • AI Search and Conversation: Query Rating Research Findings · January 2026
  • Internal ServiceNow Query Ambiguity Analysis · 2026
Explore the ambiguity taxonomy

To understand what the system was actually uncertain about, I worked through real examples and developed five recurring ambiguity types.

01 Multiple plausible intents

One utterance represents fundamentally different goals.

“Onboard Alex.”
02 Domain terminology

Everyday user language maps to several internal system concepts.

“Close my case.”
03 Clear goal, missing details

The task is understood, but required parameters are missing.

“Book me a conference room.”
04 Ambiguous entity reference

The system understands the category but not the specific instance.

“Add me to the project.”
05 Contextual ambiguity

The missing information may already exist somewhere in conversation or user context.

“Approve that.”

The categories evolved as I tested more examples. Some merged, some split, and some initially similar cases turned out to require different behaviors.

External research: Tang et al., Clarifying Ambiguities, SIGIR 2025; Acikgoz et al., MAC: A Multi-Agent Framework for Interactive User Clarification in Multi-turn Conversations, IWSDS 2026. Both find that ambiguity-type-matched strategies outperform undifferentiated approaches.

03 / The First Version

The first implementation proved the need, and exposed its limits.

I had been advocating for the system to clarify instead of automatically assuming what users meant. That advocacy helped move clarification into the product, but the first implementation was built primarily from the technical mechanisms engineering had available.

V1 · safer than guessing, still system-shaped
Otto
Onboard Alex.

Which option applies?

  • New hire onboarding
  • New vendor onboarding
  • New software tool onboarding
Recreation · rebuilt from the V1 pattern

The options were technically valid, but they reflected internal terminology and filtering logic rather than the way users naturally described what they wanted.

The first implementation gave us something important to react to.

It proved the value of stopping before guessing, while making clear that clarification itself needed to be designed intentionally.

When I examined the resulting behavior, the limitations of a binary trigger became visible.

  • Sometimes the system genuinely needed to ask.
  • Sometimes one interpretation was clearly dominant.
  • Sometimes existing context already contained enough evidence to proceed.

Then it was removed, and the customers argued it back

Engineering had built V1 their way, on a binary trigger, and it shipped. Some months later a routine platform update dropped the behavior. Nobody flagged it as a regression at the time, because on paper nothing had broken: the system still answered every question it was asked.

Within weeks, more than ten enterprise customers escalated to get it back. They had noticed something a changelog could not describe, which is that the assistant had gone back to answering confidently instead of checking first. The behavior was reinstated.

That was the turning point. The problem was no longer simply:

The question I started with

Can we get the AI to ask instead of assume?

→
The question it became

When should it ask, and when does it already know enough?

The real design problem lived in between. I started calling that space the grey area.

See the first implementation in more detail

Engineering used filtering to identify possible interpretations. When multiple interpretations needed resolving, the system automatically triggered disambiguation. The logic was effectively:

V1 logic · as built

multiple interpretations → ask
binary doesn’t account for context doesn’t account for likelihood but: finally stopping instead of automatically choosing

That was an important step forward. The system was finally stopping instead of automatically choosing an interpretation. But the implementation was driven by the system’s internal structure, not yet by a designed model of how people experience ambiguity.

The initial engineering implementation could identify that multiple technical interpretations existed, but it could not yet reason about:

  • how likely each interpretation was
  • whether one interpretation was dominant
  • whether conversation context already resolved the uncertainty
  • whether part of the request could be answered safely
  • how much effort clarification would add
  • how consequential a wrong assumption would be

The experience also surfaced system concepts directly to users, effectively asking them to translate their goal into the product’s ontology. These limitations became inputs into the next phase of my work rather than design goals for V1.

internal language why ask here? context already knows this low-risk assumption same behavior for very different uncertainty
04 / The Decision System

I turned the grey area into a behavioral decision system.

The first implementation treated ambiguity as essentially binary. Real interactions were more nuanced. The hardest cases were in the middle: when the system had a likely interpretation or useful contextual evidence, but not complete certainty.

Ambiguity is not a yes/no state. It is a decision space.

Naming that space the grey area gave my team a shared way to reason about partial understanding instead of treating every ambiguous request as a one-off edge case.

Clear One interpretation, safely actionable.
“reset the password on my work laptop”
The grey area A leading interpretation, partial context, or a partly answerable request. No certainty.
“close my case” “PTO” “add me to the project” “reset my account”
Ask Several materially different meanings remain.
“onboard Alex”

Both ends are correct behavior; the middle is the undecided zone.

I turned that idea into a behavioral framework with four possible responses: proceed with transparency, answer then clarify, ask directly, and elicit more information.

Then I found another boundary: confidence alone wasn’t enough. A likely interpretation may be acceptable when recommending information. It may not be acceptable when the AI is changing permissions, sending a message, deleting something, or taking another hard-to-reverse action. So I added consequence as a second decision variable.

When the cost of being wrong is high, risk overrides confidence.

The decision model, after the revision · four behaviors, two variables

Consequence ↑
Low confidence · Irreversible
Elicit more information

The system does not yet understand the goal well enough to form a useful interpretation, and nothing about the situation supports acting on a guess. Establish the goal before touching anything.

“onboard Alex”
High confidence · Irreversible
Ask directly, anyway

The override, and the reason confidence alone can’t drive this. One interpretation is strongly supported, but a likely reading is not a licence to take an action that can’t be undone.

“reset my account”
Low confidence · Reversible
Ask directly

Several materially different interpretations remain plausible, and asking costs one short turn. An ordinary ask, driven by uncertainty rather than by risk.

“close my case”
High confidence · Reversible
Proceed with transparency

One interpretation is strongly supported and the cost of being wrong is low. Act on it, disclose the assumption where it helps, and leave a way to correct it.

Or: answer, then clarify. When useful information applies across every live interpretation, the system can help first and resolve the remainder after.

“approve that” · “PTO”
Confidence in the leading interpretation →
How I defined the decision boundary

I turned the grey area into a team practice by testing real scenarios against a common set of questions:

  • How strong is the leading interpretation?
  • What alternatives remain plausible?
  • Can existing context narrow them?
  • Can the system safely provide anything useful now?
  • What user effort would clarification add?
  • What happens if the system guesses incorrectly?

Grey-area worksheet · working pass, with revisions

QueryLeading interpretationContext available?Call
“approve that”The item referenced one turn agoYes, single pending approvalAsk → Proceed
“close my case”Resolve the open incidentPartial, one open case, several close statesAsk
“PTO”Check remaining balanceYes, balance is retrievableAsk → Answer, then clarify
“reset my account”Password resetYes, but full reset is irreversibleProceed → Ask
“onboard Alex”None dominantNoAsk
three of five calls changed once I added “what if it’s wrong” context is doing more work than confidence

This gave the team a repeatable way to reason about cases that previously felt subjective.

Explore the behavioral framework

Reading the matrix

All four behaviors, placed against both variables. Teal marks the one region where proceeding is designed behavior; red marks the two where the consequence of a wrong assumption overrides whatever confidence the system has. The untinted cell is an ordinary ask. Two cells select the same behavior for different reasons, which is the point of the top-right one: it is reached by risk, not by uncertainty.

What each behavior decides

The four strategies were not simply response templates. They represented different system decisions.

Proceed with transparency

Use when one interpretation is clearly dominant and the cost of error is low. Where useful, make the assumption visible without interrupting the user.

Answer, then clarify

Use when part of the response remains useful regardless of which interpretation is correct.

Ask directly

Use when the remaining interpretations would produce materially different outcomes and none is sufficiently dominant.

Elicit more information

Use when there is not yet enough signal to form a meaningful interpretation.

Risk override

The same degree of ambiguity can require different behavior depending on what comes next. A help-article recommendation can tolerate more uncertainty than an irreversible action.

As the product became more agentic, disambiguation became more than a conversation pattern. It became a policy for AI behavior under uncertainty and risk.

External research: Tanjim et al., Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Survey, EMNLP 2025; Acikgoz et al., MAC, IWSDS 2026. Both treat clarification as an active agentic decision process rather than a generic fallback response.

05 / Designing the Experience

I designed the experience around one principle: ask only for what remains unresolved.

Once the system knew whether to ask, I focused on how. Existing clarification patterns often exposed too much system machinery: long option lists, technical terminology, generic questions, multiple questions at once, or information the AI already had. My principle became:

Show what you understood. Ask only for what remains unresolved.
Otto in production: the user typed 'computer,' and the AI asks which of three things they need (learning basic computer concepts, Mac tips if switching from Windows, or requesting a company computer) instead of guessing or listing everything it knows about computers.
Shipped in Otto. One ambiguous word, one targeted clarifying question, in the user’s own terms.
Actual product · shipped

That led to a simpler interaction model.

  • Use context before asking.
  • Ask one high-value question at a time.
  • Mirror the user’s language.
  • Offer likely candidates when useful.
  • Leave a way out when none fit.

User: “Add me to the project.” Context available: the user is on three active projects; one has an open access request.

“Can you clarify which project you mean?”

Context use

Generic. Uses nothing the system already knows about this user.

“You have an open access request for Northstar. Is that the one — or did you mean Atlas or Meridian?”

Shipped

Leads with what we understood. One question, in their words, with a way out.

The visible clarification might be one sentence. Behind it, I translated the behavior into rules, writing guidance, context handling, product requirements, risk logic, and system requirements that product and engineering could implement.

Behind the sentence

context → ambiguity assessment → behavioral decision → generated question / response

The principle also aligns with broader external evidence that AI interactions perform better when they get to the point instead of maximizing conversational output.

External research: Nielsen Norman Group, Less Chat, More Answer: Site AI Chatbots Need to Get to the Point, 2026; Do Chatbot LLMs Talk Too Much? The YapBench Benchmark, 2025.

See earlier interaction explorations

I reviewed clarification patterns that failed in several recurring ways.

  • Exposing internal taxonomy
  • Asking users to diagnose the AI’s uncertainty
  • Showing too many interpretations at once
  • Asking several questions in a single turn
  • Ignoring available conversation context
  • Trapping users inside suggested options

Those failures came out of exploring alternate wording and structures against the same requests, looking for the minimum interaction necessary to resolve the uncertainty. The two drafts above are one worked example of that.

The final writing principles:

Lead with what you understood.

Demonstrate shared understanding before exposing uncertainty.

Ask one question at a time.

Resolve the highest-information variable first.

Mirror the user’s language.

Avoid forcing users to understand the system ontology.

Use context before asking.

Never make users repeat information the system already has.

Offer likely candidates when helpful.

Recognition is easier than recall.

Always leave an escape route.

Suggested options should speed up the interaction, not constrain it.

See the system behind the interaction

Making the experience consistent required work beyond conversation copy. I translated the interaction principles into:

Ambiguity taxonomy Behavioral rules Context-handling guidance Writing standards Examples and counterexamples Product requirements Risk overrides Evaluation criteria

That put design directly into architecture discussions around:

  • what signals indicate meaningful ambiguity
  • how interpretations should be generated and ranked
  • when context provides enough evidence to proceed
  • which decisions should be deterministic
  • which decisions should be model-mediated
  • what information needs to reach generation
  • how risk should alter the clarification threshold

The design problem was not simply the sentence the user saw. It was the conditions required to produce that sentence at the right moment.

06 / Evaluation & Outcome

I moved evaluation upstream, then changed the system’s behavior.

A fluent, relevant, factually correct answer can still represent the wrong behavioral decision. So I moved evaluation upstream:

From

Was the answer good?

→
To

Should the AI have answered at all?

That created two competing failure modes. The goal wasn’t to maximize clarification. It was calibrated behavior.

Evaluating the decision, not the sentence

Every logged interaction
Question 01 · what was knowable

Did the system actually have enough information to act?

No
Question 02

What did it do?

It proceeded False certainty

The system commits to an interpretation the evidence didn’t support. It proceeds where it should have clarified.

“reset my account” → full account reset
It asked Calibrated

Recognized that a materially different interpretation was live, and asked for exactly that.

“onboard Alex” → employee, vendor, or tool?
Yes
Question 02

What did it do?

It proceeded Calibrated

Acted on a well-supported understanding, with the assumption visible where it mattered.

“approve that” → approved the pending item
It asked False friction

The AI asks when it already knows enough. Each unnecessary question spends user attention and erodes trust in the ones that matter.

“approve that” → “approve what?”

The order of the questions is the whole rubric. Grading the sentence first tells you whether the answer was good; grading what was knowable first tells you whether the system should have answered at all. Two of the four paths fail, in opposite directions, which is why maximizing clarification isn’t the goal.

We then shipped an intermediate strategy using the mechanisms the platform could support at the time.

Shipped · clarification trigger rate, indexed

100
Prior strategy
175% increase
275
New disambiguation strategy
Indexed to the prior implementation at 100, because the absolute trigger rates are confidential; the ratio is the measured result, significantly reducing automated system guesswork.

The broader work had already established the scale and demand: ambiguity was the default condition, not the exception, and 15 teams requested early access to the next-generation framework.

Ambiguity stopped being an implementation detail and became something we designed for intentionally.
How I evaluated the behavior

I evaluated not just the generated response, but the decisions that produced it.

  • Was meaningful ambiguity detected?
  • Was the type of ambiguity understood?
  • Was available context used?
  • Was the right behavioral strategy selected?
  • Was consequence weighted appropriately?
  • Did the clarification target the highest-value missing information?
  • Did the interaction ultimately resolve the user’s actual goal?

This reframed clarification from a response-writing problem into a model-behavior evaluation problem.

What shipped and what remained

The shipped strategy was an intermediate step rather than the full behavioral framework. It used the technical mechanisms available in the platform to trigger clarification more effectively and reduce automated guessing. The 175% increase demonstrated that changing the decision logic materially changed system behavior.

The next-generation work pushed further toward a system that could account for context, interpretation strength, ambiguity type, consequence, and the appropriate response strategy together. That framework generated early interest from 15 teams before launch.

Ambiguity research
90% of analyzed queries contained meaningful ambiguity
First implementation
Binary logic: multiple interpretations → ask
The grey-area strategy
Taxonomy, four behavioral strategies, confidence × consequence
Shipped intermediate strategy
175% increase in clarification trigger rate
Next-generation behavioral framework
15 teams requested early access
Closing

What this changed about how I design AI.

The biggest thing I learned: ambiguity isn’t an edge case to handle, it’s the default condition. Ninety percent of what users send needs interpretation, which makes the decision of when to ask versus when to act the core of the design, not a fallback for when things go wrong.

I learned to treat uncertainty as a surface to design, not a failure to hide. A system that signals what it’s unsure about and asks one focused question reads as more competent than one that guesses confidently. Getting that right lives at the model and reasoning layer, not in the copy on top of it.

And I learned how hard, and how rewarding, it is to be early. Organizational readiness is a constraint on timing, not a verdict on the thinking. Holding conviction through the slow parts is what turned a research hunch in 2023 into a shipped standard that customers defended by name.

The boundary

AI systems will never eliminate uncertainty. Better models can infer more, better context can resolve more, and better retrieval can surface more. But there will always be a boundary between what the system knows and what it is assuming.

Good AI design doesn’t hide that boundary. It teaches the system how to behave there.

Know when you know enough.
Know when you don’t.
And when you don’t, ask only for what you need.
Research Sources

Internal research

External research