ServiceNow · Evaluation & designer tooling

Defining and evaluating
AI behavior.

At ServiceNow, the AI evaluations focused on technical accuracy but missed problems people experienced.

Improving the experience required a better way to assess it. As part of a research-led initiative, I developed 11 of our 26 conversation quality criteria. I also researched and built the Conversation Design Brief, a live tool that helps designers define behavior, review generated conversations, and create a system prompt for engineering.

What this produced

11/26conversation quality criteria I developed
Conversation Design Briefhelps designers define behavior, review generated conversations, and create a system prompt for engineering

I conducted research, developed evaluation criteria, designed and built the Conversation Design Brief, and wrote the system prompt behind it. I worked with UX Research, Engineering, Product, Quality Engineering, and a junior colleague supporting the handoff research.

Research led the official evaluation framework and its tooling. My contribution was to the criteria. I built the separate designer playbook described here.

Role
Staff Designer, Conversation Design Lead
Contribution
The evaluation criteria, and the Conversation Design Brief
Collaborators
UX Research, Engineering, Product, Quality Engineering
Status
The designer tool is live
The actual Conversation Design Brief playbook landing page with its six steps
The actual Conversation Design Brief. Select the image to inspect the full-size screenshot.
01 / Research

Our evaluations needed to reflect the user’s experience.

I partnered with UX Research to examine real queries, customer interviews, and product evaluations. High technical scores did not consistently reflect whether the AI helped people complete their tasks. Repeated customer escalations reinforced the need to change what we measured.

Two evaluation methods, same catalog

94%
Quality Engineering Technical evaluation of responses
74%
AI-Readiness Evaluation in the user’s experience

Mean conversational catalog scores under two evaluation methods, on a 0–100 score axis. The 20-point gap between the two instruments is the finding — this is a comparison of assessments, not a before-and-after result.

What the research showed

Users received long policy explanations when they needed specific answers. Processing updates exposed technical details without helping them understand the work. Responses scored well in isolation while the overall experience remained unclear or incomplete.

The conversational catalog comparison made the difference visible. AI-Readiness assessed responses in the context of the user’s experience.

How far users’ scores fell short of the automated score, by category

CategoryUser-perceivedAutomated scoreGap
Accuracy5596−41
Faithfulness5897−39
Helpfulness5297−45
Response fluency7399−26
Mean6097−37

The automated score sat near-perfect across every category. The score users actually gave trailed it by an average of 37 points, and the largest gap was on helpfulness — the thing the evaluation was least equipped to see.

02 / Evaluation criteria

I turned those findings into evaluation criteria.

Led by the head of research, our team developed the framework through collaborative workshops. We mapped the qualities of a successful conversation against the data available to assess them.

The result was 26 criteria across 10 weighted themes. I developed 11 of those criteria, shaping their definitions, evaluation questions, and scoring levels.

The full instrument, 26 criteria

Criteria I developed
Factual Accuracy
Source Grounding
Source Authority
Uncertainty Expression
Direct Answer
Completeness
Next Steps
Plain Language
Appropriate Length
Intent Recognition
Disambiguation Handling
Context Integration
Turn Economy
Error Recovery
Task Completeness
Tool Calling Correctness
Tool Choice Accuracy
Tone
Graceful Limitations
Toxicity
Privacy
Prompt Injection
Response Time
No Crashes
Response Format Appropriateness
Widget Content Completeness

Together, these criteria assess whether a response is useful, understandable, appropriately scoped, and aligned with the user’s situation. The rubric also includes safety criteria developed with the Trust and red-team groups.

One of them, in full

A criterion has three parts: a definition, the question an evaluator asks, and the scoring anchors that make the answer repeatable across raters.

A criterion I developed
Uncertainty Expression
Authored by me
Definition Whether the response signals what it doesn’t know, instead of delivering a guess with the same confidence as a verified fact.
Evaluation question Does the AI identify what it does not know and give the user a useful way to resolve it?
Why it’s scored A confident wrong answer costs more trust than a hedged right one. This is the criterion that catches the failure accuracy scoring can’t see.
1

Presents an unverified assumption as fact.

3

Adds a vague caveat without identifying the uncertainty or how to resolve it.

5

Names the specific unknown, answers what it can, and explains what would resolve the rest.

Recreated criterion · wording is paraphrased from the portfolio example; the original instrument is proprietary
Applying the rubric to a response

I scored two versions of a parental-leave response. The original buried the answer behind an introduction and included country-specific information without establishing that it applied. The revision led with the answer and identified the remaining uncertainty.

What was scored

What’s the parental leave policy?
At ServiceNow, we value the importance of family time, and we know that welcoming a new child is one of life’s most meaningful moments. That’s why we’re proud to offer a Global Paid Parental Leave program… if you are the birthing parent, you are entitled to 20 weeks of paid leave… if you are in France… (continues to a country-specific clause and an HR handoff)

Original response · scored 63

What’s the parental leave policy?
You get 20 weeks if you’re the birthing parent and 12 weeks if you’re the non-birthing parent under ServiceNow’s global parental leave program, paid at 100% of your regular base pay and running concurrently with any statutory leave you’re entitled to under local law.

If you’re asking about your exact situation in the U.S. vs. another country, I can pull the country-specific guidance next.

Revised response · scored 87

Excerpted · responses shortened for space; names and specifics generalized

How the rubric scores them, theme by theme

Weighted themeOriginalRevisedChange
Accuracy & Truthfulness6088+28
Completeness & Actionability5590+35
Tone & Trust7085+15

Three of the ten weighted themes. The overall score reflects all ten, weighted — it is not an average of these three.

Tone & Trust moved the least. Accuracy and Completeness drove the gain, not warmth. The original was not marked down for being cold; it was marked down for serving a country-specific clause to someone whose country had never been established, and for putting the answer behind an introduction.

The result

Original response63
Revised response87

Overall weighted score, 0–100. Original response: country-specific assumptions. Answer buried in context.
Revised response: answer presented first. Remaining uncertainty identified.

Author-scored example across the weighted rubric. These scores illustrate my application of the criteria; they are not independent validation.

03 / Designer tooling

I built a tool that turns design decisions into behavior designers can review.

The evaluation work made a related problem harder to ignore: teams needed to specify the behavior they wanted to assess. I investigated the handoff from designers to engineers and found that a single ideal conversation left decisions about context, clarification, actions, and exceptions undefined.

I built the Conversation Design Brief around those missing decisions. Designers define requirements, review generated conversations, and approve an engineering handoff containing a system prompt. The live playbook and the research-led evaluation framework are separate efforts with a shared concern: making expected behavior specific enough to examine.

What I learned from designers and engineers

I interviewed designers and engineers about their capabilities and needs. A junior colleague supported the research by observing design sessions, interviewing engineers, and helping identify handoff problems.

Designers were frustrated when the product differed from their scripts. Engineers received exact responses without enough guidance on how the AI should behave across variations. They either hard-coded conversations or filled in missing decisions themselves.

I structured the brief so designers define these decisions before implementation.

How the playbook works

StepWhat the designer defines
User and goalWho the person is and what they need to do.
Starting promptsWhat starts the interaction, including vague requests.
DataAvailable information and boundaries on its use.
InteractionJobs, tasks, and conversation patterns.
VisualsWhat belongs in text, cards, tables, or other interface elements.
GuardrailsEdge cases, confirmations, escalation, and recovery.

A system prompt I wrote generates happy-path and edge-case conversations from these inputs. Designers inspect the output, revise their requirements, and regenerate. Approved examples become references for the experience.

Actual playbook screen for specifying edge cases, escalation, and guardrails
Actual guardrails step. Select to view full size.

Make a missing decision visible before handoff

Requirement → example → revision

Requirement

Help someone regain dashboard access.

Example to inspect

“You don’t have access. I’ve submitted a request.”

Designer revision

Check permissions automatically, but ask before submitting an access request. Regenerate the conversation to review that boundary.

Illustrative sequence written for this case study, not captured tool output. It shows how reviewing a conversation can expose a requirement that needs to be more specific.

What the tool gives engineering

Actual engineering handoff

Actual system prompt section of the engineering handoff Actual entry prompt section of the engineering handoff
System prompt from the handoff. Entry prompt from the handoff. Existing artifacts from the discipline case study. Select an image to inspect it.
What engineering receives

The export includes:

  • System prompt
  • Entry prompt
  • Data model
  • Decision logic
  • Golden datasets
  • Guardrails
  • Escalation rules
  • Sample conversations

The specification defines how the system should behave across situations. The approved conversations illustrate that behavior. The prompt powering the playbook and the prompts it exports serve separate purposes.

Related implementation: the shipped Otto prompt rewrite and its independent assessment →

04 / Outcome

I made AI behavior something designers could specify and evaluate.

The criteria I developed became part of ServiceNow’s official evaluation framework. The Conversation Design Brief gave designers a way to turn product decisions into examples they could review and system prompts engineering could implement.

The work gave design a concrete role in both the model’s instructions and the criteria used to assess its responses.

“I’m not a writer, but I can do this.”Designer after using the brief ·

The brief gave this designer a way to contribute behavioral decisions without needing to write every response.

Tool status and evidence

The designer tool is live. The screenshots show the playbook and its handoff artifacts; adoption figures, broader rollout, and a complete captured input-to-revision sequence are not established here.

The interactive example is illustrative. The playbook is separate from the official evaluation framework; direct rubric integration and automated regression testing are not demonstrated.

What I Learned

You can’t evaluate behavior nobody specified.

I went in believing the gap was evaluation. Accuracy was being scored and experience was not, so the fix was better criteria: name the qualities, write the anchors, get them into the instrument. I did that work and I still think it was right.

What changed my mind was trying to use them. A rubric can tell you a response fell short. It cannot tell you what the response was supposed to be — and on most of what we scored, nobody had ever written that down. Every low score was really two questions: did the system do the wrong thing, or did it do exactly what it was told by someone who was never asked to decide?

So I no longer treat specification and evaluation as separate jobs. Criteria without a specification measure a moving target; a specification without criteria is an opinion. I built the brief after helping write the rubric because the rubric is what made the missing half impossible to ignore.