ServiceNow Virtual Agent Health Check

A Virtual Agent health check is a review of how well the agent handles customer conversations. It asks whether customers are satisfied, whether conversations get resolved and whether escalations to people run smoothly, and it looks for the quiet failures a summary dashboard can miss.

What a Virtual Agent Health Check Covers

Virtual Agent handles customer conversations, resolving issues or escalating to humans. A health check assesses how well your Virtual Agent is working: are customers satisfied? Are conversations being resolved? Are escalations smooth? We review agent performance, conversation quality, and recommendation for improvement.

Conversation Quality Review

We analyze a sample of Virtual Agent conversations. Are responses accurate and helpful? Are customer requests being understood correctly? Are escalations happening when they should? We identify patterns: what kinds of conversations work well for the agent and what kinds need improvement.

Deflection Rate Assessment

Deflection—conversations resolved without human help—is a key metric. We calculate your actual deflection rate and benchmark it against similar organizations. If your deflection is lower than expected, we investigate why and recommend improvements.

Escalation Pattern Analysis

When Virtual Agent escalates to humans, why? Is the customer’s request outside agent scope? Is the agent not confident? Is context missing? We analyze escalation patterns to understand whether issues can be resolved by improving agent training or whether they are genuinely out of scope.

Integration Testing

Virtual Agent integrates with backend systems: databases, APIs, ticketing systems. We test that integrations are working: if agent needs customer data or needs to create a ticket, are integrations functioning? Are timeouts or errors being handled gracefully?

Routing Accuracy

When Virtual Agent escalates, it should route to the right human and the right queue. We verify routing logic is accurate: simple requests go to frontline staff, complex requests go to specialists, urgent requests escalate immediately.

Channel Performance

Virtual Agent might operate on multiple channels: web chat, email, voice, SMS. Performance differs by channel. Voice conversations are faster but more expensive. Email is asynchronous but slow. We review performance by channel and help you allocate resources appropriately.

Customer Satisfaction Measurement

We review customer satisfaction data: how satisfied are customers with Virtual Agent conversations? How satisfied with escalations to humans? Which conversation types have high satisfaction and which have low? This data drives prioritization of improvements.

Training and Improvement Recommendations

Virtual Agent learns from training data. If your training data is sparse or inconsistent for certain categories, agent performance suffers. We identify categories where agent performance is weak and recommend retraining or additional training examples.

Common Virtual Agent Issues

Many organizations overestimate how much Virtual Agent can automate. Trying to use agent on conversations that require nuance or emotional intelligence results in high escalation rates and low satisfaction. We help you be realistic about what agent can do well and where humans are essential.

Oblytech’s Approach to Virtual Agent Health Checks

We assess Virtual Agent realistically. It is great for high-volume, low-complexity conversations. It struggles with nuance, ambiguity, and sensitive issues. We help you use agent for what it does well and route other conversations to humans who can handle them better.

Conversation Analytics and Improvement

We analyze conversation patterns: what works and what does not. We provide feedback on agent responses: are they accurate? Helpful? Do they match your brand voice? We help you identify opportunities to improve: conversations with low satisfaction, common misunderstandings, requests that fail to resolve. We also help you update training data so the agent learns from successes and failures.

Integration Testing and Optimization

Virtual Agent integrates with backend systems. We verify integrations are working reliably and efficiently. Slow integrations frustrate customers (they have to wait while the agent queries a database). We help optimize integration calls: cache data when appropriate, parallelize requests where possible, implement timeouts so the agent can escalate if backends are slow.

Continuous Improvement from Conversation Data

Every Virtual Agent conversation is an opportunity to learn. We help you establish processes for continuously improving the agent: reviewing conversations, identifying patterns of failure or success, retraining the model, and measuring improvement over time. This disciplined approach to improvement means Virtual Agent gets better month over month.

Escalation Quality and SLA Compliance

When Virtual Agent escalates, the escalation should be smooth: the human should have context, the customer should not repeat information, wait times should be minimal. We monitor escalation quality: are customers satisfied with how escalations are handled? Are response times acceptable? We help optimize escalation processes and review SLAs to ensure escalations are meeting expectations.

Escalation Transparency

When Virtual Agent escalates to a human, customers should understand why. Build transparency into escalations: explain to the customer why the conversation is being passed to a person, what the next steps are, and how long they can expect to wait. Transparency builds trust even in an escalation scenario.

Diagnosing a Virtual Agent that is answering but not helping

A Virtual Agent can look healthy on the summary dashboard and still fail the people who use it. The usual pattern is a conversation that starts well and then goes quiet, a topic that fires when the user asked for something else, or a correct answer that the user cannot act on because a link is broken on their device. None of these shows up as an error. They show up as abandoned chats, repeated questions and tickets opened by people who have already tried the bot. A health check should look for these quiet failures, because they are where the agent’s reputation is decided.

This guidance is for the owner of the service channel, the conversational design lead and the service desk manager who is responsible for the agent’s day-to-day behavior. It assumes the agent has been live long enough to produce a meaningful volume of conversations.

Reading the conversations that end without a result

Start with conversations the user abandoned. An abandoned chat tells you the user gave up, but not where. Read a sample, grouping the endings by the last message the agent sent. Abandonment often clusters around three moments: a question the agent asked that the user could not answer, a long answer the user did not finish reading, and a step that required the user to leave the conversation and come back. Each cluster points to a different fix, and none of them is a model-training problem.

Topic collisions and misfires

When two topics match similar phrasing, the agent may open the wrong one. A question about changing a password might route to a topic about account lockout, and a question about a late delivery might route to an order-status topic that expects a number the user does not have. Collisions are easy to find by sorting utterances that were followed by an immediate correction from the user, such as a repeated question or a message saying no, that is not what I meant. Once you have the pairs, decide which topic should own the phrasing and which should be narrowed, rather than adding more examples to both.

Pattern in transcripts	What it usually means	Where to act
User repeats the question with different wording	The first topic matched without understanding the intent	Narrow the topic’s trigger phrasing and add the missed variants
User asks for a person straight after an answer	The answer was correct but not usable	Check the link, the form or the step the answer depends on
Conversation ends at a sign-in prompt	Identity check is failing or too heavy for the channel	Test the authentication path on each channel the agent serves
Answer cites a policy that has changed	Content owner did not review the topic after the change	Update the source content and confirm the owner’s review date

Testing the paths that depend on other systems

Integration failures can be the most damaging quiet failures, because the agent often sounds certain while the data it needs is missing. Test each path that calls another system with a real account, and make sure the agent says something sensible when the call fails. A message that says the status cannot be retrieved right now, and offers a way to raise the request, is far better than a blank reply or an invented answer. Include the paths that run in unusual conditions, such as a record that has been archived, a user with two open requests, or a case where the backend returns an empty result.

Checking the agent on every channel and in every language it serves

A topic that works in the web chat may break in a mobile app or a collaboration tool, because rich content such as buttons and attachments is rendered differently. Walk each topic on each channel at least once, and note where formatting, length or input methods change the experience. If the agent serves more than one language, check that the phrasing set covers the way people in each language actually write, not a translation of the English examples. Accessibility belongs in the same pass: confirm that the agent’s messages make sense to a screen reader and that time limits on responses do not cut off people who need longer to type.

When a health check would be premature

If the agent has handled only a few hundred conversations, patterns will be noise. A small sample invites over-reaction to one unusual session. Wait until there is enough volume for the common patterns to show, and in the meantime fix the obvious defects such as broken links and failing authentication. Run the full review once the volume gives you confidence in what you see.

Turning the findings into fixes in the right order

Rank findings by how many users they affect and how badly they fail them. A broken authentication path that stops every personal request outranks a wording problem on a rarely used topic. Assign each fix to the owner who can make it, whether that is the content owner, the integration developer or the conversational designer, and agree how the fix will be tested before it goes live. Then re-sample the same kinds of conversations after the fix to confirm the quiet failure has gone, rather than assuming it has.

Questions owners ask about agent health reviews

Is a low escalation rate a sign that the agent is healthy?

Not on its own. A low escalation rate can hide users who gave up. Check abandonment alongside escalation, because a user who leaves silently is a failure the escalation figure will never show.

Should we review the agent’s own confidence scores?

Use them as a prompt to read conversations, not as a verdict. Scores show how sure the agent was about its match, but only a read-through shows whether the match was the right one.

How do we review conversations without exposing personal details?

Limit reviewers to the conversations they need, mask identifiers in any export, and keep the review inside the service channel’s own access controls. The review is about the agent’s behavior, so personal content should be removed from the working copy.

Who should run the health review?

Someone who did not build the agent is valuable, because they read the conversations with fresh eyes. The builder should still take part in deciding fixes, since they know why each topic was written the way it was.

Next step

The Virtual Agent escalation handoff guide covers the transfer step in more depth, and the Virtual Agent optimization service page describes what that improvement work covers.