ServiceNow Virtual Agent Managed Support

Managed support for Virtual Agent is the ongoing work of keeping live topics accurate after launch, covering conversation monitoring, training updates, change control and the split of duties between client and provider. It is written for the knowledge manager, service owner and procurement lead who hold the arrangement.

Managed support for Virtual Agent treats topics like code: change control, a regression set that outlives the people who wrote it, and approval tiers that match the risk of each change. This page covers conversation monitoring and quality assessment, training data and model updates, escalation threshold tuning, customer feedback and sentiment, new conversation flows, performance dashboards and reporting, multi-channel monitoring, splitting the work between the client team and the provider, ending or transferring the service cleanly, and an illustrative case of a team that keeps finding the same gaps.

Conversation Monitoring and Quality Assessment

We continuously monitor agent conversations to assess quality and identify improvement opportunities. We measure: conversation success rate (did agent resolve question or properly escalate), customer satisfaction scores, conversation length, escalation rate. We also manually review conversations: reading low-confidence conversations where agent was uncertain, reviewing escalated conversations to understand when agent should have escalated, reviewing customer complaints. Patterns emerge: certain question types have low success rate (agent needs better training), certain users always escalate (might be preference for human, or agent isn’t understanding their phrasing), certain hours have more escalations (might indicate agent overload). We use patterns to drive improvements.

Training Data Enhancement and Model Updates

Agent accuracy improves with training data. We continuously collect and review training data: extracting successful agent conversations as positive examples, analyzing failed conversations to understand where agent missed intent. We identify training gaps: questions agent struggled with are added to training data with correct interpretation, variations of questions are added so agent recognizes different phrasings. We also update training data for changing business: if new policy is implemented, we train agent on new process; if new product launches, we add questions about new product. Regular training updates keep agent knowledge current. Testing validates updates improve accuracy: before deploying training update, we test agent against conversations that previously failed to confirm update helps. This iterative improvement typically increases agent accuracy over time.

Escalation Optimization and Threshold Tuning

Agent escalation to humans is critical customer experience moment. If agent escalates inappropriately (certain escalations are unnecessary), we adjust thresholds. If agent doesn’t escalate when it should (trying to answer questions it doesn’t understand), we adjust thresholds. We monitor escalation patterns: which question types are frequently escalated, what is escalation rate by question type, how satisfied are customers with escalations. We also track first-contact resolution (did human agent resolve issue on first contact, or did it require multiple interactions). If escalation to human doesn’t actually resolve issue quickly, that’s a problem. We analyze first-contact resolution: if human agents are struggling with certain question types, we train Virtual Agent better so it can handle them rather than escalating.

Customer Feedback Integration and Sentiment Analysis

We collect customer feedback on agent interactions: post-interaction surveys asking if agent was helpful, satisfaction ratings, free-form feedback. We also perform sentiment analysis on conversations: analyzing language to detect when customers are frustrated. Feedback and sentiment reveal where agent needs improvement: low satisfaction scores on certain question types indicate training gap, frustrated tone despite successful resolution indicates agent communication could improve. We act on feedback: questions with consistent complaints get training updates, agent response templates are revised, escalation is adjusted. Customer feedback makes improvement data-driven rather than guesswork.

New Conversation Flows and Capability Expansion

Business evolves: new services launch, policies change, new question types emerge. We continuously evaluate agent scope: should agent handle new question types, can existing question flows be improved. We prioritize based on volume and impact: high-volume questions that humans currently handle are candidates for agent automation, complex questions that frustrate customers are candidates if automation would help. We develop new conversation flows: designing workflows for new question types, training agent, testing before production deployment. Expanding agent scope over time increases automation benefit and customer satisfaction: customers can self-serve more types of requests rather than waiting for human response.

Performance Dashboard and Stakeholder Reporting

We maintain dashboards showing agent performance: conversations handled daily, success rate, escalation rate, customer satisfaction score, comparison to baseline. We provide monthly reports to stakeholders: executive summary (agent is performing well or needs attention), performance trends (improving or declining), improvement opportunities (what changes would improve performance), impact summary (how many conversations were automated, time saved, customer satisfaction improvement). Reporting maintains stakeholder engagement and justifies continued investment in agent platform. Reports also help you understand business impact: if agent handles 50 of incoming requests, that’s significant reduction in helpdesk volume.

Multi-Channel Performance Monitoring

If agent is deployed across multiple channels (portal, mobile app, Teams, email), we monitor performance by channel: does agent perform differently in Teams versus portal, are certain channels driving more escalations. Channels might have different user populations or expectations: Teams users might escalate more readily than portal users. We optimize by channel: adjusting training or configuration to improve performance in lower-performing channels, learning from high-performing channels what works well. We also manage deployment: if one channel is underperforming consistently, we investigate: is there a technical issue, is configuration wrong, is user population different.

Splitting the work between your team and the provider

An ongoing Virtual Agent service often fails over unclear ownership rather than weak technical work. The provider assumes the client will decide what the agent should say. The client assumes the provider will notice when an answer goes stale. Each side is correct about some things and wrong about others, and the gap is where incidents begin. Writing the split down before the service starts, in a table everyone signs, prevents many of these arguments.

Written for the knowledge manager, the service owner and the procurement lead who will hold an ongoing support arrangement for an agent that is already live. It assumes the agent has been in use long enough to show its own patterns.

The table is a starting point. What matters is that every recurring activity has one named owner with the authority to decide, and that owner knows it is theirs.

Change control for topics that behave like code

A topic change alters what thousands of people may be told. Treat it with the same care as a configuration change, even when the edit looks small. A common failure is a wording change to one answer that quietly alters the intent the topic is matched on, and the first sign is a rise in misfires on an unrelated subject. A simple release discipline prevents this: propose the change in writing, state which requests it should affect, test it against a fixed set of sample utterances, and release it at a time when the team can watch the result.

1. Record the change, the reason, and the content owner who approved the wording.
2. Run the change against the agreed sample set, including the utterances that previously worked and the ones that previously failed.
3. Release in a quiet period and check the first conversations on the changed topic within the same working session.
4. Retain the prior version so reverting is quick rather than a rebuild.

Building a regression set that outlives the people who wrote it

A regression set is a collection of real user phrasings, with the expected outcome for each, kept in a form anyone can run. It is often the most valuable asset a managed service produces, because it turns the memory of past failures into a test that catches them again. Keep it small enough to run quickly and include a representative spread across topics. Add to it whenever a real conversation exposes a gap, and remove phrasings only when the topic they test has been retired. Store it with the client, not the provider, so that a change of supplier does not reset the team’s knowledge.

Approval tiers that match the risk of each change

Not every change needs the same sign-off. A reworded greeting carries little risk. A change to the answer about eligibility for a benefit carries a great deal. Agree two or three tiers in advance. The lowest tier can be approved by the provider and reported afterwards. The middle tier needs the service owner. The highest tier, covering any answer that states a rule with financial, legal or safety consequences, needs the content owner’s written approval before release. Writing the tiers down avoids a debate about each change and keeps the process moving.

Ending or transferring the service cleanly

Every managed arrangement eventually ends, and the way it ends decides whether the agent survives the change. Before the service starts, agree what the client receives at the end: the topic inventory, the regression set, the change history, the configuration notes and the reports. Test that a member of the client team can run a basic cycle of topic change and regression check without the provider. A handover that has never been rehearsed often reveals its gaps during the first outage.

Illustrative case: a team that keeps finding the same gaps

Consider a service desk where the agent is consistently accurate on password questions but keeps failing on travel expense status. Each month the provider reports the failure and makes a small fix, and each month a new variant appears. The useful move here is not another fix. It is to ask whether the topic has a content owner who knows the expense process, and whether the regression set includes the phrasings that have already failed. Often the root cause is ownership, and the technical fix is the easy part.

Questions about ongoing agent support

How do we know the provider is doing enough?

Ask for the change log, the regression results for each release, and the list of open gaps. Activity without those records is hard to judge, and a provider that cannot produce them is probably not testing before release.

Should the provider be allowed to change topic wording on its own?

Only in the lowest approval tier. Wording that states a rule should always go to the content owner, however small the edit looks.

What should happen when the business changes a policy?

The content owner raises the change, the provider updates the affected topics, and the regression set is extended to cover the new rule. The policy change should trigger the update, not wait for a user to report the wrong answer.

Does a managed service replace the need for internal skills?

It should not. The client needs at least one person who can read conversations, judge an answer and approve a change. Without that person, the client cannot tell whether the service is working.

Next step

For the wider model, see the managed support services page. Improvement work that runs alongside it is handled by the Virtual Agent optimization service. The service level communication guide is a useful reference for agreeing how each party reports its performance.