ServiceNow Virtual Agent Testing and Acceptance Checklist

Test Layers for Virtual Agent

Virtual Agent testing spans configuration, conversation flows, integration, knowledge, and user experience. Configuration testing verifies that the agent is set up correctly: the right conversations are enabled, escalation rules point to the right queues, and knowledge sources are connected. Conversation flow testing confirms that the agent follows conversation logic correctly, understands user inputs, responds appropriately, and knows when to escalate. Integration testing validates that the agent can look up information from ServiceNow tables, create tickets, and pass information to live agents when escalating. Knowledge testing ensures that the agent can find relevant articles and provide useful answers. User experience testing involves actual users interacting with the agent to validate that it is helpful and not frustrating.

Each test layer requires different skills and test data. Configuration is technical. Conversation testing requires people who understand conversation design. Integration testing requires access to test ServiceNow instances. Knowledge testing requires subject matter experts who can judge if the agent’s answers are correct. User experience testing involves end users.

Entry and Exit Criteria

Entry criteria for Virtual Agent testing include that the agent is deployed to a test environment, that conversation flows are defined and imported, that knowledge sources are populated with test data, that integrations to backend systems are configured, and that test users are available. Do not begin testing until the agent responds to basic questions correctly.

Exit criteria require that configuration testing passes without deviations, conversation testing confirms correct behavior across planned scenarios, integration testing validates that the agent can perform key transactions, knowledge testing shows that the agent provides accurate information, and user acceptance testing shows that actual users find the agent helpful. The escalation path to live agents must work correctly, and performance must be acceptable even under load.

Testing Checklist

Create a checklist covering these scenarios. First, verify that the agent responds to greetings and simple questions correctly. Second, test specific conversation flows: how does the agent handle requests to create an incident, reset a password, or look up order status? Third, confirm that the agent escalates appropriately: when the user asks for something the agent cannot do, escalation to a live agent occurs smoothly. Fourth, validate that knowledge articles are returned for relevant questions and that the articles are current. Fifth, test edge cases: typos in user input, requests in different languages (if applicable), multiple questions in one message. Sixth, confirm that the agent does not answer questions outside its scope or provide information that is confidential. Seventh, validate performance: measure response time, confirm that the agent handles multiple conversations simultaneously, and test behavior under peak load.

Evidence to Keep

Document test results thoroughly. Keep conversation transcripts showing what the user said and how the agent responded. Keep screenshots of configuration settings and conversation flows. Keep logs of escalations to confirm that they route correctly. Keep performance measurements: average response time, throughput, error rates. Keep a list of user feedback from user acceptance testing: what users found helpful, what confused them. Archive this evidence for future reference and for compliance audits.

Regression Traps and Risk Areas

Watch for problems that emerge after deployment. If conversation logic changes, ensure that the new logic works correctly with escalation rules. If knowledge sources are updated, validate that the agent still finds relevant articles. If the agent integrates with backend systems, monitor for cases where the backend system is down or slow: the agent should handle these gracefully and escalate to live agents. Monitor agent response quality over time; if accuracy degrades, review recent knowledge updates or configuration changes. Users might ask questions the agent has never encountered before; monitor these failures to identify gaps in training or knowledge.

Managing False Positives in Virtual Agent Testing

Virtual Agents can create false positives: situations where the agent provides incorrect or unhelpful responses, confusing users. Testing must identify false positives before deployment. Ask specific questions that the agent is not trained to handle and confirm that it escalates appropriately. Ask variations of the same question to confirm the agent handles them correctly. A user might ask “reset my password” but another might ask “I forgot my password”. The agent should understand both.

After deployment, monitor conversations for false positives. When users report that the agent gave wrong information or misunderstood their request, investigate. Use these cases as training data: improve the agent’s understanding so that similar requests are handled correctly in the future. Virtual Agents improve through feedback; collect and act on it regularly.

Design the test pack around the ways conversations fail

Testing a conversational agent by running the questions it was built for proves very little, because the questions it was built for are the ones that were rehearsed. Real users change their wording, abandon a sentence half way, reply to a question with a different question, return the next morning, and type something no designer imagined. A useful test pack is organised around these failure modes rather than around the topic list. Each topic gets a set of happy-path cases, and each category of failure gets its own set that runs across every topic.

The near-miss family is the one teams often skip. A confident answer to the wrong question does more damage than a clarifying question, and it is also the failure users are least likely to report.

Order the test runs so the early results mean something

Run the cheapest and most informative tests first. Configuration checks come before conversation checks, because a misconfigured hand-off will make every conversation test fail for the wrong reason. Conversation checks come before integration checks, so that a failure in matching is not confused with a failure in a downstream lookup. Run the failure families after the happy paths for each topic, since a topic that cannot handle its own examples will not be improved by stress testing. Automating the regression set is covered in automated testing governance.

1. Confirm the agent, its channels and its topics are deployed to the test environment with the expected versions of the source content.
2. Run the happy-path cases for each topic and record the matched topic, the answer, and any hand-off.
3. Run the failure families across all topics, recording each unexpected match.
4. Run the integration cases against test data, checking that each record is created once and only once.
5. Repeat the failure families after any wording change, because a small edit can move a match to a different topic.
6. Hand the results to a reviewer who did not run the tests, and ask them to sample the transcripts for anything the scripts would not flag.

What a passed test should leave behind

A pass that cannot be reproduced is not evidence. Keep the input text, the topic matched, the response, the record created or not created, and the date of the test run. Keep the version of the source article each answer drew on, because an answer that passed last month may fail after an edit. Store the results where the topic owner and the acceptance approver can both read them, and do not rely on screenshots alone, since they do not show what was matched. Use the same failure definitions when you measure results after launch, as described in the KPIs and metrics guide.

Acceptance criteria that are hard to argue with

Acceptance criteria should be written so that a person can answer each one with a yes or no after reading the results. Good criteria name the failure families that must pass with no unexpected matches, the integration cases that must create records exactly once, and the hand-off fields that must be present in every test hand-off. Weak criteria say the agent should perform well, which is impossible to test. Agree the criteria before the test run, so that the result is not negotiated after it has been seen. The release side of agreeing criteria is in release acceptance criteria.

Regression risk after each change

The highest regression risk is a change to something that looks unrelated. A new synonym added to one topic can pull conversations away from another. A knowledge article rewritten for clarity can drop a phrase the agent used to match. A change to a hand-off rule can alter the queue an entire category reaches. Keep the failure families in the regression set permanently, and treat any change to shared matching or routing as a full run, even when the change looks small.

Common questions about the acceptance run

Can testers use real user transcripts?

They can, once personal details have been removed and the copies are held under the same rules as the live conversations. Real transcripts are a strong source of paraphrases that the designers did not anticipate, so build a regular step that adds new examples from them.

How many testers are needed?

Enough that each tester is not reviewing their own build. A tester who wrote a topic will read their own intent into an unclear answer, so have at least one reviewer outside the build team sign each topic.

Should the pilot be treated as a test?

The pilot is a controlled exposure, not a substitute for the test pack. Use it to find failures the pack could not predict, and feed each one back into a new test case before the topic is changed again. The support period after launch is covered in the go-live hypercare checklist.

What if a test fails only in production?

Capture the exact input and the state of the session, reproduce it in the test environment, and add it to the regression set. A failure that cannot be reproduced usually means the test data differs from production data, and that difference is worth fixing before the next release.

Next step

A test plan built only from happy-path questions often leaves the common failures untested.