Troubleshooting ServiceNow Automated Test Framework: A Practical Guide

Understanding test execution and failure modes

ServiceNow Automated Test Framework (ATF) allows Operations Managers to write and execute test scripts that verify system behavior. Test failures can be real—the system is broken—or false—the test is broken. Distinguishing between them is critical. A test that consistently fails in the same way found a real issue. A test that fails intermittently or only in certain environments is probably environment-dependent and needs adjustment. Start investigation by looking at the test execution log: the log shows each step the test executed, what it expected, and what actually occurred. If a step expected to find a button but the button was not found, the log will show this.

Test framework initialization and dependencies

Tests depend on the test framework being initialized correctly, the test browser or automation client being ready, and test data being present. When tests fail at the very start—before any real test steps—investigate initialization problems. A test that cannot authenticate to the system cannot proceed. A test that cannot find the test browser may indicate ATF is not properly installed. Verify the test environment has all prerequisites: the ATF plugin is installed, test runner is running, and the test user account exists and has permission to test. If tests fail to initialize, skip individual test troubleshooting and focus on environment setup. Update ATF components to the latest version if they are old—compatibility issues may be causing failures.

Element locators and UI changes

Tests locate UI elements using locators—selectors that identify a specific button, field, or link. When a form is modified, element IDs or classes might change. A test written against an older form version uses a locator that no longer matches any element, so the test fails. When tests start failing after a form change, inspect the form to see what locators changed. Verify test locators match current element IDs. Some locators are more brittle than others. A locator using the element ID is usually stable. A locator using CSS class names or text labels is less stable because these often change with styling updates. Write tests using stable locators. If stable locators are not available, document which tests depend on unstable locators so you can update them after form changes.

Synchronization and waits

Automated tests run faster than humans. A test clicking a button and immediately checking for a result will fail if the result takes a moment to appear. This is the most common cause of test failures: missing wait for an element or wait for a load to complete. Add explicit waits before checking results. Wait for the result element to appear, wait for a loading spinner to disappear, or wait for an asynchronous function to complete. ServiceNow provides wait commands in the ATF framework—use them. A test that works locally but fails in CI often has synchronization issues because the CI system is under load and slower. Test in the intended environment (development, staging) with the expected load to see real timing behavior.

Test data and environment state

Tests must be able to run regardless of environment state. If a test assumes a specific record exists or a specific configuration is in place, the test will fail if the environment is refreshed or records are deleted. Write tests that create their own test data rather than depending on pre-existing data. If a test needs to search for a record and verify the search works, the test should create a unique record, search for it, and delete it when done. This ensures the test does not fail due to environment state changes. Also ensure tests are repeatable: running the same test twice should produce the same result. If the second run finds test data left by the first run, the test may behave differently. Clean up test data thoroughly between runs.

Permissions and test user roles

Tests run as a specific test user with specific permissions. If a test expects to access an admin function but the test user lacks admin permissions, the test will fail. Verify the test user has the required roles and permissions for the test. If tests work for some users but not others, role differences are the likely cause. Create a test user with broad permissions so it can test many scenarios. If you need to test role-based access control—verifying that non-admins cannot access admin functions—create multiple test users with different roles and run different tests for each user.

Browser compatibility and automation client

ATF uses a browser automation client to control the browser and execute tests. Different browsers (Chrome, Firefox, Safari) and different automation clients (Selenium, Puppeteer) have different capabilities and quirks. A test that works in Chrome might not work in Firefox if it depends on Chrome-specific behavior. Run tests in the same browser your users will use. Also check whether your automation client is compatible with your browser version. Older automation clients do not work with newer browser versions. Keep the automation client updated to maintain compatibility with current browsers.

Debugging test execution

When a test fails, examine the test execution log and screenshots captured during execution. Logs show step-by-step what happened. Screenshots show the screen state at specific points. This combination often reveals the failure cause. A log showing “element not found” combined with a screenshot showing an unexpected page means the test was redirected unexpectedly. A log showing “timeout waiting for element” combined with a screenshot showing a loading spinner means the page took too long to load. Use this information to fix the test or the system. Some tests allow debugging mode—the test executes slowly and pauses at each step so you can inspect state. Use this mode for complex tests to understand what is happening.

Narrowing a failure before changing anything

The fastest way to lose a morning on an ATF failure is to start editing. A changed locator or an extra wait may make the symptom disappear without explaining it, and the next failure will look like a different problem. The more reliable method is to narrow the failure in a fixed order, changing one variable at a time, and to write down what each run showed. Four questions do most of the work: does the test fail alone, does it fail inside its suite, does it fail on another instance, and does it fail on the same build twice?

Each answer points somewhere different. A test that fails alone is often a defect in the test or the feature it checks. A test that passes alone but fails in its suite is often leaking state from an earlier test or sharing data with it. A test that passes on a second instance but not the first points at configuration or data differences between them. A test that changes outcome on the same build is a flake, and that needs a different response from a real defect.

An ordered sequence for a single red test

1. Confirm the run is against the build you think it is, and that the instance was healthy during the run.
2. Run the test alone on a clean starting state. Record the outcome, even if it matches the original failure.
3. If it passes alone, run its suite with the tests before it removed one at a time, until the failure reappears. The last removed test is the likely source of leaked state.
4. If it fails alone, read the failing step and compare the expected and actual values, rather than the step’s label, which may be generic.
5. Check the instance log around the time of the failing step for errors that the test would not show on screen.
6. Decide the failure label, and only then change the test or the product.

Deciding whether to repair, quarantine or retire

Not every failing test deserves an immediate repair. Some should be quarantined so that they stop holding up the gate while the cause is investigated, and some should be retired because the behavior they check no longer matters. The choice depends on what the failure is telling you, so write the decision down with its reason.

Keeping evidence clean when you ask for help

When a problem needs a platform specialist or a support request, the quality of the evidence decides how quickly it is solved. Send the run identifier, the build, the instance name, the failing step, the expected and actual values, and the relevant part of the log. Strip out credentials, personal data and any record contents that are not needed to show the fault. A bundle that contains a password or a customer record creates a second problem while solving the first, so check it before it leaves the team.

Checklist for closing a troubleshooting case

• The failure was reproduced alone, and the result is recorded.
• The label, the owner and the action are written on the case.
• If the test was changed, the change was reviewed and the repeat runs are on file.
• If the test was quarantined, the entry has an owner and a date for review.
• The evidence attached to any support request has been checked for sensitive content.

Questions that come up during a difficult investigation

How long should we keep investigating before quarantining a test?

Set the limit with the service owner in advance, based on how much the gate depends on the test. A test that holds up a critical release should be quarantined sooner than one that only feeds a weekly report, because the cost of a false stop is higher.

The test fails only at night. What should we check first?

Look at what else runs on the instance at that time, including scheduled jobs, refreshes and integrations. Overlapping load often explains a failure that disappears during the day, and the fix may be to move the suite rather than to change the test.

Can we debug by running the test slowly and watching it?

Yes, and it is worth doing once for a hard case. It confirms what the test actually sees, but it changes the timing, so a test that passes in slow mode may still fail at speed. Treat a slow pass as a clue about timing, not as a fix.

Is it worth raising every flake with the vendor?

Raise the ones that show a platform fault with a clean reproduction. Many flakes come from the test or the environment, and sending them upstream without evidence rarely helps.

The best practices page covers how to build tests that fail for the right reasons, and the access design notes describe who should be able to read test results and definitions. For the surrounding DevOps flow, see the DevOps best practices page.