ServiceNow Automated Test Framework Testing and Acceptance Checklist

Automated Testing Strategy and Layers

The ServiceNow Automated Test Framework (ATF) enables recording and playback of UI interactions as automated regression tests. ATF testing requires a distinct testing strategy from manual testing because automated tests are brittle; UI changes (element ID changes, field reordering) can break tests. The testing strategy should include automated tests for high-value regression scenarios: core workflows that affect many users, critical business processes, and features prone to regression. Unit testing confirms that individual steps (form fill, button click, assertion) work as intended. Scenario testing validates that sequences of steps correctly simulate user workflows. Regression testing runs after each code deployment to ensure previous functionality still works. Performance testing measures execution time; automated tests that take too long become burdensome to run and are skipped, defeating their purpose.

ATF testing requires test data setup and cleanup. A test that creates a record must clean it up afterward so that the test can be run repeatedly without accumulating test data. Use ATF setup steps to create test records and teardown steps to delete them. Test data should include edge cases: empty fields, very long strings, special characters, dates in the past and future, and numbers at boundary values. A form that expects a positive integer should be tested with zero, negative numbers, and non-numeric input to ensure validation is correct.

Test Coverage and Scope Definition

Define which scenarios should have automated tests. High-value scenarios with high regression risk should always be automated. Low-value scenarios (rarely executed, low impact if broken) may not justify automation effort. A matrix of scenarios by frequency and impact guides prioritization. Estimate automation effort: simple workflows may take less time to automate; complex workflows with data lookups may take more time. If automation effort exceeds the value of regression detection, it may be more cost-effective to manual test those scenarios. As the system matures and test step libraries grow, reusable steps reduce automation effort for new tests.

Automated Test Checklist and Validation

The following table outlines critical automated testing areas and evidence requirements:

Test Maintenance and Release Readiness

As the ServiceNow system evolves, automated tests require maintenance. UI element IDs change when forms are redesigned; test steps referencing old IDs fail. A test maintenance schedule should allocate resources to updating failing tests after each major release. Document which tests need updating and why so the next release team knows what to expect. Flaky tests (tests that pass sometimes and fail other times) are worse than no tests; identify and fix flaky tests immediately because they undermine confidence in the test suite. Causes of flakiness include race conditions (test does not wait for element to load), timing assumptions (test expects response in 1 second but it takes 2 seconds), or test isolation issues (test leaves data that affects later tests).

Continuous Integration and Automation

Integrate automated tests into the CI/CD pipeline so tests run automatically after code deployment. A deployment should not proceed if critical regression tests fail. Configure test execution to run in parallel across multiple environments to reduce execution time. Archive test results and screenshots for failed tests so test failures can be analyzed without re-running tests. If a test fails in CI but passes when run manually, investigate environmental differences (data differences, timing issues, browser version) that cause the discrepancy.

Testing the tests before they are allowed to judge a release

A test suite is itself a piece of software, and it can be wrong in both directions. It can fail a healthy release, which wastes effort and erodes trust, or it can pass a broken one, which is worse because nobody looks. Acceptance for ATF therefore has two parts: confirming that the application behaves correctly, and confirming that the test would notice if it did not. Many checklists cover only the first part. The second is the one that protects the release.

The most direct way to test a test is to break the thing it checks on purpose on a non-production instance, then confirm the suite turns red. This negative fixture approach is simple to describe and easy to skip, because it takes time and feels like deliberately causing a problem. Treat it as a standard acceptance step for any test that will gate a release.

Stages a test passes through before it gates anything

A test moves through several states. Each state answers a different question, and the evidence needed to move forward is different. The table below sets out a practical sequence. Where a state has no entry in the evidence column, the test should not move on.

Running a negative fixture properly

A negative fixture is only useful if the fault is the one the test claims to catch. A suite that checks the approval step on a change record will not catch a broken notification, however many times it runs. So the fixture should be designed from the test’s stated purpose, and it should be reverted once the run is complete.

1. Write one sentence stating the fault the test is meant to detect, in plain terms that a non-specialist could check.
2. Introduce that fault alone, on a non-production copy, and record exactly what was changed.
3. Run the test and confirm it fails at the step that should fail, not somewhere earlier or in an unrelated step.
4. Read the failure message and confirm it names the fault, so the person investigating later is not misled.
5. Revert the change, run the test again, and confirm it passes, so the fixture has not left a permanent residue.

Judging stability without guessing

A test that passes once has shown very little. Acceptance should include repeat runs, and the design of those runs matters. Running the same test back to back on an idle instance tells you almost nothing about how it behaves when the instance is busy. Run it at different times of day, and include at least one run during a period of normal load, so that timing problems have a chance to show up.

Record the number of repeats and the conditions, and do not describe a test as stable on the strength of a single green run. If a test fails even once without a clear cause, treat that as a result to investigate rather than a reason to repeat the run until it passes.

Acceptance criteria that work for the whole suite

• Every gating test has a recorded negative fixture and a red run against it.
• Repeat runs were done under more than one load condition, and the results are on file.
• The suite’s total duration on the target instance is known and fits the pipeline stage it is assigned to.
• Any test that was repaired after its first acceptance was re-accepted, not carried over unchanged.
• The acceptance record names the person who accepted each gating test and the date.

Questions about acceptance that come up in review

Do negative fixtures have to be run on every release?

Not in full. The fixture proves that a test can detect its fault, so it needs rerunning when the test changes or when the component it checks changes. For stable tests on stable components, an occasional run during the release cycle is often enough, and the record should say when that was last done.

Who should design the negative fixture, the author or someone else?

Ideally someone other than the author. The author knows which faults the test was built around, and may unconsciously choose a fault the test is good at catching. A second person is more likely to pick a realistic failure from the production history.

What if a test cannot be made to fail on a broken build?

Then the test is not doing what its description says. Either its assertion is too loose, or the step that should fail is not reached, or the chosen fault is not really part of what the test covers. Fix the test or retire it. Do not add it to the gate on the grounds that it is at least green.

Is it acceptable to accept a test with a known limitation?

Yes, if the limitation is written into the acceptance record in plain terms and the gate decision holder has signed it. A limitation that nobody has written down will be discovered during an incident, which is the worst time to learn it.

The best practices page covers how the tests themselves should be built, and the implementation roadmap places acceptance within the rollout.

Related guides: ServiceNow Release Acceptance Criteria: Prove the Outcome.

Building Automated Test Framework Tests in an Order That Catches Failures Early

The Automated Test Framework records steps and checks outcomes, but most early failures come from the order you build in, not from the recorder. Build one test end to end, prove that it can fail, and only then add the next. Teams that record a dozen tests before any of them has been checked tend to end up with a suite that passes on broken builds.

Work through these steps:

• Record one workflow that runs often and touches many users, such as a standard request, starting from a clean state.
• Read every recorded step and delete the ones that are not part of the business action, such as a stray menu click.
• Add one assertion on the saved record, not on an intermediate screen. A screen can load correctly while the save fails.
• Add setup and teardown steps that create and delete the records the test needs, using names that do not repeat between runs.

When a test fails in the first weeks, the cause is usually one of a few things. The table below pairs each symptom with its usual cause and a first check.

Keep the pipeline out of the picture until the first test is trustworthy. Then wire it into the release pipeline so that a failing gating test stops the deployment, with only tests that have already been shown to fail on a broken build. A wide gate built on unproven tests teaches a team to ignore red results.

When the suite grows, the release rules around it need their own owner. For how these gates fit into release governance, read the DevOps governance model, and for how changes reach each instance, see update sets and release management.