Common ServiceNow Automated Test Framework Mistakes and How to Avoid Them

Common Automated Test Framework mistakes are habits in test design and upkeep that make results unreliable: tests that break on cosmetic UI changes, unmanaged test data, and assertions too loose to catch real failures. The page covers each mistake's symptom, likely cause and fix, plus slow execution, coverage, maintenance burden, and why a passing suite is not proof by itself.

Brittle Tests That Fail on Minor Changes

Automated test failures often occur when the UI changes slightly (a button moves, a label is updated) even though the core functionality still works. These brittle tests create false negatives: tests fail but users are not affected. The test maintenance burden becomes so high that teams stop running or maintaining tests. The root cause is usually tests that are too tightly coupled to the UI or implementation details.

Write tests that focus on business outcomes, not UI details. Instead of testing that a button exists in a specific location, test that a required action completes successfully. Use semantic locators when possible (find by button text, field label) rather than absolute XPath that breaks if the UI reorders elements. Use Page Object patterns to encapsulate UI locators so that if the UI changes, you update one place rather than fixing every test. Separate test logic from test data: if a test depends on a specific user account or date, parameterize the test so it works with different data. This makes tests more resilient to UI changes and reduces maintenance burden.

Inadequate Test Data Setup and Cleanup

ATF tests depend on clean, consistent test data. If tests do not properly set up their preconditions or clean up after themselves, tests can interfere with each other and results become unreliable. A test that runs fine in isolation may fail when run with other tests because test data conflicts.

Create test data setup and cleanup procedures that run before and after each test. Use a dedicated test data account or role that is clearly marked as test-only so that test data does not get confused with production data. If a test creates a user, ticket, or configuration item, clean it up in teardown. If tests need specific preconditions (a configuration setting, a prior transaction), establish those conditions in setup rather than assuming they exist. Use transactions or rollback where available to ensure cleanup is reliable. Document test data requirements so that anyone running the tests understands what data must exist beforehand.

Tests With Weak Assertions

ATF tests sometimes pass even though the outcome is wrong because assertions are too loose. A test that checks “page contains the word success” may pass even if the error message says “The operation was not successful.” A test that checks a field exists may not verify that the field has the correct value. Weak assertions result in false positives: tests pass but functionality does not actually work.

Write precise assertions that verify the exact outcome. Instead of checking that a page contains certain text, check the specific message. Instead of checking a field exists, verify its value. Use multiple assertions in a test if necessary to verify all aspects of the outcome. Avoid assertions that could be true for the wrong reason: for example, checking that a field value is not null does not prove it has the correct value. Be precise: if the expected value is “the expected date”, do not accept “the expected month” and consider it close enough.

Slow and Unreliable Test Execution

ATF tests that are slow (run for minutes instead of seconds) are run less frequently or skipped because they take too long. ATF tests that are flaky (sometimes pass, sometimes fail for no clear reason) create distrust in test results. Teams start to believe test failures are false alarms rather than real bugs.

Optimize test performance: avoid unnecessary waits and sleeps, use efficient locators, minimize data creation overhead. Use data factories or bulk import to set up test data faster. If a test needs to wait for something (a record to be created, a status to change), use a polling wait rather than a fixed sleep so the test completes as soon as the condition is met. Make tests reliable by avoiding timing-dependent assertions, using explicit waits instead of implicit waits, and handling elements that may appear or disappear. If a test is flaky, investigate the root cause and fix it rather than adding workarounds like waits or retries.

Inadequate Test Coverage and Maintenance Burden

ATF tests require ongoing maintenance as the application changes. If test coverage is incomplete or tests are not maintained, the test suite becomes a liability. New developers spend time fixing broken tests that do not add value, and critical functionality is not covered by tests so changes are risky.

Focus test effort on critical paths and functionality that is prone to regression: user-facing workflows, integration points, financial calculations, security-sensitive operations. Do not test every possible combination; aim for high coverage of critical paths, not all coverage everywhere. Use a continuous integration pipeline that runs tests on every code change so failures are caught and fixed quickly. Assign test maintenance responsibility to someone on the team so that broken tests are addressed promptly. If a test is not providing value or requires too much maintenance, retire it rather than letting it accumulate as technical debt. Periodically review the test suite to ensure it still covers your current application functionality.

Mistakes that happen in how the program is run

Much of the trouble teams report with ATF is not caused by a bad locator or a missing wait. It comes from decisions about ownership, sequencing and trust that were made quietly and never revisited. These mistakes are harder to see than a flaky step, because the suite looks healthy on the dashboard while the program around it is failing. The notes below focus on that layer, so they sit alongside the technical fixes elsewhere rather than repeating them.

Symptom, likely cause and first question to ask

A useful way to catch these mistakes early is to start from what people see, rather than from a list of rules. The table pairs common symptoms with the organizational cause that often sits behind them, and the question a reviewer should ask first.

Treating a passing suite as proof of anything

A suite that passes shows that the steps it contains succeeded on the instance it ran against. It does not show that the steps were the right ones, that the instance matched production, or that the assertions would have failed for a real defect. Teams that treat green as the end of the question stop asking whether the suite could have caught the last incident. The fix is a routine habit: after each production issue that testing should have caught, ask whether the suite covers that path, and if not, add the check or record why it was left out.

Changing a test to match the new behavior without a second look

When a release legitimately changes behavior, the test should change too. The mistake is changing it so quickly that the test no longer proves anything. A steward who updates an expected value to whatever the system now returns has removed the check rather than updated it. Require a short note on every changed expectation saying what behavior changed and who agreed to the change, and review those notes at the cycle review.

Running everything in one order, on one shared state

Tests that were written one after another in a single session often assume the state those earlier tests left behind. This works until the suite is split for speed, reordered for a new pipeline or run in parallel, and then failures appear that nobody can reproduce on a laptop. A quick discipline check is to run each test in isolation on a clean state at least once when it is accepted. Tests that fail only in sequence should be rewritten to create their own starting point.

A short procedure for a mistake review

1. Pick the last three surprises, meaning a passed release with a defect or a failed release with no defect, and trace each one to the suite that should have caught it.
2. For each, record whether the gap was missing coverage, a weak check, an unclear owner or a misread result.
3. Assign one corrective action per gap, with a named owner and the next review date.
4. Check at the next review whether the action changed the outcome, not just whether it was completed.

Checklist to run before a suite is trusted

• Each test states the process or risk it protects, in a sentence a process owner would recognize.
• At least one negative check exists for each gating process, and its result has been seen.
• Expectation changes are reviewed by someone other than the author.
• Each test passes when run alone on a clean state.
• Someone is responsible for reading results between releases, and that person has a defined action when a result is red.

Questions that come up when a team looks back

Is it a mistake to have a large suite?

Size is not the problem. A large suite with clear owners and recorded purposes is easier to manage than a small one nobody understands. The warning sign is a suite whose purpose has been forgotten, whatever its size.

Why did a senior tester’s green run not catch a defect that reached users?

Often because the defect sat outside the path the suite exercised, or because the check was looser than the risk. A green run is evidence about what was checked. It does not say what was missed, which is why the question to ask is about coverage, not about the tester.

Should we stop running tests while we fix the process?

Keep running them in reporting mode while the gate is paused. Results still show drift and failure patterns, which helps the review. Just make clear, in the release record, that the suite is not currently deciding anything.

How do we avoid repeating the same mistake with a new team?

Write the decisions down in the program’s standing documents, and make the cycle review check them. New teams often repeat old mistakes because the reasons behind the existing structure were never recorded.

For how acceptance should prevent several of these patterns, see the testing and acceptance checklist.

Related guides: ServiceNow Automated Testing Governance for Safer Releases and ServiceNow Automated Test Framework Governance Model.