Moving to ServiceNow Automated Test Framework: Migration Considerations

Moving existing tests to the Automated Test Framework is a conversion project with its own inventory, design, build, stabilisation and cutover phases. Not every test converts easily, since ATF suits form validation, workflow execution and data changes better than complex API or load testing. Plan a parallel run, keep audit evidence and define rollback before cutover day.

Phase 1: Inventory and Assessment

Document all existing tests. For each, record: test name, what it tests (which workflow, business process?), test type (unit, integration, end-to-end), how often it runs, failure rate, and maintenance effort. Estimate total test volume: you might find 200+ tests across various tools and scripts.

Assess ATF compatibility. Not all tests can be easily converted to ATF. ATF is best for: form field validation, workflow execution, data creation and updates. ATF is weaker for: complex API testing, third-party system integrations, load/stress testing. Identify tests that cannot easily convert to ATF and plan alternatives: some tests might stay in their current tool, or you might need to redesign them for ATF.

Identify test dependencies. Do some tests depend on manual setup, or can they run fully automated? Do tests depend on specific data being present, and how is that data managed (reset between test runs?)? Understanding dependencies helps you design a migration that maintains test quality.

Phase 2: ATF Design and Planning

Design your ATF strategy: which tests will you convert, and in what order? Prioritize: convert high-value tests first (tests that run frequently, catch important bugs, or are currently brittle). Defer low-value tests (rarely run, trivial functionality, low maintenance burden). Some tests might be deprecated: if they test obsolete features, retire them instead of migrating.

Establish test data strategy. ATF tests need consistent data to run reliably. Do you reset the instance before each test run? Do you use a test data set? How do you ensure parallel tests do not interfere (multiple tests running at once might modify the same record)? Design a data strategy early; issues here cause months of flaky test frustration.

Plan infrastructure. Where will tests run? In a dedicated test instance? In a staging instance shared with QA? Who manages test data? Who maintains test scripts? Allocate resources before conversion begins.

Phase 3: Test Conversion and Development

Convert tests one module or process at a time. Do not try to convert all tests simultaneously. For each test, a test developer: understands the old test (what is it testing?), designs the ATF equivalent (how do we test the same thing in ATF?), writes the test script, and executes it repeatedly to ensure reliability. Conversion is often faster than writing tests from scratch, but not trivial.

Common conversion issues: old tests might have loose assertions (pass if anything happens; fail if nothing happens). ATF requires specific assertions (pass if field equals X; fail if it does not). Tightening assertions often uncovers bugs in the original tests or the system. Old tests might make assumptions about data that are not always true. ATF forces you to be explicit about setup and cleanup, improving reliability.

Phase 4: Testing and Stabilization

Test ATF tests thoroughly. Run them daily over a sustained period in your test environment. Track: pass rate (are tests reliable or do they flake?), execution time (do tests complete in acceptable time?), coverage (do you have tests for all critical paths?). Flaky tests (pass sometimes, fail sometimes) must be fixed before production use; they lose team trust.

Run ATF tests alongside old tests for a period (parallel testing). Verify ATF tests catch the same bugs as old tests. If ATF misses bugs that old tests catch, investigate the gap and improve ATF. When confidence is high, gradually stop running old tests.

Phase 5: Cutover and Stabilization

Retire old tests once ATF is mature. Turn off the old test tool or scripts. Transition maintenance to the QA team using ATF (they need training on ATF specifics). Monitor ATF performance: does it scale to your full test suite? Does it run fast enough for daily execution?

Post-cutover, the QA team owns ATF maintenance: updating tests as features change, adding new tests for new features, and investigating failures. Budget regular QA time for test maintenance.

Common Pitfalls and Prevention

Pitfall 1: converting all tests at once leads to massive rework and months of delay. Fix: migrate in phases, prioritizing high-value tests. Pitfall 2: test data strategy is poorly designed, leading to flaky tests that fail intermittently. Fix: invest time upfront in a reliable data strategy. Pitfall 3: some tests are not compatible with ATF and should not have been converted. Fix: assess compatibility early and make keep-vs-convert decisions before major effort.

Knowledge Transfer and Support

Document ATF design decisions: why certain tests were converted, why others were not, and how test data is managed. Train QA team and developers on ATF in a workshop. Have a super-user (experienced test developer) available post-cutover to support the team and coach on best practices.

Deciding when the cutover is safe, not just when it is scheduled

A migration plan often ends with a date. The more useful question is what evidence would make the cutover safe, because the date tends to arrive whether or not the evidence does. Set the decision criteria before the parallel period starts, and record them in the plan. That way the cutover is a conclusion from results, not a reaction to pressure from the program calendar.

Three kinds of evidence are often needed. The first shows that the ATF suites reach the same verdict as the legacy tests on the same builds. The second shows that the ATF suites have been stable for long enough to be trusted, which means looking at repeat behavior rather than a single clean run. The third shows that the people who will run the suites after cutover can read their results and act on a red one without help from the migration team.

Mapping old checks to ATF equivalents

Legacy tests rarely map one-to-one onto ATF. The mapping exercise is where most of the hidden work sits, and where weak checks in the old suite often come to light. The table below gives a practical starting point for the translation, with the point where it tends to need a decision.

Tightening assertions during migration will often make the ATF suite fail where the legacy suite passed. That is the point of the exercise. Each new failure should be investigated as either a real defect the old test missed, or an assumption the new assertion has wrongly encoded. Record which of the two it was, because the answer tells you how far to trust the rest of the legacy results.

Running the parallel period so the comparison means something

1. Run both suites against the same build and the same instance state, and record the build identifier with each result.
2. Compare verdicts test by test, and keep a list of every disagreement rather than only the count.
3. For each disagreement, decide whether the difference is a defect, a legacy weakness or an ATF weakness, and assign an owner.
4. Re-run the disagreements after each fix, so the comparison closes out rather than drifting.
5. Review the list with the QA lead and the platform owner before the cutover decision, not after.

Keeping the audit evidence when the old tool goes

Retiring a legacy test tool can quietly remove evidence that someone will later need. Old results may show that controls were tested in a given period, and an auditor or a post-incident review may ask for them. Before the old tool is switched off, decide how its historical results are kept, in what format, for how long, and who can read them. Export the results with their build identifiers and timestamps, and store them where access is controlled in the same way as the ATF evidence.

Do not migrate old results into ATF records as if they were produced by ATF. Keep them clearly labeled as legacy evidence, so a reader can tell which tool produced which verdict and when the switch happened.

A cutover rollback plan

Every cutover should have a documented way back. The rollback does not need to keep both tools running forever, but it does need a defined trigger and a defined path. The trigger might be a gate that cannot be trusted for a set number of consecutive cycles, or an incident that the ATF suites missed and the legacy suites would have caught. The path should name the person who can re-enable the legacy gate, the place where that switch is recorded, and the condition for returning to ATF. Rehearse the path once before cutover, on a non-production pipeline, so that the first time it is used is not during an incident.

Checklist before the old tool is switched off

• The cutover criteria were written before the parallel period began and were met.
• Every disagreement from the parallel period is closed or accepted with a recorded reason.
• Legacy results are exported, labeled and stored with controlled access.
• The rollback trigger, path and owner are written down and the path has been rehearsed.
• The team that will run ATF after cutover has run and read a full suite without the migration team present.

Questions migration teams raise once the work is under way

Should we migrate tests we already know are weak, or fix them first?

Fix the ones that protect something important, and retire the ones that protect nothing. Migrating a weak test into a new tool preserves its weakness in a more expensive form, so use the move as the moment to make the decision explicitly.

Can we keep the legacy tool for a few specialist tests?

Yes, when a test depends on something ATF cannot reasonably do. Document the exception, name its owner and review it at each cycle, so the second tool does not become a permanent and unmanaged part of the estate.

What if the parallel results disagree and nobody can say which tool is right?

Go back to the requirement the test was written for. If the requirement is unclear, that is a finding in its own right, and the cutover should wait until the expected behavior is written down.

How should the team handle the skills gap during the switch?

Pair people who know the old suites with people learning ATF, and make the first repairs joint work. Skills often move faster when a real failing test is being fixed than during a classroom session.

Cutover criteria are easier to challenge before the switch than after it.

For the wider data side of a move, the data migration and import sets hub covers the data side of a move, including staging and validation. The acceptance checklist describes the evidence a migrated suite should carry, while the audit trail article explains what an audit trail should show a reviewer.