ServiceNow Automated Test Framework Integration Patterns

Integration Architecture for Multi-Unit Deployments

Connecting the Automated Test Framework to the rest of a delivery pipeline raises the same questions whether one team or several business units share the instance: whether results are returned synchronously or published as events, how test data stays isolated between units, how failures in downstream systems are handled, and who owns the shared standards. This page works through those patterns and then the operating checklist for a connected pipeline.

What the framework exposes to a pipeline

The Automated Test Framework is not an external testing service that a pipeline calls into. It runs inside the ServiceNow instance under test, and what a pipeline can do with it is defined by the instance’s own interfaces. A delivery pipeline can start a named test suite through the platform’s continuous integration and delivery REST endpoints, receive a progress record for that run, poll the progress record until it reaches a final state, and then read the suite result record, including the outcome of each test and each step. The same endpoints can be driven from a scheduled job or a flow on another instance, which is how some teams run suites against a clone after a refresh without a pipeline tool at all. Everything a pipeline learns about a run, it learns by reading those records back from the instance.

That has a practical consequence for design. There is no separate result store to synchronise and no test farm to provision. The integration work is in three places: granting the pipeline an identity with the right roles, deciding which instance a run may target, and deciding how the pipeline consumes the result.

Where tests actually run

Server-side test steps, such as creating a record, asserting a field value or running a script, execute on the instance itself with no additional infrastructure. User interface steps, such as opening a form and setting a field as a user would, need a browser session that the framework controls. In a connected pipeline that browser is either a client test runner kept open on a dedicated machine, a headless browser started by the pipeline for the duration of the run, or the hosted runner option where the instance release provides one. A pipeline that triggers a suite containing user interface steps while no runner is available will see the run wait rather than fail, so the integration has to check that a runner is present before it starts the suite, or restrict pipeline-triggered suites to server-side steps and leave user interface suites to a scheduled run with a known runner.

Test data is cleaned up by the framework

The framework records the data each test creates or modifies during a run and rolls those changes back when the run finishes. A pipeline therefore does not need to build its own cleanup stage for records that tests create, and two suites that run in sequence on the same instance do not leave each other’s data behind. What the rollback does not cover is data the pipeline itself loaded before the run, such as a fixture import or a configuration change made to set up a scenario. Treat that setup as pipeline-owned, give it its own teardown, and keep it separate from what the tests create, so that a failed run can be diagnosed from the records that remain.

Which instances a run may target

The framework is intended for sub-production instances, and the platform property that enables test execution is off on production by default. A connected pipeline should respect that boundary rather than work around it. Each pipeline stage should have a fixed target instance, the pipeline should read the instance name from the target before starting a run, and the integration account should exist only on the instances it is allowed to test. After a clone from production, the test-execution property and the integration account are among the settings that need to be restored before the pipeline runs again, and a pipeline that fails fast on a missing account is doing its job.

Scheduled runs beside pipeline runs

Not every suite belongs in a pipeline gate. Long regression suites, suites with many user interface steps and suites that exercise integrations with other systems are usually better run on a schedule overnight, with the result read by the gate the next morning. The framework’s suite schedules handle this on the instance without any pipeline involvement. The integration pattern for scheduled runs is simply to make the result discoverable: tag the run with the build or release it was testing, and have the gate read the newest result for that tag and check its freshness before trusting it.

Ownership when several teams share one instance

On a shared sub-production instance, several teams can start suites through the same endpoints. The questions that matter are who owns each suite, which integration account each pipeline uses, and who is told when a run fails. Give each suite a named owner and a clear scope, give each pipeline its own integration account so that run history shows who started what, and route failure notifications to the owning team rather than to a shared mailbox. A central platform team should own the integration account standard, the roles it carries and the target-instance list, while each delivery team owns its suites and the decision of which ones gate its releases.

Consuming results beyond the gate

Suite and test results are records on the instance, so they can be reported on, exported or surfaced on a dashboard like any other table. Teams that want trend information, such as which tests fail most often or how long suites take on each instance, should build that reporting on the result tables rather than on pipeline logs, because the pipeline log holds only what one job saw. Where results have to reach an external system, such as a release dashboard or a ticketing tool, the cleanest route is for the pipeline to post the final outcome and a link to the result record once, after the run has reached a final state. The rest of this page is about getting that final state reliably.

Triggering suites from a pipeline without losing the outcome

Many integration problems with ATF do not come from the request that starts a run. They come from the gap afterwards, when a pipeline has fired a suite and now has to know what happened. A job that sends the request and moves on has no way to tell a passing run from a run that never started. So the first design decision is the trigger shape, because that choice sets how long the pipeline waits, what it does when a run stalls, and where the evidence ends up.

In practice three shapes come up again and again. The table below sets them against each other so the choice is made on purpose rather than inherited from whatever the first script did.

Whichever shape you choose, the pipeline should follow a short, fixed sequence. It is easy to skip one of these steps when a pipeline is first wired up, and the omissions tend to surface months later as unexplained green releases.

1. Give the pipeline its own integration account, granted only the roles needed to start suites and read their results. Do not reuse a developer’s named account, because its permissions drift as that person’s job changes.
2. Pass the build or change identifier into every run, so that each result can be matched to the artifact being promoted rather than to whichever run finished last.
3. Write the run identifier into the pipeline log the moment the run starts. When a job is retried or killed, that identifier is the only reliable handle on what is still executing.
4. Set the polling deadline from measured durations on the target instance over several recent runs, then add headroom that your service owner agrees to. A deadline copied from another team’s pipeline is a common source of false failures.
5. When the deadline passes, record the state as unknown rather than failed or passed. Query the run again before deciding, because a suite that is still executing is a different situation from one that has crashed.
6. Write the final outcome and a link to the result record into the release record, so the people approving promotion can read the evidence without asking the pipeline owner.

Idempotency, shared data and instance mix-ups

Pipelines retry. A stage that times out on the network may be re-run by the orchestrator, and a second start request can kick off a duplicate suite against the same build. The two runs then both write test data, both produce results, and it is no longer obvious which result belongs to the promotion. Before starting a run, check whether a run already exists for that build identifier, and reuse it if it is still active.

Parallel pipelines create a quieter problem. If two branches run suites against one shared test instance, and both tag their test records with a common label, then a cleanup routine written for one run will delete records that belong to the other run halfway through. Scope the tag to the run identifier rather than to the suite name, and make sure cleanup steps only touch records carrying that run’s tag.

Instance mix-ups are the third pattern worth designing against. After an instance is cloned or refreshed, a pipeline’s endpoint setting can still point at the old target, and a suite will happily run against the wrong environment while the pipeline reports success. A useful guard is to have the pipeline read the instance’s environment name before starting and refuse to proceed when the value does not match the stage it is running in.

Operating checklist for a connected pipeline

Before a pipeline is allowed to gate a promotion, confirm each of the following in a dry run that you have looked at, not one that merely returned success:

• The integration account can start and read suites but cannot change test definitions or production configuration.
• A deliberately failed suite produces a red gate, and a deliberately slowed suite produces an unknown state rather than a pass.
• A retried start request reuses the existing run for the same build identifier.
• Two concurrent runs leave each other’s test records intact after cleanup.
• The release record shows the run identifier, the outcome and a link to the result detail.

Questions that come up once the first pipeline is connected

Should a failed ATF stage ever be allowed to block a hotfix?

It should be possible, but only through a written path with a named approver and an evidence trail. A hotfix that skips the gate should record which suites were not run, why they were skipped, and when the full suite is scheduled to run against the fixed build. Without that paper trail, the exception slowly turns into the standard route.

Can one suite serve several pipelines?

Yes, provided the suite is written against behavior that every pipeline depends on and the results are tagged so each pipeline reads only its own runs. Problems start when two teams edit the same suite for different reasons. Give each suite a named owner, and split it when the pipelines need different scope rather than adding conditional steps that few people understand.

What if the results arrive after the pipeline has already moved on?

Treat that as a design fault rather than an edge case. Either the gate has to wait for the result, or the promotion has to be held in a state that a later result can release or reverse. Letting a pipeline proceed and then reporting a late failure to someone who has already deployed is the least defensible option.

Is polling enough, or do we need a callback?

Polling is often enough and is simpler to secure and debug. A callback becomes worth the effort when runs are long, when many pipelines wait on the same suites, or when the receiving system has to react to each result as it happens. Whichever you choose, the callback endpoint needs the same authentication and logging standards as the rest of the integration surface.

If you are working out how pipelines should start and read ATF runs in your environment, a conversation about the trigger design is worth having while it is still cheap to change.

Related reading: the governance model for ATF covers who signs off a gate, and the guide to release acceptance criteria explains what the gate result should be measured against. For the pipeline side, the DevOps best practices page and the access design notes for REST integrations cover the surrounding controls.