Troubleshooting ServiceNow DevOps: A Practical Guide

This guide helps a delivery or platform engineer work out why a ServiceNow DevOps deployment failed, stalled or left the target instance unchanged. It sets out the symptom, where to look and the likely causes for each common failure, then a triage order to follow before opening a support request.

Pipeline Execution Failures

Symptom: A deployment pipeline fails at a step (build, test, deploy). The pipeline logs show an error but it is unclear what went wrong.

Where to look: Check the pipeline stage that failed. Look at the step logs for error messages. Common failures: build step failed because a dependent library is missing, test step failed because a test assertion failed, deploy step failed because a credential is invalid or a network connection timed out.

Likely causes: Source code has a syntax error. A required file or dependency was not included in the commit. A test environment is not set up correctly. Credentials are expired or incorrect. Network connectivity or firewall rules are blocking deployment.

Fixes: For build failures, check the source code and commit history; revert the breaking change. For test failures, review the test and the feature; if the test is wrong, fix it; if the code is wrong, fix the code. For deploy failures, verify credentials and network connectivity. Test the pipeline with a simple change to confirm it is working before debugging a complex change.

Deployments Not Applying to Target Environment

Symptom: A change is deployed but the target environment does not reflect the change. Code was deployed to the production instance but users do not see the new feature.

Where to look: Verify that the right instance was targeted. Check the deployment logs to confirm the change was sent. Check the target instance to see if the change is actually there (search for the record, check the version, verify timestamps).

Likely causes: The deployment targeted the wrong instance (development instead of production). The change was deployed but requires a cache clear or system restart to be visible. The target instance is on a different version and does not support the feature. The deployment succeeded but was immediately rolled back by a competing deployment.

Fixes: Verify the deployment target before deploying. Add a confirmation step that lists the target instance so the deployer can confirm. After deployment, run a validation check that confirms the change is present (query for the record, verify its properties). If multiple deployments are happening simultaneously, coordinate them or implement a deployment lock to prevent conflicts.

Promotion Conflicts and Overlapping Changes

Symptom: Two developers are working on related features. Developer A’s change promotes to production. Developer B’s change is then promoted and overwrites A’s change. Both changes should be in production but only B’s change is there.

Where to look: Check the update set or change object for both changes. Look at the promotion history to see which was applied last. Check the target instance to see what is actually there.

Likely causes: Poor development discipline. Developers are not aware of each other’s work. Changes overlap and the second promotion overwrites the first. No merge logic to combine changes; instead, the second change completely replaces the first.

Fixes: Use a development workflow that tracks changes centrally (change object, branching strategy). Require that changes are logged before development starts so developers know what others are working on. Implement conflict detection: if two changes affect the same table or file, alert and require manual resolution before promotion. Use version control (Git) with branching and merging to handle concurrent development.

Environment Synchronisation Issues

Symptom: Development instance and production instance are out of sync. A feature works in development but fails in production. Or data is different between environments.

Where to look: Compare configuration (business rules, workflows, scripts) between instances. Compare data (records, tables). Check whether promotion jobs have run correctly.

Likely causes: A change was made directly to production, bypassing development. A change was made to development but not promoted. Data is out of sync because backup and restore operations are not in sync. Different release versions or plugins are active in each environment.

Fixes: Establish a promotion discipline: all changes go through development first, then are promoted to production. Prevent direct edits to production. Audit configuration regularly to detect drift. Keep environments at the same release level. Use configuration as code (store infrastructure and business logic in version control) so it can be promoted reliably.

Failed Rollback or Rollback Does Not Work

Symptom: A deployment went wrong. The team initiates a rollback, but the previous version is not restored. The broken change is still in production.

Where to look: Check whether a rollback plan was created before the deployment. Check the rollback logs to see if the rollback completed or failed. Verify the current instance to see if the rollback was applied.

Likely causes: No rollback plan was created. The rollback plan targeted the wrong instance or change record. The rollback failed because it could not be applied cleanly (for example, a database migration was not reversible). Time elapsed so long that a rollback is no longer safe (data changed since the deployment).

Fixes: Create a rollback plan before every production deployment. Test the rollback in a staging environment so you know it works. Limit the time window in which rollback is safe, and record that window for each change. For irreversible changes (database migrations, data deletions), plan an alternative recovery (restore from backup, manual data fix). If rollback is not possible, escalate to incident management and plan a forward fix instead.

Slow or Stalled Deployments

Symptom: A deployment is running but is taking much longer than expected. Pipeline is stuck on a step that normally completes quickly.

Where to look: Check the step that is stalled. Look for network timeouts, resource exhaustion (CPU, memory), or database locks. Check whether other deployments or heavy processes are running concurrently.

Likely causes: The change is larger than usual and is taking time to apply. The target instance is under heavy load. Database locks are preventing the change from being applied. A dependent service is slow or unresponsive.

Fixes: Reduce change size if possible; break large changes into smaller deployments. Schedule deployments during low-load periods. Identify database locks and resolve them (cancel conflicting transactions). Monitor the health of dependent systems before deploying. Set reasonable timeouts for each deployment step and alert if a step exceeds the threshold.

Credential or Connection Failures in Pipeline

Symptom: Pipeline fails at a step that requires credentials (connecting to an external system, deploying to a target instance). The logs show an authentication or connection error.

Where to look: Check the credentials stored in the pipeline configuration. Verify that the credentials are correct (username, password, API key). Check the firewall rules to see if the target system is reachable.

Likely causes: Credentials were updated (password changed, API key rotated) but not updated in the pipeline. Credentials are stored in clear text or in a vault that is not accessible. The target system’s IP address changed. Firewall rules were updated and the connection is now blocked.

Fixes: Use a secure vault for credentials, not clear text. Rotate credentials regularly and update the pipeline. Test connectivity to the target system before running the pipeline. Keep firewall rules updated as systems change. Use service accounts with minimal permissions for pipeline deployments, not personal credentials.

Triage order for a failing deployment

A failed deployment tempts people straight into the longest log file on the runner. That habit wastes time, and it often points at the wrong component. A faster approach is to work through the same sequence every time, so that the on-call engineer can rule out the cheap explanations before spending an hour on the expensive ones. The sequence below assumes the pipeline reports to the platform, which is the common setup; the first step is about establishing which system is actually wrong.

1. Confirm the failure is in the deployment and not in the change record. A change stuck in a pending state looks like a deployment failure but often has an approval or a schedule problem.
2. Check whether the pipeline run reached the platform at all. A run with no matching record points to the connection, not to the code.
3. Compare the artifact identifier in the run with the one the change expects. A mismatch is a common cause of a deployment that appears to succeed but changes nothing.
4. Check the target environment identifier and confirm it still matches the configuration item the pipeline was designed for.
5. Only then read the runner log for the failing stage, starting from the first error rather than the last.

Evidence to collect before opening a support request

A support request that arrives with the run identifier, the change identifier, the artifact identifier, the time of the failure in one agreed time zone and the last successful run for the same pipeline is usually solved in a fraction of the time of one that says the deployment failed. Collect these before escalating, because the logs that explain the failure are often rotated out quickly on shared runners.

Distinguishing a platform fault from a process fault

A platform fault means the tool did what it was built to do and the result is still wrong. A process fault means the tool worked and the team asked it to do the wrong thing, such as promoting an artifact before its tests finished. Treating a process fault as a platform fault can lead to a support ticket that closes with no change, and the same failure returns at the next release.

Why did the run succeed when the change was not approved?

Check whether the gate that enforces approval is attached to the stage that ran. A gate on the production stage does not protect a non-production stage that someone pointed at production by changing a variable.

Should we retry a failed deployment automatically?

Only for failures that are known to be transient and that do not change state partway through. A retry after a partial deployment can apply the same change twice, so the retry logic should check the target state before acting.

How long should runner logs be kept?

Long enough to cover the audit look-back period and the time a problem can take to surface. Shorter retention is common on shared runners, which is why the evidence list above matters.

When should we stop troubleshooting and roll back?

When the affected service is degraded and the cause is not identified within the window the service owner has agreed. Rolling back a known good state is usually cheaper than diagnosing a live fault under pressure. If a freeze window is in force, the change freeze governance guide covers how exceptions are approved, so check it before the decision is made.