Common ServiceNow DevOps Mistakes and How to Avoid Them

This guide helps a DevOps team spot the common ServiceNow release mistakes before they reach users. It covers skipped testing, missing rollback, no staged rollouts, environment mismatch, poor deployment visibility, unapproved deployments and slow incident detection, each with its symptom, root cause and fix.

Mistake: Skipping Testing in the Name of Speed

Symptom: Deployments are frequent but are followed by frequent rollbacks. Defects that could have been caught by tests make it to production. Teams spend more time fixing broken deployments than writing features.

Root cause: Teams prioritise deployment speed over quality. Testing is seen as a bottleneck that slows deployments, so it is skipped. The paradox: skipping testing makes deployments slower (due to rollbacks and hotfixes).

Fix: Invest in automated testing. Write unit tests and integration tests for critical paths. Run tests automatically before deployment. Measure defect rate: if it is high, add more tests. As the test suite matures, tests become faster and more reliable than manual testing.

Prevention: Make testing non-negotiable. A deployment that has not passed the test suite does not go to production. Measure deployment quality (defects per deployment) and track it over time.

Mistake: No Automated Rollback Mechanism

Symptom: A bad deployment is made. Rolling back requires manual steps: reverting code, re-deploying previous version, restarting services. This takes hours. Users are impacted until the rollback is complete.

Root cause: Rollback is treated as an afterthought, not part of the deployment design. There is no automated rollback script or procedure.

Fix: Design every deployment with rollback in mind. Keep the previous version available, create an automated rollback script, and test the rollback procedure regularly (not just in an emergency). Measure rollback time in each test, and set a target your service levels can support.

Prevention: Every deployment plan should include rollback criteria (at what metrics do we roll back?) and rollback procedure (what commands, who approves, how long does it take?). Test rollback quarterly.

Mistake: No Canary Deployments or Staged Rollouts

Symptom: A deployment goes to all users at once. If there is a bug, all users are affected. The blast radius is the entire user base.

Root cause: Deployments are binary: all-on or all-off. There is no middle ground.

Fix: Implement staged rollouts: deploy to a small subset of users first, monitor for issues, then expand gradually. If issues are detected, halt the rollout while investigating. This limits the blast radius and gives you time to detect issues before they affect everyone.

Prevention: Build staged rollout into your deployment pipeline. Use feature flags to turn functionality on/off for subsets of users. Monitor metrics closely during rollout.

Mistake: Lack of Pre-Deployment Environment Parity

Symptom: A deployment works in staging but fails in production. Configuration differences, data differences, or infrastructure differences cause production failures that were not caught in staging.

Root cause: Staging and production are different. Staging is outdated, has less data, or has different infrastructure.

Fix: Use infrastructure-as-code to ensure staging and production are identical. Sync production data (or a sanitised copy) to staging regularly. Test deployments in staging before production using realistic data volume and configuration.

Prevention: Automate environment provisioning so that environments are created from code, not manually configured. Use configuration management to ensure consistency.

Mistake: No Visibility into Deployment Status

Symptom: A deployment is in progress. No one knows: is it done, is it hung, did it fail? Teams are unsure what code is running in production.

Root cause: Deployment pipeline has no logging or visibility. Status is unknown until something breaks.

Fix: Log every deployment step: when deployment started, which version is being deployed, what commands are running, completion time. Use a dashboard to show current deployment status and recent deployment history.

Prevention: Build observability into the deployment process. Every deployment should be visible: to whom is it deploying, what version, what is the status, what is the expected completion time.

Mistake: Deployments Without Approval or Change Control

Symptom: Anyone can deploy to production at any time. There is no review, no approval, no documentation. A developer makes a change and deploys without telling anyone.

Root cause: Pursuit of speed without governance. No controls on who can deploy or when.

Fix: Implement change control: deployments require approval (peer review of code, approval from a release manager). Document deployments: who deployed what, when, what change was made. Use this to correlate deployments with incidents.

Prevention: Require pull request review and approval before deployment. Use a deployment tool that logs and tracks all deployments. Correlate deployment timeline with incident timeline to catch bad deployments quickly.

Mistake: Monitoring Lag and Slow Incident Detection

Symptom: A deployment breaks a critical metric (response time doubles, error rate spikes). It can take a long time to detect this and start rollback. Users are impacted for that entire period.

Root cause: Monitoring is not real-time. Metrics are collected but checked infrequently. Thresholds are not set correctly.

Fix: Set up real-time monitoring with alerting. Critical metrics (error rate, response time, availability) should alert soon after a threshold breach, with thresholds set by the platform owner. During deployments, watch these metrics like a hawk. Automate rollback if metrics degrade significantly soon after deployment.

Prevention: Monitor closely during deployments. Have on-call engineers watching deployments. Define alert thresholds based on historical baselines and test them during staging deployments.

Pipeline failures that have little to do with the code

When a release pipeline keeps failing, the instinct is to look at the build and the tests. Those are worth checking, but many ServiceNow delivery problems come from the wiring around the code: the variables that point a job at the wrong instance, the account that runs the deployment, and the record that claims a release happened when the platform says otherwise. The mistakes below are the ones that often do not appear in a standard list of testing and rollback advice, and they tend to show up only once a team has more than one application or instance in play.

Ordering problems when configuration moves between instances

Update sets and similar packages carry an order, and the order matters more than most teams expect. A set that adjusts a field can be applied before the set that creates the field, or a set can silently overwrite a change that a different team made in the target instance last week. The failure is quiet because each package installs without error. The practical defense is to record the order of application as part of the release, to check the target instance for changes that were made outside the pipeline, and to avoid bundling unrelated configuration into one package just because it is convenient to ship together.

A short review to run before the next release

A team that wants to find these mistakes in its own pipeline can run a quick review. The steps are written to be worked through in a single sitting by someone who knows both the pipeline and the instance.

1. List every environment-specific value in the pipeline definition and confirm each one comes from a variable, not from the file.
2. List the accounts that run each job and check that no account is shared across unrelated applications.
3. Pick the last successful release and compare the version recorded in the release tool with the version on the instance.
4. Find any test that passed only after a retry in the last month and assign it an owner.
5. Check that each application scope has one pipeline allowed to write to it.

Why does the release record disagree with the instance?

Often because a step after the recorded success failed, or because a manual change was made after the release. Either way the record is describing intent, not state. The fix is a verification step that reads the platform and writes its answer back to the record.

Is a shared technical account always a mistake?

It is a risk that should be justified. A shared account can be acceptable for a small team with one application, but it stops being defensible as soon as the team must show which job changed which record.

How much time should a failing pipeline get before someone looks at it?

A failing pipeline should have a named owner who sees it immediately. Waiting for the next scheduled review is how a small defect becomes a blocked release for several days.

The update sets and release management guide covers how update sets are captured and moved between instances, and the change validation evidence guide covers what a validation record should show. The approval side of the same problem is covered in the DevOps governance model.