Troubleshooting ServiceNow Upgrade Readiness: A Practical Guide
Pre-upgrade assessment and planning
ServiceNow releases new versions periodically. Upgrades bring new features, security patches, and sometimes breaking changes. CIO office analysts planning upgrades must assess readiness before committing to a date. Plan for the skipped records that the upgrade leaves unapplied on your first non-production instance. The upgrade skips customized records instead of overwriting them, and you resolve those conflicts there, so the update sets you create can be imported before the production upgrade. Also inventory custom code and integrations: third-party applications, REST API consumers, and custom flows depend on ServiceNow APIs. Check whether those applications are compatible with the target release. Vendor release notes often clarify compatibility.
Backup and recovery procedures
Before any upgrade, ensure a backup of the current instance exists and can be restored. Confirm that a usable backup of the instance exists in your own backup retention system, and check which rollback and backup options your release provides. There is no blanket rollback for a move from one release family to another. Rollback is available only between patches within the same software release, and by default only within 10 days of the most recent upgrade or plugin activation. Restoring a production instance from backup is the final option when a problem cannot be solved any other way. Develop and test a rollback procedure: if the upgrade fails catastrophically, how will you restore to the previous version? Practice the rollback in a non-production environment first. Know how long restoration takes so you can plan maintenance windows. For critical systems, having a warm standby environment ready to assume traffic can minimize downtime if the primary upgrade fails.
Staging environment testing
A staging environment running the same configuration as production should be upgraded first to test critical functionality. After the staging upgrade, run a comprehensive test plan covering critical workflows: incident creation and closure, change request approval, service catalog requests, user provisioning. Test integrations with external systems to ensure they still work. Test custom reports and dashboards to confirm they display correctly. Test authentication to ensure SSO or custom authentication still works. If any test fails, investigate before upgrading production. Failed tests in staging are likely to fail in production as well.
Custom code and plugin compatibility
Custom JavaScript, workflows, and plugins may break after upgrade if they depend on internal ServiceNow APIs that changed. Before upgrade, audit custom code for dependencies on potentially changing features. Check release notes for deprecation notices. If custom code depends on deprecated features, plan to refactor it. Some custom code may work as-is but perform differently after upgrade. Performance regression testing is important: compare transaction timing before and after upgrade for critical workflows.
Instance configuration validation
After upgrade, the instance configuration may need adjustment. Security policies may have changed. Changes that ship with an upgrade can conflict with your customizations. Email server configurations might need reconfiguration. Check release notes for configuration changes. Re-test access controls to ensure users still have appropriate permissions. Re-test notification settings to ensure emails still go to the right recipients. If you run multiple instances (development, staging, production), ensure all are upgraded consistently or have a documented reason for version differences.
Cutover and monitoring
Schedule the production upgrade during a maintenance window. Communicate the maintenance window clearly so users plan accordingly. Monitor the upgrade process closely, since unusual configurations sometimes cause issues. Have a ServiceNow support contact available during the upgrade in case problems arise. After upgrade completes, run a smoke test on critical functions before allowing users back on the system. A smoke test is a quick check that key workflows work: can users log in, can they create incidents, can they view dashboards? After users return to the system, monitor for issues. Have a clear escalation path so users can quickly report problems.
Post-upgrade issues and patches
Upgraded instances sometimes experience issues that only appear under real-world load. A workflow that works fine in testing may fail when thousands of users access it. If issues appear after upgrade, gather information: which workflow or feature is failing, what is the error, which users are affected? Compare it with the Upgrade Summary Report and the release notes, and search the known error articles to see if this is a known issue. ServiceNow releases patches to fix discovered issues, so stay current with patches. Also use release notes to understand any behavioral changes that affect users. A new default setting may change how users interact with forms or dashboards, requiring user communication and possibly training.
Start from the symptom, then choose which layer to inspect
Upgrade problems are easier to diagnose when they are sorted by what the team observes rather than by where they appear in the plan. The table groups the common symptoms by layer, so that the first place to look is set by what is failing rather than by the last test that ran.
| What is observed | Layer to inspect first | Evidence to collect before changing anything |
|---|---|---|
| The upgrade process halts or reports an incomplete step | The upgrade job itself and its step log | The step name, the time it stopped and the last successful step |
| An error appears only for certain users or roles | Access control rules and role inheritance | The affected role list and one failing record’s access trail |
| A script runs but produces a different result than before | Custom code that depends on a platform behavior that changed | The script’s input, its output on both versions and the release note that describes the change |
| Everything works but is noticeably slower | Query patterns, indexes and background loops | Timed transactions for the same action, before and after the window |
| An external system stops receiving or sending data | The integration’s credentials, endpoint and message definition | The last successful message and the response it received |
When an upgrade stalls partway through
A stalled step is tempting to re-run, and re-running is often the wrong first move. A second attempt against a half-changed instance can make the problem harder to read. The sequence below protects the evidence while the call is being made.
- Freeze the state. Stop any further changes on the instance, including fixes that someone is tempted to make in the middle of the window.
- Record the step name, the time it stopped and the last step that completed. Write these down before anyone speaks to the vendor, because the vendor will ask for them.
- Open a support case with the time-stamped evidence, and ask the support engineer to confirm whether the stalled step can be resumed or must be restored.
- Agree in advance how long the team will wait before it decides between resuming and restoring, and write that limit into the runbook so the call is not left to whoever is loudest.
- Communicate the status to the business owner at fixed intervals, including when there is nothing new to report.
When staging passes and production fails
A failure that appears only in production usually has one of four causes. Data volume differs, so a query that was instant on a small copy takes long on the full table. Concurrency differs, so a race condition that never occurred with one tester appears under real load. An environment property differs, such as a mail server name, a certificate or an endpoint address that was never copied across. Or a scheduled job runs at a time that the staging refresh did not reproduce. Start by checking which of the four applies, because the fix for each is different and the wrong one wastes the window.
When the problem is slowness rather than an error
Slowness is harder because nothing reports as failed. Compare timed transactions for the same action before and after the window, using the same user role and a similar record. If one form or list is slow in isolation, look at the queries it issues rather than the form itself. If the whole instance is slow, check background activity first, since a loop that was harmless on the old release may now consume more capacity. The platform performance guidance sets out the usual sources of this kind of load.
When to stop troubleshooting and call the rollback decision
Troubleshooting has a natural end point, and it should be named before the window opens. Call the rollback decision when a critical process cannot be made to work within the agreed time, when data integrity is in question, or when the cause is a platform defect with no workaround and no vendor date for a fix. Continuing to troubleshoot past those points tends to produce a longer outage and a rollback that is harder to perform, because the instance has accumulated changes that the restore will not reverse cleanly.
The testing and acceptance checklist for upgrades lists the processes that count as critical, and the roles view shows who makes the call. When the failure involves a workflow that moves work between teams, the guidance on coordinating a response across teams covers how to keep the response from fragmenting.
Frequently asked questions about upgrade troubleshooting
Should a restart be done after a failed step?
Not without the platform support team’s agreement. A restart can clear a transient fault, but it can also interrupt a step that was about to complete and leave the instance in a state that is harder to read. Ask before restarting, and record the answer.
How do we know a fix is complete?
Re-run the test that found the fault, then run the neighboring tests that depend on the same component. A fix that passes its own test but breaks a neighbor is a common outcome when the fault is traced to a shared setting.
Who should be on the bridge during a failed window?
The platform owner, the release lead, one developer for each failed component and a representative of the business process at risk. Keep the group small enough to make decisions quickly, and have someone whose only job is to keep the evidence log current.
Troubleshooting ServiceNow Discovery false failures after an upgrade
Discovery failures right after a ServiceNow upgrade are usually not real device problems. They typically trace to a MID Server that has not finished upgrading, a credential that changed, so Discovery retries the credential list, a customized pattern or probe that did not carry over, or a timeout the run now exceeds. Check them in that order.
Start with the MID Server, because Discovery scans run through it. After an upgrade it normally upgrades itself to match the instance, and a run started during that window can fail healthy devices. Confirm each MID Server is up, validated and on the instance’s version before reading a failure.
Then pick one device that failed but was discovered cleanly before the window, run Discovery against it alone and read its log. The error text sorts the problem.
| What the log shows | Likely cause | First check |
|---|---|---|
| Authentication failed on a device that worked before | A device’s credential was changed, so Discovery retries the available credentials | Which credential the log shows being tried |
| Device found but classified as a generic type | A customized pattern or probe was not applied | The skipped changes recorded during the upgrade |
| Timed out or no response | The network path or run length changed | The MID Server’s connection to that subnet and the time limit |
| Duplicate or missing configuration items | An identification or reconciliation rule changed | The rule set, tested on one record |
Widen the test only after that device passes. Re-run one subnet and compare the devices found with the last clean run. A gap of a few devices points to individual cases; a gap across a subnet points to the MID Server or the credential.
The Discovery guide explains credential and schedule settings, the MID Server guide covers its health and version, and Discovery and Service Mapping delivery shows how these runs are validated. If failures spread beyond Discovery, see platform performance and upgrade readiness.
Can a MID Server cause Discovery failures right after an upgrade?
Yes. A MID Server that is still upgrading, has lost its validation or runs a different version from the instance can fail healthy devices, so confirm its status and version first.
How do I tell a false failure from a real one?
Test the same device by a method that does not use Discovery, such as a direct sign-in with the same account from the MID Server’s network. If that works and Discovery fails, the cause sits in the Discovery configuration. If both fail, the device or the network has changed.
Should I re-run the full schedule to see whether the failures clear?
Not first. A full run on a half-understood problem adds noise and can change the records you want to compare. Fix the single device, confirm one subnet, then run the full schedule.
Prove Discovery false failures after an upgrade with a pre-upgrade baseline
A Discovery false failure after an upgrade is a scan that reports a fault the device does not have. The quickest way to prove it is to compare the failing scan with a baseline taken before the upgrade, for the same device, through the same MID Server. Without it, the team argues from memory.
Build the baseline in the quiet period before the upgrade. Choose six to ten devices that represent the estate: a laptop, a server, a network device, a printer, a virtual machine and one device that has been noisy in the past. For each one, record the scan outcome and the attributes the scan wrote back, then save the list outside the instance so it survives a rollback. Use the same MID Server and credentials that production uses, otherwise the comparison measures the path rather than the upgrade. If you are unsure how those credentials are scoped, read how to govern and scope Discovery credentials first.
Avoid two traps when you build it. The first is a device that is already failing, which gives a baseline with no value, so check that each chosen device scans cleanly before you record it. The second is changing the MID Server version in the same window as the platform upgrade. When both change together, you cannot tell which one caused a difference, so move the MID Server on its own schedule where you can.
After the upgrade, scan the same devices in the same order and compare them to the baseline field by field. Three patterns are worth checking first. If the scan fails on every device, look at the MID Server and its path to the network first. If it fails on one device class only, look at the probe that handles that class. If the scan succeeds but the attributes differ, the problem is in the data, so compare the changed values with the release notes before deciding whether the change is a regression.
Record each comparison in the change ticket, not in someone’s notes. One line per device is enough. Keeping this record also gives the next upgrade a reference point, since the first comparison shows which devices behave differently. If the comparison exposes a probe you cannot repair in house, Discovery and service mapping support is the next step to scope.
Protect the CMDB while Discovery false failures after an upgrade are being sorted out
While you are chasing Discovery failures after an upgrade, the first job is to stop the failed runs from changing your configuration data. Where a related-CI deletion strategy is set to Delete rather than the default Keep, a scan that still finds the main CI but cannot find that related CI can remove it from the CMDB, and that removal can outlast the failure itself.
The reason is simple. A device that Discovery could not reach looks, to the data, the same as a device that has gone. If anything downstream treats a missed scan as evidence of retirement, a login problem on a MID Server becomes a data problem across the whole estate.
Hold the actions that read a failure as a fact
- Pause any scheduled job that marks records stale, retires them or removes them, until the first clean run after the window is confirmed.
- Do not accept bulk requests from teams who see “not discovered” on their servers and want records deleted or recreated by hand. Manual copies are a common source of duplicates once Discovery recovers.
- Keep a note of which schedules ran during the window, so that every failure in that period can later be labelled as suspect rather than real.
Decide in advance who may declare a failure real
In most implementations nobody owns this call, so it falls to whoever is loudest. Name one person from the platform side and one from the infrastructure side. A failure counts as real only when both agree. Until then it is recorded as unconfirmed and left alone.
Reconcile after recovery, not during it
Once a clean run completes, compare the records it touched with the state before the window. Look for devices that now show two records, for fields that were blanked by a partial scan, and for relationships that disappeared because one end of the link was not reached. Duplicate handling is covered in the guidance on reconciliation and duplicate control, and the wider routines for keeping records trustworthy sit under CMDB data quality and governance.
Measure the damage rather than guessing at it. Counts of records by last-discovered date, before and after, show quickly whether the window left a gap. The CMDB metrics that matter explain which of those counts are worth tracking afterwards.
Applying a vendor patch or workaround to production after an upgrade needs its own change record
A vendor patch or a documented workaround for a post-upgrade fault should reach production only after it has passed staging, been tested against the original failure, and been approved as a change in its own right. Applied as a quick fix mid-window, it often adds a second fault, and the team cannot tell which change caused which symptom.
The risk is interaction. A patch edits code on a platform the upgrade has already changed, so the two can meet in ways the release notes may not describe. A workaround, such as an altered configuration value or a temporary business rule, can clear the symptom it targets while changing a process another team relies on.
Before anything is applied, write down four things: the symptom it addresses, the test that proves the symptom is gone, the neighbouring tests that share the same component or setting, and the person who approves the change. If the team cannot name the neighbouring tests, the staging run has not yet covered the fault. Check the patch’s release note against the release your instance runs.
Should a workaround go in before the patch is confirmed?
Generally only when a business process is blocked and no other route exists. Put it in as a separate, documented change with a named owner and a date for removal. A workaround with no removal date can stay in place long after the patch that made it unnecessary has arrived, and then it becomes the thing the next upgrade breaks.
- Apply the patch to staging and rerun the test that found the fault.
- Run the neighbouring tests before the production window opens.
- Record the change with its rollback step written out in full.
- Schedule the change so the team can watch the first real transactions afterwards.
Where your organisation already runs formal release control, the post-upgrade patch should follow the same route as any other production change. The change governance and release management approach sets out the approval steps, and the guide to troubleshooting update sets covers what to check when a configuration change does not move cleanly between instances.
Finally, close the loop. Once the patch is in and the neighbouring tests pass, remove any temporary workaround and note in the upgrade record which fault each change closed. That note is what a future upgrade review will want to see.
Frequently asked questions
What should we do if the Upgrade Health Dashboard shows high-severity issues?
Resolve high-severity issues before upgrade. They often indicate custom code or configurations incompatible with the new release. Work with ServiceNow support to understand the issue and fix it in staging before upgrading production.
How do we rollback if the upgrade fails?
ServiceNow provides backup snapshots taken before upgrade. If the upgrade fails, restore from the backup. Test this procedure in a non-production environment first to know how long restoration takes.
Why might custom workflows break after upgrade?
Custom workflows may depend on internal ServiceNow APIs that changed. Check release notes for deprecation notices. Audit custom code before upgrade to identify potential issues. Refactor code that depends on deprecated features.
Related guides
Related Oblytech services
Where this topic turns into delivery work, these are the Oblytech services that cover it.
Planning, running or fixing a ServiceNow programme? Tell us where you are and we will suggest a sensible next step.
Contact us