Troubleshooting ServiceNow Event Management: A Practical Guide
This guide works through the symptoms that break Event Management: events that stop arriving, alert floods, incidents that fail to correlate, escalations that misfire and incidents left open. It shows where to look first and gives an ordered procedure for a missing or misrouted incident.
Symptom: Events Stop Flowing Into ServiceNow
Events are normally flowing into ServiceNow, then suddenly stop. First, verify that monitoring systems are still generating events: check monitoring dashboards to confirm that events are being created and alerts are firing. If monitoring is generating events but ServiceNow is not receiving them, the issue is in the ingestion path.
Check the integration between monitoring and ServiceNow: are API credentials still valid, or have they expired? Is the integration account disabled or permissions revoked? Verify network connectivity from monitoring system to ServiceNow: is there a firewall rule blocking the traffic, or has network configuration changed?
Check ServiceNow Event Management configuration: is the event ingestion rule enabled? Has someone accidentally disabled it during maintenance? Check the event import logs to see if there are error messages indicating why events are not being imported. If API errors appear, address the root cause (credentials, permissions, network) and restart the integration.
Symptom: Too Many Events, Alert Fatigue
Events are flowing in, but volume is overwhelming: thousands of events per minute creating incidents faster than teams can resolve them. This is a data quality problem, not a ServiceNow problem.
Start at the source: are monitoring systems generating too many events? Check monitoring configuration: are thresholds too sensitive (alerting on minor deviations)? Is polling frequency too high (creating duplicate alerts)? Work with monitoring teams to reduce the volume of events at the source.
If you cannot reduce volume at source, add filters in ServiceNow: create ingestion rules that drop low-severity or informational events, or that deduplicate events received within short time windows. However, filtering in ServiceNow should be a last resort; better to fix monitoring configuration.
Symptom: Incidents Not Correlating Correctly
Events are flowing in but are not correlating as expected: either too many incidents are being created (events not correlating that should), or events are being over-correlated (unrelated events grouped together).
Review the correlation rules: are they appropriate for your environment? Test a specific rule by creating sample events that should match it and verify that correlation occurs. If correlation does not happen, the rule expression may be incorrect or the event attributes may not match expected values.
Check event attributes: do incoming events have the fields your correlation rules expect? If a rule correlates on service name but incoming events do not have a service name attribute, correlation fails silently. Log sample events and inspect their structure to ensure rules match event structure.
Symptom: Escalation Not Happening or Happening Incorrectly
Incidents are created but escalation rules are not triggering or are escalating to the wrong people. Check the escalation rule configuration: is it enabled? Are the conditions (incident age, priority, assignment group) matching the actual incidents?
Create a test incident that matches the escalation criteria and trace whether escalation occurs. If it does not, the rule condition may be too strict. If escalation occurs but to the wrong group, the escalation assignment logic may be incorrect.
Also verify that incident notifications are working: even if escalation creates an assignment, the assignee must be notified. Check notification rules and verify that the escalated user is receiving notifications.
Symptom: Incidents Remain Open Indefinitely
Incidents are created and assigned but are never resolved: they accumulate, and incident lists become cluttered with old tickets. The issue is usually that no one is responsible for closing incidents or no automatic closure rule exists.
Check incident closure workflow: when an incident should be resolved, what triggers closure? Is there an automatic closure rule (if there is no activity for a set number of hours, auto-close), or must incident handlers manually close? If automatic closure exists, verify it is working: check if incidents meeting closure criteria are being auto-closed or if they are stuck.
If automatic closure is not configured, consider adding it: incidents from monitoring systems that have not been updated within an agreed period can be auto-closed. Add a reassignment step before closure so incidents do not auto-close without review, but automate the routine closures to prevent accumulation.
Symptom: Performance Degradation When Importing Large Event Volumes
Event ingestion is slow when large volumes arrive simultaneously, or incident creation is lagging behind event flow. This suggests insufficient capacity or inefficient processing.
Check the event ingestion job: how long does it take to process a batch of events? If it is slow, check the Transform Map and business rules executing during import—are they efficiently coded? Are they querying large result sets? Optimize the import configuration.
Also check system capacity: is the ServiceNow instance reaching CPU, memory, or database limits during high event volume? If so, scaling the instance or optimizing database queries may be necessary.
Work the event trail backwards from the missing incident
When an on-call engineer opens a monitoring question and finds that no incident was created, the instinct is to open the event rules and start changing things. That habit often makes the problem harder to find. A faster route is to walk the trail the platform leaves behind, beginning with the thing the person can see and moving toward the source. Each step either confirms that a stage worked or shows exactly where the chain broke. The sequence below assumes an administrator with read access to the event, alert and task tables, plus access to the logs on any MID Server that collects for the affected source.
The order matters. A missing incident often sits at the end of a chain whose earlier links are healthy, so checking the incident first only restates what you already know. Starting at the raw event tells you whether the problem is an absence of data, a data problem, or a processing problem. Those three call for different owners and different fixes, and mixing them up is a common reason changes get made during an outage that do not help.
An ordered procedure for a missing or misrouted incident
- Pull the raw event records for the affected host or service across the window when the alert should have appeared. If none exist, the fault sits upstream of ServiceNow and the remaining steps can wait until the source is confirmed to be sending.
- Compare the event timestamps with the time the source system recorded the fault. A gap of a few minutes points to a queue or collection delay. A gap that runs for hours often points to a stalled connector, an expired credential or a MID Server that stopped checking in.
- Find the alert each event should have produced. An event with no alert has been filtered out, merged into another alert, or failed rule evaluation before it reached the alert stage.
- Read the event rule and alert rule evaluation for that event. Note which rule touched it, which fields it changed, and whether a later rule overwrote the result.
- Only then check the task. A task that exists but points at the wrong configuration item is a matching problem, and it belongs with whoever owns the CMDB data, not with whoever owns the alert rules.
Matching the evidence you find to the likely cause
The table below turns the trail into a short decision aid. It is deliberately coarse. The point is to decide who to bring in, not to diagnose the exact defect from a single row.
| What you find on the trail | What it usually means | Who to involve first |
|---|---|---|
| No raw events for the window | Collection stopped, or the source never sent the data | The team that runs the source integration, and the MID Server owner if one is involved |
| Raw events with late timestamps | Queue delay, a slow connector, or clock drift on the source host | The platform administrator |
| Events present, no alert | Events were filtered, deduplicated into an existing alert, or dropped by a rule | The Event Management administrator who owns the rules |
| Alert present, no task | Alert conditions were not met, or task creation is switched off for that alert type | The Event Management administrator |
| Task present, wrong configuration item | Matching on name, address or class did not resolve to the record the team expected | The CMDB data owner |
What not to change while the outage is still running
Under pressure, the temptation is to loosen a filter so the missing alert appears. Done during an outage, this floods the queue with the noise the rules were written to suppress, and it buries the one event that matters. Make any change in a non-production copy where you can, or write down the reason and the revert step before you touch a production rule. If nobody can explain why a rule exists, leave it alone and escalate. That rule may have been written to stop an earlier outage, and removing it can recreate that outage. The wider case for controlling alert volume is covered in a practical post on alert noise, which is worth reading before the next incident rather than during it.
Deciding when to stop troubleshooting and escalate
A troubleshooting session should have a stopping point. Agree one with your service owner before the incident starts, and keep it in the runbook so the on-call engineer does not have to decide it alone at night. Escalate when the raw event trail is missing for a source that carries a critical service, when the answer depends on credentials or network paths you cannot see, or when a fix would change how an entire class of events is handled. Each of those is a decision for someone with wider authority, and an event rule is rarely the right place to make it.
A handover is only useful if the next person can pick it up without repeating your checks. Include the following in the note you leave:
- The time window and the affected sources, stated in one line.
- The last stage of the trail you confirmed as healthy, with the record identifier you looked at.
- Every rule, connector or credential you changed during the session, and the exact revert step for each.
- What you could not check, and who holds the access needed to check it.
Checking whether the change held on a real event
A fix is confirmed when the same class of event produces the expected alert and task on a fresh occurrence. Clearing a backlog only shows that the queue moved, and it says nothing about whether the next real event will be handled correctly. If your team has a test source, replay a known event through the chain and walk the trail again. Otherwise, mark the fix as provisional and check the trail on the next genuine occurrence. Trend data on alert and task volume, covered in the metrics guide for event management, can show whether the change held over a longer period than a single incident.
Should the connector be restarted before the logs are read?
Not as a first move. A restart clears in-memory state and can remove the evidence you need to explain why events stopped. Read the connector and MID Server logs for the window first, record what you see, and restart only when the logs show a fault that a restart would clear. If a restart is unavoidable during a live outage, note the time and the state beforehand.
Why does an event exist but never become an alert?
The event record is written before the rules run, so its presence proves only that data arrived. The usual causes are a filter that treats the event as informational, a deduplication rule that attached it to an alert opened earlier, or a rule that was edited and no longer matches the event’s fields. Compare the fields on the event with the conditions on the rule that should have matched it. A field that is empty or spelled differently is a common reason for a silent miss.
Who should own the troubleshooting runbook?
The Event Management administrator should own the steps that touch rules, and the platform team should own connector and MID Server steps. Give each section a named owner and a review date that someone is accountable for. A runbook nobody reviews after a rule change will send the next responder down a path that no longer exists.
Talking through a recurring fault with a specialist
If the trail keeps breaking at the same stage, the pattern often points to a design issue rather than a bad night. A short review of sources, rules and ownership often recovers more time than another round of individual fixes. Event correlation work and the MID Server troubleshooting guide are the pages for the adjacent stages.
Frequently asked questions
How do we test Event Management correlation rules without impacting production?
Create a test table (Event Test) mimicking event structure but not flowing into production incidents. Write test events to it and verify transformation and correlation logic. Once validated, enable production ingestion. This allows safe testing before impacting real incident workflows.
What should we do if an event ingestion rule fails?
Check the event import log for errors: invalid structure, expired credentials, permission denied, or business rule exceptions. Address the root cause and retry. Consider adding error handling that routes failed events to a queue for manual review rather than silently dropping them.
How do we measure whether Event Management is working well?
Track metrics: events per minute, incidents per minute, average incident age, correlation ratio, and resolution time. Trends in these metrics indicate whether Event Management is functioning properly. Increasing incident age or decreasing correlation ratio indicate problems in configuration or capacity.
Related guides
Related Oblytech services
Where this topic turns into delivery work, these are the Oblytech services that cover it.
Planning, running or fixing a ServiceNow programme? Tell us where you are and we will suggest a sensible next step.
Contact us