ServiceNow Automated Test Framework Requirements Workshop Questions

These workshop questions settle five things before any Automated Test Framework test is written: what the tests are meant to protect, where test data and environments come from, who owns and reviews the suite, how results gate deployments, and whether performance and scale are in scope. Each theme names who should be in the room, and the closing sections turn the answers into design decisions.

Theme: Test Strategy and Scope

These questions establish what automated testing is supposed to accomplish.

Q: What are the top three application risks you worry about today? Answers might include: configuration changes breaking core workflows, integrations failing silently, user access permissions being unintentionally changed, performance degradation, or data corruption. The answer tells you what automated tests should focus on.

Q: What percentage of today’s testing is manual? If most of testing is manual, Automated Test Framework is high-ROI. If testing is already highly automated via non-ServiceNow tools, be careful not to duplicate effort or create conflicting test suites.

Q: Which ServiceNow modules are in scope for testing? Testing everything is too broad. Focus on modules that have high failure impact or high change frequency (e.g., Incident Management is usually higher priority than a rarely-changed reporting module).

Q: What is the acceptable test coverage? the majority coverage (testing the majority of common scenarios) is a common target. Full coverage (testing every edge case) is usually not cost-effective. Discuss where you want deep coverage (critical workflows) versus shallow coverage (nice-to-have features).

Q: How often should tests run? After every code deployment is ideal but may be impractical. Daily, three-times-weekly, or weekly are common alternatives. The answer depends on deployment frequency and risk tolerance.

Theme: Test Data and Environments

These questions address how to set up and maintain test data, which is often the largest effort in test automation.

Q: Do you test against production data, a refresh of production data, or synthetic test data? Production data is realistic but raises privacy concerns (test data may contain real customer information) and can’t be modified during testing. Production refreshes (a copy made weekly or monthly) are a middle ground. Synthetic data (test data created specifically for testing) is clean and safe but must be maintained and may not catch edge cases in real data. Discuss the trade-offs.

Q: How often can you refresh the test environment? If configuration in the test environment diverges from production (different field definitions, different workflows), tests pass in test but fail in production. Lock the test environment to match production weekly, or accept the risk. If you can only refresh monthly, plan tests to avoid data conflicts.

Q: Do tests need to clean up after themselves? If Test A creates an incident and Test B also creates an incident, and both incidents remain in the system, tests may interfere with each other (search results include previous test data). Decide whether tests clean up (delete their test data at the end) or the test suite resets the environment between runs. Cleanup is faster but riskier; reset is slower but safer.

Q: How do you handle tests that need data from external systems? If a test needs to verify a ServiceNow-to-Salesforce integration, it may need to create test data in Salesforce, verify sync, then clean up. Decide whether tests can modify external systems or should use mocked data instead.

Theme: Test Organization and Governance

These questions clarify who owns testing and how test decisions are made.

Q: Who owns the test suite? Is it the platform team, the application team, or a dedicated testing team? This affects how tests are updated (when a feature changes) and who determines test coverage.

Q: What is the process for updating tests when a feature changes? If a team member modifies a workflow and tests fail as a result, who fixes the test? The team member who made the change, or the test owner? Establish a clear ownership model.

Q: How do you handle tests that fail in automation but pass manually? These are often environment-related issues (test data missing, timing issue) or tools issues (browser automation clicking the wrong button). Decide on a triage process: how long do you investigate before declaring the test too fragile to maintain?

Theme: Deployment Integration and Guardrails

These questions address how test results affect deployment decisions.

Q: Can code be deployed if tests fail? Ideal answer: no. If a small minority of tests fail, deployment is blocked and the team must either fix the code or fix the test before proceeding. This prevents broken code from reaching production. However, if many tests are flaky (fail randomly), blocking on test failure becomes a burden. Decide on your tolerance for flakiness.

Q: What is the approved escalation if a production deployment breaks? If tests were supposed to catch an issue but missed it, you need a rapid response: rollback, hotfix, or configuration reversal. Define what recovery looks like in advance so the on-call team can execute quickly.

Q: Are there tests you want to run more frequently than others? Critical-path tests (the most important workflows) should run on every deployment. Nice-to-have tests (reporting improvements, cosmetic changes) can run less frequently. Create test suites by frequency to optimize feedback speed.

Theme: Performance and Scalability Testing

These questions identify performance concerns that automated testing should address.

Q: What performance SLAs do you need to meet? Examples: incident creation should complete in under 2 seconds, searches should return results in under 5 seconds. These targets drive test design.

Q: What is your peak load? If 1,000 users log in simultaneously at 8 a.m., does the system stay responsive? Performance tests should simulate expected peak load, not just average load.

Q: When should load testing be run? Load tests are heavy and time-consuming. Run them before major deployments, or monthly to detect performance degradation. Don’t run load tests on every change or the feedback loop becomes too slow.

Common Design Decisions from Workshop Answers

If most answers emphasize preventing configuration mistakes, prioritize tests that verify core workflow behavior and user permissions. If data quality is a concern, emphasize integration tests that validate data correctness end-to-end. If performance is a concern, invest in load testing. Your test strategy should reflect organizational priorities, not generic best practices.

Document the decisions made in the workshop so new team members understand why tests are organized a certain way.

Turning workshop answers into design decisions

A question list only helps if the answers are written down as decisions the build team can act on. A common failure in workshops is a long discussion that ends with everyone agreeing the topic was important and nobody recording what the suite should now do. Plan the session backwards from the outputs: a short list of scope choices, a named owner for each one, and a date for the first review.

Sequence the session so that each topic feeds the next. Scope comes first, because it decides which questions are worth asking about data and cadence. Governance comes next, because ownership decides who can answer for test data. Gating comes last, because it is the decision that people are most tempted to rush.

How answer patterns point to design choices

Certain patterns in the answers are more informative than any single answer. The table below shows the signals the room should watch for and the design choice each one often points toward. It is a starting position for discussion, not a rule that replaces the team’s own judgment.

Running the session so the outputs are usable

1. Send the questions to facilitators in advance and ask each one to bring the evidence they rely on, such as recent change records or a list of recent defects.
2. Open with the top risks answers, and ask the room to rank them. Ranking forces trade-offs that a list of equal items hides.
3. Capture each decision in a single sentence with the reason beside it, and mark anything unresolved as an open item with an owner.
4. Stop before gating is designed. Deciding what blocks a release needs the scope and ownership answers first, and a rushed gate decision often gets reversed within a cycle.
5. Circulate the decision record within the working week, while the answers are still fresh.

What the workshop should not decide

A requirements workshop sets direction. It should not try to settle the exact content of every test, the choice of individual locators, or the tooling for reporting. Those decisions belong to the build team, who will learn more in the first few weeks of writing tests than the room could predict. Keep a parking list for them, and make sure the parking list is reviewed at the first build checkpoint.

Checklist for closing the workshop

• Every scope choice has a one-sentence decision and a stated reason.
• Every test suite proposed has a named owner or an open item for finding one.
• Data and environment questions are linked to the decisions they affect.
• The gating position is recorded as a draft, with the conditions under which it will be revisited.
• The decision record is stored where the build team can find it.

Questions people ask about running the workshop

Should business users attend, or only technical people?

Both, but for different parts. Business process owners are the right people to rank risks and name the workflows that must not break. Technical owners are better placed to say what the platform can reliably test. Give each group the section where its knowledge matters, and bring them together for the scope ranking.

What if the answers contradict each other?

Record the contradiction as an open item rather than choosing the answer you prefer. Contradictions often reveal two teams who hold different views of the same process, and resolving them is a governance task, not a design task.

How long should the decision record be?

Short enough that someone new can read it in one sitting. A long record that nobody opens is worth less than a page that answers why the suites are shaped the way they are.

Can the same questions be reused for a second product area?

The questions transfer, but the answers rarely do. Reuse the structure of the session and rebuild the answers with the second area’s owners, because data sensitivity and change frequency often differ sharply between areas.

Teams preparing this session often ask for the question set to be adapted to their platform before it goes out.

For the design work that follows, the access design notes take up the data decisions the workshop raises, and the governance model sets out how ownership is recorded. The testing strategy article covers the evidence a release should carry.