[OR] [P2] [S4] [ST] [C9] Setting Testing Scope and Boundaries Introduction

[P2] [S4] Chapter 9

Setting Testing Scope and Boundaries

Introduction

Scenario testing must be carefully scoped to ensure that it is both focused and meaningful, while still capturing the complexity of real-world operations.

Without clearly defined boundaries, testing can become either too narrow—failing to uncover critical risks—or too broad—resulting in impractical and unmanageable exercises.

Setting the scope and boundaries of scenario testing is, therefore, a critical step in the operational resilience framework. It determines what is tested, how extensively it is tested, and under what conditions, ensuring alignment with organisational priorities, regulatory expectations, and resource constraints.

This chapter outlines how organisations can define the scope of scenario testing in a structured manner, ensuring that testing activities are targeted, realistic, and aligned with impact tolerance objectives.

Purpose of the Chapter

The purpose of this chapter is to define what will be tested in scenario testing by establishing clear scope, boundaries, assumptions, and success criteria aligned with Critical Business Services (CBS) and impact tolerances.

Selection of CBS and Sub-CBS

The starting point for defining testing scope is the selection of Critical Business Services (CBS) and their supporting sub-components.

Prioritising CBS for Testing

Not all CBS can be tested simultaneously. Organisations should prioritise based on:

• Criticality to customers and stakeholders
• Regulatory importance
• Financial and systemic impact
• Historical incidents or known vulnerabilities

High-priority CBS—such as Payments & Funds Transfer Services or Deposit and Account Services—should be tested more frequently and in greater depth.

Selection of Sub-CBS

Within each CBS, organisations should identify relevant sub-CBS to be included in the scenario.

Examples:

• Payment initiation
• Authentication and authorisation
• Transaction processing
• Settlement and reconciliation

This ensures that testing:

• Covers end-to-end service delivery
• Identifies vulnerabilities at a granular level
• Aligns with dependency mapping outputs

Coverage Strategy

Organisations may adopt different coverage approaches:

• Full CBS testing: End-to-end validation of all components
• Targeted sub-CBS testing: Focus on high-risk components
• Rotational testing: Different CBS tested over time

A structured coverage strategy ensures balanced and sustainable testing programmes.

Determining Scope (Unit, Enterprise-Wide, Ecosystem-Wide)

Scenario testing scope can vary significantly depending on organisational objectives and maturity.

Unit-Level Testing

• Focuses on a specific business unit or function
• Tests isolated processes or systems
• Typically used for initial or component-level validation

Advantages:

• Easier to manage
• Useful for early-stage testing

Limitations:

• Does not capture end-to-end interdependencies

Enterprise-Wide Testing

• Involves multiple business units and functions
• Tests end-to-end CBS delivery across the organisation
• Includes coordination between operations, IT, risk, and management

Advantages:

• Provides a holistic view of organisational resilience
• Identifies cross-functional dependencies

Limitations:

• More complex to plan and execute

Ecosystem-Wide Testing

• Extends beyond the organisation to include third parties, partners, and external stakeholders
• Tests inter-organisational dependencies and systemic risks

Examples:

• Payment network disruptions involving multiple banks
• Cloud service provider outages are affecting multiple systems

Advantages:

• Reflects real-world interconnected risks
• Aligns with increasing regulatory expectations

Limitations:

• Requires coordination with external parties
• May involve regulatory oversight

Choosing the Appropriate Scope

The scope should be determined based on:

• Testing objectives
• Maturity of the resilience programme
• Regulatory expectations
• Resource availability

Organisations should progressively evolve from unit-level testing to enterprise and ecosystem-wide testing.

Defining Test Assumptions and Constraints

Every scenario test operates within a set of assumptions and constraints that define its boundaries.

Test Assumptions

Assumptions simplify the scenario and provide a controlled testing environment.

Examples:

• Certain systems remain operational
• Key third parties are available or unavailable
• Data integrity is preserved or compromised

Assumptions should be:

• Clearly documented
• Realistic and justifiable
• Consistent with scenario objectives

Test Constraints

Constraints define the limitations of the test, including:

• Time constraints (e.g., limited test duration)
• Resource constraints (e.g., availability of participants)
• Operational constraints (e.g., avoiding disruption to live services)
• Regulatory constraints (e.g., restrictions on live testing)

Constraints ensure that testing is:

• Practical and manageable
• Conducted without unintended operational impact

Balancing Realism and Practicality

Organisations must strike a balance between:

• Realistic scenarios that reflect actual risks
• Controlled environments that allow safe and effective testing

Overly restrictive assumptions reduce realism, while overly complex scenarios may become unmanageable. Effective scoping ensures an optimal balance.

Establishing Success Criteria Aligned to Impact Tolerance

Defining success criteria is essential to evaluating the outcome of scenario testing.

Alignment with Impact Tolerance

Success criteria must be directly linked to impact tolerance thresholds, such as:

• Maximum tolerable downtime (MTD)
• Maximum tolerable data loss (MTDL)
• Acceptable service degradation levels

This enables objective assessment of:

• Whether CBS remained within tolerance
• Where breaches occurred
• The severity and duration of disruptions

Quantitative Metrics

Examples include:

• Time taken to restore service
• Volume of transactions processed during the disruption
• Percentage of service availability

Qualitative Metrics

Examples include:

• Effectiveness of decision-making
• Quality of communication and coordination
• Adherence to escalation protocols

Defining Clear Evaluation Criteria

Success criteria should be:

• Measurable (quantitative where possible)
• Aligned with business and regulatory expectations
• Agreed upon before testing begins

This ensures that results are:

• Objective and defensible
• Comparable across different tests
• Useful for continuous improvement

Setting the scope and boundaries of scenario testing is a critical step in ensuring that testing activities are targeted, effective, and aligned with operational resilience objectives. By carefully selecting CBS and sub-CBS, determining the appropriate scope, defining assumptions and constraints, and establishing clear success criteria, organisations can ensure that their scenario testing delivers meaningful and actionable insights.

A well-defined scope enables organisations to balance depth and breadth, ensuring that testing is both practical and comprehensive. It also provides a clear framework for evaluating outcomes and driving continuous improvement.

Ultimately, effective scoping transforms scenario testing from a broad conceptual exercise into a focused and disciplined capability, enabling organisations to validate resilience where it matters most—at the level of critical services and real-world disruptions.