[OR] [P2] [S3] [ITo] [C12] Testing and Validation of Impact Tolerances

[P2] [S3] Chapter 12

Testing and Validation of Impact Tolerances

Introduction

Defining impact tolerance is only meaningful if it can be demonstrated, tested, and validated in practice. Regulators increasingly expect organisations to provide evidence that they can operate within defined tolerances under disruption—not just document them.

Testing and validation transform impact tolerance from a theoretical threshold into a proven capability. Through structured exercises and scenario-based testing, organisations can measure actual performance, identify gaps, and refine both their tolerances and resilience strategies.

This chapter focuses on how to test and validate impact tolerances using practical approaches aligned with operational resilience implementation.

Purpose of the Chapter

The purpose of this chapter is to:

• Validate impact tolerances through structured testing and exercises
• Apply scenario testing aligned with Operational Resilience Phase 2 – Stage 4 (OR-P2-S4)
• Use tabletop and simulation exercises to test decision-making and response
• Conduct end-to-end CBS testing to assess real service resilience
• Measure actual performance against defined tolerance thresholds

Scenario Testing (OR-P2-S4)

Scenario testing is the primary mechanism for validating whether an organisation can remain within its defined impact tolerances.

Role of Scenario Testing

Scenario testing enables organisations to:

• Assess whether CBS can operate within tolerance under stress
• Identify weaknesses in dependencies and recovery strategies
• Validate assumptions made during tolerance setting
• Provide evidence for governance, audit, and regulatory review

Alignment with OR-P2-S4

Within the operational resilience methodology:

Scenario Design Principles

Effective scenario testing should:

• Be severe but plausible
• Reflect realistic operating conditions
• Test end-to-end service delivery
• Include multiple dependencies (technology, people, third parties)
• Simulate time-based escalation of impact

Tabletop and Simulation Exercises

Testing can be conducted at different levels of complexity and realism.

Tabletop Exercises

Tabletop exercises are discussion-based sessions where stakeholders walk through a disruption scenario.

Key Features:

• Facilitated workshops involving cross-functional teams
• Focus on decision-making, escalation, and coordination
• No live system disruption

Benefits:

• Low cost and easy to organise
• Effective for testing governance and communication
• Identifies gaps in roles, responsibilities, and procedures

Simulation Exercises

Simulation exercises are more advanced and may involve:

• Partial system testing
• Simulated data or transaction flows
• Real-time response activities

Key Features:

• Higher realism compared to tabletop
• Tests operational execution and response capability
• May include controlled disruption scenarios

Benefits:

• Provides more accurate validation of capabilities
• Tests both technical and operational resilience

Comparison

CBS End-to-End Testing

End-to-end testing is critical to validating impact tolerance.

What is End-to-End Testing?

End-to-end testing evaluates the ability of the organisation to deliver a CBS across the full service chain, including:

• Front-end customer interaction
• Internal processing
• Technology systems
• Third-party dependencies
• Output delivery (e.g., transaction completion)

Why It Matters

Testing individual components is insufficient because:

• Failures often occur at integration points
• Dependencies are interconnected
• Recovery of one component does not guarantee service restoration

Example

For Payments and Funds Transfer Services, end-to-end testing may include:

1. Payment initiation via digital channel
2. Authentication and validation
3. Routing through the payment gateway
4. Clearing and settlement
5. Customer notification

The test must confirm that the entire process can operate within impact tolerance thresholds.

Measuring Actual vs Defined Tolerance

Testing must include quantitative measurement of performance against defined tolerances.

Key Measurement Areas

Example Comparison

Interpretation

• Within tolerance: Capability validated
• Near tolerance: Risk identified, monitoring required
• Breach: Immediate remediation required

Identifying Gaps and Weaknesses

Testing reveals gaps that may not be visible during planning.

Common Gaps

Root Cause Analysis

Each gap should be analysed to determine:

• Why was the tolerance breached
• Whether assumptions were incorrect
• Whether dependencies were underestimated
• Whether controls are inadequate

Remediation and Improvement

Testing results must lead to actionable improvements.

Example Actions

Continuous Improvement Loop

Testing feeds into a continuous improvement cycle:

1. Define impact tolerance
2. Test through scenarios
3. Measure outcomes
4. Identify gaps
5. Implement improvements
6. Re-test and refine

Governance and Reporting

Testing and validation must be supported by strong governance.

Reporting Requirements

• Test objectives and scope
• Scenario description
• Results vs defined tolerances
• Identified gaps
• Remediation actions
• Timeline and ownership

Escalation

• Breaches must be reported to Senior Management
• Critical issues may require Board-level visibility
• Regulatory reporting may be required in certain cases

Common Challenges

Best Practices

• Test end-to-end CBS, not isolated systems
• Use severe but plausible scenarios
• Combine tabletop and simulation exercises
• Measure performance using quantitative metrics
• Involve cross-functional stakeholders
• Document results and maintain audit evidence
• Integrate testing into regular resilience programs

Testing and validation are critical to ensuring that impact tolerances are credible, achievable, and aligned with real-world conditions. Through structured scenario testing, tabletop exercises, simulation activities, and end-to-end CBS validation, organisations can assess whether they can truly operate within defined thresholds.

By measuring actual performance against defined tolerances, organisations gain valuable insights into their resilience capabilities, identify weaknesses, and drive continuous improvement. This process not only strengthens operational resilience but also provides the evidence required to satisfy regulatory expectations.

Ultimately, testing transforms impact tolerance from a static definition into a living, validated capability, ensuring that organisations are prepared to manage disruption effectively and maintain critical services within acceptable limits.