[OR] [P2] [S4] [ST] [C10] Metrics, Evaluation & Continuous Improvement

Chapter 10

Metrics, Evaluation & Continuous Improvement

Introduction

Testing and exercising only deliver value when organisations can measure performance, evaluate outcomes, and drive improvements. Without a structured approach to metrics and evaluation, exercises risk becoming one-off activities that generate limited organisational learning.

Within Business Continuity Management (BCM), Crisis Management (CM), and Operational Resilience (OR), metrics and evaluation serve as the bridge between:

• Testing activities → What was done
• Performance insights → What was learned
• Improvement actions → What will change

In an Operational Resilience context, the ultimate measure of success is clear:

Can the organisation deliver its Critical Business Services (CBS) within defined impact tolerance during disruption?

Purpose of the Chapter

This chapter aims to:

• Define key metrics for testing and exercising across BCM, CM, and OR
• Provide a structured framework for evaluating exercise performance
• Establish methods for continuous improvement and maturity development
• Align metrics with Critical Business Services and impact tolerance
• Highlight best practices, challenges, and governance considerations

The Role of Metrics in Resilience Testing

Why Metrics Matter

Metrics enable organisations to:

• Objectively assess performance
• Identify gaps and weaknesses
• Track progress over time
• Demonstrate compliance with regulators
• Support decision-making at senior levels

From Activity to Outcome

Traditional metrics focus on:

• Number of tests conducted
• Participation rates

Modern resilience metrics focus on:

• Service continuity outcomes
• Decision-making effectiveness
• Impact on customers and stakeholders

Key Categories of Metrics

BCM Metrics (Recovery-Focused)

Crisis Management Metrics (Decision & Communication)

Operational Resilience Metrics (Service-Centric)

Cyber & Technology Metrics

• Mean Time to Detect (MTTD)
• Mean Time to Respond (MTTR)
• System recovery performance
• Data integrity validation

Evaluating Exercise Performance

Evaluation Framework

Evaluation should be structured across three levels:

Operational Performance

• Was the disruption managed effectively?
• Were recovery objectives met?

Management & Coordination

• Were decisions timely and effective?
• Was coordination across teams successful?

Strategic Outcomes

• Was CBS maintained within impact tolerance?
• Was the customer and regulatory impact minimised?

Evaluation Methods

• Observers and evaluators during exercises
• Real-time monitoring and logging
• Participant feedback sessions
• Post-exercise reviews (After Action Reviews)

After Action Review (AAR) Process

Purpose

The AAR ensures that lessons learned are:

• Captured systematically
• Translated into actionable improvements

Key Components

• Summary of exercise objectives
• Key observations and findings
• Strengths identified
• Gaps and weaknesses
• Recommended actions

Root Cause Analysis

Go beyond symptoms to identify:

• Process gaps
• Resource limitations
• Decision-making issues
• Dependency failures

Continuous Improvement Framework

Improvement Cycle

A structured improvement cycle includes:

• Test & Exercise

• Evaluate Performance

• Identify Gaps

• Implement Improvements
• Retest and Validate

Integration with BCM and OR

Continuous improvement must:

• Update Business Continuity Plans (BCPs)
• Enhance Crisis Management frameworks
• Strengthen Operational Resilience capabilities

Maturity Progression

Organisations evolve from:

• Compliance-driven → Basic testing
• Capability-driven → Measurable performance
• Resilience-driven → Continuous improvement and optimisation

Reporting and Governance

Reporting to Senior Management

Reports should include:

• Exercise outcomes
• Key performance metrics
• Identified gaps and risks
• Improvement actions

Board-Level Reporting

Focus on:

• Impact on Critical Business Services
• Alignment with risk appetite
• Overall resilience posture

Regulatory Reporting

Demonstrate:

• Evidence of testing and exercising
• Continuous improvement
• Alignment with resilience requirements

Common Challenges in Metrics and Evaluation

Over-Reliance on Quantitative Metrics

Ignoring qualitative aspects such as leadership effectiveness.

Lack of Standardisation

Inconsistent metrics across exercises.

Weak Follow-Through

Failure to implement improvement actions.

Siloed Evaluation

Separate evaluation of BCM, CM, and OR without integration.

Best Practices

• Align metrics with Critical Business Services and impact tolerance
• Combine quantitative and qualitative evaluation
• Use standardised evaluation frameworks
• Ensure executive visibility of results
• Track improvement actions to closure
• Continuously refine metrics based on evolving risks

Case Illustration

Scenario: Payment System Disruption

Metrics Observed:

• RTO achieved: 3 hours (target: 2 hours)
• CMT activation time: 20 minutes
• Customer impact: High

Evaluation Findings:

• Delay in decision-making
• Communication gaps

Improvement Actions:

• Enhance escalation protocols
• Improve communication templates

Metrics, evaluation, and continuous improvement are critical to transforming testing and exercising into measurable resilience capability. They ensure that organisations do not merely conduct exercises, but learn, adapt, and improve continuously.

By adopting a structured and integrated approach, organisations can:

• Measure what truly matters—service continuity and impact
• Identify and address weaknesses
• Strengthen decision-making and coordination
• Enhance overall resilience maturity

Ultimately, resilience is not a static state—it is a continuous journey of improvement. Metrics and evaluation provide the compass that guides this journey, ensuring that organisations remain prepared for an ever-evolving risk landscape.

C1	C2	C3	C4	C5	C6	C7
C8	C9	C10	C11	C12	C13

More Information About OR-5000 [OR-5] or OR-300 [OR-3]

To learn more about the course and schedule, click the buttons below for the OR-300 Operational Resilience Implementer course and the OR-5000 Operational Resilience Expert Implementer course.

	If you have any questions, click to contact us.