[OR] [P2] [S4] [ST] [C16] Continuous Improvement and Lessons Learned

[P2] [S4] Chapter 16

Continuous Improvement and Lessons Learned

Introduction

Scenario testing does not end when the exercise concludes. In fact, the most critical phase begins after execution—learning, improving, and embedding changes across the organisation.

Without a structured approach to continuous improvement, scenario testing risks becoming a repetitive exercise that fails to enhance resilience over time.

Operational resilience is inherently dynamic. As organisations evolve, so do their risks, dependencies, and vulnerabilities.

Continuous improvement ensures that insights from scenario testing are systematically captured, translated into action, and embedded into the organisation’s resilience framework.

Purpose of the Chapter

The purpose of this chapter is to ensure ongoing resilience enhancement. It outlines how organisations can conduct post-exercise reviews, update CBS mapping and impact tolerances, incorporate findings into strategy and controls, and build a continuous feedback loop into resilience programmes.

Post-Exercise Review and Lessons Learned

A structured post-exercise review (PER) is essential to capture insights and ensure that lessons are not lost.

Conducting the Post-Exercise Review

The review should be conducted promptly after the scenario test and involve all key participants, including:

• Business units responsible for CBS
• Technology and support teams
• Risk and compliance functions
• Crisis Management Teams (CMT)
• Observers and evaluators

The objective is to gather a comprehensive and balanced view of performance.

Key Areas of Review

The PER should focus on:

• Achievement of test objectives
• Performance against impact tolerance
• Effectiveness of response and recovery actions
• Decision-making and escalation processes
• Communication and coordination

Lessons Learned Identification

Lessons learned should be:

• Clearly documented
• Linked to specific observations and evidence
• Categorised by themes (people, process, technology, third-party)

Good Practice: Structured Lessons Learned Template

A structured template typically includes:

• Observation
• Impact
• Root cause
• Recommendation
• Owner
• Target completion date

This ensures consistency and accountability.

Encouraging Open and Constructive Feedback

Organisations should foster a culture where participants:

• Share honest feedback
• Highlight weaknesses without fear of blame
• Focus on improvement rather than fault-finding

This is critical for meaningful learning.

Updating CBS Mapping and Impact Tolerances

Scenario testing often reveals gaps in how Critical Business Services (CBS) are understood and defined.

Refining CBS Mapping

Insights from testing may highlight:

• Missing or underestimated dependencies
• Weaknesses in process flows
• Overlooked third-party relationships
• Inaccurate assumptions about connectivity

Organisations should update:

• Dependency maps
• Process and resource mapping
• Interconnections across systems and stakeholders

Reassessing Impact Tolerances

Scenario testing provides real-world validation of impact tolerance assumptions. Organisations should evaluate:

• Whether thresholds are realistic and achievable
• Whether tolerances are too lenient or too stringent
• Whether new risks require revised thresholds

Aligning Mapping and Tolerances

Updated CBS mapping and impact tolerances should remain aligned, ensuring that:

• Dependencies support resilience objectives
• Recovery capabilities meet tolerance requirements
• Risks are accurately reflected in resilience planning

Continuous Validation

CBS mapping and impact tolerances should not remain static. They should be:

• Reviewed after each major scenario test
• Updated to reflect organisational and environmental changes
• Validated through subsequent testing cycles

Incorporating Findings into Strategy and Controls

Scenario testing insights must translate into tangible improvements in organisational strategy and control frameworks.

Enhancing Resilience Strategies

Findings may lead to strategic changes, such as:

• Increasing system redundancy
• Diversifying third-party providers
• Enhancing crisis management capabilities
• Investing in technology resilience

Strengthening Controls

Control improvements may include:

• Updating policies and procedures
• Enhancing monitoring and detection capabilities
• Improving escalation and communication protocols
• Strengthening third-party risk controls

Prioritisation of Actions

Not all findings can be addressed immediately. Organisations should prioritise based on:

• Impact on CBS
• Likelihood of recurrence
• Regulatory expectations
• Cost and feasibility

Integration with Risk Management

All findings should be:

• Recorded in risk registers
• Linked to existing or new risk categories
• Tracked through risk management processes

This ensures that improvements are formally managed and monitored.

Building a Feedback Loop into Resilience Programmes

Continuous improvement requires a structured feedback mechanism that integrates scenario testing into the broader resilience lifecycle.

Establishing the Feedback Loop

A robust feedback loop involves:

• Testing – Conduct scenario exercises

• Evaluation – Assess performance and identify gaps

• Improvement – Implement corrective actions

• Validation – Re-test to confirm improvements

This cyclical approach ensures ongoing enhancement.

Integration Across Resilience Pillars

The feedback loop should feed into:

• Operational Risk Management (ORM)
• Business Continuity Management (BCM)
• Crisis Management (CM)
• Cyber Resilience
• Third-Party Risk Management

This ensures that improvements are holistic and coordinated.

Governance and Oversight

Continuous improvement should be supported by governance structures that:

• Review lessons learned and action plans
• Monitor progress of remediation actions
• Escalate unresolved issues
• Ensure accountability at senior levels

Tracking and Measurement

Organisations should track:

• Completion of remediation actions
• Improvements in key metrics (e.g., recovery time)
• Reduction in risk exposure over time

This provides evidence of progress and effectiveness.

Embedding into Organisational Culture

To sustain improvement, organisations should:

• Promote a culture of learning and resilience
• Encourage proactive identification of weaknesses
• Integrate lessons learned into training and awareness programmes

Continuous improvement becomes effective only when it is embedded in daily operations and behaviours.

Advancing Scenario Testing Maturity

Continuous improvement also involves evolving the sophistication of scenario testing.

Increasing Scenario Complexity

• Introduce multi-event and cascading scenarios
• Expand to ecosystem-wide testing
• Simulate emerging risks (e.g., cyber threats, systemic failures)

Enhancing Measurement and Analytics

• Use advanced metrics and dashboards
• Apply data analytics to identify trends
• Benchmark performance against industry standards

Strengthening Integration

• Align scenario testing with strategic planning
• Integrate with digital transformation initiatives
• Embed resilience considerations into business decisions

Moving Towards Predictive Resilience

• Use insights from testing to anticipate risks
• Develop early warning indicators
• Enhance proactive risk management

Advancing maturity ensures that scenario testing remains relevant and forward-looking.

Continuous improvement and lessons learned are the foundation of effective operational resilience. Scenario testing provides valuable insights, but its true impact lies in how those insights are captured, analysed, and translated into action.

By conducting structured post-exercise reviews, updating CBS mapping and impact tolerances, incorporating findings into strategy and controls, and building a robust feedback loop, organisations can ensure that scenario testing drives meaningful and sustained improvements.

Ultimately, continuous improvement transforms scenario testing from a periodic activity into a dynamic capability—enabling organisations to adapt, evolve, and strengthen their resilience in an ever-changing risk landscape.