Operational Resilience Audit

What is Scenario Testing?

Scenario Testing aims to test the organisation's ability to remain within impact tolerances in severe but plausible disruption scenarios, focusing on recovery and response arrangements rather than preventative measures.

1. Scenario Testing Planning

• Has a scenario testing plan been developed outlining the objectives, scope, and methodology?
• Are the scenarios relevant to the organization's critical business services and potential threats?
• Has the testing plan considered various disruption scenarios, including natural disasters, cyberattacks, and system failures?

• Verify the existence of a scenario testing plan that outlines objectives, scope, and methodology.
• Assess the relevance of the scenarios to the organization's critical business services and potential threats.
• Ensure the testing plan considers various disruption scenarios, including natural disasters, cyberattacks, and system failures.

2. Scenario Development

• Has a range of realistic scenarios been identified for testing operational resilience?
• Do the selected scenarios cover a variety of potential disruptions and stress events?
• Have scenarios been designed to test different aspects of operational resilience, including people, processes, technology, and facilities?
• Are the selected scenarios aligned with the organisation's risk profile and potential impact on critical business services?
• How were the scenarios developed? Were they based on historical incidents, industry best practices, or internal risk assessments?
• Are the scenarios realistic and representative of the organisation's potential threats and disruptions?
• Have relevant stakeholders, including management and subject matter experts, reviewed and approved the scenarios?

• Review the scenario development process and ensure it is based on historical incidents, industry best practices, or internal risk assessments.
• Evaluate the realism and representativeness of the scenarios concerning potential threats and disruptions.
• Confirm that relevant stakeholders, including management and subject matter experts, have reviewed and approved the scenarios.

What is Lesson Learnt?

The key to improving "Lesson Learnt" when implementing Operational Resilience or OR is for an organisation to promote a continuous learning and improvement culture.   It is essential to improve and communicate remediation and vulnerabilities after scenario testing.

Leadership Commitment

• Is there a visible leadership commitment to promoting a culture of continuous learning and improvement?
• Do leaders actively support and participate in scenario testing and incident review processes?
• Are leaders accountable for implementing recommendations and lessons learned from scenario testing and incidents?
• Is there a communication strategy emphasising the importance of continuous learning and improvement for all employees?

Learning Framework

• Is there a documented framework or process for capturing and analysing lessons learned from scenario testing and incidents?
• Does the framework include mechanisms for identifying and documenting root causes and contributing factors?
• Are there standardised templates or tools for collecting and organising lessons learned information?
• Is there a designated team or individual responsible for managing the lessons-learned process?

Incident Review and Analysis

• Is there a structured process for reviewing and analysing actual incidents?
• Are incidents thoroughly investigated to identify root causes and contributing factors?
• Are incident review findings documented and shared with relevant stakeholders?
• Is there a mechanism to track and monitor the implementation of corrective actions resulting from incident reviews?

Scenario Testing Evaluation

• Is there a process for evaluating the effectiveness and impact of scenario testing exercises?
• Are scenario testing results analyzed to identify areas for improvement and enhancement?
• Are there mechanisms to capture feedback from participants and stakeholders on the scenario testing process?
• Is there a feedback loop to incorporate insights from scenario testing into future exercises?

Knowledge Sharing and Communication

• Is there a platform or mechanism for sharing lessons learned and best practices across the organisation?
• Are lessons learned and best practices communicated to relevant teams and departments?
• Are there regular communication channels, such as newsletters or internal portals, to disseminate information on operational resilience and continuous learning?
• Is there a process for capturing and sharing success stories and examples of continuous learning and improvement?

Training and Development

• Is there a training program in place to enhance employees' knowledge and skills related to operational resilience?
• Are employees trained on incident response, scenario testing, and lessons learned?
• Are there opportunities for employees to participate in specialised training or workshops related to operational resilience?
• Is there a process to evaluate the effectiveness of training programs and incorporate feedback for improvement?

Metrics and Performance Monitoring

• Are there defined metrics and indicators to measure the effectiveness of the continuous learning and improvement initiatives?
• Is there a process to track and monitor the organization's performance in implementing lessons learned and recommendations?
• Are performance metrics used to identify areas of success and areas that require further attention?
• Is there a mechanism for reporting and communicating performance metrics related to operational resilience readiness?

Continuous Improvement Culture

• Is there a culture of continuous improvement embedded in the organisation's values and behaviours?
• Are employees encouraged and empowered to share insights, ideas, and suggestions for improving operational resilience?
• Are there mechanisms to capture and evaluate employee suggestions, such as suggestion boxes or innovation platforms?
• Are there recognition and reward mechanisms for individuals or teams that contribute to continuous learning and improvement?

External Benchmarking

• Does the organisation seek opportunities for external benchmarking and learning from other organisations?
• Are there partnerships or networks established to share experiences and best practices in operational resilience?
• Is there a process to review and incorporate relevant industry standards and guidelines into the organisation's practices?
• Are there mechanisms to learn from regulatory changes, industry trends, and emerging risks?

Governance and Oversight

• Is there a designated governance body or committee responsible for overseeing and promoting continuous learning and improvement?
• Are there regular reporting and updates provided to senior management or the board of directors on the organisation's operational resilience readiness and continuous improvement efforts?
• Are clear accountability and responsibilities assigned for implementing and monitoring continuous learning initiatives?
• Is there a process to review and assess the effectiveness of the organisation's continuous learning and improvement initiatives?

Questionnaires and Checklist "Implement" Phase

Set Impact Tolerance

Conduct Scenario Testing

Improve Lesson Learnt

What is Gap Analysis in OR?

A gap analysis is a method of assessing the performance of a business unit to determine whether operational resilience requirements or objectives are being met and, if not, what steps should be taken to meet them.

A gap analysis is called a needs analysis, needs assessment or need-gap analysis.

• Has a structured process been defined for conducting the gap analysis?
• Are the objectives and scope of the gap analysis clearly defined?
• Is there a designated team responsible for conducting the gap analysis?
• Are the necessary resources allocated for conducting a thorough analysis?
• Has a timeline or schedule been established for completing the gap analysis?
  
  

• Has the current state of the operational resilience program been accurately assessed?
• Are the program's key components, processes, and controls identified and documented?
• Has the maturity level of each component been evaluated?
• Are there any gaps or deficiencies identified in the current state?
• Have relevant stakeholders been involved in the identification process?
  
  

• Verify the accuracy and comprehensiveness of the assessment of the current state of the operational resilience program.
• Evaluate the documentation of key components, processes, and controls.
• Assess the methodology used for evaluating the maturity level of each component.
• Identify and document any identified gaps or deficiencies in the current state.
• Confirm the involvement of relevant stakeholders in the identification process.
  
  

• Has a risk assessment been conducted to identify the risks of closing the gap?
• Are the identified risks prioritized based on their potential impact and likelihood?
• Has a mitigation strategy been developed for each identified risk?
• Are the resources and efforts allocated appropriately based on risk prioritization?
• Have appropriate stakeholders reviewed and approved the risk assessment and prioritization?
  
  

• Is there a well-defined governance framework and structure for operational resilience?
• Are individuals or teams responsible for operational resilience assigned clear roles, responsibilities, and accountabilities?
• Is there a mechanism to ensure oversight and monitoring of operational resilience activities?
• Are there regular reporting and escalation processes to senior management or the board of directors?
• Are there mechanisms to review and update the governance framework and structure as needed?
  
  

• Review the documented governance framework and structure for operational resilience.
• Evaluate the clarity and effectiveness of assigned roles, responsibilities, and accountabilities.
• Assess the mechanisms in place for oversight and monitoring of operational resilience activities.
• Verify the existence of regular reporting and escalation processes to senior management or the board.
• Determine if there are mechanisms to review and update the governance framework and structure.
  
  

• Are there documented plans and procedures for testing the effectiveness of business continuity and resilience measures?
• Is there a schedule for conducting regular testing and exercises?
• Are different scenarios and levels of disruptions considered during testing?
• Are testing results analyzed and used to identify areas for improvement and corrective actions?
• Are there mechanisms to track and follow up on implementing corrective actions identified during testing?

• Is there a process to identify and address gaps and deficiencies in the operational resilience program?
• Are there mechanisms to capture and document lessons learned from incidents, tests, and exercises?
• Is there a feedback loop to ensure that identified improvements are implemented?
• Are there metrics and performance indicators to measure the effectiveness of the operational resilience program?
• Is there a culture of continuous improvement and learning within the organization?

Questionnaires and Checklist "Plan" Phase

Develop Strategy Roadmap

Confirm Risk Appetite

Develop and Embed Governance

What is the Capability and Maturity Model?

Capability and maturity are models against which an organisation’s operational resilience performance can be measured and improved.

These capability and maturity models describe the essential elements of effective operational resilience processes and organisational work. The completeness of the continuation of business services is highly influenced by the rigour and quality of the method used to develop it.

• Has a structured process been defined for conducting the gap analysis?
• Are the objectives and scope of the gap analysis clearly defined?
• Is there a designated team responsible for conducting the gap analysis?
• Are the necessary resources allocated for conducting a thorough analysis?
• Has a timeline or schedule been established for completing the gap analysis?
  
  

• Has the current state of the operational resilience program been accurately assessed?
• Are the program's key components, processes, and controls identified and documented?
• Has the maturity level of each component been evaluated?
• Are there any gaps or deficiencies identified in the current state?
• Have relevant stakeholders been involved in the identification process?
  
  

• Verify the accuracy and comprehensiveness of the assessment of the current state of the operational resilience program.
• Evaluate the documentation of key components, processes, and controls.
• Assess the methodology used for evaluating the maturity level of each component.
• Identify and document any identified gaps or deficiencies in the current state.
• Confirm the involvement of relevant stakeholders in the identification process.
  
  

• Has a risk assessment been conducted to identify the risks of closing the gap?
• Are the identified risks prioritized based on their potential impact and likelihood?
• Has a mitigation strategy been developed for each identified risk?
• Are the resources and efforts allocated appropriately based on risk prioritization?
• Have appropriate stakeholders reviewed and approved the risk assessment and prioritization?
  
  

• Is there a well-defined governance framework and structure for operational resilience?
• Are individuals or teams responsible for operational resilience assigned clear roles, responsibilities, and accountabilities?
• Is there a mechanism to ensure oversight and monitoring of operational resilience activities?
• Are there regular reporting and escalation processes to senior management or the board of directors?
• Are there mechanisms to review and update the governance framework and structure as needed?

• Are there documented plans and procedures for testing the effectiveness of business continuity and resilience measures?
• Is there a schedule for conducting regular testing and exercises?
• Are different scenarios and levels of disruptions considered during testing?
• Are testing results analyzed and used to identify areas for improvement and corrective actions?
• Are there mechanisms to track and follow up on implementing corrective actions identified during testing?

• Review the documented plans and procedures for business continuity and resilience testing.
• Evaluate the adequacy of the testing schedule and the consideration of different scenarios.
• Assess the analysis and use of testing results for improvement and corrective actions.
• Verify the existence of mechanisms to track and follow up on the implementation of corrective actions.
• Determine if there is a process to document lessons learned from testing and exercises.

• Is there a process to identify and address gaps and deficiencies in the operational resilience program?
• Are there mechanisms to capture and document lessons learned from incidents, tests, and exercises?
• Is there a feedback loop to ensure that identified improvements are implemented?
• Are there metrics and performance indicators to measure the effectiveness of the operational resilience program?
• Is there a culture of continuous improvement and learning within the organization?

Questionnaires and Checklist "Plan" Phase

Develop Strategy Roadmap

Confirm Risk Appetite

Develop and Embed Governance

1. Understanding Critical Business Services

• Verify the documentation of critical business services and their dependencies.
• How well does the organization understand the interdependencies and dependencies of critical business services?
• Are the definitions and boundaries of critical business services clearly defined?

• Verify the documentation of critical business services and their dependencies.
• How well does the organization understand the interdependencies and dependencies of critical business services?
• Are the definitions and boundaries of critical business services clearly defined?
  
  

3. Identification of Impact Tolerances

• What is the process for identifying impact tolerances for each critical business service?
• Were critical stakeholders involved in determining the impact tolerances?
• Are the impact tolerances based on a thorough analysis of potential impacts, considering various scenarios and threat vectors?

• Review the methodology and approach to determine impact tolerances for critical business services.
• Assess relevant stakeholders' level of involvement and engagement in setting impact tolerances.
• Evaluate the robustness and comprehensiveness of the analysis conducted to establish impact tolerances.

4. Quantitative and Qualitative Measures

• Does the organisation use quantitative and qualitative measures to set impact tolerances?
• Are specific quantitative measures, such as recovery time (RTOs) and recovery point objectives (RPOs), included in the impact tolerances?
• How are qualitative factors, such as customer perception and brand reputation, incorporated into setting impact tolerances?

• Verify whether measurable and subjective criteria are considered in setting impact tolerances.
• Assess if the impact tolerances include measurable criteria for recovery time and recovery point objectives.
• Evaluate if subjective factors are adequately considered in the establishment of impact tolerances.

5. Documentation and Communication

• Are the impact tolerances for critical business services adequately documented?
• How precise and accessible are the documented impact tolerances?
• How are the impact tolerances communicated to relevant stakeholders, including management, operational teams, and third-party vendors?

• Verify the existence and completeness of documentation for the established impact tolerances.
• Assess the clarity and availability of the documented impact tolerances to relevant stakeholders.
• Evaluate the communication process and mechanisms disseminating impact tolerances to relevant parties.

6. Alignment with Business Continuity Plans

• How well do the impact tolerances align with the organization's business continuity (BC) plans?
• Do the BC plans address the identified impact tolerances for each critical business service?
• Is there evidence of testing the BC plans against the impact tolerances?

• Assess the alignment between the established impact tolerances and the corresponding measures in the BCPs.
• Verify if the BCPs incorporate specific provisions to address the established impact tolerances.
• Assess if the BCPs have been tested to ensure their effectiveness in meeting the impact tolerances.

7. Monitoring and Reporting

• How is the performance of critical business services against the impact tolerances monitored?
  Are regular assessments and measurements conducted to track adherence to the impact tolerances?
  How are the results of impact tolerance monitoring communicated to relevant stakeholders?

• Evaluate the mechanisms and processes in place for monitoring the performance of critical business services.
• Assess the frequency and comprehensiveness of assessments to evaluate adherence to the impact tolerances.
  Evaluate the reporting framework for sharing the outcomes of impact tolerance monitoring with relevant stakeholders.

8. Continuous Review and Adjustments

• How often are impact tolerances reviewed and adjusted?
• What triggers a review or adjustment of impact tolerances?
• Are there mechanisms to capture lessons learned from incidents and near misses to inform adjustments to impact tolerances?

• Assess the frequency and timeliness of reviews and adjustments to impact tolerances.
• Determine the circumstances or factors that prompt a review or adjustment of impact tolerances.
• Evaluate if there are processes to incorporate insights from incidents and near misses into adjusting impact tolerances.

Questionnaires and Checklist "Implement" Phase

Set Impact Tolerance

Conduct Scenario Testing

Improve Lesson Learnt