John Lathrop, Irem Dikmen, Emma Soane and Terje Aven helped create the Risk Analysis Quality Test and discuss how risk analysis can be improved in an organisation.
A risk analysis can sometimes just sit on a shelf. But we want risk analyses to effectively guide risk management decisions. To strive for that goal, we spent six years and many emails and meetings among 19 authors to create the Society For Risk Analysis’ Risk Analysis Quality Test (RAQT), a battery of 76 questions based on risk analysis fundamentals and the shortfalls experienced by the authors.
“Quality” in the test’s title means effectiveness in guiding risk management decisions. Each question tests for a risk analysis shortfall. For example the first question: “Is the goal of the risk analysis clearly described and announced?” If yes, good news. If no, that’s a shortfall. So the RAQT is both a definition of risk analysis quality and a spotter of shortfalls. The test can be applied to guide planning a risk analysis project, or to review a completed one. In both uses, the RAQT can spot shortfalls, which can then be disclosed to planners and decision-makers, along with their implications.
Five findings
The biggest one
It’s all about context. It’s about how large the scope and considerations of a risk analysis should be if it is to guide risk management decisions better. We found that in addition to getting the analysis right, it’s as important to embed it in the complete risk management sequence of operations and considerations, from data collection upstream to risk management decision advice downstream. Many of those upstream and downstream elements are organisational or matters of formatting and can be tested by asking simple challenging questions. We present this finding in several ways here, starting with brief overviews: How well is the risk analysis engaged upstream? (See “Frame it”.) How well is the analysis engaged downstream? (See “Engage it”).
Frame it
Identify the actual goal and design accordingly. For example, if it’s to compare the risk aspects of alternative actions, is the net cast broad enough for the alternatives and for the scenarios? If it’s to defend a decision already made, use the same “is the net cast broad enough” questions, but also “is the ‘defend goal’ openly stated?” If stakeholder input is called for, is it elicited? Are all available applicable data collected and applied?
Engage it
Will the analysis results be formatted such that decision-makers, including ones lacking formal training in risk analysis, will be able to understand and correctly apply the results? Will the results be presented to the people in the organisational chart with the appropriate authority, early enough for them to advise the decisions to be made? Is the organisation genuinely committed to having the results of the risk analysis affect its decisions as called for? If not, are all aspects within the control of the risk analysts brought to bear to encourage that commitment?
A more complete look at scope
We can expand on the “biggest finding” above, the large scope called for of risk analysis operations and considerations. The easiest way to do that is simply to list the 15 categories of the 76 questions in the RAQT:
1. Framing the analysis and its interface with decision-making
2. Capturing the risk generating process (RGP).
3. Communication
4. Stakeholder involvement
5. Assumptions and scope boundary issues
6. Pro-active creation of alternative courses of action
7. Basis of knowledge
8. Data limitations
9. Analysis limitations
10. Uncertainty
11. Consideration of alternative analysis approaches
12. Robustness and resilience of action strategies
13. Model and analysis validation and documentation
14. Reporting
15. Budget and schedule adequacy
This list of categories may make it seem that testing risk analysis quality calls for a massive effort, but in fact in our three case study applications of the RAQT reported here, each one only took from one to several hours.
Know your limits and cope with them
In our work to develop and apply the test looking at the real world, we encountered risk management situations with many shortfalls (as identified by the 76 questions), which analysts could not remedy. In those cases, the risk analysis reviewer, manager or analyst should argue strongly for prominent statements in the risk analysis report disclosing those shortfalls and explaining their possible consequences in positive and constructive language. That is, applications of the RAQT are not meant to create a flawless risk analysis world, but rather to identify and disclose shortfalls and their consequences.
Our three applications
Three of us performed applications of the Risk Analysis Quality Test.
Dr Dikmen applied the test to evaluate the risk reporting within a very large infrastructure project and highlighted its importance to test for and improve communication among analysts and decision-makers, thus improving transparency and so trust by the decision-makers in the analysis. Trust is key to the acceptance and use of risk analysis results by decision-makers.
Dr Aven applied the test to identify and describe two main problems with traditional risk matrices as result formats. He posited solutions to those problems and considered why risk matrices are widely used in spite of their flaws.
Dr Soane investigated the relationship between applying the RAQT and organisational culture. She found that the test provides a structure and a language with which to detect, describe, and address issues of risk, and so provides a basis for developing an organisational culture that fosters risk awareness and management. In addition, the application of the test is an organisation’s signal to all its members that it values risk analysis quality.
From another perspective, those applications described extensions in the scope of operations and considerations in risk analysis to advise risk management beyond the scope described by the 15 categories.
Dr Dikmen identified the role of managers’ trust in risk analysis as affected by degree of disclosure.
Dr Aven identified the role of risk analysis result formats in effectively communicating risk analysis results to decision-makers.
Dr Soane identified the role of organisational culture in two directions. The culture must be willing to seek and use analysis results in its decision-making in order for the RAQT to be applied and used. At the same time, applying the test can play a role in developing a culture of risk management by asking the 76 questions, with the ensuing discussions.
Takeaways and recaps
Note here how much we learned simply by defining risk analysis quality in an operational way. We learned by developing the test, listing shortfalls and developing corresponding questions. Then we found out more by applying it.
Our takeaways include:
- Risk analysis quality is not only a matter of getting the data and calculations right. We have described a much larger scope for risk analysis than data and calculations. Of the 76 questions, only about a third specifically consider data and calculations. Two thirds consider the other categories listed.
- Transparency is key. Risk analysis users need to understand as much of its workings and causal chains as they can, so that they trust the results and so will use them.
- The RAQT can only work effectively in an organisation with a culture that supports the use of risk analysis to inform decisions and behaviour.
- In the other direction, applying the test, posing its questions and causing discussions can promote a risk management culture.
- The RAQT provides a language with which to describe strong and weak points of risk analysis formats.
The three applications we describe here are examples of three “fronts” in a “campaign” to improve how risk analysis supports risk management: analysing risk and communicating the findings effectively to decision-makers; evaluating and improving formats used to characterise risk; and improving an organisation’s risk management culture.
Sign up for our weekly newsletter here.
- This blog post is based on Defining and assessing risk analysis quality: insights from applications of the SRA risk analysis quality test, Journal of Risk Research. (This is not an open access link. If you need more information from the paper, please contact Dr Emma Soane at e.c.soane@lse.ac.uk)
- The post represents the views of the author(s), not the position of LSE Business Review or the London School of Economics and Political Science.
- Featured image provided by Shutterstock
- When you leave a comment, you’re agreeing to our Comment Policy.
