Further thoughts on the utility of risk matrices

This is a new paper, co-authored with colleague John Watt, and published in the journal Risk Analysis (2013, Risk Analysis 33(13): 2068-2078). It examines that widely-used tool the risk-consequence matrix, sometimes simply referred to as a risk matrix. These devices are everywhere and there are hundreds of versions around which vary mainly in the number of cells in the matrix (these commonly range in size from 2×2 to 10×10) and the colour scheme used to denote the different combinations of risk and consequence. Exponents of these devices believe, amongst other things, that they help users prioritise their activities.

Risk matrix
An example of a Risk Matrix

We do not share this view. Along with a growing number of authors we find these matrices to be misleading and verging on or worse than useless. As the abstract says:

“Risk matrices are commonly-encountered devices for rating hazards in numerous areas of risk management. Part of their popularity is predicated on their apparent simplicity and transparency. Recent research, however, has identified serious mathematical defects and inconsistencies. This article further examines the reliability and utility of risk matrices for ranking hazards, specifically in the context of public leisure activities including travel. We find that (a) different risk assessors may assign vastly different ratings to the same hazard, (b) that even following lengthy reflection and learning scatter remains high, (c) the underlying drivers of disparate ratings relate to fundamentally different worldviews, beliefs and a panoply of psycho-social factors which are seldom explicitly acknowledged. It appears that risk matrices when used in this context may be creating no more than an artificial and even untrustworthy picture of the relative importance of hazards which may be of little or no benefit to those trying to manage risk effectively and rationally.”

Risk-Benefit Assessment: The Latest (August 2014)

With colleagues Tim Gill and Bernard Spiegal I have been working on ‘A short guide to implementing risk benefit assessment’ on behalf of the sponsors who are Play Wales, Play Scotland, PlayBoard Northern Ireland, Play England and other members of the Play Safety Forum. This guide has now been completed and is available here.

 

Reflections on the assessment and management of risk

The following is an article which appeared in the Activity Industries Mutual (AIM) Newsletter in autumn 2013, written by D. J. Ball and L. Ball-King, and which we reproduce here with permission of AIM.1

We are fortunate in being able to devote some of our work time to things in which we enjoy participating, namely, adventure activities. Consequently, over the last few years, we have had the rare luxury of being able to research the conduct of risk assessment and how it is applied and interpreted, not just in shops and factories, the finance sector, and the courts, but also adventure pursuits. This has identified a series of what we believe to be deep issues affecting a surprisingly broad swath of life beyond the conventional factory fence.

First, we would like to draw attention to continuing disconnects over the meaning of that, for us, crucial word – ‘risk.’ We define risk as the likelihood that some particular thing will happen as a result of exposure to some hazard. So, for instance, in the aftermath of some accident, the immediate question which springs to mind is ‘What was the prior risk of that event?’ leading to the follow-up question which is ‘Was that level of risk acceptable in the circumstances?’ Furthermore, we are not thinking here about the subjective risk (‘it looks dangerous’), but the objective risk (‘it might look dangerous but what is the reality?’).

Experiences in court, however, have shown that expert witnesses and enforcement agencies are not consistently applying this approach. Instead, the logic sometimes followed is what Professor Ragnar Löfstedt has called a hazard-based approach. Basically, this seeks to identify hazards and then to eliminate or control them. In contrast, a risk-based approach would identify hazards, assess the risk, and then decide what control is warranted in the circumstances. Which approach is correct? Mark Hoban, Minister for Employment, summarised the position in February 2013 when, in the context of the government’s reforms of health and safety including the Löfstedt review, he said: We need to implement the remaining measures in both reports as well as continue to identify areas where further reforms are needed to create a modern, simplified, risk-based framework for health and safety in Great Britain.”2

We take it as unassailable that the risk-based approach is fundamental to the national risk control philosophy. It is, after all, enshrined in the requirement of the Health and Safety at Work etc Act to do what is reasonable practicable. Second, assessing risk of some hazardous activity is, contrary to advice often given, not easy. As Professor John Adams has said “It’s not rocket science. It’s much harder.” We agree, and recent research supports this contention. Risk assessors, especially when parachuted in from some external setting, find it very difficult to quantify consistently the risk of harm from activities and situations. The primary means of quantifying risk include use of accident statistics (if they exist) and personal experience through immersion in the sector of interest. Thus, if you are an outsider, you will be in a weak position to assess risk. This is one of the reasons why AALA has been held in such high regard. As initially set up, it was staffed by sector experts who had been immersed in the industry throughout their lives. The danger was, when AALA was subsumed by the HSE, that a hazard-based factory culture of risk control would take over which was also comparatively uninformed about actual risk levels and more driven by perception. If your experience derives from shopping malls and office environments, the natural world can appear an horrendously dangerous place.

Third, we have observed a tendency in litigious settings for the prosecution, and their experts, to demand evidence of proper governance and appropriate management systems. The question, seldom asked, is ‘What is proper governance and a proper management system in the circumstances?’ You don’t need to be an expert in management to know that there are different management styles and one size does not best fit all circumstances.

Over the decades, health and safety has developed an increasingly autocratic, command-and-control
structure with numerous written protocols to be followed. While this may work in some situations the downside is that it shifts responsibility away from front-line professionals. We think that what is most effective in managing the safety of adventure participation is the training and experience of leaders who need more autonomy than might be expected of a factory environment.

These thoughts lead us to a number of considerations:

• it appears imperative to us that any new and future regulatory body for adventure activities be populated by persons with contributory expertise and thus drawn from the adventure sector
• that a risk-based philosophy must be pursued
• that it not be pretended that risk assessment is easy (that’s one reason why you need in-house experts)
• that there is a fundamental difference between managing risk in factories and the natural environment, the latter being undertaken in order to realise benefits3 and which implies the need to trade-off risk of injury against gains in things like health, enjoyment etc
• in seeking support or advice via external agencies it is important to ensure that they have the relevant insight into your sector

1. http://www.activitiesindustrymutual.co.uk/sites/default/files/uploads/misc/On%20Target%20Autumn%202013.pdf).
2. https://www.gov.uk/government/uploads/system/uploads/attachment_
data/file/92634/progress-report-health-safety-reforms-feb-2013.pdf
3. See article by James Willis, On Target, Autumn 2012.

Barbican Conference on Health & Safety Reform

Public Service Events ran a one day conference entitled ‘Health and Safety Reform – reducing the burden of red tape,’ sponsored by ARCO at the Barbican on 26 April 2012. The conference was chaired by David Ball and had an impressive line-up of speakers including Andrew Miller MP (member of Löfstedt review panel), Judith Hackitt (Chair of HSE), Luise Vassie (Policy Director, IOSH), Thomas Martin (joint-MD, ARCO) and Lynda Armstrong (Chair of Trustees, BSC). In the afternoon session panel members Stephen Williams (Director of Operation Strategy and 2012 Olympic Games, HSE), Roger Bibbings (RoSPA) and Gerard Forlin QC (Cornerstone Barristers) led the debate.

Messages to emerge included strong support for the Löfstedt review findings, notably that the underlying legislation was sound but that its interpretation was sometimes problematic and could depart from good sense and proportionality. In the words of Andrew Miller, the regime was overly complex with too much emphasis on paperwork. What was to be done? HSE’s contribution to the ambitious reform programme recommended by the Löfsted review included a comprehensive review of guidance with the aim of simplification while maintaining standards. In addition the ‘Myth Buster Panel’ had been set up, chaired by Judith Hackitt herself, and had already received its first batch of over twenty myths for scrutiny. One newly-emerged myth was summarily dealt with on the spot, namely someone’s proposal on safety grounds that holly trees be removed from school environments because of the (mild) toxicity of their berries. Judith’s response was an unreserved negative!

So the pervasive message was for a risk-based and proportionate approach to safety. But how to do this? As Luise Vassie described, what is needed is scientific evidence of the magnitude of risks in order to inform priorities, and secondly to check the effectiveness of H&S interventions.

From a personal perspective the overall drift of the conference was encouraging. The need to retain the concept of ‘reasonableness’ and ‘reasonable practicability,’ the need to prioritise on the ‘real’ risks (however defined), and the need to simplify perceived administrative demands including paperwork trails and tick-boxes which were more to do with secondary (liability) risk management than the primary focus which is safety and health. However, the task is not trivial. It requires a deeper understanding of many things which are seldom discussed or alluded to in the practicing safety world, including the nature and utility of evidence, the meaning of practicability, the behaviours of people, and the role of values in decision making. Many of these things have of course been intensively analysed in academia, but it has long been the case that the flow of academic thinking into the ‘real’ world has been erratic and sluggish.