Further thoughts on the utility of risk matrices

This is a new paper, co-authored with colleague John Watt, and published in the journal Risk Analysis (2013, Risk Analysis 33(13): 2068-2078). It examines that widely-used tool the risk-consequence matrix, sometimes simply referred to as a risk matrix. These devices are everywhere and there are hundreds of versions around which vary mainly in the number of cells in the matrix (these commonly range in size from 2×2 to 10×10) and the colour scheme used to denote the different combinations of risk and consequence. Exponents of these devices believe, amongst other things, that they help users prioritise their activities.

Risk matrix
An example of a Risk Matrix

We do not share this view. Along with a growing number of authors we find these matrices to be misleading and verging on or worse than useless. As the abstract says:

“Risk matrices are commonly-encountered devices for rating hazards in numerous areas of risk management. Part of their popularity is predicated on their apparent simplicity and transparency. Recent research, however, has identified serious mathematical defects and inconsistencies. This article further examines the reliability and utility of risk matrices for ranking hazards, specifically in the context of public leisure activities including travel. We find that (a) different risk assessors may assign vastly different ratings to the same hazard, (b) that even following lengthy reflection and learning scatter remains high, (c) the underlying drivers of disparate ratings relate to fundamentally different worldviews, beliefs and a panoply of psycho-social factors which are seldom explicitly acknowledged. It appears that risk matrices when used in this context may be creating no more than an artificial and even untrustworthy picture of the relative importance of hazards which may be of little or no benefit to those trying to manage risk effectively and rationally.”

Risk-Benefit Assessment: The Latest (August 2014)

With colleagues Tim Gill and Bernard Spiegal I have been working on ‘A short guide to implementing risk benefit assessment’ on behalf of the sponsors who are Play Wales, Play Scotland, PlayBoard Northern Ireland, Play England and other members of the Play Safety Forum. This guide has now been completed and is available here.

 

The sport of orienteering and safety – the debate

The following article was written by myself and son Laurence and was published in the orienteering journal CompassSport with whose permission it is reproduced here.

The article originated from a blog which appeared after the last minute cancellation of an orienteering event in 2012. The blog contained some contrasting views about the cancellation e.g.

Safety at events is a matter for officials and not individuals.

Anyone unwilling to take responsibility for their own safety should not come to my events.

The last minute-cancelation of Southampton Orienteering Club’s 2012 ‘November Classic’ after heavy rain prompted a lengthy online debate during which many viewpoints were expressed. The cancellation resulted primarily from genuine concerns about the risk associated with crossing a swollen brook. What interests us here is not the decision by SOC, but the ensuing debate. Our curiosity is heightened because we both work in the risk field and the issues raised by orienteers, like those italicised above, reflect a wider ongoing debate about health and safety (H&S) and public life.

In this example the primary contest is seemingly represented by just two viewpoints. One is largely supportive of the impact of new H&S procedures on the sport (e.g. safety should always have priority), whereas the other is doubtful if not alarmed (e.g. O is becoming a health and safety-driven sport and is more prone to cancellation). In the following Table we have paraphrased the sentiments expressed, with the left column containing those inherently supportive of H&S initiatives and the right column somehow questioning their appropriateness.

Accepting Questioning
Safety always takes priority O by its nature is a challenging and risky sportCompetitors know this and expect to face hazards continuously and should accept participation at their own risk
Accidents are foreseeable and need to be tackled Some accidents are inevitable
The sport has changed in many ways, mainly for the better O is turning into a health and safety driven sportCancellation of events is more likely than it used to be
The primary safety mechanism is the management of the eventCompetitors should only take responsibility for small risks What keeps you safe is your own common senseIf management is seen as creating a hazard-free environment competitors may compensate by taking more risks
Written rules and protocols are beneficial Proliferation of written rules and guidance can undermine safety in specific situations
Safety at events is a matter for officials and not individual participants Anyone unwilling to take responsibility for their own safety should not come to my events
Risk assessments are fit for purpose New risk assessment procedures are problematic
Event officials should not be questioned on decisions Decisions may be questioned
Liability has to be considered Liability is a secondary issue

From this summary a familiar situation emerges. The views in the left column fit well with standard H&S beliefs as held in the workplace environment, from whence the beliefs largely originate. In those industrial settings the general and arguably worthy aim has become one of minimising risk of injury. Those on the right, however, are more akin to views of people who organise public events including adventure sports, or run venues open to the public including parks, forests, wilderness areas, waterways, sites of cultural heritage, seaside locations and the like.

Temperatures rise in situations where exponents of workplace-originated H&S procedures, including risk assessment and management systems, maintain that their primary goal of minimising risk should necessarily hold sway in beyond-the-workplace settings, say, adventure sports and sites of cultural heritage, and that industry methods of assessing and managing risk are universally applicable, regardless of the location or activity. Temperatures may rise further when this is justified by statements such as ‘safety is paramount.’

Understandable though the pro-safety position is, for who wants to be cavalier about injuries, a counterpoint is now being made by many agencies dealing with these kinds of more public activities. This is that safety (from injury) is but one of many legitimate goals and that it should be a matter of policy which is paramount. But what are these other goals? Examples given include: the opportunity to have fun; to experience challenge and risk; the pursuit of health, whether physical, emotional or social, through leisure pursuits; the preservation of sites of natural beauty or cultural heritage without alteration; and, above all, the freedom to choose what one does subject to one’s own common sense decisions.

While sympathetic to the pro- H&S position which is largely appropriate in its traditional settings e.g. industry, when it comes to public life we nonetheless cannot help gravitating towards the position of those seeking to maintain a balance between managing the risk of injury, and the other goals of promoting health-giving and resilience-building activities like adventure sports, and public life more generally.

Many organisations are now taking this debate forward as a matter of some urgency. The reason is that to accept unconditionally the injury-minimisation perspective leads into a cul de sac, one which will remorselessly seek to eradicate any safety hazards, and which does not contemplate that exposure to hazards may be unavoidable or, in some cases, even desirable, or that injury prevention measures may have unintended consequences.

From our experience we have found that the debate is driven by the following counter-beliefs, which we here adapt to the context of orienteering:
• The benefits of O, physical mental and social, are legion and need to be recognised and promoted
• Safety from injury is not automatically paramount and needs to be considered as one of a number of goals
• Adventure sports, including O, present risks of injury to all participants however managed
• In many sports and leisure activities exposure to risk is an integral part of the activity
• Risk exposure cannot be assumed to be undesirable
• Formal risk assessment of sports like O is very subjective, driven by undisclosed value judgements, and of uncertain benefit
• The main determinant of competitor safety is not some imposed management system but the competitor
• Competitors who think an event is ‘safe’ because of some imposed system may endanger themselves
• Participants in adventure sports do so because of the perceived benefits, including health, and the chance to confront challenge
• Risk assessment protocols which ignore benefits need to be reformulated

This sport can never be safe from injury
This sport can never be safe from injury

Fellow orienteers might also like to try an experiment. Next time you are out on a course, try consciously thinking about how your brain is handling the endless stream of hazards, from trip hazards to slopes and overhanging branches, with which you are confronted. You will soon tire of this, but may find, with some relief, that your brain is automatically carrying out its own risk assessments during every step you take, a process which has been learnt from day one of your life and may even reside in your genes. Perhaps the ultimate question, seldom asked, is to what extent and in what ways does the current paper-driven approach augment this innate process?

Reflections on the assessment and management of risk

The following is an article which appeared in the Activity Industries Mutual (AIM) Newsletter in autumn 2013, written by D. J. Ball and L. Ball-King, and which we reproduce here with permission of AIM.1

We are fortunate in being able to devote some of our work time to things in which we enjoy participating, namely, adventure activities. Consequently, over the last few years, we have had the rare luxury of being able to research the conduct of risk assessment and how it is applied and interpreted, not just in shops and factories, the finance sector, and the courts, but also adventure pursuits. This has identified a series of what we believe to be deep issues affecting a surprisingly broad swath of life beyond the conventional factory fence.

First, we would like to draw attention to continuing disconnects over the meaning of that, for us, crucial word – ‘risk.’ We define risk as the likelihood that some particular thing will happen as a result of exposure to some hazard. So, for instance, in the aftermath of some accident, the immediate question which springs to mind is ‘What was the prior risk of that event?’ leading to the follow-up question which is ‘Was that level of risk acceptable in the circumstances?’ Furthermore, we are not thinking here about the subjective risk (‘it looks dangerous’), but the objective risk (‘it might look dangerous but what is the reality?’).

Experiences in court, however, have shown that expert witnesses and enforcement agencies are not consistently applying this approach. Instead, the logic sometimes followed is what Professor Ragnar Löfstedt has called a hazard-based approach. Basically, this seeks to identify hazards and then to eliminate or control them. In contrast, a risk-based approach would identify hazards, assess the risk, and then decide what control is warranted in the circumstances. Which approach is correct? Mark Hoban, Minister for Employment, summarised the position in February 2013 when, in the context of the government’s reforms of health and safety including the Löfstedt review, he said: We need to implement the remaining measures in both reports as well as continue to identify areas where further reforms are needed to create a modern, simplified, risk-based framework for health and safety in Great Britain.”2

We take it as unassailable that the risk-based approach is fundamental to the national risk control philosophy. It is, after all, enshrined in the requirement of the Health and Safety at Work etc Act to do what is reasonable practicable. Second, assessing risk of some hazardous activity is, contrary to advice often given, not easy. As Professor John Adams has said “It’s not rocket science. It’s much harder.” We agree, and recent research supports this contention. Risk assessors, especially when parachuted in from some external setting, find it very difficult to quantify consistently the risk of harm from activities and situations. The primary means of quantifying risk include use of accident statistics (if they exist) and personal experience through immersion in the sector of interest. Thus, if you are an outsider, you will be in a weak position to assess risk. This is one of the reasons why AALA has been held in such high regard. As initially set up, it was staffed by sector experts who had been immersed in the industry throughout their lives. The danger was, when AALA was subsumed by the HSE, that a hazard-based factory culture of risk control would take over which was also comparatively uninformed about actual risk levels and more driven by perception. If your experience derives from shopping malls and office environments, the natural world can appear an horrendously dangerous place.

Third, we have observed a tendency in litigious settings for the prosecution, and their experts, to demand evidence of proper governance and appropriate management systems. The question, seldom asked, is ‘What is proper governance and a proper management system in the circumstances?’ You don’t need to be an expert in management to know that there are different management styles and one size does not best fit all circumstances.

Over the decades, health and safety has developed an increasingly autocratic, command-and-control
structure with numerous written protocols to be followed. While this may work in some situations the downside is that it shifts responsibility away from front-line professionals. We think that what is most effective in managing the safety of adventure participation is the training and experience of leaders who need more autonomy than might be expected of a factory environment.

These thoughts lead us to a number of considerations:

• it appears imperative to us that any new and future regulatory body for adventure activities be populated by persons with contributory expertise and thus drawn from the adventure sector
• that a risk-based philosophy must be pursued
• that it not be pretended that risk assessment is easy (that’s one reason why you need in-house experts)
• that there is a fundamental difference between managing risk in factories and the natural environment, the latter being undertaken in order to realise benefits3 and which implies the need to trade-off risk of injury against gains in things like health, enjoyment etc
• in seeking support or advice via external agencies it is important to ensure that they have the relevant insight into your sector

1. http://www.activitiesindustrymutual.co.uk/sites/default/files/uploads/misc/On%20Target%20Autumn%202013.pdf).
2. https://www.gov.uk/government/uploads/system/uploads/attachment_
data/file/92634/progress-report-health-safety-reforms-feb-2013.pdf
3. See article by James Willis, On Target, Autumn 2012.

Barbican Conference on Health & Safety Reform

Public Service Events ran a one day conference entitled ‘Health and Safety Reform – reducing the burden of red tape,’ sponsored by ARCO at the Barbican on 26 April 2012. The conference was chaired by David Ball and had an impressive line-up of speakers including Andrew Miller MP (member of Löfstedt review panel), Judith Hackitt (Chair of HSE), Luise Vassie (Policy Director, IOSH), Thomas Martin (joint-MD, ARCO) and Lynda Armstrong (Chair of Trustees, BSC). In the afternoon session panel members Stephen Williams (Director of Operation Strategy and 2012 Olympic Games, HSE), Roger Bibbings (RoSPA) and Gerard Forlin QC (Cornerstone Barristers) led the debate.

Messages to emerge included strong support for the Löfstedt review findings, notably that the underlying legislation was sound but that its interpretation was sometimes problematic and could depart from good sense and proportionality. In the words of Andrew Miller, the regime was overly complex with too much emphasis on paperwork. What was to be done? HSE’s contribution to the ambitious reform programme recommended by the Löfsted review included a comprehensive review of guidance with the aim of simplification while maintaining standards. In addition the ‘Myth Buster Panel’ had been set up, chaired by Judith Hackitt herself, and had already received its first batch of over twenty myths for scrutiny. One newly-emerged myth was summarily dealt with on the spot, namely someone’s proposal on safety grounds that holly trees be removed from school environments because of the (mild) toxicity of their berries. Judith’s response was an unreserved negative!

So the pervasive message was for a risk-based and proportionate approach to safety. But how to do this? As Luise Vassie described, what is needed is scientific evidence of the magnitude of risks in order to inform priorities, and secondly to check the effectiveness of H&S interventions.

From a personal perspective the overall drift of the conference was encouraging. The need to retain the concept of ‘reasonableness’ and ‘reasonable practicability,’ the need to prioritise on the ‘real’ risks (however defined), and the need to simplify perceived administrative demands including paperwork trails and tick-boxes which were more to do with secondary (liability) risk management than the primary focus which is safety and health. However, the task is not trivial. It requires a deeper understanding of many things which are seldom discussed or alluded to in the practicing safety world, including the nature and utility of evidence, the meaning of practicability, the behaviours of people, and the role of values in decision making. Many of these things have of course been intensively analysed in academia, but it has long been the case that the flow of academic thinking into the ‘real’ world has been erratic and sluggish.