Further thoughts on the utility of risk matrices

This is a new paper, co-authored with colleague John Watt, and published in the journal Risk Analysis (2013, Risk Analysis 33(13): 2068-2078). It examines that widely-used tool the risk-consequence matrix, sometimes simply referred to as a risk matrix. These devices are everywhere and there are hundreds of versions around which vary mainly in the number of cells in the matrix (these commonly range in size from 2×2 to 10×10) and the colour scheme used to denote the different combinations of risk and consequence. Exponents of these devices believe, amongst other things, that they help users prioritise their activities.

Risk matrix
An example of a Risk Matrix

We do not share this view. Along with a growing number of authors we find these matrices to be misleading and verging on or worse than useless. As the abstract says:

“Risk matrices are commonly-encountered devices for rating hazards in numerous areas of risk management. Part of their popularity is predicated on their apparent simplicity and transparency. Recent research, however, has identified serious mathematical defects and inconsistencies. This article further examines the reliability and utility of risk matrices for ranking hazards, specifically in the context of public leisure activities including travel. We find that (a) different risk assessors may assign vastly different ratings to the same hazard, (b) that even following lengthy reflection and learning scatter remains high, (c) the underlying drivers of disparate ratings relate to fundamentally different worldviews, beliefs and a panoply of psycho-social factors which are seldom explicitly acknowledged. It appears that risk matrices when used in this context may be creating no more than an artificial and even untrustworthy picture of the relative importance of hazards which may be of little or no benefit to those trying to manage risk effectively and rationally.”

The risks of long distance running

This year’s London marathon saw the tragic death of Claire Squires. Subsequent media stories revealed that Claire’s was the eleventh such death since the event began in 1981. From this one can (in a sense) gauge the relative riskiness of marathon and long distance running as a sport. From 1981 until 2012 there have been about 850,000 competitors. Taking 4 hours as a rough race time means that competitors have collectively spent around 3.4 million hours on the course. A sometimes used risk statistic for comparative purposes is the death rate per 100 million hours of an activity, known as the FAR (Fatal Accident Rate). From this:

FAR for London marathon ≈ (100/3.4) x 11 ≈ 320 fatalities per 100 million hours

Budapest marathon 2011

How does this compare with other sports and other activities? Back in 1998 I made a study of fatal and non-fatal accident rates for a range of sports ranging from mountaineering to badminton (J. Sports, Exercise and Injury, 1998; 4:3-9). At that time the data showed the most dangerous sport was that categorised as ‘air sports’ which included aerobatics, gliding, hang-gliding, micro-lights, paragliding etc, and which had a FAR in the region of 200. Mountaineering was next highest coming in the range of 30-60, but has since been displaced into third place by caving which comes in at about 160. Water-related sports such as swimming, boating and fishing all have FARs of around 10 to 20, and horse riding comes in at 10. Sports such as rugby, soccer, hockey, cricket and badminton lie in or close to the range of 1 to 3.

From this (see histogram) it can be seen that marathon running is at the high risk end of the spectrum in terms of the FAR yardstick when compared with other sports. This is also true if compared even with industries operating in challenging environments, such as offshore oil and gas, where the FAR as reported by the International Association of Oil and Gas Producers is in the range of 5 to 10.

More generally, within the UK, occupational fatality rates can be estimated from HSE statistics. The average FAR for all workers is around 15, rising to around 70 in construction and 240 in agriculture. Likewise, Department for Transport statistics report FARs for various modes of travel: car 9.8; motorcycle 430; pedal cycle 38; pedestrian 15; bus or coach 0.63; rail 1.5.

So if marathon running is so risky why do it? The answer is that participation brings huge rewards in terms of physical fitness and health, psychological benefits, and social ones too. Even businesses benefit. But the debate over the relative merits and demerits of participation in these kinds of relatively extreme pursuits has been with us for thousands of years. When the London marathon was first proposed by Chris Brasher and John Disley in the 1970s there were objections, for example, that entrance should at least be restricted to club athletes. And during the earlier 1967 Boston marathon a race official attempted to physically eject Kathrine Switzer from the supposedly all-male event on the grounds that “Anything long like 800m, or even longer, God forbid, was considered dangerous ………” for women.

Times, and views, have clearly changed, and to some extent this has been unavoidable given the accumulating evidence of the health benefits of sport. Thus, in the nineteenth century, the view of Reverend Charles Wadsworth, with reference to the Oxford and Cambridge boat race, that ‘no man in a racing boat could expect to live to the age of thirty,’ was gradually proven wrong by epidemiological research which showed life expectation to be greater for rowers than non-rowers by several years.

For a concise and enthralling account of the history of beliefs surrounding the benefits or otherwise of physical activity, see Domhnall MacAuley’s ‘A history of physical activity, health and medicine’.

The debate, of course, continues. But it does illustrate the importance of collecting evidence and not being overly reliant upon subjective opinion and prior beliefs.

Public Safety and Risk Assessment book: Available now

Public Safety and Risk Assessment: New book published in September 2011 by EARTHSCAN/Routledge.

This book is the outcome of over 30 years of involvement with the public, public sector bodies, regulators and the academic community. It explains swathes of academic research from numerous disciplines including risk, economics, psychology, philosophy and decision making and applies them simply and coherently to the intensely practical matters faced by public and private sector bodies responsible for the provision of public space and activities. A second crucial input is first-hand experience of the law pertaining to safety based on numerous cases undertaken as an expert witness.

The purpose of the book is to explain the basics of risk and safety from first principles and take you to a deep understanding of what risk assessment is about, and what it can and cannot deliver. The authors believe that if you do not understand the underlying philosophy of public safety and risk you will forever be vulnerable to competing ideologies which will occupy the slack space bequeathed to them.

Prior to publication the book was reviewed by Sir Chris Bonington, Lord Hoffmann, and Professor John D. Graham. We are immensely grateful for their support. This is what they said:

“So many worthwhile activities are banned or rejected in the name of Health and Safety. Everyone should read this book to see that very often these bans are not justified.” – Sir Chris Bonington

“This book strips away the mysticism and jargon from health and safety, subjecting it to rational analysis. It shows that safety precautions always have to be paid for, in money and lost opportunities, and involves choices which cannot be left to experts but should concern all members of a democratic society.” – Lord Hoffmann

“Safety is important, but it is not paramount. This book exposes the myths and reinstates the choices we have in determining our lives. Trade-offs are unavoidable. We need to make them with as much care and understanding as we can muster. Read how.” – Professor John D. Graham, Dean, School of Public and Environmental Affairs, Indiana University

Since publication Professor Ragnar Löfstedt of King’s College London and author of ‘Reclaiming health and safety for all’ has said in that selfsame document (page 93):

“Ball and Ball-King’s recent book helpfully summarises the key issues surrounding the risk assessment process in the context of public safety.” – Professor Ragnar Löfstedt

Click here to order the book on Amazon.

About the Authors

David Ball is Professor of Risk Management and Director of the Centre for Decision Analysis & Risk Management at Middlesex University, UK. Previously he was Director of the Centre for Environmental and Risk Management at the University of East Anglia, and before that worked as a scientist in local and central government and the private sector in Britain and the USA. He is a regular consultant to government departments and regulators as well as international agencies. Laurence Ball-King has a Masters degree in risk management and a BA in economics and politics. He has worked in credit risk management within financial services and on a variety of non-financial risks including adventure activities and public safety more generally.

The Löfstedt review of 2011

From our perspective this report by Ragnar Löfstedt is both immensely welcome and dramatically  important. As was said in our evidence (by myself and Laurence Ball-King) to his Review Panel, the problems being experienced in relation to the safety of the public and the impact of safety measures on public life were not in our view attributable to our legislation (primarily the Health and Safety at Work etc Act 1974 (HSWA), and the Occupiers’ Liability Acts (OLA)), but rather to their misinterpretation. We went so far as to say in our evidence that the underlying philosophy of these Acts was even something to die for. So we are delighted that Löfstedt has drawn the same conclusion.

Why is the legislation so important? Because, as explained in our book, the UK philosophy is not about eliminating risk but essentially about doing what is reasonable, and what, we say, could be more reasonable than that?! The implication of this is that if Tom, Dick or Harry comes up with some ridiculously expensive or inordinately troublesome safety measure, you are not required by law to implement it because it would not be reasonable. So a good question to ask if you are presented with a proposition which has a whiff of over-the-topness about it, is to see the evidence that it is reasonably practicable (this should contain evidence on how effective it will be in reducing risk as well as information on costs and difficulty of implementation and any unintended consequences which the proposed intervention might have).

Also of monumental significance is Löfstedt’s total support for evidence-based health and safety measures, and for proportionality. Although both of these are obvious requirements for any rational approach to anything at all, there has been a feeling that the UK has been drifting towards a philosophy of hazard-based thinking, as opposed to the kind of risk-based thinking inferred by the HSWA.

The essential difference between hazard-based and risk-based thinking can be illustrated by a simple example. The picture below shows a traditional canal-side lock. These are used by the public for recreational purposes. They clearly expose users and casual  walkers to a number of hazards, in this case of unfenced drops and serious injury or drowning. From a strict hazard-based perspective you would be required to do something about this, maybe to put up railings or warnings or both, or even ban access. From a risk-based perspective, however, you might be able to leave it as it is if the risk of falling off can be shown to be very low, and/or the benefit of retaining the historical authenticity of the unmodified lock outweighs the risk of harm.

In fact, we think the Löfstedt report and our book tread a remarkably similar path. The main difference is that Löfstedt’s remit was primarily the effects of H&S activities on business, whereas our emphasis is upon the effects of H&S upon the provision of public space and public activities. The stories, though, are remarkably close.

The Manchester Hole Case

This involved the tragic death by drowning of a young school boy while on an adventure holiday in Yorkshire. The school party was exploring an underground cave in Nidderdale known as Manchester Hole when the cave experienced an highly unusual flooding event. The case was brought by the Health and Safety Executive who sought to prosecute North Yorkshire County Council who ran the outdoor centre. I was involved as one of several expert witnesses. The case was in court for six weeks, with ultimately the HSE losing, much to its chagrin since it seemed to challenge some of its fundamental beliefs.

A fuller account of the case can be found in Horizons volume 51 published by the Outdoor Learning Institute in December 2010 (http://www.outdoor-learning.org/Default.aspx?tabid=137), but the main issues raised in the case from my perspective as a risk professional were:

  • Should young people be deliberately exposed to risk?

HSE’s published position on this was decidedly ambiguous. According to its 1999 publication ‘Adventure activities centres: five steps to risk assessment’:

“Adventure activities aim to allow young people to develop by meeting challenges they do not necessarily face every day and to experience a sense of achievement in overcoming them. Some degree of risk is unavoidable if the sense of adventure and excitement is to be achieved. However, it is important to remember that adventure activities should only create a sense of adventure and excitement and not cause harm.”

This statement presents a field day for policy analysts, for on the one hand it implies that young people need to meet real challenges and in so doing some risk is inevitable, but then informs that adventure should be fake (only sensed) and  not cause harm (HSE’s emphasis). Apart from its obvious inconsistencies, this contrasts sharply with HSE’s previous support of the Play Safety Forum’s policy statement which says quite simply that “Children need and want to take risks…”.

  • Should risk be negligible?

The Prosecution argued that an acceptable level of risk of serious injury or death is that it should be “negligible.” Since the HSWA requires an employer to ‘ensure’ the health and safety of employees etc, so, prima facie, if an accident occurs s/he has failed. The burden then falls upon the employer to prove that they had done everything which was reasonably practicable to ensure health and safety. The HSE alleged that because an accident had happened this proved that all reasonable measures had not been taken and NYCC had to prove otherwise, which is a curiously circuitous argument for an agency which purports to have a risk-based philosophy.

Another issue of considerable interest was whether the concept of reasonable practicability should encompass consideration of the benefits of such activities. In a civil case it likely would, but this was a criminal case where the answer was less clear cut. However, Justice Wilkie, in directing the jury, explicitly identified as a material consideration ‘the benefits of conducting the activity,’ just as Lord Reid had done in a much earlier civil case.

  • Should adventure activities be managed like factories?

Oral evidence by the HSE in court was in part along the lines that health and safety systems should be designed, implemented and managed in a local authority environment in exactly the same way as in any other organisation, and frequent reference was made to one of HSE’s best-selling texts, known as ‘Successful health and safety management’ or ‘HSG65’, described on HSE’s website as an “over-arching guide on the essential philosophy of good health and safety.” HSG65 is, however, primarily written for industry, not public life, though it could be concluded from this case that HSE sees no difference. But there are differences. Local authorities are providing and managing public services for their public benefits. HSG65 was written in a benefits vacuum where such things are barely acknowledged if at all. This might conceivably be alright in an industrial environment – you wouldn’t expect factory employees to have to walk along a plank to get to the canteen, or office workers to abseil to the toilet, but in public life pitting yourself against these challenges can have benefits.

  • Have you got the right management system?

Another aspect of the HSE’s case against NYCC was targeted on alleged deficiencies of NYCC’s management system. To this end HSG65 was also frequently cited. The fact is, though, that HSG65 is an advisory document, the advice given is based on some unknown person’s opinion, the evidence-base that it is cost-effective is unidentified, and there is no legal requirement to abide by that opinion. It was not at all obvious that the kind of management system described therein was suitable for an education authority or a county council.

  • What and who make adventure activities as safe as they are?

Much time was allocated in court to arguments about the technicalities of risk assessment and who should be involved in performing this required task.

On the technical side, the prosecution’s witnesses described the essential intricacies and interlinking of generic, site-specific, and dynamic risk assessment, which, it was said, comprised a ‘three-legged stool.’ The intended inference was that all three legs had to be equally robust to avoid catastrophic failure, but there was also an impression created that this stool was the rock upon which the safety of adventure activities rested. From a personal point of view, and as I said in court, if my children were going on an adventure activity, I should not want to rely upon such a device. Far more important would be the skill, knowledge and competence of the team leaders!

With respect to who should be doing the risk assessments, HSE’s position was that professional health and safety people employed in NYCC’s headquarters should have had a greater role in compiling the adventure activity risk assessments, whereas defence witnesses, from around the country, poured varying levels of scorn upon this proposition.

The research on expertise, with which I have become familiar in recent years, has led me to believe that there are different kinds of expertise, some of which are appropriate for certain tasks, but not for others. At the bottom of the heap is ‘beer mat knowledge’ – the sort of wisdom you gain from reading about bird flu on the back of a beer mat! In sharp contrast, interactional expertise is what you get from, say, years of participation in canoeing. The implication here is that unless there is strong evidence that anyone on NYCC’s central health and safety team had maximal ‘interactional expertise’ as in potholing experience – through years actually potholing themselves or through years and years actually discoursing with cavers about technical matters – then the only deep expertise about the dangers and so forth would have lain with the adventure leaders themselves. This position accorded with the views of the adventure leaders who gave evidence in court.

Risk-benefit assessment

The move to get risk-benefit assessment recognised as a suitable and sufficient form of risk assessment is gathering momentum but still has hurdles to overcome. It seems that some agencies do not want to concede that the benefits of public space and activities should be a primary consideration in determining how safety from injury decisions should be made. This may be because this would result in a transfer of power because, to make balancing decisions, you would need to know about both the risks and the benefits of some place or activity and traditional H&S exponents may know little or nothing about benefits.

The case for RBA was strongly put in 2008 by Play England and the government in its ‘Managing risk in play provision – implementation guide’ (http://www.playengland.org.uk/resources/managing-risk-in-play-provision-implementation-guide.aspx). The case has more recently been expressed by the National Tree Safety Group in its ‘Common sense risk management of trees’ published by the Forestry Commission (http://www.forestry.gov.uk/pdf/FCMS024.pdf/$FILE/FCMS024.pdf), as well as by the long-standing Visitor Safety in the Countryside Group (http://vscg.co.uk/). Other organisations with strong interests include the Association of Heads of Outdoor Education Centres and the English Outdoor Council.

Overall, there is a widespread desire to get the benefits of public life back on the agenda of health and safety. Paradoxically, one of these benefits is health. But as the Trades Union Congress has put it, “Sadly we have a society that seems to see preventing injury as being more important than preventing illness.”